Revolutions

Because it's a long weekend: Who's at your party?

If you're planning a house party this long weekend, you might want to take a cue from Everett Hiller to liven up your party snaps with a bit of Bill Murray:

And what party is complete without Tom Cruise on a piñata?

Yes, these are photoshops -- but awesome photoshops. (Everett clearly has some skills with lighting and composition.) You can see more celebrity substitutions at Twisted Sifter.

We'll be taking a break at the blog for the Memorial Day holiday; we'll be back will on Tuesday. Enjoy the weekend, and for those of you in the US have a great long weekend.

Twisted Sifter: This is What Happens When You Photoshop Celebrities Into Your Holiday Party

Facebook-class social network analysis with R and Hadoop

In computing, social networks are traditionally represented as graphs: a connection of nodes (people), pairs of which may be connected by edges (friend relationships). Visually, the social networks can then be represented like this:

Social network analysis often amounts to calculating the statistics on a graph like this: the number of edges (friends) connected to a particular node (person), and the distribution of the number of edges connected to nodes across the entire graph. When the graph consists of up to 10 billion elements (nodes and edges), such computations can be done on a single server with dedicated graph software like Neo4j. But bigger networks — like Facebook's social network, which is a graph with more than 60 billion elements — require a distributed solution.

Marko A. Rodriguez, a graph consultant with Aurelius, shows in a blog post how to use R and Hadoop (integrated with Revolution Analytics' RHadoop packages) to analyze Facebook-scale social networks. He first simulates a social network (shown at the top of this post) using R's igraph package, and then distributed the network in the Hadoop cluster with to.dfs function (from the rhdfs package). He then used the mapreduce function (from the rmr package) to write a simple map-reduce algorithm in R to count the number of edges associated with each node:

degree.V <- mapreduce(edge.list, 
    map=function(k,v) keyval(v[2],1), 
    reduce=function(k,v) keyval(k,length(v)))
from.dfs(degree.V)[[1]]

From there, it's another simple map-reduce job to calculate the connectivity statistics for the entire network. For more details on how Marko used RHadoop to perform this analysis, see the entire blog post linked below.

Aurelius blog: Graph Degree Distributions using R over Hadoop

Data Science shows maturity at 2012 Summit

As a discipline, Data Science is growing up fast. That's my key takeaway from the 2012 Data Science Summit.

At the inaugural 2011 Data Science Summit (you can see some highlights in this recap video), the focus was on the Big Data part of Data Science: issues with streaming data, how to store big data, technology platforms, that kind of thing. This year's summit was much more focused on the "Science" part of Data Science: applications of Big Data, and statistical issues related to the analysis of Big Data. A few examples:

• Nate Silver (political forecaster for the NYT) talked not just about building models and making predictions, but also the importance of, in his words, "embracing uncertaintly". A prediction often isn't useful without an assessment of its uncertaintly (or risk). He gave this real-life example: a flood-level prediction of 49 feet doesn't mean a city can rest easy because the levees are 51 feet high. The weather service failed to mention that there was a plus-or-minus 9 feet margin of error to that prediction, or about a 50-50 chance the city would be flooded. (It was.)
• Michael Chui (author of the McKinsey Big Data report) said that schools should be teaching more Statistics, and less Calculus, so that graduates have a better grasp of issues like sampling and selection bias.
• Michael Brown (CTO of ComScore) talked about the need to understand the impact of recall bias and outliers.
• Jeremy Howard (Chief Data Scientist of Kaggle) warned of the dangers of observation bias inherent in "data exhaust", and extolled the benefits of statistical experiments to distringuish between causality and correlation.
• Tony Jebara (co-founder of Sense Networks) expressed the need for the focus of predictive analytics to graduate from mere accuracy to making models interpretable, and predictions actionable.
• Hadley Wickham (R package author and educator) described the variety of application areas for Data Science, from cheesemakers to airport designers, and from sports teams to cruise lines. 

These are all important statistical issues, which until recently have had a back-seat to the technological and operational issues of data science. It's great to see the practice maturing, and this new focus will lead to data applications which are not just more powerful, but more reliable and more impactful as well. Data Science has come of Statistical age.

NYT charts the Facebook IPO with R

In conjunction with Facebook's record-setting IPO last Thursday, the New York Times created an infographic to put the size of the offer in context with other recent IPOs. A detail of the graphic as it appeared in the print edition appears below:

ChartsNThings gives a fascinating peek into the weeklong process that went into creating this chart, where about a dozen "sketches" of charts were tried and considered until the final chart was selected when Thursday's deadline arrived. It's a great look at the developmental process that goes into creating the quality infographics that are the hallmark of the NYT team led by graphics editor Amanda Cox. It's also a testament to the value of working in the R language (used to create all of the prototype charts), which is expressly designed to encourage experimentation and creativity in data analysis and visualization. Plus, the highly expressive nature of the language allows such prototypes to be created quickly, allowing the NYT team to iterate through many alternatives as the deadline loomed.

The post also highlights an interesting angle related to the differing requirements of the web version and print version of NYT infographics: 

If you’ve seen the web version, though, you know it doesn’t look like this. [Amanda thinks print graphics can be smarter than web graphics.] For one, the browser window doesn’t give us this kind of space. But the medium itself plays a part too. Online, if you’re not engaged in 10 seconds, you’re not going to stay on the page, so they needed to keep it fun.

The prototype below was considered for the print version, but was instead ultimately adapted for the web:

The fun aspect here is that as you switch between the different view on the web, it's not just a slideshow; each IPO "bubble" moves into it's new position as the view changes. Like a motion chart, it's a great way to track comparisons of individual data points between data views. (Try it out here.)

There are lots more details about the process of creating these charts at the post linked below; the whole thing is a great read.

ChartsNThings: Amanda Cox and countrymen chart the Facebook I.P.O.

The grade level of Congress speeches, analyzed with R

As widely reported by CNN, the Huffington Post, Talking Points Memo, the sophistication of speeches by US politicians has declined in recent years, dropping from an 11th-grade level in 2005 to a 10th-grade level today. The reports are based on an analysis by the Sunlight Foundation, based on textual analysis of congressional speeches given since 1996 provided by the Capitol Words API. You can see where your favourite legislator ranks in this table. Picking out a couple of famous names, Sen. Rand Paul (R-KY) ranks near the bottom with an 8th-grade speaking level; near the top of the ranks is retiring senator Olympia-Snowe (R-ME) who speaks at a Grade 14 level. 

A regression analysis detailed in the article teases out some of the contributing factors. While the speech levels of members of both parties have declined over the past couple of years, the chart below shows that for Republicans, the more ideological extreme the congressperson, the less sophisticated the speech.

It's clear that the data visualizations were created using R's ggplot2 package; presumably the statistical analyses were peformed with the R language as well. For the infographic circulated to the media, the ggplot2 charts have been cleaned up using an editing tool like Illustrator (easy if you export the chart to PDF from R).

Read the details of the analysis in the full report from the Sunlight Foundation linked below.

Sunlight Foundation: The changing complexity of congressional speech

A visual data summary for data frames

If you want to get a quick numerical summary of a data set, the summary function gives a nice overview for data frames:

> require(ggplot2)
Loading required package: ggplot2
> data(diamonds)
> summary(diamonds)
     carat               cut        color        clarity          depth           table      
 Min.   :0.2000   Fair     : 1610   D: 6775   SI1    :13065   Min.   :43.00   Min.   :43.00  
 1st Qu.:0.4000   Good     : 4906   E: 9797   VS2    :12258   1st Qu.:61.00   1st Qu.:56.00  
 Median :0.7000   Very Good:12082   F: 9542   SI2    : 9194   Median :61.80   Median :57.00  
 Mean   :0.7979   Premium  :13791   G:11292   VS1    : 8171   Mean   :61.75   Mean   :57.46  
 3rd Qu.:1.0400   Ideal    :21551   H: 8304   VVS2   : 5066   3rd Qu.:62.50   3rd Qu.:59.00  
 Max.   :5.0100                     I: 5422   VVS1   : 3655   Max.   :79.00   Max.   :95.00  
                                    J: 2808   (Other): 2531                                  
     price             x                y                z         
 Min.   :  326   Min.   : 0.000   Min.   : 0.000   Min.   : 0.000  
 1st Qu.:  950   1st Qu.: 4.710   1st Qu.: 4.720   1st Qu.: 2.910  
 Median : 2401   Median : 5.700   Median : 5.710   Median : 3.530  
 Mean   : 3933   Mean   : 5.731   Mean   : 5.735   Mean   : 3.539  
 3rd Qu.: 5324   3rd Qu.: 6.540   3rd Qu.: 6.540   3rd Qu.: 4.040  
 Max.   :18823   Max.   :10.740   Max.   :58.900   Max.   :31.800  

But if you'd prefer a visual overview of your data, Andrew Barr suggests the tableplot function (included in the tabplot package) for a graphical version:

tableplot(diamonds, cex = 1.8)

Andrew explains how to use the tabplot function in the post linked below.

W. Andrew Barr's Paleoecology Blog: Quickly Visualize Your Whole Dataset (via @JacquelynGill)

Because it's Friday: Game theory

Game Theory is the mathematical study of how agents in a system make choices for their actions, in light of the fact that other agents are also making competitive choices of their actions. As the name suggests, the "system" is often some kind of game and the "agents" are players, but game theory is also used to explain the behaviour of crowd motion, business dealings, foreign relations, and even the evolution of altruism. (There's a excellent chapter involving game theory in The Selfish Gene.)

The textbook example of game theory is the Prisoner's Dilemma. The UK game show "Golden Balls" includes a form of the Prisoner's Dilemma where the stake is prizemoney instead of freedom. Each player may independently choose to "split" or "steal" the prize, but if both steal each goes away empty-handed. You might think there aren't many strategic options to guarantee a win in a game like this, but one player found a way:

 

It's an elegant strategy. Nick is apparently altruistic (or at least risk-averse), and wants to share the prizemoney. But altruistic actions are open to subversion by a selfish opponent. Ibraham, unable to discern Nick's apparent "mutually assured destruction" strategy, has no option but to behave altruistically. I wonder if this strategy has ever been attempted again on the show; but now that this strategy is in the "gene pool" of strategic options, it would likely be less effective the next time around. And that's one of the subtle beauties of Game Theory.

via Business Insider: British Gameshow Contestant Puts On Badass Display Of Game Theory

R is to SAS as Java is to COBOL

An interview with Revolution Analytics CEO Dave Rich was published this week by BeyeNetwork. During the interview, Dace was asked about how the statistical modeling platforms have changed over the decades:

People have been doing statistical modeling and predictive analytics for 50 years now, SAS and SPSS have been around since the early ‘70s. What’s different now -- what’s making this move toward other statistical and “big data” areas?

David Rich: Well, I think obviously SAS and SPSS have been around, as you pointed out, for decades. We call that sort of the first generation of analytics and insight-driven solutions. In my perspective, having been in the business for more than three decades, it reminds me a bit of what COBOL was back in the day relative to business software. I see R as the more modern language. In this analogy, R would represent Java or C++. What happened in the middle of the nineties when the shift occurred is very similar to where we are now with R. Open source is a worldwide collaboration innovation. It’s a way to tap into that channel for research, and I think the role that Revolution Analytics can play – very similar to what Red Hat did back in the Linux days – is to be the conduit between the community and enterprise deployment.

The conversation also touched on the future of big data analytics, impact of advanced analytics on business, and the benefits of R and Revolution R Enteprise to reduce costs and expand the scope of possibility with big data analytics. For the complete interview, follow the link below.

BeyeNetwork: Advanced Analytics, Big Data and the Power of R: A Q&A Spotlight with David Rich, CEO of Revolution Analytics

In Mexico, more marriages ending in divorce, and sooner

R user Diego Valle analyzed the rate of divorces in Mexican marriage since 1993 (the earliest date for which data are available) and found that not only have more marriages ended in divorce over time, but marriages that do end are ending sooner:

This chart is a bit complicated, but it bears close inspection. Each line you see is a cohort of all of the marriages in a given year: 1993, 1994, all the way up to 2009. The vertical height of each line is proportional to the total number of divorces in each subsequent year within each cohort (expressed as a fraction of all marriages in the cohort year). Cleverly, the cohort lines are all arranged not by calendar time, but by years since marriage: the leftmost point represents divorces in the first year (relatively few), then divorces in the second year, and so on. 

More residents of Mexico married in 1993 saw their 10th wedding anniversary than those married in 1998. Overall, the trend is clear: more weddings that take place now will end than those from previous years, and they're likely to end sooner as well. Although there's not much historical data for recent marriage, the steady progression of divorce rates over time allows Diego to create a forecast (using a linear mixed-effects model in the R language) of the outcomes of recent marriages. He predicts, for example, that 11% of marriages registered in 2007 will have ended in divorce by 2022. By contrast though, that's about the same rate as US marriages from the fifties.

If you want to do a similar analysis, Diego provides R code in his post linked below, and at his github.

Diego Valle-Jones: Proportion of marriages ending in divorce

Where's Waldo? Image Analysis in R

R user Arthur Charpentier attempts to use the raster library and R functions to find Waldo in a "Where's Waldo" image:

Sadly, it turned out that Waldo was a bit too tricky to spot using these techniques. But Arthur did have more success identifying the US flag in a shot from the Apollo mission, and identifying answers in the form for a multiple-choice test. All of the R code is provided at the link below, so that's a great place to start if you're looking to do some image analysis in R yourself.

Freakonometrics: Finding Waldo, a flag on the moon and multiple choice tests, with R