My data is many thousands of “leads”, for which I’ve constructed hundreds of predictive features (mostly 1/0, a few numeric) each. I can plug this data into any number of common statistical and machine learning systems which will crunch the numbers and provide a black box that can do a pretty good job of separating more-valuable leads from less valuable leads. That’s great, but now I have to communicate what I’ve done, and how valuable it is, to an audience that struggles with relatively simple statistical concepts like correlation. What can I do?

I’m generally interested in finding better ways to build clean, intuitive, and informative visualizations of data, especially when the visualizations can leverage intuitions and skills that everyone has. For example, almost everyone has a surprisingly good approximate number sense, the ability to quickly identify about how many items are in a largish group. For example, if shown a photo of 30 oranges and a photo of 20 oranges, you would be able to immediately say that there were more oranges in the first photo, and you would happily say that that photo had a few dozen oranges in it. This psychological skill can be used to make more effective visualizations of certain types of data. Instead of comparing two quantities by lines in a chart, or even a number in a table, it may be useful to compare visual density.

How can this be used to make better visualizations of prediction quality? Consider the standard ways that predictive model quality is reported. I have obfuscated the test set data from the problem I mentioned above, and placed it in a public Dropbox in Rdata format. I’ve also put together an R script to demonstrate various ways of looking at the predictions and put it in a Github gist. Follow along if you’d like.

First, take a look at the data frame and some summary statistics:

> head(pred.df)
      predicted actual actual.bin
7379  0.6020833    yes          1
5357  0.5791667    yes          1
7894  0.5791667    yes          1
5893  0.5604167    yes          1
16093 0.5541667    yes          1
2883  0.5520833    yes          1

> summary(pred.df)
   predicted        actual       actual.bin
 Min.   :0.000000   no :7785   Min.   :0.0000
 1st Qu.:0.004167   yes: 366   1st Qu.:0.0000
 Median :0.016667              Median :0.0000
 Mean   :0.040827              Mean   :0.0449
 3rd Qu.:0.041667              3rd Qu.:0.0000
 Max.   :0.602083              Max.   :1.0000

The mode predicts about 4% of the items will be in the “yes” category, which is similar to the 4.5% that actually were. Using the very flexible ROCR package, I can quickly and easily convert this data frame into an object that can then be used to calculate any number of standard measures of predictiveness. First, I calculate the AUC value, which has a very intuitive interpretation. Consider sorting the list of items from most-predicted-to-be-”yes” to least. If the predictions are good, most of the “yes” values will be relatively high in the list. The AUC is equivalent to asking, if I randomly pick a “yes” item and a “no” item out of the list, how likely is the “yes” item to be higher on the list? If the list was randomly shuffled, it would 0.5; if it were perfectly shuffled with 20/20 hindsight, the AUC would be 1.0.

> # convert to their object type (labels should be some sort of ordered type)
> pred.rocr <- prediction(pred.df$predicted, pred.df$actual)
> # Area Under the ROC Curve
> performance(pred.rocr, 'auc')@y.values[[1]]
[1] 0.8237496

In this case, it’s about .82, which is probably valuable but far from perfect. Another common way of looking at this type of predictions comes from business uses, where the goal is to identify leads (or whatever) that are likely to convert to purchases. From this point of view, the goal is to lift the leads higher in the list, so that you can focus on the top of the list and got more benefit from sales effort with less work. Two common ways of looking at lift are with a decile table, which shows how much value you get by focusing on the top 10%, 20%, etc. of the list, sorted by the predictive model, and the lift chart, which visualizes the same thing by showing how much benefit over random guessing you get by looking at more or less of the sorted list. Here they are for this data:

# decile table
dec.table <- ldply((1:10)/10, function(x) data.frame(
    decile=x,
    prop.yes=sum(pred.df$actual.bin[1:ceiling(nrow(pred.df)*x)])/sum(pred.df$actual.bin),
    lift=mean(pred.df$actual.bin[1:ceiling(nrow(pred.df)*x)])/mean(pred.df$actual.bin)))
print(dec.table, digits=2)

   decile prop.yes lift
1     0.1     0.61  6.1
2     0.2     0.69  3.4
3     0.3     0.76  2.5
4     0.4     0.80  2.0
5     0.5     0.84  1.7
6     0.6     0.90  1.5
7     0.7     0.92  1.3
8     0.8     0.95  1.2
9     0.9     0.99  1.1
10    1.0     1.00  1.0

# Lift Curve
plot(performance(pred.rocr, 'lift', 'rpp'))

This graph shows, not particularly intuitively in my view, that if you focus on the top 10% of the data, you get more 5 times the bang for the buck than if you focus evenly on the whole set of items. The decile table shows the same thing — the top decile is lifted by a factor 0f 6.1, and in fact you get 61% of the “yes” items in that top 10% of the data. These are very useful numbers to know, but I think there are considerably more intuitive ways of showing how the predictive model pulls the “yes” values away from the 5% base rate.

These more intuitive ways are not the standard graphs used in statistics and machine learning, such as the sensitivity/specificity curve and the ROC curve. Those graphs, shown below, illustrate trade-offs between accepting false positives and false negatives. Useful, yes, but to understand them you have to think about the ways you could set a threshold and what effect that threshold would have on the nature of your predictions. That’s not particularly intuitive, and the visualization doesn’t visually contrast two things, so it’s difficult to get an intuitive understanding of what has been gained.

I’ve put some thought and some tinkering into potentially better ways of visualizing the output of predictive models. The key, I think, is to use a visualization that builds on the scatter graph. Scatter graphs are great for less-technical audiences, because you can tell them that every individual dot is a customer (widget, whatever). They can immediately see the number of items in question, and if you can plot the points on axes that make sense to them, they can go from “that dot there represents one person with this level of X and this level of Y”, to “this set of dots represents a set of people with similar levels of X and Y”, to “this graph represents everyone, and their respective levels of X and Y.” And because of skills like the approximate number sense and the ability to quickly understand visual density, scatter graphs can give a vastly better understanding of the range of a data set than summary graphs that just plot a line and maybe some error bars.

Here are several versions of a graph that illustrates how the predictive model smears out the set of dots from the 5% base rate, disproportionately pulling the “yes” items to the right, separating at least some of them from the much larger set of “no” items. One key change from a basic scatter graph is to jitter the Y position of each point randomly, which I think makes these graphs look a little like a PCR gel image.

This first approach is built around a basic scatter graph, where the X axis is the predicted likelihood of being a “yes”, and the Y axis is 0 for actual “no” and 1 for actual “yes” items. On top of that is an orange line representing the base rate of about 5%, a blue line showing the smoothed ratio between “yes” and “no” items at each level of prediction, and a thin grey line showing where the blue line ought to be. In this case, the model tends to underestimate the likelihood that some items are to be “yes” items. At 50%, half of the items should be “yes” and half should be “no”, but it’s more like 3:1.

I like this graph as it intuitively lets people see the extent to which the predictive model is separating the categories, and how much better it does than just assuming the base rate. My second approach at this combines the “smears” with another way of visualizing lift.

In this graph, the smeared real data is at the bottom of the graph, and the black line represents the lift, or how much better you are at identifying “yes” items by using the predictions. It’s also an intuitive way of motivating the need to draw a boundary to focus effort. When trying to convert the points at the 25% level or above, you may be ineffective 75% of the time, but you’re also more than 10 times more efficient than you would be otherwise.

My final attempt worth sharing is this one, which combines the dual smear approach with the cumulative value numbers from the lift table.

Now, in addition to being able to see the density of yes and no items for various levels of the prediction, you can see what proportion of the potential “yes” values exist to the right of each level of the prediction. For example, at a threshold of 25%, you capture 40 or 45% of the “yes” items. At a 5% threshold you capture more than 70% of the “yes” items.

I’d love some feedback on these graphs! Do you agree with my assertion that scatter graphs are more visually intuitive and  easier to motivate to non-technical audiences? Do these variations on lift charts seem clearer or more valuable than traditional alternatives to you? Have I re-invented something that should be cited?

The R code for these graphs is available in the Github gist. I used ggplot2, naturally, which is an essential tool for exploring the space of possible visualizations without being tied down by traditional graph structures.

Incidentally, for people interested in building graphs that leverage people’s innate visual capabilities, I recommend Kosslyn’s book, Graph Design for the Eye and Mind.

Also incidentally, the question of how to communicate or visualize the potentially incredibly complex sets of rules/weights/whatever inside the categorization black box is another fascinating issue, the subject of ongoing research, and maybe something I’ll write about soon.

And a few thoughts about PAW. The conference was well-run (although I have my gripes with the hotel and its location!) and there were an interesting and eclectic lineup of speakers, from a variety of industries. Compared to academic conferences I’ve attended, I missed having all the grad students around. At PAW, I felt rather young, which had not been true at academic conferences in quite a long time! The content of the conference focused on people using predictive methods (statistics, data mining, machine learning) at the individual-customer level, for marketing or retainment or other purposes. That’s not my primary interest right now — my work is focused at a slightly higher operations-research-y level, trying to make sure that customers in the aggregate have good options. But I enjoyed learning about what other people are doing using somewhat similar methods. Next year, though, I think I’ll try to go to a different conference, perhaps UseR! in the UK, or INFORMS’ applied conference…

The ggplot package, written by the overachieving and remarkable Hadley Wickham, is based on earlier more theoretical work by Leland Wilkinson. Wilkinson abstracted the process of putting data onto an image, and created a Grammar of Graphics, which describes how the data maps to the parts of a graph, rather than describing the final graph itself. For example, here’s how to create a pie chart, clipped from Wilkinson’s book:

Don’t worry about the details, but briefly, a pie chart is just a stacked bar graph (summary.proportion) plotted in polar coordinates (polar.theta). If you took the time to learn this grammar, you would realize that the hierarchical structure of a graph on a page (elements have positions and labels and visual properties like color, each of which have their own abstract structure) maps cleanly to the hierarchical structure of the grammar, and that variables in the grammar map cleanly to the linear structure of the data. As a user of this system, you would be able to see all three key representations at once: the data, the grammatical mapping from data to graph, and the graph itself.

Now consider ggplot, the implementation of the Grammar of Graphics in the R programming language. Does ggplot maintain three visible representations, all straightforwardly mappable to each other? Sadly, it does not. Instead, users of ggplot must map among four representations: the data (a standard data.frame object), the R syntax for ggplot2 (which has some quirks), an underlying ggplot object (similar to the Grammar of Graphics, but vastly more complex and impossible to examine directly), and the generated graph.

Consider the simple pie graph, below.

This chart is generated in ggplot2 by the following R code:

> zz <- data.frame(cat=c("a", "b"), val=c(5,3))

> zz
 cat val
1   a   5
2   b   3
> pp <- ggplot(zz, aes(x="", y=val, fill=cat)) + geom_bar(width=1) +
        coord_polar("y")
> print(pp)

The print() function is optional within an R interpreter session, but I include because it illustrates a point that’s not initially obvious to many users. Unlike the built-in R plotting tools, the ggplot() function and its associated functions don’t plot anything on the screen, they just construct an object of type “ggplot”. Almost all of the actual work of mapping the data to stuff on your screen occurs when you print that object, using print() or ggsave().

So what does that object look like? If you type str(pp), you’ll get an answer, but it’s about a hundred lines of undecipherable hierarchical object and list structure, not intended to be examined by mere mortals. But there’s something critically important about that structure — like the original Grammar of Graphics, and unlike the R syntax above, it’s hierarchically structured.

In the R syntax, you create a base ggplot structure with the ggplot() call, then you abuse the “+” operator to make changes to that structure. The geom_bar() function adds a layer to the ggplot() object, where a layer is just what it sounds like, a set of information about one of potentially many overlaid layers of content that will be put on the graph. So you construct a ggplot object by first initializing everything about the basic plot, then tack on layers with +, right? Actually no, because the coord_polar() call doesn’t create or modify a layer at all, it modifies the base object! Even if you’ve acquired the nonobvious intuition that ggplot objects are hierarchical and are created by concatenating layers, you now have to break the analogy again to fully understand what + is doing!

There is a way to partially see the structure directly, but it’s not well thought-out from the point of view of someone trying to learn how to use the package. The summary() method on ggplot objects tells you about things you didn’t specify (faceting?), it’s incomplete, and it doesn’t map well to the R syntax. If something in your plot isn’t working the way you want it to, summary() won’t help you.

> summary(pp)
mapping:  x = , y = val, fill = cat
faceting: facet_grid(. ~ ., FALSE)
-----------------------------------
geom_bar:
stat_bin: width = 1
position_stack: (width = NULL, height = NULL)

Another shortcut that leads to conceptual problems by ggplot beginners is the use of qplot(). The qplot() function is a wrapper around ggplot(). Unlike ggplot(), you can give qplot() data that is not in the form of a data.frame, and the syntax is somewhat different. There’s nothing wrong with some syntactic sugar to make life easier, but in this case, learning ggplot by starting with qplot is like trying to learn a foreign language by starting with contractions and slang. You may be able to say a few essential things on your vacation, but you won’t be able to creatively construct new sentences as new situations arise. The brilliance of the Grammar of Graphics is exactly that it’s a grammar — you can construct new graphs and new types of graphs as new situations arise! But tutorials that start with qplot, with the ggplot book an unfortunate (but in other ways excellent) example, send their learners down a linguistic garden path. To fully use the power of the system requires unlearning the conceptual structures that map the slang to charts on a screen, and starting over with learning the new, more powerful ggplot() grammar and hierarchical representations.

I’d like to conclude this overlong rant with two notes. First, just today a new graphics package for R was introduced. jjplot uses many of the ideas of the Grammar of Graphics and ggplot2, but seems to avoid at least a few of the conceptual problems. The + operator is not overloaded in conceptually confusing ways, and there is no distracting qplot function to mislead new users. Additionally, a quick look at the source code finds it much, much simpler than ggplot2′s source, which will likely lead to a more active base of contributors. I look forward to trying jjplot and watching its continuing development, and hope the authors learn from both the remarkable successes and frustrating failures of ggplot. Second, I use ggplot extensively in my work. It’s simply the best available tool for quickly generating elegant graphs of data in R, especially if that generation needs to happen automatically in code. Hadley Wickham deserves extensive praise for the amount of effort he has put into developing and popularizing the Grammar of Graphics. If you want to be maximally effective when visualizing data in R, take the time to learn ggplot2, but do so while keeping in mind that the learning process will be easiest if you skip qplot and other shortcuts, think hierarchically, and prepare for some frustration. Fortunately, the support communities on the ggplot mailing list and Stack Overflow are extremely helpful, as is Hadley himself.