Launch, grow, and unlock your career in data

September 12, 2013

Visualizing statistical significance in survey results

My colleagues and I at Facebook had the honor of presenting at the 2013 Tableau Customer Conference alongside Kate Treadwell of Interworks, who was subbing for Mike Roberts. The topic of our talk was "Data Discovery at Facebook: Why Culture & Flexibility Matter for Modern Analytics". We based the session around four of Facebook's company values:
  • Move fast
  • Be open
  • Be bold
  • Focus on impact
During the "Be open" section of the talk, Heather Torres, who leads HR & Recruiting Analytics for us, revealed a method for encoding statistical comparisons of survey results. The basic premise is for a manager to be able to view their survey results compared to the company in total, but also understand the statistical significance of the results. Heather had created something similar to this in her past life at AOL in Excel and she challenged me to build something similar in Tableau. It didn't end up being very complicated, once the data was in the correct format of course.

I need to pause for a moment and thank two Tableau Zen Masters for their help and inspiration.

First, thank you to Jonathan Drummey for his help in getting both the manager and company results into the same view.  Jonathan taught me about Data Source Filters, which I had never used before.  We ended up aggregating the data so that we didn't have to have multiple data sources, but without his help early on, we wouldn't have gotten there.

Second, we need to thank Steve Wexler for his awesome work in visualizing survey results.  We based many of our other reports off of work that he's done.

And now back to our regularly scheduled program...

In order to help aid with understanding, here are some tips for how to read and interpret the results:
  1. The dark bar represents the company in total.
  2. The colored bar represent the chosen manager's results.
  3. The width of the bars represents the statistical variation. The manager bar will always be wider than the company bar because there are less results for a manager, thus more less statistical significance.
  4. The color of the manager bars represents how statistically significantly different they are vs. the overall company results.
  5. If you see a gap between the company and manager bars, then the difference is statistically significant.
  6. You can filter by Manager.
  7. You have three chart types to choose from (more below about them).
  8. You have three sorting options, which allow you to answer different questions depending on what you want to compare.
Note: All of this data has been altered randomized, so it doesn't reflect our actual survey.  The manager IDs are all fake, etc.

The view that Heather showed in our session was the Double Gantt (TM) option.  I've provided two other alternatives as well: Candle Gantt (TM) and Dots Gantt (TM).

The Candle Gannt is more or less the same as the Double Gantt, but the company bar is thinner and there are reference lines for the ends.  The idea here was to make it look somewhat like a candlestick chart.

The Dots Gantt is nice because it clearly shows the outer ends of the statistical range without concern for the range itself.

I had no idea what to call these types of charts, so the names are simply a combination of the chart types (And they aren't really trademarked.  Maybe I should TM them). These are all dual axis charts. Download the workbook and pull the charts apart if you're interested in seeing how they're built.

For those of you that came to our session, we owe you a heartfelt thank you.  We appreciated all of the questions, comments and great interaction.


  1. This was my favorite session of the week. Excellent job presenting, and excellent content.

    1. Thanks Kyle! How much did Dan pay you to say that?

  2. Nice work. Great for managers to be able to have a reality check as to just how awesome or how bad they really are. Great that FB HR is as nerdy as you guys and values this kind of analysis.

  3. I have done something similar to this using employee engagement results. This is a very effective way to visualize this information. Thanks for sharing!

  4. Andy, can't wait to dive into this and to watch the recorded session. Also, I'm a big Heather Torres fan and was blown away by her presentation at TCC09. Please say hello to her for me.

  5. Thanks. A blog would be great!. In the meantime, I will take a stab at what you described. Thanks for your help. All your other workbooks and examples have been extremely helpful to me!