Resolution

2012-10-09 04:10:00 +0000

This population density map of the United States (you can see more of them here) made me think of some of the challenges we face in working with big data. Look at the map and what immediately stands out is how divided a nation we really are. East of the Mississippi is a tightly woven web of cities and human activity; West of the Mississippi are vast expanses of empty. Pockets of people are connected by long and thin tendrils—the web of civilization is woven much thinner. 

My point is simply this: when we are working with big data, it can be so easy to get sucked right into the data. We live and breathe and think data, and being immersed in it is what makes us giddy. It’s seductive. But the best insights are often gleaned from a distance. Turning the resolution way down on our analysis let the bigger patterns emerge. 

These bigger patterns are what truly matters, and they can guide us in our quest for ever better answers. Tune the resolution way down to see the big picture, then gradually increase it again as you ask ever more probing questions of the data.