The Rise of Big Data

NRC President Tom Miller reviews the article, "The Rise of Big Data, How it’s changing the way we think about the world” from this edition of Foreign Affairs. This article has sparked a great deal of conversation in our office - would you care to chime in?

Cukier and  Mayer-Schoenberger argue that technology now permits us to capture, not samples of useful data, but ALL of it. By having all the data – from sources like translated transcripts, satellite images, weather reports – we can find patterns and associations that might never have occurred to us – (I’m making these up) – how to teach Chinese, where power plants are likely to be sited, when municipalities will decide to build indoor farmers’ markets. Here are a few highlights of the article:

·         Small data sets of the past (really, the present) have to be ‘curated’ so data are ‘pristine’

·         Big data compensates for dirty data by the quantity.  It’s ok to live with dirty data if there are enough.

·         Forget theories or hypotheses. No need to know in advance what we are looking for. With big data, we crunch until we find a useful relationship.

·         Forget caring about cause. Causes are often illusory or ephemeral anyhow. Care about correlation. Correlation is enough to predict what will work, even though we won’t know why.

Survey data for local governments are not big data by Cukier and Mayer-Schoenberger standards, but they still hold potential for insights about what is working in a jurisdiction. NRC researchers, local staff and other stakeholders will help answer the why and what actions should follow.

For further review:

http://www.youtube.com/watch?v=bYS_4CWu3y8 40 minutes of Cukier on Big Data

“The Rise of Big Data, How it’s changing the way we think about the world.” Kenneth Neil Cukier and Viktor Mayer-Schoenberger, Foreign Affairs, May/June, 2013, pp28-41.