The standard example of this was the discovery that people who buy diapers often buy beer at the same time, so shop owners would do well to put those items next to each other. It’s a clear example, but it’s also a terrible example because, according to a November 2016 article by Steve Swoyer in Upside, the discovery wasn’t made through advanced statistical analysis, but via a simple SQL query by someone looking to see what else was in market baskets with diapers. While it’s not clear how real or reliable the beer–diaper correlation was, it’s a clear example of how a useful surprise might be just waiting to be discovered in Big Data.
Today’s AI age
And now we’re living in the fifth age of data: the Age of AI (or, more exactly, machine learning). This, I think, represents an especially profound shift, since it presents data as a hidden source of not just surprises, but secrets.
For example, AI finds heart health information in retinal scans, and no one knows how. We don’t know exactly how ChatGPT constructs its sentences. That is normal for AI, which then means these secrets are actually mysteries.
Data in each of these ages lights up the world differently. Mainframe data revealed the world as manageable by being reduced to standardized data. PC data revealed the future as incapable of being captured in standardized, normalized tables. The future is not just something we can predict, but something we can play with. The unstructuring of data in the Internet Age highlighted the chaotic and complex nature of the world we were born into and especially the social world we’ve created and that has created us. Big Data’s data showed a world that contains secret relationships waiting to be unearthed. In the Age of AI, data’s ability to reveal secrets in ways we can’t understand highlights the mysterious, overwhelming complexity of our world.
I personally think that these ages of data have each deepened and made more realistic how we think about the world. Perhaps this latest phase in the history of data will bring us to accept inexplicable complexity as a property of the world. We could view this as pure chaos, but thanks to having lived through the past four ages in rapid succession, we might instead recognize that chaos as being rich with endless mysteries we will never uncover completely.
Those mysteries are that which is given.