We are swimming in data. Or possibly drowning in it. Organizations and individuals are generating and storing data at a phenomenal and ever-increasing rate. The volume and speed of data collection has given rise to a host of new technologies and new roles focused on dealing with this data, managing it, organizing it, storing it.
copyright by www.oreilly.com
But we don’t want data . We want insight and value . We can think about these new technologies and roles, and the way they help us move from data to insight and value, through the lens of something called the wisdom hierarchy . The wisdom hierarchy is a conceptual framework for thinking about how the raw inputs of reality (signals) are stored as data and transformed first into information, then into knowledge, and finally into wisdom. We might call these steps in the hierarchy “levels of insight.”
To illustrate, starting with the input of specific error codes coming out of a software system, we can draw out an example of each step in the hierarchy:
Data: All the error logs from that system.
Information: An organized report about the error codes.
Knowledge: The ability to read a specific error report, understand it, and possibly fix the problem.
Wisdom: An understanding of how different and seemingly unrelated errors are actually connected symptoms of a larger, underlying problem.
Whatever esoteric connotations we might associate with the word “wisdom,” we can define it here as deep subject matter expertise. From this kind of wisdom comes decision-making, business value, and leadership.
Getting the levels right
From a human perspective, the levels of this hierarchy aren’t absolute, though. The line between each level is fuzzy, and the growing insight within a particular organization doesn’t usually follow a direct path; the journey circles back on itself. For example, as knowledge about the value of data increases, the nature of data collection might change. Insights gleaned from data analysis become data themselves. Additionally, insight doesn’t simply move up the hierarchy, but comes in from all directions. There are many inputs aside from information that create knowledge, and many inputs aside from knowledge that create wisdom and business value. From a machine learning and AI perspective, the hierarchy undergoes a weird recursion process. While humans might be said to climb the hierarchy, machines flatten it. No matter how advanced a computer system is, it is really only dealing with data. The inputs to an AI system are data; the processing of those inputs is data centric; and the outputs are, from the computer’s perspective, simply more data. But the outputs can be interpreted by humans at each level of the hierarchy. We can understand the results in terms of information, knowledge, or—if the AI is sufficiently advanced—wisdom. […]