In 2022, the lives of many who survive will be sufficiently different that it will be difficult to recall how we moved through life without real-time data. We have long welcomed the tools that will deliver this world to us. Our cellphones are a nest of analog sensors, as are our streets, our televisions, and our shoes, and yet these sensors have not fulfilled their most powerful promise. Sir Francis Bacon wrote with prescience opening his Novum Organum, originally published in 1620:

He chose to call these methods the Anticipation of the Mind and the Interpretation of Nature.

Real-time data integrates both the algorithmic Anticipation of the Mind and a shared Interpretation of Nature into the human condition. We will become new beings augmented by an understanding of the present moment that is fed by an awareness of disparate space-time that has never before been presented to a single perceptual construct. We will make decisions, for better or for worse, in a fundamentally different way than we do today. There will be rush of opportunity to manage the transition from our socially based economy to a data driven one.

The growth of the vitality of computing as it relates to economics, scientific study, and even personal enjoyment withers away the transaction costs of changing your mind. Computation combined with sophisticated storage and recall of information reduces the time, and as a proxy for time, the money, involved in trailblazing paths through complex decisions. Instead of detonating atom bombs, we simulate their detonation ad nauseum. Instead of cutting up strips of film we toss around light represented as a string of numbers.

For the most part, we in this room have spent much of our lives waiting in shrinking periods of time for a small group of people to dissolve the considerable friction along the route from data collection to data visualization. In the words of Montgomery Scott, “It’s like trying to hit a bullet with a smaller bullet, whilst wearing a blindfold and riding a horse.” In 2010, we approach the end of this tunnel. We see a world where data remains in motion from the sensory apparatus from which its gathered, through an arbitrary number of silos in which it could be interpreted, to the sensory experience of person who is also in motion. In 2022, we will have the horsepower in our media servers to hit that bullet.

The massive datasets that sit locked in the vaults of our decision makers who have no idea how to pull them apart will soon find connections to each other and outgrow their cages. Small vendors such as Mint.com will entice large vendors such as Mastercard to reinterpret the value of their services. The consumer will demand a way to opt in to systems that deliver their data back to them. Foursquare understands this model but is no match for your cell phone provider, who already knows where you are at any given moment without checking in. AT&T also happens to have access to the speed at which you walk, the number of times you check your Twitter, your burn rate on battery life, the ambient noise you’re surrounded by, and how many times you call your mother. In 2022, the providers of your mobile connection will be the first filter that skims off the data warning everyone of your impending heart attack. Your bank will engage with you based on whether you went to Starbucks at your usual time or six hours after that. Through data mashups, companies will excavate information about your behavior, and they will realize that they will lose you as a customer to anyone who can serve it up in a better way. Product lines and capitalism itself are about to evolve.

Patterns of human behavior emerge in the remixes of organic data, and these patterns are far more valuable to their creator than they are to the marketing MBA. As the transaction cost of calling up a data visualization at will approaches zero, we reap a cognitive surplus that is invested in the ideas which capture our interest. We will see choices for our cash flow painted on top our upcoming events when we’re actually thinking about them, walking from the subway to the office. We will rely on the automatic execution of small decisions, such as choosing a brand of milk based on the carbon footprint of a farming method, to feel better about our participation in society. We will have an elaborate scoring system attached to many types of decisions that will rank the efficiency of action, the relevance to a goal, the likelihood of a risk, the trust of a data source, the quality of the data, the accuracy of the prediction, the standard deviation from the mean of previous decisions, and the concurrency with your social network. The boundaries of a path set by series of fossilized choices – that have data streams which have not varied outside of a set range – will guide us into realizing what we truly care about.

But the future is not all tasty peaches. These peaches will ripen and might rot. There will be three pitfalls in data streaming: becoming obsessed with the past, the present, or the future. According to Frances Bacon, salvation lies in alternating between awareness of abstracted structures of data and understanding the particles which construct them. Subtlety in the ability to balance past observations with confirmations of nowcasts and weighted predictions of the future will be a way to gain advantage over your trading partners. You will be measured not only by the products you consume, but also by your contributions to data streams. The leaders of men and women will become those who are capable of manipulating the data stream to support their ideas. Some will thrive on amplifying grotesque ideas. Some will thrive on balancing the diversity in data sources. We as consumers will attempt to heavily filter information that we find tedious, but the suppliers of information will attempt to prevent us from doing so. The point for them is not to know the truth but to get as many people as possible to believe the same idea by reiterating selected evidence. The data streams will often become Ouroboros, the serpent eating its own tail. You will easily believe that the world is exactly as you are, but you will be wrong. You will find optimal solutions but then you will repeat them over and over again. This is the problem faced by the makers of Coca Cola, but their solutions often rely on their customer having less than perfect information or fairly bad information recall. There will be a tension between the feeling of control and the loss of control as the opportunity for real deviations from a highlighted chain of events are presented to you for the first time in your life.

The important thing to remember is that data streams can only be evaluated by experience. There will certainly be frustration around the limitations of the ability to measure things, the tendency to measure superficial things, and the gaps the exist between measurements. Furthermore, faith in the ability to make a useful decision out of real-time data relies on trusting three people: the person who defined the data, the person who cleaned the data, and the person who presented the data. Your social network is an excellent way to ground your data streams in trust, but because useful data inevitably comes from far flung sources, data streams should be treated with skepticism. Early predictions will likely fail until Adam Smith’s invisible hand steps in to correct them through feedback loops. This creates another kind of tension between the instinct to react to your sensory experience and the suggestions of a visualization giving contrary advice. Evaluation of the data stream can either become a reflection of self-awareness or a distraction from it.

A third kind of tension will arise between the demand to pay for information, the opportunity to steal it, and the Faustian bargain to accept a manipulated data stream in exchange for promotional awareness. The global economy will be modeled on creating barriers and artificial bottlenecks that monetize streams of data. Clearinghouses and packagers of information will become the definition of the first world. Ad hoc networks to transfer stolen data will be the third world. Many of these steams will flow through open source analysis. We saw an example of this last week, where a very vocal group of smart people insisted that Twitter was suppressing any trend of Wikileaks based on their observations of Twitter’s public data stream. Consumers will opt in to systems that compete on their granularity, breadth, personalization, and honesty. The most detailed streams will be proprietary, but leaks will counterbalance that model. Social class will be indicated by the quality of information and the aesthetics of a visualization. Too few streams will feel like poverty. Too many streams will feel like stupidity. Countries whose first networks are real-time networks will deliver subsidized streams that are diluted to the lowest common denominator. Villages where the closest stream is within walking distance will receive information, but information that is garbled through word of mouth. Many of us in the middle will be caught up in new types of depression and euphoria arising out of constant comparison to your friends.

Finally, real-time data will change the intimacy of your social network. Sharing data is a sign of intimacy. The lack of sharing is an obstacle to intimacy but perhaps a necessary one. You will accept a role in your household to pay attention to certain streams, much as one of the partners in a marriage pays attention to the checkbook, and you will negotiate with your family and friends about which personal data streams you will allow them to see. You will feel a stronger need to keep others unaware of certain details of your life, and yet you will feel a stronger temptation to break into the details of the lives of the people that you love.

Every so often, we may just need to go be alone. We may need to step out of the stream and rely on our unaided intuition or our prayer rather than our planning. And when this step occurs, we will know that real-time data is an inseparable part of us.

Given the forces driving the expansion of access to information in the 12 years spanning 1998 to 2010, we can expect that progeny of these forces will affect our lives in 2022. In general, across the world but particularly in the developed world, it is predetermined that our ability to find information will increase. Whether increased access to information impacts our capacity for knowledge is uncertain, but it is likely that our access to information will have a deep impact on the ways in which information is used.

Four converging trends drive increased awareness of the immediate state of being:

1. The volume of data and metadata on human behavior and activity is rising
2. The willingness to collect, aggregate, and share many different streams of personal data is rising
3. The cost of processing power and data storage is falling
4. Predictive models of human behavior based on collected data are becoming more accurate

These trends converge in a future where the transactional cost, time, and complexity of decision making is reduced in accordance with our ability to eliminate latency in processing time and apply appropriate filters in visualization. Nowcasting – the accurate prediction and presentation of information representing the current state of being – lies near the center of our data informed lives. Real-time data lies at the center of nowcasting.

Macroeconomists developing econometrics have encountered frictions that might delay the widespread adoption of nowcasting. Dean Croushore, a macroeconomist at the Federal Reserve, defines real-time data analysis as, “research for which data revisions matter or for which the timing of the data releases is important in some way.” (Croushore, 2008) The transition from data that is repeatedly revised, such as GDP, to data that is immediately factual, such as clickthroughs, is met with several challenges. In a paper published before he was Chairman of the Federal Reserve, Ben Bernanke wrote, “Research departments throughout the Federal Reserve System, as in other central banks, monitor and analyze literally thousands of data series from disparate sources, including data at a wide range of frequencies and levels of aggregation, with and without seasonal and other adjustments, and in preliminary, revised, and ‘‘finally revised” versions.” (Bernanke and Boivin, 2003) The effects and predictability of data revisions are the chief investigation of macroeconomists exploring real-time data. The chart below demonstrates the movement in a single, common statistic published in 138 different vintages of data.

Real consumption growth for the second quarter of 1978 is not a number that remains stationary after its published but a number that is revised when new data becomes available or old data is reevaluated. The predictive power of current models is often based on the ability to separate “news” from “noise” in successive data vintages. Predictive models also face a ‘jagged edge’ problem defined as the loosely periodic release of different data streams at different times. For example, the table below describes the timing release of 35 different data streams relevant to setting monetary policy.

Understanding the timing of data releases helps to explain the motivations behind a decision made at a specific moment in time, as researchers often grapple with evaluations that contain more data series than data observations. Fortunately, many of the challenges with using real-time data have been overcome, or the power of the data despite the challenges has proven to be greater than traditional or simple models.

Macroeconomists commonly agree that manipulating highly varied real-time data can improve our awareness of the business cycle used to set monetary and fiscal policy. “We find, in brief, that the scope of the data set (the number and variety of series included) matters very much for forecasting performance, while the use of revised (as opposed to real-time) data seems to matter much less.” (Bernanke and Bolvin, 2003) Applying techniques such as averaging traditional and real-time forecasts or introducing noise reduction mechanisms such as the Kalman filter to real-time data produce more useful results than the best results the Federal Reserve Bank has customarily published in its Greenbook.

Nowcasts are immediately valuable in their ability to uncover recessions more quickly than information the National Bureau of Economic Research has often relied upon. Their accuracy may also rival the well-specified econometric models currently used to forecast monthly indicators (Castle, Fawcett, and Hendry, 2010). For example, Google has built a real-time price index that adds granularity to the US Government’s Consumer Price Index. The driving force behind using nowcasts in macroeconomic decision making seems to be building momentum, and we can easily imagine a future in which the application of nowcasts spreads beyond the Federal Reserve.

Hardware manufacturers and software developers have begun to produce robust products that overcome the barriers blocking production of accurate and relevant nowcasts. Foremost is the ability to move data as quickly as possible from collection to visualization, a process IBM describes as stream computing or “a futuristic technology that can detect insights within data streams still in motion, that is, before they are saved into databases.” (Rea and Mamidipaka, 2010) Stream computing is enabled by removing the bottlenecks along the paths of data and reducing calculation overhead by performing highly parallel computations concurrently on related streams, such as those informing a stock trade.

Stream computing offers a better way to conceive of gleaning information.

Very low-latency systems are currently employed by financial services industries and used as competitive advantages among stock exchanges. The London Stock Exchange recently touted its 126 microsecond average latency as twice as fast as its main international competitors. IBM has deployed its stream computing solution, InfoSphere System S, in manufacturing, health monitoring, and energy trading experiments. ATI and Nvidia have invested in general purpose graphics processing unit (GPGPU) computation, a touchstone of stream computing that may bring it into the households of 2022.

Our world is being instrumented (Adrian, 2009). Many smartphone owners carry at least four more analog sensors with them than they did 12 years ago. The ability to run these sensors in the background without battery life concerns, along with other types of sensors, is predetermined for 2022. Increasing activity of self-published data is highly probable. The translation between silos of self-aggregated data into information motivating a single decision, like that of the stock trade presented above, is low hanging fruit. Applying this process to track consumption, like equating purchases to carbon footprints, are the most obvious but perhaps least interesting implications.

Given the right balance of filtering relevant to the current moment based on the methods being developed in macroeconomics, a stream computing solution that facilitates nowcasting will improve the interaction of a person with his environment. Rather than reducing the human condition to algorithms, algorithms will combine to improve the human condition in ways that are difficult to imagine (in much the same way that the uses of Twitter are difficult to define). Going beyond real-time platforms like Twitter, Facebook, Mint.com, Nike+, and others, stream computing will create a nowcast of hybrid information woven into a cocoon from which the possibility of acquiring knowledge may undergo a metamorphosis. Certainly, in 2022, we will use our increased capacity to monitor ourselves and decreased cost of remixing information to improve upon our mediated intake of data.

Bibliography

Background Notes