by Mark Ridinger

Predicting the long range and far reaching effects of disruptive, emerging technologies is the focus of many organizations, including CReST. Different phases can be identified, that run the gambit from improving efficiencies within an established industry to disrupting that industry entirely, or creating entirely new, previously unimagined industries. The PC and its word processing “killer app” at first augmented the efficiency of secretaries, for example, but ultimately essentially ended that profession all together, shifting writing documents and memos to the executive or manager. The World Wide Web has been even more disruptive. Ripe for the Internet “reaper” has been “the middleman”, for example. Cut him or her out, and save the customer and seller time and efficiency. Case in point: the travel agent, driven to near extinction by Expedia et al. Successful and accurate prognostications are exciting, but studying—and learning from—examples of projected Internet produced chaos and disruption that has failed to materialize is equally invaluable.

Case in point: the real estate agent. By all accounts, this middleman industry should have been essentially eliminated by the Web, and was widely predicted to be so by savvy investors and pundits alike. A 6% commission on a very large sum of money—indeed the largest purchase or investment most folks will ever make-- is a lot of money to part with. Add to that the bursting of the real estate bubble in 2006 and the financial crises and ensuing Great Recession of 2008-9, not to mention substantial venture capital backing of startups trying to take over this huge industry, it was all but inconceivable that the demise of the broker did not occur. With so many aligned financial incentives, it seemed to be a slam-dunk prediction. It was a perfect storm.

But it wasn’t. In fact, real estate agents are thriving. Bloomberg reports that only 9% of homes were sold without a broker in 2012, down from 13% in 2008.

So what happened? What went wrong? And how did so many get it wrong? At CReST one of the books we are reading is Radical Evolution, by Joel Garreau. In it, the author addresses this point at a high level. For example, some categories to which bad or failed technological predications fall into: underestimating complexity, inadequate cost/benefit ratio, the emergence of an even newer/more disruptive technology, prior bad experiences with similar technology, and a fundamental misunderstanding of human behavior. The last category explains why the real estate agent demise prediction went wrong.

The successful Internet real estate startups—now substantial companies—recognized this; people wanted their hand held, and were willing to pay for it. Far from displacing real estate agents, Zillow, Trulia and Realtor.com [the main players] have become essentially advertising companies for brokers. Many of the things brokers had to do for clients—show pictures, comparable sales, neighborhood and school information to name a few—are done by these internet firms for free. What is left, namely title search and legal closing documents primarily (which admittedly require expertise), could be outsourced to an attorney for a fraction of what a home seller is paying in commissions. Yet that hasn’t occurred. Redfin, the startup that set out to eliminate brokers and their commissions, has been on death’s door as a company for years, but has finally switched its model as well.

Technology does not advance merely for the sake of technology, nor change for the sake of change. The missing link is often the human element. In Social Physics, another book we are reading, the author, Big Data guru Sandy Pentland, contends that “people prefer trusted and personalized relationships,” likely an evolutionary remnant, and one that remains very much intact even in the era of social media. How Big Data and the Internet might be used to exploit those relationships is in part one of the focuses of the emerging field of Cognitive Security.

We need to look at—and learn from—failed predictions as much as successful ones, in order to improve our ability to make successful science and technology forecasts, and ultimately policy recommendations. Often, when we get things wrong, it is because we fail to accurately account for human behavior and desires. In short, we need to understand ourselves better to become better forecasters.