Serendipity and the 'Aha' Moment – Unexpected Insights in the Innovation Process

May 14, 2014

Disruptive innovation and displacement is happening so fast that many innovation and product development leaders are looking to accelerate and systematize the hard-to-fulfill ideation and discovery phase of the innovation process—the innovation funnel. To that end, for more than 20 years, organizations, researchers and computer scientists have been examining the recurring role that serendipity has played in successful innovations. These innovation drivers have sought to not only assess the role of serendipity, but to also determine if serendipity can indeed be systemically enabled through big data, analytics and visualization. Indeed, it can be.

Simply defined, serendipity is “an aptitude for making desirable discoveries by accident.” Key findings on serendipity in innovation are detailed in a paper, “Discovery Is Never by Chance: Designing for (Un)Serendipity,” by experts at the University of Southampton, UK and  Microsoft Research. The paper describes how computer scientists have been generating “serendipity-inducing systems.”

IT experts have focused on developing “discovery recommender” technology to recommend something interesting and previously unknown—or at least something unknown within the domain involved. These systems can enhance serendipity as a foreground activity in innovation and can foster behavior change in looking for, internalizing, and applying insights. The authors suggest it is the particular type of unknown and unexpected data that creates value in a Big Data-type recommender system. In particular, tapping other domain “knowns” can be very valuable. 

Importantly, it has also been shown that partially relevant or iterative nuggets of data or results may play a key role in informing and generating new directions in the discovery-seeking and harvesting processes. The authors of the paper cite the importance of publishing discoveries so many serendipity-hunting agents can find them. They also cite the importance of structurally dispatching this insight to appropriate domain experts who may be able to make something of it. They stress that knowledge about the encountered information or resource—as well as knowledge about the task the person is engaged in—are both critical dimensions within serendipity and recommender-system outputs.

Combining the value of big data for serendipitous insights and preparing recipients to harvest those insights, the authors unearthed several key attributes for potential systemic replication, including:

  • Structured data gathering and analytics to unearth what would be viewed as serendipity—particularly insights from previous “unknown” and “delightful” data.
  • Supplementing that data gathering with elements to tap and increase collective wisdom to ascribe merit and potential to a serendipitous discovery (which could include the use of data gathering processes and “containers” as well as collaboration processes to share and work information across organizations, ecosystems, unrelated disciplines and industries)
  • B uilding capacity to internalize serendipitous discovery into innovative insight and action, including through networks that can marshal resources to extend innovative insights and ideas into tests and eventually full production.

Enter the Super-Encounterers

Another key finding that presents implications for organizations, innovation managers and teams is the existence of so-called “super-encounterers.” These are people who regularly and repeatedly encounter—and harvest—unexpected information, even counting on it as an important “expected” element in information acquisition. Finding, retaining, nurturing, equipping, and leveraging super-encounterers who can receive, discern, and share information is the key to establishing a culture that can produce serendipity-enabled discoveries and innovations.

One of today’s most recognized, successful and enabling super-encounterers is the big data innovation leader, Stephen Wolfram. His innovativeness using big data and analytics was behind the renowned computer language Mathematica. Wolfram also publishes one of the most leveraged computational knowledge engines, Wolfram|Alpha, which supports Apple’s Siri, Microsoft’s Bing, Facebook’s personal analytics, the CIA World Factbook and the independent search engine DuckDuckGo. Written with over 15 million lines of Mathematica code, Wolfram|Alpha has helped people make powerful, unanticipated connections across multiple databases and operations.

“The number one thing I probably contribute is making connections to other things,” Wolfram said. “As a CEO, I get different people in different parts of our company to learn about what’s happening in other parts of the company. It’s somewhat successful, but ultimately I’m usually the one who has to tell people to make this or that connection.”

Yet, fostering serendipity (or at least the potential for it) is possible throughout an organization, not just with these super-encounterers. It boils down to expanding knowledge and connections to create fertile ground for serendipitous insights. In a 2011, Forbes contributor Deborah Mills Scofield wrote that the randomness that emanates from our networks is a key element in disruptive innovation. According to Scofield, “What you know depends a lot on who you know. … If you stay within those confines, your network remains fairly constant and self-selected. … It’s when you venture outside of that circle that your network, and knowledge, starts to expand – you ‘know’ more people so you ‘learn’ more which leads to knowing more people and on and on.”

The importance of chance encounters in the workplace can be frustrated by a world where telecommuting is increasingly in vogue. Not so at Yahoo!, which banned telecommuting in 2013. Of this challenge, Greg Lindsay wrote in The New York Times about the importance of forcing collaborations among colleagues and filling corporate “structural holes.” Lindsay noted: “As Yahoo and Google see it, serendipity is largely a byproduct of social networks. Close-knit teams do well at tackling the challenges in front of them, but lack the connections to spot complementary ideas elsewhere in the company…but are hallway collisions really the best way to stoke innovation?”  

The Beauty of Sagacity

Innovations that have been enabled via serendipity have required an equally important aspect—“sagacity” (the breakthrough connection of those findings to relevant perspectives—or wisdom—to generate the truly “aha” discovery). Indeed, equally important to appreciating serendipity is finding those parties who have the sagacity to accept the insights referred by the serendipity-hunting systems and developed by the super-encounterers—especially if those encounterers are not the final say regarding an idea’s worth.

A 2012 paper from scholars at Sam Houston State University, “Leadership Sagacity and Its Relationship with Individual Creative Performance and Innovation,” affirms the importance of sagacity in leadership when guiding an idea through full innovation implementation. The authors assert that it is essential for leaders with authority for allocation of resources to have a high level of sagacity, including a high level of discernment, wisdom, and judgment necessary to decide which ideas should be championed toward innovation.

As an example of the importance of sagacity in innovation leadership, consider Ernest Duchesne, who first documented Penicillin in 1897. Duchesne’s findings, however, were rejected by the Institut Pasteur, reportedly because of his youth. It would take another 30 years before Alexander Fleming would accidentally create Penicillium mold, the first step down the road to today’s antibiotics.

There other examples of potentially serendipitous discoveries missed for lack of sagacity, essentially because people were incapable of drawing or accepting the necessary connections. Thus, a complementary challenge to spurring innovation through serendipity must involve purposefully and systematically enhancing the discoverer‘s domain knowledge to enhance the likelihood that they will be able to appreciate a serendipitous connection when they stumble across it.