Data-Driven Innovation in Retail

September 22, 2015
Managing Partner, Made by Many


We understand DDI is good for our customers and our business, but how do we actually enable it?


This essay is part of a series of articles relating to the Internet of Everything project. Read more at

By Leslie Bradshaw

No one would disagree that data allows retailers to flexibly and quickly convert and retain consumers. Nor would the same audience disagree that there are powerful benefits for the consumer like highly personalized recommendations and tailored deals for the right products at the right moments in their lives. But the practicality of enabling data-driven innovation (DDI) is much harder than it looks.

Smart partnerships, more open systems, and leveraging not just the tech trends du jour, but the right tech trends for your business and your customers are critical for data-driven innovation to help realize a retailer’s full potential of better serving their customers and increasing their revenues.


The future of retail is using data to inform, connect, optimize, and continually innovate the way an organization services its customers. It’s as simple and as complicated as that.

The simplicity lies in this equation: retailers who create, maintain, and evolve the connective tissue between their consumer, their site, and their physical stores will win big. These same winning retailers know how to gather and leverage the input from their consumer, their site, and their physical stores to offer better products, produced in smarter ways, at more competitive prices. This in turn leads to more customer satisfaction, lower acquisition and retention costs, and ultimately increased revenues. Nothing a retailer wouldn’t want. Nothing we haven’t all read before. And nothing controversial.

However, the hard part lies in what it takes to evolve the systems—from mindset to methodology to actual technological systems—that make these relationships, insights, and feedback loops possible. Even if CEOs at every major retailer said to their teams, “I want us to drive more innovation through data,” there is inertia and status quo holding the organization hostage. At the same time, the steps that move an organization from Traditionalist, to Venturer, to Pilgrim, to (finally) Data-Driven Organization are linear and build on one another. This puts a slowed pace to the otherwise “future now” ambitions of the organization.

To complicate things further, while a growing number of data-driven startups are taking on some of the top issues in retail—from the software that runs the machines at the checkout, to the systems that help maintain an accurate picture of each customer’s history and preferences, to the systems that connect how, what, when, and where we buy—these upstarts are either seen as or in reality are: (1) not stable enough to provide reliability at scale; (2) too modern to integrate into legacy code bases; and/or (3) threaten the investments made to date into large IT buildouts.

We can host events and write papers ad infinitum about the Minority Report’ization of retail, but until we solve the conundrum of “we want a more data-driven future now but are stuck with the technology equivalent of Stone Age tools,” we will not fully realize the potential of data-driven innovation. Said another way: the near future of retail is less about innovating the interfaces through which consumers experience commerce and more about the radical innovation the underlying systems to these consumer experiences desperately need.

So what would it look like to put the right tools and methods in place to achieve uber-personalized and highly efficient outcomes for retailers and their customers? And how can we get there not just theoretically but practically? And what about all of the roadblocks and hurdles long-standing retailers are facing? They aren’t going away anytime soon; the investments are too large and the inertia too heavy.

The answers to all of these questions will be the result of combining three activities: (1) knowing when, how, and with whom to partner—and then doing it consistently and well; (2) creating secure “doors” in and out of your systems where data can flow and external technologies can plug-in their capabilities so you aren’t a closed, isolated system; and (3) leveraging the right macro technology trends that make sense for your business and your customers.  

A Retail Innovator’s Dilemma

Context: This is a real-world case study from a Fortune 500 business. It highlights the difficulties retailers face when they try to better serve their customers through data-driven innovations, like personalization and recommendation algorithms. In my 10 years of working in digital innovation, I have found that every business leader faces these challenges in one form or another and will continue to do so until their methods and tools are upgraded. 

Consider a story from Priyanka, one retail company’s digital leader. [Name changed to protect the confidentiality of the employee, company, and project.] Priyanka explained that her company’s systems weren’t able to sync up data from a customer’s in-store credit card purchase history and data from that same customer’s online purchase and browsing history. She also lamented that the in-store experience was poorly tracked and understood, leaving the company “blind and dumb” because it only provided insights if a credit card was used and if camera data was analyzed (which it wasn’t). How many people came into the store, why they came, why they did or didn’t buy something after they tried it on, and if they paid with any method other than credit card are all data points that were never collected.

And that says nothing of what the consumer wants and needs out of the situation. When asked, Priyanka’s core customers wanted three things: less lines, better deals, and a heads up when something they would truly like has just been released (or went on sale). I emphasize the truly because I’ve also worked with clients on algorithms that were more about featuring “bumper crop” inventory than serving up what the end customer truly would like.

To make matters worse, despite the fact that Priyanka’s company collected millions of e-mail addresses from its customers over the years, they weren’t able to match it with any on- or offline purchase history or preferences. This further hindered her ability to meaningfully segment her company’s customers in a way that allowed her to deliver personalized messaging and offers—not to mention learn about things like product trends that could influence “fast fashion” decisions at the production level.

Let’s take stock for a moment of the data and tools Priyanka had—and didn’t have—at her disposal:

●      Insufficient collection tools

  • Unable to collect in-store behavior data.

●      Insufficient connection tools

  • Unable to map meaningful relationships between in-store, online, and e-mail.
  • Unable to create personalized, omnichannel experiences for consumers

●      Sufficient collection tools

  • Able to collect in-store credit card purchase data
  • Able to collect online purchase history data
  • Able to collect online browsing history data – on-domain
  • Able to collect online browsing history data - cookie’d

●      Sufficient connection tools

  • Able to map meaningful relationships between various online behavior data (e.g., purchase history, on-domain browsing history, off-domain browsing history.

With this as a starting point, Priyanka pursued a variety of initiatives, including building and implementing a loyalty app where she and her team had hoped to put data to work in the name of customer service, inventory management, and yes, of course, revenue goals. However, in addition to the challenges already enumerated, Priyanka had no way of dynamically accessing the organization’s vast catalog of product shots she envisioned serving up to these same customers in customized ways through a yet-to-be-developed loyalty app. And if she couldn’t match any pre-existing insights with registrants of the app (which knowing what we know from the above, she couldn’t), at least she could track preferences once a customer created a login…right? Wrong.

As for said loyalty app, while Priyanka spent six months iteratively learning from customers, testing low-fidelity prototypes, and validating value propositions with a “side projects” team, when it came time to build, she found that working with deeply ingrained systems was like working with an curmudgeon board at City Hall: you can’t fight it as an upstart and win without more air cover and more capital. This introduced an additional set of collection and connection insufficiencies into the mix.

●      Sufficient innovation tools

  • Able to identify, prototype, and validate value-driving propositions

●      Insufficient implementation tools

  • Unable to port catalog content through an API
  • Unable to integrate innovative digital products into proprietary IT

The outcome of the project? The loyalty app was never created. A deeper relationship with customers was not established. These same customers never received the kind of geolocation, temporal, or product preference information they were asking for—and getting from tangential brands like Netflix, Amazon, and Rent the Runway.

Instead, a few additional features were bolted onto an existing mobile optimized website but done so in such way that they were not easy to find.

To paraphrase the old adage: if a new feature is added into an experience and no one knows about it, is it even useful?

This move also violated the sine qua non of software development: good user experience is not about more features; it is about more focused features. Users don’t want all the things to sort of do some things well. They want a few killer things to perfectly solve their one or two main problems. Furthermore, the point of sale system is slated to be upgraded in the next two years and integrated with CRM data. But unlike the personalization promise heralded by clickbait headlines, the Holy Grail of data-driven innovation remains elusive for this retailer. And thousands more like them.

A Retail Innovator’s 3-Pronged Solution

Priyanka’s unfortunate tale is more the rule than the exception. For anyone who has faced similar challenges, here are three achievable solutions to consider.

1) The future belongs to those who partner best

The world has reached a point of complexity that without a war chest like Apple’s, doing everything in-house is near impossible (and even that technology behemoth depends on a cadre of external partners to design and produce their wares). What’s more, there are also upstarts and boutiques who have made it their sole focus to be the best in the world at one thing, which, let’s face it, you can never match as long as you have dozens of priorities. With these two bookends in mind, partnering has never been more important and more possible than in this moment

Let’s look at a few examples of successful pairings that are advancing a retailer’s ability to drive innovation and deliver better, faster, and stronger for their customers.

One way Macy’s is infusing innovation into their organization is through their quarterly request for proposal (RFP) process. The RFPs pose a new retail-related challenge every three months to relevant startups. From social shopping to in-store behavior tracking, they’ve been able to learn from external partners while also providing real-world scale and scenarios for startups that are still trying to crack their product market fit. How might you take the biggest challenges you are facing and turn them into open-ended briefs for startups to respond to, present on, and possibly even execute against?

Another way to bring innovation into the fold is to buy the upstarts doing it better than you are. For example, retail giant Bed, Bath & Beyond (BB&B) recently announced their acquisition of indie darling Of a Kind, a beloved brand that “sells the pieces and tells the stories of emerging fashion, home, and jewelry designers.” Market research informed BB&B that the next generation of shoppers isn’t looking for cookie-cutter items that all match but rather is interested in the back stories, ethics, and exclusivity of what it buys—and Of a Kind delivers this model in spades.

Finally, an unexpected partner to bring in earlier and at a deeper level is “Big IT.” Too often, innovation projects are developed in isolation and without consideration of what’s possible not just on current IT systems but what they could be in the future. Major IT providers like SAP and Oracle are working hard to innovate, and the more real-world requests they get from important clients, the more likely they are to prioritize the right feature upgrades and systems revamps.

2) Build for and with Application Programming Interfaces (APIs)

Historically, retail systems have at best been connected by proprietary protocols—and at worst remained unconnected and in siloes.

To overcome these challenges, more progressive technologists and software companies are turning to open systems that allow for others to leverage and share tools, data, and protocols. The entry points to these systems are called application programming interfaces (APIs) and act just as the name implies: they allow for an interface from which (and to which) programming and building applications can occur. In turn, innovation cycles are sped up and the work is distributed across thousands—and even sometimes millions—of other minds, computers, and talent types.

Consider again the example of Priyanka’s company. One of the most effective things her organization could do is build APIs and algorithms for each dataset that enable logic-based requests to recall and relate information. In addition to having APIs and algorithms for things like customer size preferences and purchase history, they are great tools for visual content-heavy product catalogs and corresponding pricing, sizing, and availability data.

APIs are also essential for accurate inventory availability data and competition. Highly expectant consumers who shop in a hyper-competitive retail environment expect the items they want are in stock—and if they are not, then they have no problem going to another site or possibly even voiding the purchase altogether. In light of these needs, consider again the limitations of siloed systems. With data stuck in separate “vessels,” it is hard to reconcile duplicative data (e.g., the same e-mail address being tied to multiple purchase mediums like online, through partner retailers online, and in-store at multiple locations). With siloed systems, it is also hard to deduce any patterns or make any connections because you don’t have all the data combined in one holistic picture (e.g., similarities between when and where certain customers buy which type of products). This is why APIs are so critical. They marry data, they flag and clean up duplicates, and they ultimately give you a 360-degree view of your data and your customer.

A related concept to building APIs within your own systems is building APIs into which outside developers can tap. At first glance, this might feel like creating a security nightmare, but when done properly, a well-documented API actually encourages innovation from external sources. It is one of the single biggest reasons why the early Twitter ecosystem exploded: their open, well-documented API allowed developers to collect and analyze data, mash its data up with other sources, and build applications that displayed data in interesting ways. APIs also accelerate the ability of partner systems (such as inventory and fulfillment) to gain visibility into purchasing patterns and supply chain demands.

3) Understand and leverage macro trends in technology

One of the greatest things a retailer can do to innovate actually has little to do with invention and everything to do with overlaying a comparative advantage with the right macro technology trend. These trends are being driven by advances in technology, advances in devices, evolution in design techniques, and new interface experiences.

Customers are expecting to have transparency in time and distance, while also having full control making requests and contacting a real person to aid them. This is something seen in the fast-growing business, Uber. If this is what consumers are coming to expect when it comes to travel, what is the Uber equivalent for the retail business model?

Is it about tracking an order beyond just the delivery point and instead seeing its actual position on a map? Or is it about the customer having a more direct route to the delivery truck or store that has their item in their site—not just some call center thousands of miles away? Or some of all of the above?

What about the Netflix-i-zation of content? Their model has evolved from DVD subscriptions to streaming content subscriptions—and now they are even in the original content game. They’ve not only set the bar for all other media companies—which they have—but they are also able to make smart original content investments based on the massive dataset they’ve collected on user preferences, behaviors, and trends. What is the Netflix equivalent for the retailer? Is it about vertically integrating and cutting out distributors? Or is it about powerful recommendation engines? Or some of all of the above?


Top 6 Tech Trends for Data-Driven Retail

To further stir your thinking, here are six macro technology trends, examples of top players leveraging the trend, and a snapshot of the role innovation and data are playing with each.

1 - Enabling physical retail stores to simultaneously serve as warehouses and store fronts

●      Why: Net shipping costs are outpacing shipping fees for many online-only retailers, chief among them the powerhouse of all powerhouses, Amazon. In 2014, they charged $3.1B in shipping fees and spent $6.6B in transportation costs to deliver. Macy's doesn't have Amazon's transportation cost problem. In addition to having smartly invested $2B on technology and e-commerce, their physical stores are acting as "ship to store" and "online in-store pickup" locations while still serving as good-old-fashioned storefronts, earning revenue-per-square foot, too. To decrease their transportation costs, Amazon needs to make a brick and mortar acquisition. NYU Professor and L2 Digital founder Scott Galloway suggests RadioShack, a gas station chain, or the U.S. Post Office.

●      Who: Any retailer who has only click, only brick, or click and brick models.

●      Data’s role: Understanding the final mile of shipping is sine qua non in retail fulfilment. There are so many data points tracked here from utilization, to routes, to pounds of inventory, to what’s selling best and where.

2 - The rise of subscription and bundled services

●      Why: Sign up once and get what you need at the frequency you need it. It also helps company’s project and plan for fixed revenues, increase the lifetime value of a customer, and cut down on friction and transaction costs for both the buyer and the seller.

●      Who: Amazon Prime,, Walmart Subscription, Trunk Club, Birchbox, Sephora, Blue Apron

●      Data’s role: For consumer product good (CPG) companies, data reveals which purchases are made with some regularity and what that timeframe looks like. For example, women buying mascara know the “safe zone” is getting a new one every 90 days to avoid diseases like pink eye. Recognizing this, Lancome offers a 15% discount to sign up for its subscription service. What do you offer that needs to be regularly replenished? You have an opportunity to track habits, purchasing patterns, and external factors (like weather) to determine what can be offered in a bundle and/or as a subscription—from highly disposable items like food to staple items in our wardrobe that wear over time.

3 - Groceries, content, and chores delivered immediately and at scale through an app

●      Why: These are the ultimate enablers of acting on impulses, having near-instant convenience (Zeel will send a masseuse to you within two hours!) and saving the time-and-hassle normally associated with securing things like groceries, a movie rental, or even disrobing outside of your home for a massage.

●      Who: Freshdirect, Netflix, Zeel,, Seamless

●      Data’s role: While these companies may seem like they are simply providing a good (like Freshdirect does for groceries) or a service (like in-home concierge service Alfred does for busy professionals)—they are all actually data companies. They are collecting every possible bit of information about your demographics and what people like you want, when you want it, and how often you want it. The more data they have, the better they can target and recommend options to you, which as we know lowers cost of acquisition of a customer thanks to referrals, retention, and lower churn rates. It creates deeper loyalty to the brand, and as a result, the retailer enjoys a higher lifetime value per customer.

4 - The rise of niche and aggregate marketplaces

●      Why: Googling something is no longer enough—the sheer amount of information online is overwhelming. Niche sites are bringing order to this chaos by surfacing deals, aggregating luxury items, and offering unique items in one place. It is also providing an unprecedented opportunity for independent artists, namely through sites like Of a Kind and Etsy, to produce in small volume and still make a living.

●      Who: Etsy, 1stdibs, Shoply, Bib & Tuck, ArtFire, Zibbet,, Overstock, One Kings Lane, Gilt Groupe

Data’s role: Never before have retargeting and recommendations been more powerful. Smart retailers in this space are using data to determine what people are searching for and posting in social media—and then targeting them with the right item and right deal at the right time. Data is also being used to understand and optimize for the right keywords and content placement for high-ranking results in search engines (namely Google). This is allowing smaller niche sites and larger aggregate marketplaces that are not Amazon to become known for being the “ultimate destination” for a particular good (e.g., the way Overstock has done with furniture).

5 - The rise of social shopping and data-driven fashion production

●      What: Peers and role models have always played a role in determining what’s hot (or not) each season. And brands are perpetually playing catch up with design and production cycles that lag behind the latest trends. This phenomenon is now on steroids thanks to technology and data. Technology-enabled platforms like Instagram and Tumblr are decentralizing the power of advertising, as these outlets are the place for models, cool hunters, and tastemakers to share their of-the-moment styles. In turn, magazines, editorial columns, and out-of-home ad buys have less and less pull. They are still a part of the media mix that influence a purchase, but they are now one of a dozen touchpoints—not the only one.

●      Why: There is more interest in following and buying what models, celebrities and social icons are promoting than there is with traditional mediums like fashion magazines and ads. Much like the rise of blogs decentralized and disrupted news, Instagram and Pinterest have decentralized and disrupted the fashion industry—and by proxy, the retailers who market their wares.

●      Who: Instagram, Pinterest, Tumblr, “hover states” within reputable editorial sources

●      Data’s role: There are two important roles data is playing—one at the initial “which styles to produce phase” and one at the post-production “how to market, drive preference, and purchase” phase.

  • Regarding the production phase, retailers like Adidas are making investments in major fashion hubs like Brooklyn to embed their teams with the teenagers driving new looks. Not only does osmosis come into effect, but they are able to study adjacent purchasing behaviors and related data to help them determine what they should be producing. Zara has traditionally done a phenomenal job of shortening the cycle from street trend to production—also known as “fast fashion”—by closely monitoring trends with the help of outside companies.
  •  As for the marketing phase, a dream scenario is unfolding for retailers in which a customer’s organic passions for certain items, colors, and styles can instantly be satisfied with digital clicking and one-touch impulse buying online and through their phones. Love what you are pinning on Pinterest, don’t just know where to go to buy it, have a hover state that allows for immediate conversion. This is what the second screen experience has been trying to achieve between tablet, phone and TV but has yet to cut down on the steps between see it, love it, buy it.

6 - Enabling short-term consumption and one-off ownership

●      Why: There are a few interesting drivers of this trend. Some relate to technology advances, such as more advanced geo-positioning systems, ability to make micro-payments, lower cost and higher quality cellular bandwidth technology, and the ubiquity of free Wi-Fi. Other drivers have more to do with shifts in mindset; for example, studies show that millennials have a new definition of success that values experience over possession—and other generations are starting to make the shifts, too. Whether it is because the technology is now available or because of human factors like the financial crisis of 2008 turning us off the idea of going into debt to own a lot of stuff, the fact remains: there will only be more and more instances of this type of consumption. I know I am personally looking forward to the day that I can rent an entire season’s worth of on-trend clothing—and return it three months later.

●      Who: Rent the Runway, Bag Borrow or Steal, iTunes, Uber, Lumoid, Hulu, Audiobooks

●      Data’s role: In addition to basic human psychology driving a preference for experiencing over owning, the Internet age has produced accessibility, impermanence, and rapid change like no other moment in time. Valuing experiences means we don’t want just one thing; we want a little bit of everything. Businesses launching or adapting to this are smart to do so. And there is plenty of data to help light the way.

Conclusion: Time to Turn it Into Action

Let’s do a quick back of the napkin exercise to start bringing these concepts together. Go ahead and take out a pen and draw three separate circles, filling them in with these contents:

●      One should be labeled underserved customer need and under that label list at least 2 to 3 customer insights you’ve gathered from the front lines of talking with customers (and not just having them passively fill out surveys). Is waiting in line an issue? Special sizing a problem? Too many choices and not enough guidance on what to buy? Dig in, shop along, and record.

●      The next circle should read our comparative advantage, under which you list off the 3 to 5 things you are known for and/or have special systems in place that others don’t. Perhaps it’s about your loyalty program or your supply chain or maybe your partnerships and endorsements. Think: what do you have that would be extremely hard for a startup in a garage in Silicon Valley to replicate in less than a year.

●      And the last one should say macro technology trend, which is where you examine the six items listed above and pick 1 or 2 that really fit your business. The best way to determine this is to look at the best in class “Who” examples and try to find a similar company to yours—even one of your competitors.

Now that you have these three circles, it’s time to make a Venn diagram. There will be a point in the middle where all three overlap—what customers need but don’t yet (fully) have; what you uniquely provide and/or have access to; and the larger trends in technology that are driving customer behaviors. This is where you explore deeper with your team and your partners.

And as we’ve learned, it’s one thing to have grand ideas about how to use data and technology to better serve the end customers, and it’s quite another to actually implement these innovations. That’s why you are going to write down the 3 to 5 most important people at your company who have authority in the areas of IT, digital/mobile/social, and customer relationship management (CRM). Get to know them, the technologies they use and are exploring, and find ways to build social capital and excitement with them before you have a big data-driven innovation ask.

Next, it’s time to look at who the right external partners might be. You will hopefully begin to populate a list from the meetings you take with your tech colleagues and to it, add companies from and that are in and around the space you occupy. Also keep your eye on non-establishment trend-watching retail blogs like and They are great at calling attention to killer apps and upstart retail companies that would make for great partners or even acquisition targets.

Finally, if you already haven’t de-siloed your data, the time has come to put APIs in place that will enable innovation from the inside out and the outside in. The human relationships you forge with internal partners will ideally pave the way for this to be a reality, and the relationships you forge with external startups will provide you with “beta” partners who can leverage your data in interesting, value-driving ways. It won’t be easy, but it will be worth it. Winning the future of retail requires the right partnerships, the right systems, the right data, and the right methods. Let’s replace those “Flintstone” tools with the “Jetsons” ones we know we need to get there.

And above all, let’s remember why we are doing this in the first place: better knowing and serving the end customer. Their lives are increasingly complex, and they are turning to technology to make sense of it. The smart companies are the ones shifting their offerings and modalities to meet the underserved needs of the modern consumer. And the less smart companies are hoping that if they close their eyes and double down on their existing infrastructure, it will all be over soon like some sort of bad dream. Because in the end, what’s good for the customer has more times than not proven to be good for the business, too.

Leslie Bradshaw is managing partner of Made by Many, an accelerator that helps businesses identify market opportunities and bring digital products to life. 

The views expressed here are those of the author and do not necessarily reflect those of the U.S. Chamber of Commerce Foundation, U.S. Chamber of Commerce, or their affiliates.