9 Ways Retailers Are Using Big Data and Hadoop 

July 26, 2016 by Alex Woodie

The big data revolution is changing how business gets done in all industries. That includes the massive retail market, which drives $2.6 trillion in business in the U.S. and employs 42 million Americans. The use of advanced analytics and predictive modeling is changing the face of retail, and helping us all get what we want, when we want it.

Here are 9 ways retailers are using big data technology to create an advantage in the retail sector.

1. Recommendation Engines

This is one of the classic use cases of big data tech in retail (albiet mostly in ecommerce settings). Based on a customer’s purchase history, and the histories of others like him, what is that customer likely to purchase next? By training machine learning models on historical data, the savvy retailer can generate accurate recommendations before the customer leaves the Web page.

“When you think about recommendations, everybody wants to beat Amazon,” says Eric Thorston,Hortonworks‘ GM for consumer products. “Love them or hate them–most retailers hate them–Amazon makes from 35 percent to 60 percent revenue uplift on recommendations, and everybody is saying, 'How can we get a piece of that?' ”

This is one of the “low hanging fruit” opportunities because it is implemented relatively easily and can have an immediate impact on revenue and profit. According to Thorston, retailers are finding they can improve revenue uplift by generating recommendations on Hadoop, instead of using standard recommendation engines.

“One customer was using a third-party recommendation engine,” he says. “They brought that in house, let the merchants and developers build up some logic, and deployed it. They raised revenue uplift to 24 percent. So you’re still not getting an Amazon-like response, but you’re getting a huge boost.”

2. Customer 360

Forrester analyst Mike Gualtieri often talks about how consumers today want the “celebrity” treatment when they engage with companies. We all want to be treated like Taylor Swift (or perhaps Kanye West) when we enter a store. We expect companies to anticipate our needs, to have the products we want on-hand, to communicate with us in real time (via social media), and to adapt to their needs as they change.

This is a tall order for any retailer to achieve, but it would be practically impossible to do without some sort of Customer 360 initiative. And considering how many customers a retailer must interact with, and how many data sets are involved with getting there, big data technology and real-time processing is critical to making that happen.

Companies in lots of industries would like to have a 360 degree view of their customers. However, few actually obtain it. In the cutthroat world of retail, where profit margins are razor thin, developing a customer 360 system may be a matter of survival.

3. Market Basket Analysis

Market basket analysis is a standard technique used by merchandisers to figure out which groups, or baskets, or products customers are more likely to purchase together. It’s a well-understood business processes, but now it’s being automated with big data.

“We’ve been doing basket analysis the old fashioned way for decades, if not hundreds of years, but now have an opportunity to add mechanization to that,” Hortonworks’ Thorston says. “We have a fast food chain that’s doing that with us. They’re able to analyze what they’re selling when something is on sale or when something is given away in a promotion.”

Platforms like Hadoop help with basket analysis by enabling a retailer to analyze more data. The capability to store many year’s worth of receipts, instead of just one or two year’s worth, for many different items can help retailers get better confidence in their analyses.

“I’m seeing basket analysis a lot,” Thorston says. “Basket analysis is big.”

4. Path to Purchase

Analyzing how a customer came to make a purchase, or the path to purchase, is another way big data tech is making a mark in retail.

(6kor3dos/Shutterstock)

While marketing executives have studied path-to-purchase techniques for many years, the advent of big data and big data tech is enabling them to get much more out of this type of analysis. The rise of multi-channel marketing and retailing and omni-channel selling is creating a large number of different paths that customers can take to buying a product.

Tools like Hadoop, Spark, and machine learning libraries can be instrumental in eliminating a “shotgun” approach to understanding customer buying patterns and instead zeroing in on exactly what works in the real world. The rise of social media (people love to talk about what they just bought) also helps because it generates much more data to work with on path-to-purchase projects.

5. Social Listening for Trend Forecasting

As a retailer, if you’re not at least listening to social media at this point—let alone actively engaging with them on Instagram or Twitter–then you’re missing out on a slew of free and potentially invaluable information that can help you spot trends.

Platforms like Hadoop were designed to facilitate the handling and analysis of large amounts of unstructured data, such as Facebook posts or Pintrest pins. The use of natural language processing (NLP) to extract information from social media, and machine learning to make sense of them, can give the social enterprise an edge over the competition.

But care must be taken in using social listening, Hortonworks’ Thorston says. “Social listening is critical for understanding the target, and it’s very important in confirming what you have,” he says. “But the minute you make a wrong move, you lose. The obligation is to use it judiciously. That prevents the misuse and that also preserves and supports and aligns to the ultimate goal, which is customer intimacy, customer loyalty, increased revenue, and increased margin.”

6. Price Optimization

Having the right price on a product can mean the difference between making a sale and losing a customer. But what is the right price? That’s the million-dollar question merchants have struggled with for millennia. But retailers who approach this problem with big data tools may have an advantage over those that don’t.

(Sergey Nivens/Shutterstock)

In many cases, setting the right price requires knowing what your competitors are changing. In the past, retailers would employ “secret shoppers” to gather this intel. That data can be collected electronically using daemons that crawl competitors’ website to get detailed info about product pricing.  “That’s not new news,” Thorston says. “But the machine learning helps you by saying ‘Here’s what we think will resonate in this market at this time for this buyer.'”

Hortonworks helped one customer by building an automated price-optimization routine that features to a Web-based dashboard interface. “When the person who manages the category shows up, she can say, ‘Yes I accept the recommendation,'” Thorston says. “It’ll go out and reach into the structural system, change the price, which will flood out to stores. Then she can launch a promotion along with that.”

7. Workforce and Energy Optimization

What’s the single largest cost for retailers? If you said “labor,” then give yourself a big red star. While it’s true that big data tech can deliver benefits on the marketing and merchandising side, it can also help big retailers optimize their spending on human capital, which can have a sizable impact on the bottom line.

Figuring out where and when to staff people can have a big impact on the bottom line, according to Alexander Gray, Ph.D., the CTO and co-founder of machine learning automation company Skytree.

“One of the engagements we’re doing is around workforce optimization,” Gray tells Datanami. “An organization discovered that if they put different amount of manpower on different parts of the store, they can actually…have a large impact on revenue.  That’s another lever that’s opened up by having more powerful data discovery capabilities.”

Retailers can also save big bucks by using big data tech to analyze their energy usage, which is another factor on the cost side of the ledger. “We’re working with a supermarket where energy costs are very high,” Gray says. “Optimization the running of those machines, as well as preventative-maintenance type problems, turns out to be a big concern for them.”

8. Inventory Optimization

Inventory optimization is a complicated thing that touches many aspects of the consumer goods supply chain, and often requires close coordination among manufacturers and distributors. But with the rise of omni-channel fulfillment, retailers are increasingly looking for ways to improve the availability of in-demand products.

“I have one customer who is fulfilling to the store saying ‘Hey I’m out in the back room. Where am I going to fulfill from? Am I going to go to the store next door, to the fulfilment center, to the distribution center, or to the vendor,” Thorston says. “That logic is driven by a Hadoop cluster.”

Time and distance matter in this equation, and so does another factor that’s not so obvious: seasonality. For example, if a retailer is trying to locate a particular dress for a customer in Florida, the retailer may have to choose between shipping the dress from their Boston or Los Angeles location.

“Are they going to ship from Boston, where maybe you’ll have two months to sell that dress? Or are you going to ship from LA, where you might have a window of 8 months,” Thorston says. “You need to make sure you plan your seasonality in the shipping algorithms.”

9. Fraud Detection

Retail fraud is a huge problem, accounting for hundreds of billions of lost dollars every year. Retailers have tried every trick in the book to stop fraud, and now they’re turning to big data tech to give them an edge.

Big data analytics can help retailers fight fraud in a number of ways. For starters, they can use predictive capabilities to create a baseline sales forecast at the SKU level. If a product deviates noticeably outside of that range, it could indicate some fishy business.

Fraud committed by employees can be tough to stop. But with the power of big data tech, internal controllers may be able to create more transparency into internal activities.

Retailers have some work to do to beef up their anti-fraud activities, according to Robert Fowlie, a partner in the Forensic practice of Deloitte Canada. “Many retail companies have significant amounts of data at their disposal, captured daily through operations,” Fowlie says in a Wall Street Journal story. But turning that data into insight through continuous monitoring and real-time feedback remains a challenge.”