Applying AI/ML to the Fatal Flaw in Online Grocery:
Injecting quality into order fulfillment

Two weeks of receiving zero bananas in my online grocery pickup is a crime in my banana bread-obsessed household. I suspected a COVID-related supply chain snafu but called the store to check. The online grocery website requires customers to select either green or yellow bananas. When one type of banana is out of stock, the surrogate grocery shoppers do not routinely substitute green bananas for yellow ones, or vice versa, because so many people bellyache when the bananas aren’t their preferred ripeness. They suggested I phone the store each week a couple of hours before my pickup to remind them about my preference for bananas of any ripeness.

Since I was motivated to get my bananas, I swallowed the low-tech ridiculousness and phoned each week. At the same time, I planned to restart post-vaccination in-person shopping to get what I wanted without these hassles.

Ask anyone who’s ordered groceries online and they’ll have their own rendition of an errant banana tale. My banana embargo is a minor annoyance compared to the early days of the pandemic when merely procuring a delivery window was a bloodsport.

So, kudos to the grocers for leveling up their online shopping apps so rapidly. Now that the pandemic is waning, grocery stores are becoming crowded again and online grocery shopping is decreasing.

Why Go Back to the Grocery Store?

There are plenty of reasons why people choose to go back to the market instead of continuing to shop online:

  1. Enjoyment of the sights and smells of the grocery store
  2. Independent people dislike paying others to shop when they can do it themselves
  3. Objection to the costs of grocery delivery or pickup
  4. Time window violations for deliveries or long wait times for pickups
  5. Frozen food arrives thawed / concerns about adequate refrigeration for perishables / damaged items
  6. Dissatisfaction with online item availability
  7. Dissatisfaction with online item substitutions
  8. Quality of the fresh food selected by a picker

In her 6/25/2021 Progressive Grocer article, A Drop-off in Pickup: New Research, Lynn Petrak cites the impending increase of in-store shopping and the decrease of online grocery shopping:

A survey from ChaseDesign shows that about half of grocery shoppers turned to online ordering for store pickup during lockdowns and restrictions, but only about half of those consumers will continue to shop that way in the future.

Before grocers throw in the towel and deconstruct the massive infrastructure they’ve just built for online shopping, it’s important to note that most of the problems cited in the list above are relatively simple operational fixes.

The One Big Fixable Problem

Paying others to choose our food and dissatisfaction with the results is an immense problem that suggests a new approach to grocery order fulfillment.

Petrak corroborates this point with data:

In-person shopping is seen as especially important in fresh foods and perishables, with nearly half of shoppers saying they won’t buy meat or seafood online for pickup and 40% indicating that they avoid dairy, produce and frozen products. Additionally more than a third (35%) say they won’t order deli or bakery via e-commerce for store pick-up.

Everyone’s different — this is the challenge of purchasing food by proxy. We each have our own preferences, superstitions, and oddities about selecting fresh food — meat, fish, deli, bakery, and produce.

For example, here are just some of my produce preferences:

  1. Broccoli that’s firm, deep green with no yellowing.
  2. Grape tomatoes that are bright red, but check the bottom to ensure that none are soft.
  3. Bananas that are 6″ and more green than yellow. If green bananas are out of stock, give me yellow ones.
  4. Cilantro with a pungent smell. If I can’t smell the cilantro, I don’t want it.
  5. Watermelon with a large yellow flat side indicating it peacefully ripened in the field instead of being picked prematurely and tossed into the back of a truck. The rest of the skin should be a slightly dull variegated green.
  6. Avocados that are soft but not too soft. Ditto pears, lemons, and limes.
  7. Red grapes must be very firm, preferably round. If oblong, the grapes must not be too flabby. The stems must be green, not brown.
  8. Blueberries should be firm, unwrinkled, and plump. Check the underside of the box to ensure all the berries are intact.

Granted, ranting about the firmness of my avocados is a decidedly first-world problem. Shouldn’t we be grateful that grocery delivery worked so well during a global pandemic? Absolutely. It’s remarkable that the industry turned on a dime and enabled consumers to receive food safely. I’m eternally thankful to the front-line workers who risked their lives on my behalf and tipped generously to show my gratitude. But the aggregation of picayune complaints like “these limes are too hard” prevents online grocery shopping from fully satisfying its customers.

For the most part, people employed to pick groceries are compensated for speed. More deliveries fulfilled = More $$$ for the grocery pickers. These employees have no patience, nor should they, for sample narratives like this:

I want four heads of broccoli crowns if they’re large and seven if they’re small. The broccoli must be a deep green with no yellow on the crowns and no brown in the stems. The stems must be firm, not limp. If the broccoli doesn’t meet these standards, check the organic broccoli and use the same criteria. If the organic broccoli isn’t up to snuff, check the broccolini. If all else fails, just grab me a head of cauliflower, but make sure it’s white and firm with no brown spots.

Most surrogate shoppers will not accommodate this level of When Harry Met Sally food exactitude. But these complicated decision trees are precisely how many of us shop.

Sally being difficult in a diner. Watch to the end to see the frustrated waitress who may or may not satisfy Sally’s wishes.

Instacart offers a more personal shopping experience, but additional fees and product markups violate the third reason why people abandon online grocery shopping — Objection to the prices of grocery delivery or pickup. Besides, educating a proxy shopper each time about one’s frustrating and complicated fresh food predilections becomes tiresome for both parties, making it simpler to DIY.

Today’s Handling of Item Substitutions

Virtually all online grocery websites currently support item substitutions when one’s first choice isn’t available. Some grocers ask customers to either allow or reject substitutions without specifying the details of the substituted item. In this case, the grocer leaves the substitution entirely up to the surrogate shopper, a recipe for customer dissatisfaction.

This grocer supports just a binary Yes/No for substitutions and the surrogate shopper chooses what they think best.

More advanced grocery website substitutions allow customers to choose a specific item. If the substitute is available, this approach provides greater customer satisfaction than the simpler binary approach reflected in the screen capture above.

More advanced grocery websites allow customers to specify the substitution.

Even the second, more detailed substitution selection is too crude because it ignores quality. The mere presence of a fresh item on a grocer’s shelf doesn’t mean a consumer necessarily wants it — the item must also meet the customer’s quality standards. Refer to my sample broccoli narrative above.

Bring on the Technology

If online grocers figure out how to heed their customers’ complicated preferences and selection criteria, busy people who can afford it will undoubtedly continue to use the service. I love in-person grocery excursions more than most, but I’d happily continue online shopping and save myself an hour in the store if the surrogate shopper does as good a job as I do myself.

Between robotics, natural language processing, and machine learning, the technology is available right now to solve this problem. Humans can certainly play a role but item selection and decision-making require technological assistance.

Avoiding Natural Language Colloquialisms with Software Design

A customer needs a way to express a decision tree for fresh food selection, much like my sample broccoli narrative. Unlike humans, computers do not get frustrated with marvelously complicated food selection instructions. However, Dr. Melanie Mitchell illustrates in her fascinating 2020 book, Artificial Intelligence: A Guide for Thinking Humans, how natural language processing mangles colloquialisms and sarcasm with the following passage:

“A man went into a restaurant and ordered a hamburger, cooked rare. When it arrived, it was burned to a crisp. The waitress stopped by the man’s table “Is the burger okay?” she asked. “Oh, it’s just great,” the man said, pushing back his chair and storming out of the restaurant without paying. The waitress yelled after him, “Hey, what about the bill?” She shrugged her shoulders, muttering under her breath, “Why is he so bent out of shape?”

French Translation (Google): Un homme est entré dans un restaurant et a commandé un hamburger, cuit à point. Quand il est arrivé, il était brûlé. La serveuse s’arrêta à la table de l’homme. “Est-ce que le hamburger va bien?” elle a demandé. “Oh, c’est génial”, dit l’homme en reculant sa chaise et en sortant du restaurant sans payer. La serveuse lui cria: “Hé, qu’en est-il de l’addition?” Elle haussa les épaules, murmurant dans un souffle. “Pourquoi est-il si déformé?”

Translation back to English: A man entered a restaurant and ordered a hamburger, cooked to perfection. When he arrived, he was burned. The waitress stopped at the man’s table. “Is the burger okay?” she asked. “Oh, that’s great,” the man said, pulling back his chair and leaving the restaurant without paying. The waitress shouted, “Hey, what about the bill?” She shrugged, murmuring in a breath. “Why is he so deformed?”

Rather than accepting free-form customer narratives, thoughtful user interface design with checkboxes and dropdown menus ensure a more bulletproof method of communicating with AI-based food selection systems. Also, selecting photos depicting acceptable and unacceptable choices provides a good visual baseline for Machine Learning models. If this sounds complicated or arduous for the customer, these are one-time preferences a user enters and associates with their profile.

Replicating Human Senses

In my produce preferences at the beginning of this article, I rely on sight to inspect the quality of broccoli, tomatoes, watermelon, blueberries, grapes, and bananas. I use touch to determine the proper ripeness of avocados, pears, lemons, and limes. I smell to assess the freshness of cilantro.

The other two senses, sound and taste, are less important in fresh food selection. There are better ways to assess melons’ ripeness than tapping on them and listening for an echo. Scofflaws may taste the grapes before buying them, but this doesn’t make it right.


Computer vision is the most advanced Machine Learning/Artificial Intelligence (ML/AI) of the technologies required for grocery picking. If a shopper identifies acceptable and unacceptable produce like the photos below, an AI can be trained against a large image data set to select these items based on the user’s stated preferences. This good/bad produce problem is a softball for the AI/ML field.

It’s easy to train an ML model to choose the dark green broccoli on the left, not the yellowed broccoli on the right. Images by Arina Krasnikova from Pexels (l) and Bridgette Lynn from Pexels (r)


Photo by sydney Rae on Unsplash

Feeling fruits and vegetables is an easy use case for tactile sensing. Robots already have tactile capabilities at this level. The Robot Report recently published a report about ML researchers building tactile-sensor robots.


Photo by Gustavo Fring from Pexels

Smell is essential in selecting fresh produce, but most surrogate human shoppers are probably too hurried to use this sense. This Forbes article (5/10/2021) provides information on the state of digital olfaction and odor analytics. Startups like Aryballe, the French odor analytics company, and California-based Aromyx use biosensors to enable digital olfaction to mimic human smell. The potential uses for this technology, like the detection of food spoilage in consumer appliances, are more ambitious than merely detecting the presence of cilantro’s odor or the scent of an orange. A digital olfactory freshness/ripeness application would likely match against a library of smells of various fruits and vegetables.

Revival of Picking Warehouses

Grocery stores aren’t arranged for efficiency. Grocery stores are arranged for inefficiency.

Behavioralists plan store layouts that force people to walk the entire store, seducing them with alluring sights and smells along the way, all but begging shoppers to deviate from their lists. Shoppers typically enter a store in a sensually arresting produce section and traverse the entire store to find the milk. Highly visible endcap displays sometimes showcase sale items but just as often house the items a store wants to move. The checkout stations often stock impulse items at childrens’ eye level.

Once grocers utilize technology and overcome shoppers’ resistance to online ordering, professional shoppers inefficiently scurrying around these immense stores filling orders may become quaint relics of the past. The injection of technology into order fulfillment and the ensuing influx of online shoppers may finally pave the way for cost-effective picking warehouses — an idea abandoned by Webvan and its ilk following the dot-com bubble. Kroger and Ocado have already partnered, operating state-of-the-art AI-based automated food warehouses. People like me who enjoy the in-person shopping experience may ultimately lose out.

Continued Innovation in Grocery Shopping

The technology already exists to vastly improve web-based grocery shopping customer satisfaction. The COVID-19 pandemic proved that grocers could nimbly pivot to support the massive influx of online shopping. Rather than blindly assuming customers are flocking back to grocery stores because they miss the brick-and-mortar experience, it’s better to explore why customers feel the need to go back. Applying innovation to fresh item selection will ultimately be profitable for grocers and delightful for consumers.