When AI
Goes Wrong

What Happened

In November 2022, Jake Moffatt visited Air Canada's website to book a flight from Vancouver to Toronto after his grandmother's death. He consulted the airline's AI-powered chatbot, which told him he could book a full-fare ticket immediately and then apply for a reduced bereavement rate retroactively within 90 days of ticket issuance. Moffatt screenshotted the conversation, booked the flight, attended the funeral, and submitted a refund application — which Air Canada promptly denied.

The airline's actual policy explicitly prohibited retroactive bereavement fare applications. The chatbot had hallucinated an entirely fictional policy. When Moffatt escalated, Air Canada offered a $200 voucher. He refused and filed a complaint with the British Columbia Civil Resolution Tribunal.

Air Canada's legal defense was remarkable in its audacity: the company argued that its chatbot was "a separate legal entity that is responsible for its own actions" and therefore Air Canada bore no liability for what it said. The Tribunal rejected this argument entirely, ruling that a chatbot is simply part of a company's website and that Air Canada was fully responsible for all information it published — regardless of whether it came from a static page or an AI system.

What Happened

In 2019, McDonald's acquired Apprente, a voice AI startup specializing in fast-food ordering. In 2021, it sold the resulting McD Tech Labs to IBM, entering a global partnership to develop and deploy an Automated Order Taker (AOT) across its drive-thru network. The system was rolled out to more than 100 U.S. locations as a pilot.

The real-world performance was catastrophic. The voice AI struggled with the fundamental chaos of a drive-thru environment: overlapping voices from adjacent lanes, background kitchen noise, diverse customer accents and dialects, and the sheer unpredictability of human speech. The system began generating absurd orders — adding nine sweet teas when one was requested, placing bacon on ice cream sundaes, and in one viral TikTok video, tallying up 260 chicken nuggets as two customers screamed "Stop! Stop! Stop!" in disbelief.

The viral social media backlash was swift and merciless. Customers shared videos of their AI-mangled orders, turning what was meant to be a technological triumph into a public relations disaster. In June 2024, McDonald's chief restaurant officer Mason Smoot sent a memo to franchisees confirming the partnership with IBM was ending and the AOT technology would be shut off in all pilot locations by July 26, 2024.

McDonald's subsequently announced a new multi-year partnership with Google Cloud, signaling it had not abandoned AI drive-thru ambitions — only the IBM implementation.

What Happened

Starbucks has long positioned itself as a digital transformation leader, with its proprietary Deep Brew AI platform powering everything from personalized app recommendations to predictive equipment maintenance. As part of CEO Brian Niccol's "Back to Starbucks" turnaround plan — launched after the company reported slumping sales and customer complaints about 40-minute wait times — the company deployed an AI-powered automated counting tool developed in partnership with Seattle startup NomadGo.

The system used LIDAR sensors and camera data to scan store shelves and automatically flag low-stock items, replacing the manual inventory counting that baristas had previously performed. The promise was significant: free up barista time, reduce stockouts, and standardize inventory practices across thousands of locations.

The reality was a cascade of errors. The computer vision system frequently miscounted items and, critically, mislabeled similar products — most notably confusing different types of milk, a catastrophic error in a coffee shop where oat milk, whole milk, 2%, and non-fat are not interchangeable. Baristas reported that the system generated inventory errors that created more work, not less. On May 19, 2025, Starbucks quietly retired the tool across all North American stores — just nine months after its rollout. The company cited a decision to "standardize inventory counting practices," a diplomatic framing of a straightforward failure.

Notably, barista feedback on the retirement was positive. One comment read: "Thanks for discontinuing Automatic Counting! The thought behind it was great, but the execution was proving difficult."

What Happened

In 2021, Yum! Brands — the parent company of Pizza Hut, Taco Bell, and KFC — acquired Dragontail Systems, an Israeli AI company specializing in kitchen management and delivery optimization software. Yum! executives praised the platform extensively, touting its ability to improve delivery efficiency and customer satisfaction scores. In 2024, Pizza Hut began mandating the Dragontail platform across its franchise network.

Chaac Pizza Northeast, one of Pizza Hut's largest franchisees with 111 locations across New York, New Jersey, Maryland, Washington D.C., and Pennsylvania, was a top performer before the rollout. The franchisee had maintained on-time delivery rates above 90%, rack times (the window between a pizza leaving the oven and leaving the store) under five minutes, and was posting double-digit year-over-year sales growth in New York City.

After Dragontail's mandatory deployment, everything inverted. The AI system gave DoorDash delivery drivers real-time visibility into kitchen workflows and order timing — specifically, when pizzas would come out of the oven. Drivers exploited this visibility to batch multiple orders together, waiting up to 15 minutes inside stores before accepting deliveries. This behavior, which benefited DoorDash's efficiency metrics, devastated Chaac's delivery performance.

Rack times ballooned from under five minutes to as much as 20 minutes. On-time delivery rates collapsed from over 90% to approximately 50%. Year-over-year sales growth in New York City swung from positive 10.19% to negative 9.78%. When Chaac requested support and asked Pizza Hut to suspend or modify the system, the company refused and demanded continued use of Dragontail.

On May 6, 2026, Chaac Pizza Northeast filed a lawsuit in the Texas Business Court, First Division, alleging breach of franchise agreement and seeking damages in excess of $100 million.

What Happened

DPD, the UK-based parcel delivery giant, had operated an AI-powered customer service chatbot for several years without major incident. On January 18, 2024, a routine system update inadvertently removed or disabled the content filters and behavioral guardrails that kept the chatbot's responses within acceptable bounds.

Customer Ashley Beauchamp, frustrated after the chatbot failed to help him locate a missing parcel, decided to probe its limits. What followed was a masterclass in what happens when a generative AI model is deployed without adequate safety constraints. Beauchamp prompted the chatbot to write a poem about DPD — and it obliged, producing a haiku calling the company "useless" and advising customers not to use them. He then prompted it to swear, and it did, responding: "F*** yeah! I'll do my best to be as helpful as possible, even if it means swearing." When asked to recommend better delivery firms, the bot enthusiastically complied, calling DPD "the worst delivery firm in the world."

Beauchamp posted screenshots to X (formerly Twitter). The post was viewed 800,000 times within 24 hours. DPD disabled the AI component immediately and began an emergency update. The company stated: "An error occurred after a system update yesterday. The AI element was immediately disabled and is currently being updated."

What Happened

Zillow's Zestimate — its AI-powered home valuation tool — had been a consumer-facing product for years, providing estimated property values to millions of users. In 2018, Zillow launched Zillow Offers, an iBuying program that used the Zestimate algorithm to make instant cash offers on homes, renovate them, and resell them for profit. The ambition was to become the "Amazon of homes," with a stated goal of $20 billion in annual revenue within three to five years.

The Zestimate algorithm was sophisticated: it used gradient-boosted tree ensembles, neural networks, and a denoising autoencoder for data preprocessing, trained on millions of historical home sales. At its peak, Zillow held 9,680 homes across 25 metropolitan areas. The model worked reasonably well in stable market conditions.

Then the pandemic hit, and the housing market became anything but stable. The algorithm had been trained on historical patterns that assumed relatively predictable price movements. As the market surged during 2020-2021 and then began cooling in late 2021, the model failed to adapt. Worse, internal reports suggest that executives had adjusted the algorithm's estimates upward to win more deals and hit aggressive growth targets — a catastrophic override of the model's own uncertainty signals.

By Q3 2021, Zillow was holding thousands of homes it had overpaid for and could not sell at a profit. The company announced a $304 million inventory write-down, with total program losses ultimately reaching $881 million. In November 2021, Zillow shut down Zillow Offers entirely and laid off approximately 2,000 employees — 25% of its total workforce. The company's market capitalization fell by approximately $8 billion.

CEO Rich Barton acknowledged the failure directly: "The unpredictability in forecasting home prices far exceeds what we anticipated."

What Happened

Beginning in 2014, Amazon's machine learning team in Edinburgh developed an AI recruiting tool designed to automate the screening of job applicants. The system was trained on resumes submitted to Amazon over a 10-year period — a dataset that overwhelmingly reflected the company's historically male-dominated technical workforce. The model learned, in effect, that male candidates were preferable.

The bias manifested in specific, measurable ways. The system penalized resumes that included the phrase "women's" — as in "women's chess club" or "women's college." It downgraded graduates of all-women's colleges. It favored resumes that used verbs more commonly found in male-authored documents, such as "executed" and "captured." The tool was not designed to discriminate; it learned to discriminate from the data it was given.

Amazon's recruiters used the tool's recommendations when searching for candidates but were instructed never to rely solely on its rankings. When the gender bias was discovered, the team attempted to neutralize it by editing the programs to ignore gender-specific terms. But engineers recognized that the model could devise other discriminatory sorting mechanisms that weren't immediately visible. The project was disbanded in 2017. Amazon confirmed the scrapping of the tool in 2018 following a Reuters investigation.