How Amazon uses AI to detect damaged products before shipping to customers

At the heart of Amazon’s distribution centers across North America, millions of products of all kinds-from dog food to phone cases, T-shirts to books-pass through scanning tunnels, where an artificial intelligence (AI) model called “Project P.I.” (a.k.a. “private investigator”) uses detective tools to scan items for defects. The goal? To ensure that customers are always satisfied with every order they receive.

How Amazon’s “Private Investigator” works.

Leveraging a combination of generative AI and computer vision technologies, Project P.I. can detect defects such as damaged products or problems with color and size before the items reach customers. In addition, the system can help identify the root causes of such problems, allowing preventive measures to be taken upstream to prevent their recurrence.

At sites where the system is available, Project P.I. has proven very effective in sifting through the millions of items that pass through the tunnels each month, accurately identifying any product problems.

Before an item is shipped to the customer, it goes through a scanning tunnel, where Project P.I. uses computer vision to analyze the product and detect any defects, such as a bent book cover. If a defect is found, Amazon isolates the product so that it is not shipped to the customer and investigates further to determine if it is a larger problem affecting similar items.

Amazon employees who review items reported by Project P.I. then decide whether the item is eligible to be resold at a discount on Amazon’s Second Chance site, whether it should be donated or put to another use. The model acts as a “second pair of eyes” for Amazon employees and is already helping to improve manual inspections in several North American distribution centers. The technology is expected to expand to other sites during 2024.

Ensuring a more sustainable customer experience

Project P.I.’s work is not only part of Amazon’s customer-centric culture, but also one of the many ways the company is using AI innovation to help integrate its efforts to address climate change into the customer experience.

Preventing damaged or defective items from reaching customers is critical to a positive experience, but it is also critical to the planet. In fact, inadvertently shipping imperfect items can lead to unwanted returns, resulting in wasted packaging and unnecessary carbon emissions from additional transportation.

“Amazon is using AI to meet our sustainability commitments with the urgency required by climate change, while improving the customer experience,” says Kara Hurst, vice president of Worldwide Sustainability at Amazon. “AI is helping Amazon ensure that we are not only delighting customers with high-quality items, but we are extending this customer obsession to our sustainability work, preventing less-than-perfect items from leaving our factories and helping us avoid unnecessary carbon emissions from transportation, packaging, and other steps in the returns process.”

Preventing future mistakes

In parallel, Amazon teams are leveraging a generative AI system using a Multi-Modal LLM (MLLM) to investigate the root causes of negative customer experiences. When we learn of a customer-reported defect that we were unable to identify, we use this feedback to understand the cause and continuously improve the system.

The system first examines customer comments, then analyzes images captured by Project P.I. in distribution centers and other data sources to confirm the causes of the problem. For example, if a customer contacts Amazon because they ordered single sheets but received double ones, the system cross-references that feedback with images from the distribution center and asks questions such as “Is the product label visible in the image?” and “Does the label show single or double?”

This same technology is poised to help Amazon’s sales partners by making defect data more easily accessible. For example, if a sales partner has accidentally applied labels with the wrong size on a product, Amazon will report the problem to prevent the mistake from happening again.

More than 60 percent of sales in Amazon’s store come from independent sellers, mostly small and medium-sized businesses, who offer a wide selection of outstanding products, competitive prices, and convenience for consumers. By reducing the number of defective products shipped to customers, we are also reducing the overall number of returns. Project P.I. is a great example of our focus on improving the customer and sales partner experience.

Article source here.