Steve Commisso

Case study · Amazon

Modernizing Amazon Grocery's ML forecasting


Three years on the platform that forecasts for over 600 stores: Whole Foods Market, Amazon Fresh and dark stores for delivery. Spark ETL cost fell 90.5% from its peak, labor-forecast runtime fell 81%, people could trust the tests again, and the item-level pipeline went from a daily manual upload to an automated, daily ensemble. In October 2024, ML forecasting for Amazon's US Same-Day Delivery network launched on the same platform.

Role
Owner of the three-year architecture plan, hands-on across 60+ repositories
When
2021 – 2024
  • −90.5%Spark ETL cost, from peak
  • −81%labor-forecast runtime
  • −42%item-level inference cost
  • 60+repositories worked in

Problem


Amazon's grocery businesses plan from forecasts at two grains, for over 600 stores: Whole Foods Market, Amazon Fresh and dark stores for delivery. Labor and capacity forecasts say how much volume each site will handle, so it can be staffed. Item-level forecasts say how much of each item each site will sell, so in-stock managers and buying systems order the right amount.

Both ran on an in-house forecasting platform that had been built quickly, on technology that was new to the team: Spark on AWS Glue, Step Functions and SageMaker. When I joined in September 2021 it was showing the strain in three places.

  • Cost

    Spend on the Glue data pipelines was growing 39% a month, and more businesses and regions were about to be onboarded. The prevailing view was that Glue itself was inefficient and had to come out of the codebase first. That meant a rewrite.

  • Tests

    The inference integration tests took three to four hours and passed about one time in ten, mostly because of timeouts. People re-ran them or overrode the pipeline. Without someone babysitting it, a build took roughly a week to reach production.

  • A manual last mile

    The item-level forecast landed as files in storage. Every day, in-stock managers downloaded them, added their overrides, and uploaded them by hand to the planning portal.

None of this was a knock on the team. It is what shipping a product quickly on unfamiliar technology looks like before the optimization pass.

Constraints


  • No pause button

    The platform had to keep producing forecasts every day for live businesses while it was being fixed.

  • New to Amazon, new to AWS

    I was new to the team and new to AWS. I had to earn trust quickly with the people who owned the code.

  • No API on the planning side

    Until the planning organization shipped an intake platform in 2023, its portal accepted manual uploads only.

  • Models owned by scientists

    The Bayesian models belonged to the science team. Engineering changes had to be model-agnostic: memory, data feed, cluster sizing, partitioning.

Approach & architecture


I worked the same loop on every problem: measure it, identify the bottleneck, audit and update the code, validate with the same instruments, and repeat for the next problem area. Then share what was learned, through talks, wikis and shared libraries, so the fix spreads beyond one team.

The method

  1. Measure

    Where is the money going?

  2. Identify

    Where is the time going?

  3. Audit

    Read the code against known best practices, and update it.

  4. Validate

    The same instruments, after the change.

  5. Repeat

    The next problem area. Then share what was learned.

The Spark bill

The cost kept climbing as the tables grew, and that was the clue. Our jobs filtered data with Scala lambdas. Spark cannot see inside a lambda, so it cannot push the filter down to storage. Every run read the whole table, which grew every day, and filtered it on the driver while the executors we were paying for sat idle.

One filter, written two ways

Before: a Scala lambda

val today = spark
  .table("example_db.daily_events")
  .as[Event]
  .filter(event => event.eventDate == runDate)

Spark cannot see inside the lambda, so it reads every partition and filters afterwards.

12.7 billion rows scanned

After: a column expression

val today = spark
  .table("example_db.daily_events")
  .filter(col("event_date") === lit(runDate))
  .as[Event]

Spark pushes the filter down and reads only the partition it needs.

47.7 million rows scanned

A generic reconstruction. The table and column names are invented.

On one job that change took the scan from 12.7 billion rows to 47.7 million, the run from 23 minutes to 46 seconds, and the cost down 99.6%. The rest was the same kind of work.

Data-pipeline spend, 2021

Stylised curve of data-pipeline spend, indexed to its peak. It rises steadily from May 2021 to a peak in mid-August, holds through September, drops sharply after fixes were deployed in late October, and is flat and low by late November. Spend ended 90.5% below the peak and stayed flat for the following year.

Illustrative shape, indexed to peak. Not to scale.

Glue was fine, and we did not rewrite anything. Spend on those pipelines fell 90.5% from its peak and stayed flat for the following year.

Tests people could trust

The integration tests were failing for one reason. Six parallel Glue jobs, across eight parallel test runs, were all repairing the same table's partitions at the same moment and hitting storage rate limits. I replaced the blanket repair with targeted partition adds, added shared helpers, and took the repair helper out of the shared library so it could not be misused again.

25 → 8minutes from build to deploy

Tuning how Spark ran inside unit tests cut their runtime by 58% and took build-to-deploy from 25 minutes to 8.

Labor-forecast inference

I profiled the labor-forecast inference workflows one at a time, removed storage contention between their steps and fixed the parallelism in pre-processing. Runtime fell 81%, transitive test failures went from more than 90% to about zero, and infrastructure spend for that pipeline fell 51%.

The item-level pipeline, year by year

The item-level forecasting pipeline changed the most, over three years. Its skeleton never moved: sources at the top, a feature snapshot store, the models, a publish step, the planning portal, a distribution service, and the buying systems at the bottom. What changed was everything around that skeleton.

Most of what the diagram shows was built by several teams and a science group. My parts were the three-year architecture plan, and hands-on work along the path: in 2022, tuning the steady-state model's jobs so it could scale from pilot to every site and item pair, automating the processing steps someone had been doing by hand, and building a stopgap upload bridge. In 2023, extending anomaly detection to item-level forecasts and scaling inference.

Item-level forecasting, 2022 to 2024

  1. 2022: Pilot and paperwork

    Two models, one of them still a pilot, and a person in the middle of the path.

    • Two models: a cold-start model for new items and stores, run every two weeks, and a Bayesian steady-state model in a limited pilot covering only some site and item pairs.
    • The forecast is published as files. Every day an in-stock manager downloads them, adds overrides and uploads them by hand to the planning portal.
  2. 2023: Automate the last mile, add a guardrail

    The hand-off becomes a feed, and the pipeline starts checking itself.

    • Removed: the manual hand-off. Forecasts publish daily through an automated feed to a forecast data lake and to the planning platform's model intake. In-stock managers now enter overrides in the portal.
    • Changed: the Bayesian model is out of pilot and runs daily for every site and item pair, beside the cold-start model.
    • New: anomaly detection compares each forecast with the one before it.
  3. 2024: Richer signals and an ensemble

    More inputs, and more than one model per forecast.

    • New signals: planogram (shelf layout), causal events, promotions, and an in-stock flag derived from inventory.
    • New: an ensemble. The Bayesian model, for high-velocity items, and a LightGBM model, for low-velocity items, run in parallel and are blended into one forecast.
    • New: the pipeline also serves Same-Day grocery.

Redrawn with generic component names. Colour marks what changed each year.

In May 2023 I scaled item-level inference to Whole Foods Market, ten times the size of Amazon Fresh, with changes that did not depend on the model: less memory per run and a better data feed into it. The cost of a run fell 42% and its wall-clock time 14%.

From platform to product

The last step was reuse. In 2023 I analyzed, and in 2024 prototyped, running demand planning for Amazon's US Same-Day Delivery on the grocery forecasting platform instead of building a second one. Same-Day had no historical data store, only finance spreadsheets, so I reverse-engineered those to backtest the candidate models. The team that owns Same-Day planning took the prototype to production, and their forecast launched in October 2024.

Results


  • −90.5%

    Spark ETL cost, from peak

  • −81%

    labor-forecast runtime

  • −51%

    labor-forecast infrastructure spend

  • −42%

    item-level inference cost

  • −14%

    item-level inference wall-clock time

  • −58%

    unit-test runtime

  • −68%

    build-to-deploy time

  • −20%

    high-severity tickets

The ticket figure came from data quality, not from pipelines: a self-healing data-quality mechanism cut high-severity tickets by 20%.

Two pieces of this work, anomaly detection and demand remapping, were adopted as shared architecture. And the item-level hand-off went from a person uploading files every day to an automated feed.

Why it mattered


The pipeline and test fixes saved money, but they also turned the integration tests from something engineers bypassed into a gate they trusted. I think that mattered more: it changed morale, and it made on-call less of a nightmare.

The reason to care about any of it is what the forecasts drive. Item-by-site forecasts decide what gets ordered for every grocery site. Better ones mean lower costs for customers and less wasted food.

The Same-Day launch showed the platform had become something other teams could build on. Same-Day Delivery had been planning from a semi-automated process built on historical averages. In October 2024 its team launched a machine-learning demand forecast for US Same-Day Delivery planning on the grocery platform's time-series model. My part was the prototype and the case for reusing the platform. They ran with it and made it a product.

Lessons


  1. Measure before you rewrite

    The team was ready to remove Glue. The real problem was how we were using Spark, and the billing data and executor logs showed it within days.

  2. Cost that grows with table size is a code smell

    When the bill tracks the size of a table, look for full scans.

  3. Reliable tests meant more releases

    Going from a one-in-ten pass rate to a reliable one took releases from one or two a week to several a day.

  4. Take the risky tool away instead of writing a warning

    Taking the repair helper out of the shared library kept the problem from coming back.

  5. Design the stopgap as a drop-in

    The upload bridge I built in 2022 was a browser-automation script, written to be replaced by the planning platform's API. When that API arrived in 2023, the change was a substitution, not a rewrite.

  6. Anomaly detection paid off on its first day

    Anomaly detection, running in shadow mode, found two real bugs on its first day.

  7. Reuse beats rebuild

    The Same-Day launch reused the grocery platform instead of standing up a new one.

  8. Spread the fix

    This could have ended with one team's bill going down. Talks, wikis and shared libraries took it across the forecasting organization.