DoorDash

What the interviewer is probing

Whether the candidate can position LLMs as a data-enrichment layer inside a classical retrieval architecture rather than as a replacement for it: generate normalized, descriptive item profiles once (offline, cacheable), embed those into the existing vector machinery, and blend with behavioral signals as they accumulate. Also probing evaluation maturity — cold-start improvements hide in slices that aggregate metrics wash out — and clear-eyed handling of hallucinated attributes in a system that makes claims to users.

Strong answer outline

• The core move: use the LLM offline to transform sparse, inconsistent metadata into a normalized rich profile per merchant/item (cuisine, attributes, occasions, descriptive summary) — enrichment plus standardization, which is what makes downstream embeddings comparable across a heterogeneous catalog. Then encode profiles with an embedding model into the same vector infrastructure retrieval already uses.
• Why not embed raw metadata: thin and inconsistent fields embed inconsistently; the LLM's contribution is inference (a taqueria's menu implies attributes no field states) and canonical vocabulary.
• Cost architecture: generation is one-time per item + refresh on meaningful catalog change (menu updates), not per query; batch it, version prompts, cache aggressively; per-query path touches only vector similarity — LLM latency never enters serving.
• Hallucination containment: constrain generation to grounded fields where truth matters (dietary claims validated against source data or suppressed), use profiles for retrieval and ranking features rather than user-facing claims, and audit a sample per generation batch; measure downstream harm via complaint/refund proxies on affected surfaces.
• Blending with behavior: cold items ride semantic embeddings; as interactions accumulate, blend (weighted or learned gate on interaction volume) toward behavioral signals, which encode what descriptions can't (quality, conversion propensity). Replacing behavioral embeddings outright throws away your best signal for the head of the catalog.
• Evaluation designed for cold start: offline — retrieval recall and ranking metrics sliced by item tenure and interaction count, since aggregate metrics are dominated by the head; online — experiment with cold-start-focused readouts (new-merchant first-week conversion, long-tail impression share) plus overall guardrails. Flat-overall/ better-in-slice is the expected success shape.
• The flat-A/B probe: (1) cold items are a small traffic share, so the win is real but diluted — check the slice; (2) retrieval improved but the ranker, trained on behavioral features, buries the new candidates — retrain or add semantic features to ranking; (3) offline gains were popularity-bias artifacts of the eval set.

The underlying concept

Cold start is a missing-signal problem: collaborative/behavioral representations are functions of interaction history, so new entities sit at the origin. Content-based representation is the classical answer, and LLMs upgrade it by converting messy, heterogeneous metadata into normalized, inference-enriched text — effectively transferring world knowledge into items that lack local data. The architectural principle is separation of timescales: expensive generation lives offline where it amortizes; serving stays vector-fast. And the evaluation principle is that improvements targeted at a minority slice must be measured in that slice — aggregate metrics are head-of- distribution metrics, and cold start is by definition the tail.

Source

Derived from Using LLMs to Build Content Embeddings for Search and Recommendations

What the interviewer is probing

Whether the candidate understands adaptive experimentation as a trade — bandits minimize regret (cost of showing losers) at the price of clean inference and added machinery — rather than as a strict upgrade to A/B testing. The delayed-feedback probe is the discriminator: naive bandits over-exploit whatever converts fastest, and strong candidates reach for batched updates, pessimistic priors on immature cohorts, or delay-corrected reward estimates. Knowing when not to use a bandit is worth as much as knowing how.

Strong answer outline

• Frame the objective difference: A/B optimizes for learning (precise effect estimates at fixed allocation); bandits optimize for earning (minimize cumulative regret while learning). Many-variant, short- lived, low-stakes decisions like promo copy sit squarely in bandit territory.
• Why greedy fails: always-play-the-leader locks onto early noise and never gathers the evidence to escape it. Exploration is the price of correctable beliefs.
• Thompson sampling: maintain a posterior over each variant's conversion rate (Beta-Binomial for binary rewards); each allocation period, sample one draw per variant and route traffic proportionally to how often each variant wins the sampled comparison. Exploration emerges from posterior uncertainty and self-anneals as evidence concentrates — no tuning knob like epsilon.
• Delayed/batched rewards, the real-world core: updating on immediate signals biases toward fast-converting variants, not best ones. Mitigations: update in batches aligned to the conversion window; score only matured cohorts (or model the delay distribution and correct immature counts); keep a minimum-allocation floor per variant so late bloomers keep collecting evidence; slow the allocation cadence relative to the feedback delay.
• When to refuse the bandit: guardrail-heavy or high-blast-radius changes needing precise CIs, treatments with learning/novelty dynamics (early performance misleads by construction), interference- prone marketplace levers (allocation shifts change the market), and anything requiring a defensible effect size for a launch review.
• Post-hoc inference honesty: adaptive allocation breaks the fixed- sample assumptions behind naive estimates — variants get traffic correlated with their own noisy history. Report via inverse- propensity weighting on logged allocation probabilities, or accept that the bandit's output is a decision, not an unbiased measurement, and say so.

The underlying concept

The explore-exploit trade-off is the recognition that every allocation decision spends traffic on two goods at once: immediate reward and information. Fixed experiments buy pure information; greedy policies buy pure (apparent) reward; Thompson sampling prices the two automatically by sampling from posterior beliefs — probability matching that explores exactly as much as uncertainty warrants. Delayed feedback breaks the loop's core assumption (act, observe, update) and re-opens the door to premature convergence, which is why production bandits are mostly engineering around feedback latency. And adaptivity taxes inference: data collected under a policy that reacts to outcomes is not a random sample, so measurement either corrects for the collection policy or downgrades its claims from "effect" to "choice."

Source

Derived from Accelerating Experimentation at DoorDash with a Multi-Armed Bandit Platform

What the interviewer is probing

Rare-event forecasting judgment: the candidate should recognize this as a small-data problem embedded inside a big-data one, where a monolithic model drowns a handful of holiday examples in hundreds of thousands of ordinary days. The expected shape of the answer is decomposition — model the baseline, then model event effects separately with pooling — plus honest evaluation design for rare slices and graceful handling of never-observed combinations.

Strong answer outline

• Why the flag feature fails: gradient descent on the full dataset allocates capacity proportional to data; a few dozen holiday rows can't move a model dominated by ordinary days, and each holiday's effect differs (Valentine's ≠ Thanksgiving), interacts with weekday, and varies by market — one coefficient can't carry that.
• Cascade design: layer 1 forecasts baseline demand trained on non-holiday data (or with holidays masked); layer 2 models the holiday effect — realized demand relative to the baseline prediction (a multiplier or residual) — trained only on event days across all holidays, markets, and years. Publishing forecast = baseline × predicted effect.
• Making 5 observations work: pool across the hierarchy — holiday family (major feast vs. minor observance), market cluster, weekday interaction — with hierarchical/shrinkage estimation so Thanksgiving in a new market borrows from Thanksgiving elsewhere and from similar-holiday behavior, with uncertainty widened accordingly.
• The quiet advantage of decomposition: the everyday model is untouched — no risk of degrading 350 good days to fix 15 bad ones — and holiday-effect estimates are inspectable numbers operators can sanity-check ("we predict +38% Mother's Day lunch"), preserving trust.
• Evaluation for rare slices: leave-one-holiday-out backtests across years; report error on holiday days specifically and the cost- weighted version (staffing error cost), never blended MAPE where 350 easy days launder 15 hard ones; track baseline and effect layers separately so misses attribute cleanly.
• Never-seen combinations: fall back up the hierarchy (holiday family × weekday-shift priors), widen intervals, and flag for the human adjustment channel — the cascade makes "what we don't know" explicit rather than silently interpolated.

The underlying concept

Rare structured events break the i.i.d. comfort of large-scale forecasting: the information about holidays lives in a tiny data slice with its own structure, and monolithic training dilutes it. The classical remedy is decomposition — separate the abundant-data problem (baseline) from the scarce-data problem (event effects) — paired with hierarchical pooling, which is the general answer to "many related small problems": share strength across the hierarchy, let data earn specificity, and express ignorance as widened uncertainty. Evaluation must mirror the decomposition, because aggregate metrics are majority-class metrics and the minority days are the ones that carry the operational cost.

Source

Derived from How DoorDash Improves Holiday Predictions via a Cascade ML Approach

What the interviewer is probing

Whether the candidate has a decision framework rather than a favorite design: the choice among user-level, cluster/geo, switchback, and long-window designs is driven by two properties of the treatment — does it touch a shared resource (interference), and does its effect develop over time (learning/carryover) — traded against power. This is DoorDash's home turf; strong candidates match each scenario to its failure mode and can articulate the power price of every unit coarsening.

Strong answer outline

• The framework first: ask two questions of any treatment. (1) Does it affect units beyond the treated one — via shared supply, shared queues, social exposure? If yes, the randomization unit must contain the interference. (2) Does the effect take time to develop or persist after removal — learning, habituation, carryover? If yes, the design needs long exposure windows and washouts.
• Checkout flow: no shared resource, effect immediate → user-level A/B is correct; maximum power, no reason to pay for more.
• Dasher pay display with learning: Dasher-level randomization is fine on interference (weak coupling through display alone) but the effect develops over weeks → long-running Dasher-level experiment with effect curves by exposure time; switchbacks are wrong here — alternating conditions faster than Dashers learn measures a blend of transient states.
• Dispatch algorithm: treated and control assignments compete for the same Dashers → user- or Dasher-level randomization leaks treatment effects into control (typically flattering the treatment). Design: switchbacks over region × time windows, or geo-cluster randomization; washout/burn-in periods at boundaries for queue carryover.
• The power ledger: effective sample size collapses from millions of users to hundreds of region-time units or dozens of geos — expect longer runs, variance-reduction (covariate adjustment, stratification), and cluster-robust inference; state this cost explicitly when recommending the design.
• Post-hoc leak detection: compare boundary vs. interior switchback periods (carryover signature), test control-group metric shifts correlated with local treatment intensity, and monitor market-level invariants.
• The pushback answer: interference is a design problem; analysis-time corrections require the very model of spillovers you don't have — randomize at the level the mechanism operates or accept a biased answer fast.

The underlying concept

Experimental design is the art of choosing a randomization unit whose boundaries contain the treatment's mechanism. Interference (one unit's treatment touching another's outcome, via shared resources) violates SUTVA and forces coarser units — clusters in space or blocks in time — while temporal dynamics (learning, carryover) force longer and buffered exposure. Every coarsening pays in power because inference runs on the number of independent randomization units, not raw observations. There is no universally correct design — only a mapping from treatment mechanism to unit, with power as the currency; the professional skill is naming the failure mode of the cheap design before the data does.

Source

Derived from Balancing Network Effects, Learning Effects, and Power in Experiments

What the interviewer is probing

Judgment about human-in-the-loop forecasting: the naive positions ("trust the model" / "trust the operators") are both wrong, and the mature answer is structural — convert judgment into measurable, attributable inputs. Strong candidates propose backtesting the adjustment layer separately from the base model, designing structured channels for operator knowledge (event calendars, capacity flags), and preserving an audit trail so every published number decomposes into model + adjustment with separate accuracy accounting.

Strong answer outline

• Diagnose before deciding: backtest overrides against the counterfactual — for every adjusted forecast, compare realized demand against both the model's original number and the adjusted number, sliced by operator, adjustment size, and stated reason. Typical finding: event-driven adjustments help; sentiment-driven ones add noise.
• Reframe the architecture: overrides replace the model's output and destroy attribution; structured inputs inform it. Build channels for the knowledge operators actually have — an event calendar (festivals, closures, weather ops), promo schedules, capacity constraints — that enter the model as features with learned effects.
• Keep a principled adjustment layer for what can't be featurized: adjustments logged with reason codes, bounded in magnitude, applied as explicit deltas on top of the model output — never silent edits.
• Accountability by decomposition: publish forecast = model + named adjustments; score each component's error separately and report it back — operators see their own adjustment accuracy, which self-corrects behavior better than mandates.
• Evaluation discipline: the base model is evaluated pre-adjustment; the system (model + adjustments) is what supply planning consumes and is evaluated end to end; both tracked over time so the adjustment layer's shrinking value is visible.
• Retirement path: when a class of adjustment (e.g., holiday effects) becomes learnable — enough labeled instances — promote it into the model, then watch that adjustment category's frequency fall; the goal is migrating knowledge from heads into features, not banning judgment.

The underlying concept

Judgmental forecast adjustment is one of the oldest findings in forecasting research: domain experts add real information about discrete, foreseeable events and subtract value everywhere else — so the design goal is to channel judgment, not to choose between human and machine. Treating human input as data — structured, logged, attributable, and separately scored — turns an untestable override culture into a measurable model component. The organizational insight mirrors the statistical one: accountability requires decomposition, because a single blended number lets both the model and the human disown its errors.

Source

Derived from Why Good Forecasts Treat Human Input as Part of the Model

What the interviewer is probing

Whether the candidate understands that a zero average can mask offsetting heterogeneous effects, and that finding responders is a causal estimation problem, not a segmentation exercise — post-hoc subgroup mining is multiple-testing on noise. Strong candidates reach for heterogeneous-treatment-effect estimation on the randomized data, know why individual effects are fundamentally unobservable (one potential outcome per user), and insist on validating the resulting targeting policy with a fresh experiment.

Strong answer outline

• Name the possibility space first: a flat average means (a) truly no effect for anyone, (b) small effects below detection, or (c) offsetting heterogeneity. Distinguishing them is the job.
• Why naive slicing fails: dozens of segments × noisy subgroup estimates = guaranteed false discoveries; effects found by searching the same data that generated them won't replicate. Pre-registered segments or held-out validation are the minimum discipline.
• HTE estimation: use the experiment's randomization as the engine — meta-learners (S/T/X-learners) or causal forests estimate conditional average treatment effects as a function of user covariates (pre-treatment only). The fundamental problem: each user reveals one potential outcome, so models estimate conditional averages, never individual truths.
• Evaluation without ground truth: uplift curves / Qini coefficients on a held-out slice of the experiment — rank users by predicted uplift, verify realized treatment-control gaps concentrate in the top ranks; calibration of predicted vs. realized subgroup effects.
• Policy validation: a new experiment where treatment = "model-targeted promotion" vs. control = status quo (and ideally an arm with the old untargeted policy). The original test validated the treatment; this validates the decision rule — different estimand.
• Decay hypotheses: novelty effects in the original data, covariate drift, feedback loops (targeting changes the population's behavior), and cannibalization as competitors/other promos adapt. Monitoring: periodic holdouts to re-measure the policy's incremental value.

The underlying concept

An average treatment effect is exactly that — an average — and decision-relevant heterogeneity lives underneath it. The conditional average treatment effect (CATE) is estimable from randomized data because randomization holds within every covariate stratum, but the fundamental problem of causal inference (one potential outcome observed per unit) means uplift models predict group-conditional effects, and their evaluation must be rank- and calibration-based rather than per-user accuracy. The deeper discipline is separating estimand layers: an experiment validates a treatment; an uplift model proposes a policy; only a policy experiment validates the policy. Skipping that last step ships a multiple-testing artifact with a model wrapped around it.

Source

Derived from Leveraging Causal Modeling to Get More Value from Flat Experiment Results

What the interviewer is probing

Operational decomposition under time pressure: can the candidate break the promise-to-door timeline into stages (quote, prep, assignment, travel, wait-at-store, last mile), distinguish "we predicted badly" from "the system performed worse," and use the multi-city pattern to prioritize hypotheses (simultaneity suggests a common cause: model release, app version, incentive change, weather system)?

Strong answer outline

• Definition first: late relative to quoted time — so lateness rises if quotes got tighter OR operations got slower. Check whether the ETA model/quote policy changed; a quote-side change explains multi-city simultaneity instantly.
• Timeline decomposition: order → merchant confirmation → prep → Dasher assignment → travel to store → wait at store → pickup → travel to consumer. Attribute the extra minutes: which stage(s) grew?
• Common-cause scan (because several cities moved together): model or app releases, dispatch/batching parameter changes, incentive/supply changes, weather systems, holidays/events; overlay release calendars on the trend.
• Segment cuts in order of information: by stage (above), by city vs. control cities, merchant type (prep-time issues concentrate in specific cuisines/chains), batched vs. solo deliveries, Dasher tenure (new-Dasher influx slows pickup), hour-of-day.
• Immediate mitigations: widen quote buffers where lateness concentrates (protects promises at slight conversion cost), cap batching aggressiveness, targeted supply incentives at peak.
• Durable fix: whichever stage drove it — recalibrate ETA/prep models with recent data, add wait-at-store telemetry, dispatch parameter guardrails, and a lateness decomposition dashboard so the next spike self-localizes.

The underlying concept

"Late" is a relation between a promise and an outcome, so its investigation always forks immediately: did promises tighten or did performance slip? Treating the delivery as a sum of stage durations turns a vague metric move into an attribution problem — the same telemetry-decomposition instinct as latency debugging in distributed systems. And correlated onset across units is itself evidence: simultaneity points to shared infrastructure (models, releases, policies) rather than local operations, which is why release calendars are the first dataset a good investigator pulls.

Source

Distilled Prep canon — curated from DoorDash's public work on delivery operations and ETA prediction.

Source: DoorDash engineering blog

What the interviewer is probing

Three-sided metric thinking plus a specific trap: bikes get assigned short, dense, small-basket orders, so naive bike-vs-car comparisons measure the assignment policy, not the mode. Strong candidates design at the market level (geo or phased rollout), decompose incremental capacity vs. substitution, and treat existing car Dashers' earnings as a first-class guardrail.

Strong answer outline

• Define success across sides: consumers — delivery time, lateness, food quality, cost; Dashers — earnings per active hour for both modes (cannibalization shows up as car-Dasher earnings decline); merchants — order volume, wait-at-store; platform — cost per delivery, fulfillment rate at peak.
• Name the selection problem: dispatch will route bikes to short/dense orders — never compare bike orders to car orders directly; compare markets (or time periods) with and without the bike program.
• Design: staged geo rollout with matched control markets or synthetic control; within-market switchbacks are polluted by supply persistence (Dashers don't appear/disappear hourly), so market-level inference is the honest unit.
• Incrementality decomposition: total supply-hours before/after by mode; survey/behavioral evidence on whether bike Dashers are new workers or car Dashers switching; capacity value measured at peak (did fulfillment improve when it was binding?).
• Interaction effects: weather (bike supply collapses in rain — check reliability under adverse conditions before calling success), geography (dense cores vs. suburbs), and time-of-day mix.
• Decision framing: expand where bikes add peak capacity at lower cost per delivery without degrading car-Dasher earnings past a threshold; hold or adjust incentives where substitution dominates.

The underlying concept

When a new supply type enters a marketplace, the platform's own assignment algorithm immediately confounds any unit-level comparison — the treatment (mode) is entangled with the routing policy. The clean counterfactual lives at the market level, which is why geo experiments and synthetic controls dominate marketplace launch evaluation. The second discipline is decomposing gross additions into incremental capacity vs. substitution: marketplaces care about the net supply curve, and total-hours accounting plus peak-constraint analysis is how you see it.

Source

Distilled Prep canon — curated from DoorDash's public work on marketplace experimentation.

Source: DoorDash engineering blog

What the interviewer is probing

Whether the candidate reaches for idempotency keys as a design principle rather than a buzzword: where the key is generated, what gets stored, the atomicity requirement between "check" and "act", and TTL trade-offs. The second part tests degraded-mode reasoning — do they fail open (risk duplicates) or fail closed (risk lost orders), and can they connect that choice to the business cost of each failure?

Strong answer outline

• Client generates the idempotency key (per logical order attempt, not per HTTP request) so all retries of one intent share a key.
• Server: atomic check-and-set (e.g. unique constraint or conditional write), not read-then-write — the race between two concurrent retries is the classic bug.
• Store the response with the key so retries return the original result, not just a "duplicate" error.
• TTL discussion: too short re-enables duplicates, too long bloats storage; tie it to realistic client retry windows.
• Degraded mode: articulate fail-open vs fail-closed and pick using domain cost — a duplicate food order costs a refund; a dropped one costs a customer. Reasonable answers differ; unreasoned answers don't.

The underlying concept

Exactly-once delivery doesn't exist between distributed parties; what systems actually implement is at-least-once delivery plus idempotent processing, which is indistinguishable from exactly-once from the caller's perspective. The idempotency key turns a side-effecting operation into something safely retryable by giving the server a durable memory of intents it has already honored. Every payment and ordering system you've used is built on this pattern.

Source

Hand-written seed example in the style of DoorDash's platform reliability posts.

Source: DoorDash engineering blog

What the interviewer is probing

ML judgment under label noise and asymmetric costs: ready-time is rarely logged directly (proxies: pickup time minus wait, merchant confirmations when present), errors are asymmetric (early Dasher = paid waiting; late Dasher = cold food and lateness), and the model's value is realized only through dispatch decisions — so calibration and uncertainty matter more than point-accuracy bragging rights.

Strong answer outline

• Label construction: where merchant "ready" signals exist, use them with de-noising; elsewhere infer from Dasher arrival/wait/pickup telemetry (ready ≈ pickup when Dasher waited; censored when food waited). Model the censoring rather than pretending labels are clean; keep a small ground-truth panel for calibration.
• Loss design: asymmetric cost — under-prediction (Dasher early) costs Dasher time; over-prediction (food ready, no Dasher) costs quality and lateness. Use quantile regression or explicit asymmetric loss; dispatch consumes a distribution (e.g., P50 + P90), not a point.
• Features without leakage: merchant historical prep by item/basket size/hour, current confirmed queue depth, kitchen throughput proxies, time/weather; real-time features (queue, recent confirmations) via online feature store; strictly exclude anything timestamped after the prediction moment in training joins.
• Cold start: hierarchical backoff — item → menu-category → cuisine → market priors; shrink toward the merchant as their data accumulates; widen predicted uncertainty for new entities so dispatch pads conservatively.
• Serving: prediction at order-confirmation and re-prediction on signal updates; latency budget small; feature freshness monitored; training/serving parity via shared feature definitions.
• Evaluation tied to decisions: offline — calibration and quantile loss by segment; online — Dasher wait minutes, food-sit minutes, lateness, cost per delivery via experiment; monitor drift by merchant cohort (menus and staffing change constantly).

The underlying concept

This is decision-focused ML: the model's product is not a number but a calibrated distribution consumed by an optimizer whose costs are asymmetric — so loss functions, uncertainty, and calibration are the design surface, not afterthoughts. Label noise and censoring are the second theme: real operational labels are inferred from behavioral traces, and modeling their generation process beats wishing they were clean. Hierarchical priors solve cold start the same way they solve every sparse-entity problem: borrow strength, then let data earn independence.

Source

Distilled Prep canon — curated from DoorDash's public work on ETA and prep-time prediction.

Source: DoorDash engineering blog

What the interviewer is probing

Interference-aware design: batching decisions consume shared Dasher supply, so treated and control orders in one market at one time contaminate each other. This is the canonical switchback scenario, and the interviewer wants the candidate to reason to it — plus the analytic maturity to frame cost savings against lateness as a non-inferiority test with pre-set margins.

Strong answer outline

• Why order-level randomization fails: an order batched in treatment removes a Dasher from the shared pool that control orders draw on — control outcomes absorb spillover, biasing effects (typically making treatment look better than reality).
• Design: switchbacks — randomize (zone × time-window) units between algorithms; windows long enough for the system to reach steady state (dispatch queues clear) but short enough for many units; buffer/burn periods at boundaries to limit carryover.
• Analysis on switchbacks: cluster-robust inference at the window-zone level; expect far less power than order-level tests — plan duration accordingly; check residual carryover by comparing boundary vs. interior periods.
• Metric frame: primary — cost per delivery (superiority); guardrails — lateness rate, food-temperature proxies/complaints, cancellations, Dasher earnings per hour, merchant wait congestion (non-inferiority with pre-registered margins, e.g., lateness within +0.5pp).
• Heterogeneity: batching pays only above an order-density threshold; pre-plan analysis by density tier and hour; rollout policy is likely conditional (batch aggressively in dense zones at peak, minimally elsewhere), not global.
• Decision rule stated upfront: ship the density-conditional policy if cost improves where guardrails hold; iterate the algorithm where quality margins fail.

The underlying concept

Marketplace experiments fail SUTVA whenever treatment consumes a shared resource; the fix is to randomize the level at which the resource is shared — space-time blocks rather than orders. Switchbacks buy validity at the price of power and carryover risk, which planning (steady-state windows, buffers, clustered inference) manages. The metric lesson: when an intervention trades cost against quality, the honest statistical frame is superiority on the target plus non-inferiority margins on guardrails, all pre-registered so the trade-off is decided before the data can argue.

Source

Distilled Prep canon — curated from DoorDash's public work on switchback experimentation and dispatch optimization.

Source: DoorDash engineering blog