Daily Note: Debugging a Production API Flow
Integration test green, staging green, prod failing for 1% of callers. The bug was in a place I would never have guessed.
Today was one of those.
API returns 200 in staging, 500 for ~1% of prod traffic. CloudWatch shows the Lambda timing out at 29.9s. Every integration test green. Every load test green.
Spent two hours blaming the database. It wasn't the database.
Turned out: one of our upstream SaaS integrations started returning responses with a specific header that our HTTP client silently retried on. Three retries × 10s backoff = 30s = Lambda timeout. The 1% was the subset of customers whose data happened to route through a specific vendor edge.
# before
session = requests.Session()
session.mount("https://", HTTPAdapter(max_retries=Retry(total=3, backoff_factor=1)))
# after
session = requests.Session()
session.mount("https://", HTTPAdapter(max_retries=Retry(
total=3,
backoff_factor=1,
respect_retry_after_header=False, # the vendor lies
allowed_methods=frozenset(["GET"]), # never retry POSTs
)))
Lesson I keep re-learning: retry policies are a system boundary. Default values are almost always wrong for your system.
Going to bed with a headache but a green dashboard.
Written by
Adrian Romo
Senior Backend Engineer building scalable Python APIs, AWS Lambda architectures, voice systems, and enterprise integrations.
Related
Keep reading
Seconds of Work, Hours of Residency
My morning briefing started failing. Ollama was up and returned HTTP 500, because a 21-second image render was still holding 6.6 GB of VRAM hours later.
What Deserves Attention Today
My homelab produces one verdict each morning: something needs you, or nothing does. Getting the second half honest was much harder than the first.
The Alert That Named a Thing and Called It Evidence
Five high-severity alerts from my own monitoring. All five were the same defect: a record that names something being read as an observation of a property it never measured.
Keep going
Where to next?
Browse more technical writing, see the engineering case studies, or reach out directly.