Skip to content

Recover failed API workflows with exponential backoff and dead-letter queues

Stop losing orders and leads whenever a third-party SaaS throws a 502 Bad Gateway or 429 Rate Limit—distinguish transient blips from permanent bugs automatically.

Copy the prompt and paste it into the AI assistant you already use.

Four pillars of production-grade workflow error recovery

  1. 01

    Classify transient vs non-retryable HTTP and AI errors

    Retry network timeouts, 429s, and 5xx outages automatically while halting 400/401/403 permanent errors immediately.

  2. 02

    Checkpoint completed side-effect steps

    Persist step outputs so when Step 4 recovers on Attempt #3, Steps 1–3 never send a second email or create duplicate records.

  3. 03

    Apply randomized jitter to prevent thundering herds

    Spread retry wake-up times randomly when 500 queued runs recover from a vendor outage simultaneously.

  4. 04

    Park exhausted runs in an inspectable Dead-Letter Queue

    Move items that fail all retry attempts into a structured DLQ with full input payload, error stack, and 1-click replay.

How it works in practice

Step 3/4 failure during NetSuite maintenance window

Run #run_49102: Step 1 (Parse PO) OK -> Step 2 (CRM Lookup) OK -> Step 3 (NetSuite POST) HTTP 503 Service Unavailable at 02:01 UTC

Checkpointed backoff recovery trace

Attempt 1 (02:01:04): 503 -> Scheduled Retry #1 (+30s + jitter) Attempt 2 (02:01:38): 503 -> Scheduled Retry #2 (+3m) Attempt 3 (02:04:41): 503 -> Scheduled Retry #3 (+15m) Attempt 4 (02:19:49): HTTP 201 Created (Steps 1–2 skipped from checkpoint) -> Step 4 Slack confirmation sent.

A 15-minute vendor maintenance window at 2:00 AM heals itself completely before anyone wakes up.

Common questions

Why is step-level checkpointing critical when retrying multi-step workflows?

Re-running an entire workflow from Step 1 when Step 5 fails duplicates Slack alerts, burns double LLM tokens, or creates twin CRM records.

Why shouldn't HTTP 400 or 422 validation errors retry five times?

A malformed email address or expired API key will fail identically five times; failing fast to the DLQ alerts engineers immediately without wasted API quota.

How long do items wait in the Dead-Letter Queue if an API stays down all day?

Dead-lettered payloads persist durably for weeks until you inspect, patch credentials or mapping, and replay them in batch.

Learn more

Related guides and background articles for this task