Background jobs fail in ways web requests never do: workers restart mid-task, brokers redeliver messages, and external APIs time out at the worst moment. Reliability comes from designing for those failures up front, not from hoping they will not happen.
Make every task idempotent
A task that runs twice must produce the same result as a task that runs once. The cheapest way is a unique job key stored in Redis: before doing the work, check the key; after finishing, set it.
import redis
r = redis.Redis()
def run_once(job_id: str, fn):
if r.set(f"job:{job_id}", "done", nx=True, ex=86400):
fn() # only the first delivery executesRetries with backoff, not hope
Celery retries should be explicit: capped attempts, exponential backoff, and jitter so a fleet of workers does not stampede a struggling API at the same second.
@app.task(bind=True, autoretry_for=(TimeoutError,),
retry_backoff=True, retry_jitter=True, max_retries=5)
def sync_payment(self, charge_id: str):
...Dead-letter the rest
After the last retry, move the job to a dead-letter queue and alert. A failed job you can inspect is an annoyance; a failed job that vanished is a refund. This pattern is baked into our boilerplates precisely because everyone rediscovers it the hard way.