Distributed Task Execution Engine
Fault-tolerant distributed job execution engine with Redis-based dispatch, priority scheduling, lease-based recovery, and full Prometheus/Grafana observability.
Problem
Long-running background jobs fail in messy ways: workers die mid-task, retries hammer downstream services, and duplicate executions corrupt state. This engine treats failure as the default case — jobs are claimed with leases, retried with backoff, and dead-lettered when they exhaust recovery.
Impact
- Fault-tolerant execution with lease-based crash recovery
- Priority scheduling across concurrent workers
- Retries with exponential backoff and a Dead Letter Queue
- Duplicate-execution prevention via job claiming
- Real-time dashboards for throughput, latency, failures, and DLQ events
Architecture
flowchart LR
P[Producer] -->|enqueue| R[(Redis Dispatch Queue)]
R -->|priority pull| W1[Worker 1]
R --> W2[Worker 2]
R --> W3[Worker N]
W1 --> C{Job Claimed?}
C -->|lease acquired| E[Execute Task]
E --> DB[(PostgreSQL: state + audit)]
E -->|failure| RT[Retry w/ Exponential Backoff]
RT -->|exhausted| DLQ[(Dead Letter Queue)]
E --> M[Micrometer Metrics]
M --> PR[Prometheus]
PR --> G[Grafana]Mermaid flowchart source — renders as a diagram when embedded in docs.
Data model
- jobs (id, type, payload, priority, status, max_retries, next_run_at)
- job_leases (job_id, worker_id, leased_until, attempt)
- job_events (id, job_id, event, worker_id, created_at)
- dead_letter_jobs (job_id, final_error, attempts, failed_at)
- workers (id, hostname, last_heartbeat, active_jobs)
API design
| Method | Path | Purpose |
|---|---|---|
| POST | /api/jobs | Enqueue a job with type, payload, and priority |
| GET | /api/jobs/:id | Job status, attempt count, and event history |
| DELETE | /api/jobs/:id | Cancel a pending or leased job |
| POST | /api/jobs/:id/retry | Requeue a dead-lettered job |
| GET | /api/workers | Live worker registry and heartbeats |
| GET | /actuator/prometheus | Metrics scrape for Prometheus |
Features
- →Redis-backed dispatch with priority scheduling across worker pools
- →Lease-based job claiming — expired leases make crashed jobs recoverable
- →Retries with exponential backoff and jitter
- →Job cancellation and Dead Letter Queue handling for unrecoverable jobs
- →Duplicate-execution prevention so two workers never run the same job
- →Prometheus + Grafana dashboards for throughput, latency, retries, DLQ, and health
Engineering decisions & tradeoffs
Leases over locks
A distributed lock held forever is a deadlock when its holder dies. Leases with an expiry make failure self-healing: a crashed worker's jobs become claimable again without manual intervention, and the lease table doubles as the audit trail of who ran what.
Redis for dispatch, PostgreSQL for truth
Redis gives low-latency queueing and priority pulls; PostgreSQL is the source of truth for job state and history. If Redis loses data, PostgreSQL can rebuild pending work — the two are deliberately separated.
Exponential backoff with jitter
Plain retries at fixed intervals synchronize failing workers into thundering herds. Backoff spreads retries out, and jitter desynchronizes them — small additions that keep a failing dependency from being DDoSed by its own retry logic.
DLQ instead of silent drops
Jobs that exhaust retries go to a Dead Letter Queue with their final error and attempt count, so failures are inspectable and requeueable rather than silently lost.
Challenges & lessons
- — Coordinating multiple workers without duplicate execution — solved with atomic job claiming and lease renewals during long tasks.
- — Recovering jobs stranded by a worker that died mid-execution, without re-running ones that actually completed.
- — Tuning lease duration so recovery is fast but a slow task isn't falsely reclaimed and run twice.
- — Designing metrics that surface real problems — retry storms vs. isolated failures — not just raw counts.
Future improvements
- — Exactly-once semantics via idempotency keys on job side effects
- — Distributed tracing across producers, queues, and workers
- — Rate limiting per job type to protect downstream dependencies
- — Horizontal worker autoscaling driven by queue depth metrics