Orchestration pipeline & control plane
Async workflows over third-party APIs that fail constantly
- Role
- Backend, infrastructure, admin UI
- Since
- 2024
- Status
- private
- Stack
-
- Python
- FastAPI
- asyncio
- Tortoise ORM
- Postgres
- Nuxt
Some systems are hard because the algorithm is hard.
This one was hard because everything it depended on was unreliable.
External providers time out, return half-states, and change their mind about a result you already handled.
01 How I thought about it
Assume every external call fails
The naive version calls an API, gets a result, moves on. In production that shape breaks within an hour. I modelled every provider interaction as a state machine over their status codes, so an unexpected reply moves the record to a state I have already thought about instead of throwing.
Retries are not a loop
A retry without backoff turns one outage into a self-inflicted denial of service against a provider that is already struggling. Exponential backoff with a real timeout, and a bounded number of attempts before the job is parked for a human.
Long-lived sessions must survive restarts
Sessions are persisted, so a deploy or a crash does not mean starting over and re-paying for work already done.
If you cannot see it, you cannot run it
The pipeline got a control plane: a FastAPI service behind JWT auth and rate limiting, and a dashboard showing state per record. Operations stopped being "read the logs and guess".
02 Architecture
03 Decisions, with the reason attached
-
State machine over provider status codes
Every reply, including the ones I did not expect, maps to a defined state. Unknown input becomes a state transition rather than an exception.
-
Exponential backoff with a hard timeout
Bounded failure. The system degrades in a way you can read, instead of hammering a dying provider.
-
Versioned migrations from day one
Schema changes on a running system with real data need to be reversible. Migrations are the only honest way to do that.
-
Rate limiting on the control plane, not just auth
Auth answers who you are. Rate limiting answers how much damage one authenticated mistake can do.
04 Where it landed
Async orchestration with retries, backoff and self-healing sessions running against providers that fail daily.
An admin control plane behind JWT auth and rate limiting, with per-record state visible instead of buried in logs.
Concurrency is handled with polling and state transitions rather than a real queue. With more volume I would put a proper broker in front of the workers — that is the first thing I would change.
Building something like this — or putting AI agents into production?
I design AI systems, orchestrate the agents that run them and own the backend underneath. Founding-engineer and systems-architect work, and advisory for teams that need it to run unattended and stay within budget. Abu Dhabi (ADGM) / Remote.