insights.fashion
Economic indicator pipeline and the product on top of it
- Role
- Data pipeline, Laravel backend and admin
- Since
- 2024
- Stack
-
- Python
- APScheduler
- Postgres
- Laravel
- Inertia
- React
A research report needed live numbers, not a spreadsheet someone updates by hand.
Twenty-three economic indicators — oil, currencies, freight, electricity — sourced from central banks and government statistics agencies.
Every one of those sources fails differently.
01 How I thought about it
Every source gets a fallback chain, not a single URL
Each metric declares a primary source and an ordered list of fallbacks in a YAML config. The service tries them in order, so one agency being down does not blank a number on the report. Adding a source is a config change, not a code change.
Trusting the freshest number is a trap
The pipeline picks whichever candidate has the newest date. That sounds obviously right and it is where the worst bug came from — described below. Freshness is only a useful signal if the date is honest, so every value now passes sanity bounds and a maximum age before it is allowed to compete.
Every source keeps its own row, not just the winner
A published number is only worth what you can say about it. `metrics` holds the value that won for a date; `metrics_by_source` holds what each source returned for that same date, with the provider URL, the publication date and a flag for whether it was seen live; `metrics_history` logs every change to a published value; `run_log` records each pass of the pipeline. When someone asks why a figure moved, the answer is a query, not an investigation.
Freshness is enforced, not hoped for
Every metric declares a maximum age. Past it, the value is flagged stale rather than published quietly, and a silent fallback — a source that answered but with something old — raises an alert instead of passing for a live reading. The composite index recomputes from the database on every tick, so it can never drift from the series underneath it.
The database is the contract between the two halves
The Python side writes to Postgres. The Laravel app reads the same Postgres through Eloquent and renders through Inertia — no REST layer between them, because a second API would be a second thing to keep in sync for no benefit at this size.
The commerce side is a compliance problem wearing a checkout
I own the backend and the admin here, not the front end. That is the half with rules attached: a B2B checkout that validates company requisites and enriches them from the state register by tax number, a merged contract-invoice built to the legal template with the amount written out in words, the act, and the transactional mail around them. Personal data is stored by country of residence — Russian buyers in a Russian store, European buyers in a European one — kept off the shared analytics database, with the consent audit and the version of the offer recorded on each order.
The bug worth catching was in the price, not the code
Before the report went on sale, I tested every plan-and-price combination a buyer could send and moved the rules to the server, because a price rule that lives in the browser is a suggestion.
02 Architecture
03 Decisions, with the reason attached
-
Idempotent upserts on every write
The scheduler retries and backfills. Re-running a day must not duplicate rows or move a number that was already correct.
-
Sanity bounds per metric
Every value is checked for plausibility before publication. Anything a source returns that is not a plausible value gets rejected at write time rather than surfacing on the published report.
-
Removed a source that was technically the freshest
A cached intermediary stamped today’s date onto days-old snapshots, so it beat the authoritative source on the newest-date rule. It published a stale value as current, which I confirmed against three independent references before I touched anything. Cut the cache out of the chain, went direct, and put a staleness alarm on the metric.
-
Order state is a state machine, not a text field
Money changes state in one direction, and an admin should not be able to drag an order backwards by hand. The status transitions are enforced in code, the payment webhook is idempotent so a retried callback cannot double-fulfil, and a verification failure returns an error the gateway will retry rather than a success that quietly loses the payment.
-
Residency decides where a person is stored
Two jurisdictions, two obligations. Buyer data is written to the store that belongs to the buyer’s country rather than to whichever database was convenient, and it is kept off the shared analytics spine entirely — that database holds an opaque identifier and nothing that names anybody. Emails and phone numbers are masked in every log channel, access to a document is recorded, and erasure on request runs as a command instead of a support ticket.
-
Values that are not live say so
Two of the twenty-three have no public API and arrive as an upload. Rather than let them pass for real-time, each value carries a liveness flag and the interface labels them historical or manual. A number the reader can calibrate is worth more than a number that looks fresher than it is.
-
Backfilled the history before trusting the series
A metric with no past is a number, not an indicator. Series were backfilled from their authoritative sources — sixty months where the agency publishes that far back — so a chart is not born on the day its data pipeline was switched on.
-
Separate hourly and daily jobs
Only seven of the twenty-three metrics move intraday. Polling the rest hourly would be load on public APIs for no new information.
-
One container, one process
A blocking scheduler in a single container is the least infrastructure that does the job. No queue, no workers, nothing to orchestrate until the volume asks for it.
04 Where it landed
Twenty-three indicators plus a tagged news feed updating on schedule in production.
A Laravel product on the same database with auth, payments, CMS and transactional email.
The fallback chains are configured per metric by hand, and I would rather they were derived from observed reliability. The AI-search fallback tier is specified but not implemented — the direct APIs have not yet failed often enough to justify it.
Building something like this — or putting AI agents into production?
I design AI systems, orchestrate the agents that run them and own the backend underneath. Founding-engineer and systems-architect work, and advisory for teams that need it to run unattended and stay within budget. Abu Dhabi (ADGM) / Remote.