Network Pulse

Data ingestion

BODS, every fifteen seconds

One process polls the DfT’s national bus feed on a fixed clock, matches every vehicle it finds to a timetabled journey, and writes the result into two tables that behave in opposite ways. Everything else reported here is derived from those two, overnight.

Reading the feed…

Four clocks, not one

“How often does it update?” has a different answer for each part of the data. This is the distinction worth carrying away: the bus positions are seconds old, the reported figures are a night old, and the timetable they are both measured against is loaded by hand.

Where the feed lands

Drawn from the same snapshot as the figures under it, so the picture and the evidence cannot drift apart. A solid lane is carrying data right now; a dashed one is reference data, replaced wholesale when it is imported rather than streamed.

What one poll does

The same six steps, on repeat, for as long as the service is up. It is a fixed-rate loop rather than a fixed delay — it sleeps the interval minus however long the work took — so the cadence holds even when the feed is slow to answer.

    Two tables, opposite behaviour

    Both are written by the same poll, a few milliseconds apart, and they are the most common thing to get wrong about this system. One is a photograph that is retaken every fifteen seconds; the other is the film.

    The last 24 hours

    Stop departures recorded per hour. The feed itself never stops — the poller runs all night at the same interval. What falls away is the buses, which is why a dark chart at 03:00 is normal and a dark chart at 11:00 is an incident.

    The nightly chain

    Four jobs between 03:10 and 03:50 turn the raw observations into the tables every report reads. Each re-runs the last two or three days, so a late-arriving observation still gets counted.

    Runs Writes What it produces Latest day State

    The order is held by the clock alone. Three of the four read stop_observations, and the one that deletes from it runs second. There is no systemd dependency between them — only the fifteen-minute gaps — so a job that overran badly could in principle read a table another had already pruned.

    What lands, and how long it stays

    The raw feed is the expensive part and the shortest-lived. The derived tables are small enough to keep indefinitely, which is what makes a five-week retention on the raw observations affordable.

    Table Rows On disk Written Kept

    The whole database, in families

    Every table in the schema, grouped by what it is for and measured as it stands. The lines are the joins the queries actually perform, drawn family to family rather than table to table, and labelled with the key they run on.

    If BODS stopped naming the journey

    Everything above hangs on one field: the GTFS trip_id the feed supplies per vehicle. Franchised on-board equipment will not carry one, and neither does SIRI-VM — both report the operator’s own name for the work. The journey can still be identified, from a description of it rather than from an id, and the two routes meet at the same place.

    On the BODS trip id — today On a description of the journey

    What this doesn’t cover

    Each of these gets asked. Better to have the answer on the page than to be caught out by it.