Moving it off the laptop: what we need to ask for
Status: a shopping list, written 2026-09-08. The proof of concept runs on a laptop and works. This is what it would take to run it properly, why each item is on the list, and what happens if it is left off. Nothing here is a design decision that has to be made now — it is the set of things only somebody else can grant.
Why the laptop is the limit
Not because it is slow. It captures, matches, measures and publishes exactly as a server would, and everything below is about when and for whom rather than whether.
Capture only runs while somebody is logged in. On 2026-09-08 the poller started at 09:29, when the laptop was logged into. Of the 12,583 scheduled journeys touching the capture area that day, 2,893 were already past before capture began — filed as not watched, correctly, because nothing observed them. That is the morning peak, every day, and it is the part everyone asks about first. The feed only ever serves now: a minute not captured is gone permanently and cannot be bought, requested or backfilled later.
The dashboard is live for one person. It serves on 127.0.0.1, so
everyone else gets files copied into a shared folder on a timer. That is
adequate for showing people what exists and inadequate as a service.
Power BI cannot reach a file share live. DirectQuery needs a database; against files, a viewer pressing Refresh re-renders the last dataset refresh rather than pulling anything new, and scheduled refresh floors at 30 minutes on Pro. Live figures for other people need something always on, holding the data, that Power BI can query.
The ask
| What | Why | Without it |
|---|---|---|
| A Windows Server VM, always on | 24/7 capture, including the morning peak | Capture follows somebody's working day |
| 4 vCPU, 16 GB RAM | The poller is trivial; the archive joins are what use memory | 8 GB works, queries get slower as history grows |
| ~1 TB data volume | Raw capture is ~650 MB/day compressed for one region | Retention becomes a decision made by running out of disk |
| A service account to run it | The scheduled task must run with nobody logged on | Same problem as the laptop, in a nicer rack |
| "Log on as a batch job" for that account, and a password that does not expire | A silent password expiry is the likeliest way capture dies | Capture stops, and nothing says so |
Outbound HTTPS to data.bus-data.dft.gov.uk |
The feed itself | Nothing to capture |
| Outbound HTTPS to CRAN, or an internal package mirror | Installing and updating R packages | Every package change becomes a ticket |
| R installed per-machine (current 4.x) | The whole stack is R | — |
| Auto-start at boot, restart on failure | Patch reboots are the other way capture dies | A gap after every Tuesday reboot |
| An alert when no data has been written for 15 minutes | Silence is the failure mode | We find out days later, from a hole in a chart |
| A backup of the data volume, or a written decision not to | The raw capture cannot be re-fetched from anywhere | One disk failure ends the history |
If other people are to read it live
| What | Why |
|---|---|
| A SQL Server database (or Postgres) | Power BI DirectQuery, and anything else that wants to query rather than download |
| An on-premises data gateway | What lets the Power BI service reach a database inside the network |
| A hostname, a certificate, and IIS in front of the dashboard port | An intranet URL people can open, instead of files on a timer |
The database is a one-line change at our end — storage already sits behind its own script for exactly this reason — but it is not free to them, so it is worth asking only if live-for-everyone is actually wanted.
Sizing, with the arithmetic shown
Raw compressed feed capture measured at roughly 650 MB per day for one region at 20-second polling. That is ~20 GB a month, ~240 GB a year. A 1 TB volume is about four years at the current area; doubling the area captured roughly doubles the rate. The derived files the queries actually read are a fraction of that, and are rebuildable from the raw at any time — worth measuring after a week on the server rather than guessing now.
Nothing here needs a licence. R and everything it uses are free, and the store runs inside the R process — there is no database service to install, patch or licence unless the Power BI question above says otherwise.
What we are not asking for
Worth saying, because it keeps the ask small and answerable:
- No admin rights on anybody's laptop. None were needed to get this far.
- No inbound internet. The server calls out; nothing calls in.
- No new software licences.
- Not journey planning. Planning a route needs a different piece of software with a large Java memory footprint. It is a separate machine and a separate conversation, and nothing in this list depends on it.
Questions we need answered, not decided by us
- Who owns and patches the box — the IT platform team, or us on a VM they hand over? This changes who is called when it stops.
- Does the audience for the figures have Power BI Pro? If yes, the database and gateway route is the better one. If no, we host the page and the ask is a hostname and a certificate instead.
- How much of the network are we capturing? The list above is sized for one region. Wider is mostly disk.
- Is the raw capture kept indefinitely? It is the only thing that cannot be recreated, and the only reason a fix to how something is measured can be replayed over past months rather than only improving future ones.
What it would change on day one
Capture starts at midnight instead of at logon, so the morning peak stops being unmeasurable. Delivery and punctuality gain a full day's denominator. The dashboard gets a URL. And the daily figures stop depending on somebody remembering to leave a laptop open.