In February of this year, the UK Government launched its Fuel Finder API, a data source of fuel prices, submitted by fuel station operators, within 30 minutes of changes. The goal being for drivers in the UK to be able to find the cheapest fuel prices wherever they might be in the country.
It’s a commodity product, but pricing is hyper-local, because it can be. A station on a quiet rural A-road and one across from a Tesco are selling the same fuel at very different prices for reasons that have nothing to do with the fuel.
Regardless of what you may think of the UK Government, the Government Digital Service has been doing great work for years, and I’ve long wanted to work with the public data APIs the UK Government publishes. I like building software that genuinely helps people, so this seemed like the perfect opportunity.
So, I opened Xcode and got started. I didn’t have a strong idea of exactly what I wanted to build. I felt that the app needed to meet at the intersection of what the data was telling us and what the user needed to hear. And, so I would need to build by feel, more than design. I started out, roughly, with a full-screen map, and a bottom-attached sheet, like Apple’s Maps app.
I built the basics, like listing the stations on the sheet, and displaying markers on the map. All with fake data to build out a UI prototype. Apple had, obviously, verified that the UI concept worked for Maps, so I wanted to make sure that it still made sense for this use specific case. The answer was almost certainly going to be yes, of course, but I’d rather not make assumptions.
Very quickly I reached a point where I needed the actual data to begin to steer the user experience in the direction of the fuel price comparisons, at what would be the heart of the app. So, I logged into the developer website, to obtain whatever credentials I may need, and to look at the shape of the API.
My heart sank. I’m not sure what I was expecting exactly, but the API I found wasn’t it. I suppose on some level I was hoping I could access the API directly from user’s device. This is geospatial data, the API would reflect that, no?
No. The API provides two endpoints. One to fetch fuel station data, in batches of 500. And another, to fetch fuel price data, in batches of 500. The API is rate limited to 100 requests per hour, per credential set. And there are just south of 8000 fuel stations in the UK. I obviously couldn’t call these endpoints from the app directly. I would need a server. Crumbs. I have built and administered servers in the past, but it’s not my favourite thing to do, I’ll be honest. I couldn’t see a way around it, though.
So, I set about building a server. At this point, in my mind, I was imagining an API where the app requests all fuel stations in a radius from a coordinate, or all fuel stations within some bounding box. That’s going to be so many requests. The server is going to melt. Or cost me an arm and a leg.
I tried to put it out of my mind, and get on with the business of building the agent that would interact with the Fuel Finder service. I started with more of a script than a server, that fetched the data, to get my head around the order of operations, including authorisation, and re-authorisation. It was fairly straight-forward, no hiccups. And luckily, as the service was in beta, it was throwing errors, so I was able to test it failing, against production, quite thoroughly.
At this point in the proceedings, I was fetching the fuel stations in batches of 500, until I had exhausted the set, then fetching the fuel prices, in batches of 500, until I had exhausted that set. I was then merging the data together, so every fuel station (JSON) object contained its own prices. I was also successfully obtaining an (OAuth) token, refreshing the token, and starting all over again when the token couldn’t be refreshed any more.
I was feeling good. Mostly. The thought of building an efficient geospatial API was creating anxiety (dread?) in me. Postgres with PostGIS? How big will the server need to be? How many servers will I need to scale to?.
I tried to delay the next step, and started messing around with the data. I had been persisting the data, temporarily, in a SQLite database, to avoid keeping the data in memory, as I didn’t know how much data there would be. So, I decided to check. What if I read all the data into memory and create a single JSON file? I had been persisting the data with a simple schema, with an id, supplied by the data source, and a JSON blob. So, this was trivial. And not as much data as I thought.
12MB. For the whole dataset. Huh. At the time, there were just shy of 7000 fuel stations in the data set. Do I need a geospatial API? The API documentation says that we, the developers, must do our best to keep the data fresh, given that prices could change every 30 minutes. So, my first thought was whether the app could download 12MB every 15 minutes, say? No, that’s too much. I couldn’t let the thought go, though. I think, out of desperation.
Ah-hah! I’ll Brotli encode it. I’m a genius. 7MB. Now, that’s a great reduction, but it required setting the encoding to maximum, and using the text mode option. It is a great reduction, but it’s not enough. The desperation was still strong in me, though.
Feeling a little deflated, I started poking around the data, to deflect my attention from defeat. I realised that I would need to transform the JSON objects to something more suitable for the app. There were obviously many opportunities to collapse, merge and combine data, going from a general API format for everyone to one for a specific app.
So, once I’d made all the changes, I was happy that the data ingestion in the app would be simpler, and, that it had reduced the total dataset to 5MB. Much less, but still too much to download frequently.
A little happy, a little sad, I went back to building the server. I realised that the server wouldn’t be the agent fetching the data from the Fuel Finder Service. That would be the script I had written, that would likely run on a CRON job, every 15 minutes or so.
I figured it would write the data to the database, and then the server would read the database to fulfil the requests. I quite liked that idea. The writes and the reads, separated by process. The time had then come, to start building the server. I had no choice at this point. Everything else was done.
But how? How was I going to build this server, and have it be remotely efficient? It seemed like there would be too many requests? As the user panned and zoomed the map, I would need to cancel existing requests, and make new ones. Crumbs. Or I could wait until the map settles to make the requests, but then the app feels unresponsive. But, there will be a request delay anyway, so the app is going to feel unresponsive. Double Crumbs.
I could cache requests, I suppose? Or “normalise” the bounding box boundaries, so the app is fetching “tiles” of requests. WAIT. Map tiles. (Software) Maps are made of tiles. Instead of making requests around coordinates the app could fetch, and cache, tiles of stations. The stations aren’t moving. Nor are the coordinates of the UK.
So, the server would pre-compute map tiles, and calculate which stations fall into which tiles. The app would then need similar logic to determine which tiles are visible, and therefore to fetch.
I had never built a system like this before, so I needed to give it a little thought first. It occurred to me, that I could actually generate these tile files at the point that the script fetches and ingests the data. The server would then be a static file server. In fact, it could keep the files in memory, and watch for file changes.
The relief was falling away. This wasn’t going to be the epic task that I thought it was going to be. Or maybe I was celebrating too soon, as, at this point, I hadn’t actually built the system yet.
There was a thought, though, that was niggling away in my mind. I’ve forgotten to do something.
It took longer than I’d like to admit but I realised that I had not Brotli encoded the dataset that contained my transformations.
450kB. Kilo. Bytes. 450 of them.
Wait. Hold on. What? The user could actually download the entire dataset every 15 minutes. That’s completely acceptable to me.
It took a minute for that thought to set in, and for the implications to become clear.
I wouldn’t need the tiles. I would only need the script to ingest the data, and a file server for the app to download the entire dataset. No requests in the app (except for the dataset). No coordinate data leaving the device. No latency. Markers immediately rendered in the app, in real time.
The relief was immeasurable. I had stumbled my way, by accident more than by design, into the ideal API:
- Fetch, merge, transform and compress the Fuel Finder data.
- Serve a single file, with cache headers.
I began simplifying everything. And, in doing so it hit me I’m an idiot. Why on earth do I need a server for a single file?! The script can upload the file to a CDN. No more server. Well, I need somewhere for the script to run. Okay, fine. But, it won’t have user requests, and it will have a small and finite workload. And then it hit me. I really might be an idiot. I don’t need a server for this. AWS’s Lambda, Cloudflare’s Workers, or something similar, is all I need.
I don’t need a server at all.
I began simplifying everything. Again.
I decided upon using Cloudflare, as the R2 service (and CDN) does not charge for bandwidth, and their Workers platform has the ability to run the scripts on CRON schedules.
So, I rewrote the script to do what it had been doing before, but storing the credentials in the Workers KV data storage, and uploading the final data file to R2. I tested it locally with Wrangler, and everything worked perfectly.
Until I deployed it, and ran it on the production servers.
I had been using my original script, and testing the new Cloudflare version, from the UK. So, when the script was run on Cloudflare’s infrastructure in various places in the world, I was blocked.
Why is this happening to me?
I wasn’t 100% sure, this was the problem. The requests just were blocked from the Fuel Finder infrastructure. I tried running the Worker script in the UK, but you can’t really specify that. The options are a little more hand-wavy. You can specify AWS-equivalent regions, for example, but you can’t ask Cloudflare to run the Worker in specific countries.
I needed to know if this really was the issue. So, I created an AWS Lambda function in their London datacenter, to forward requests from my Worker script, wherever it was running in the world.
I’m not sure whether I was happy or sad to learn that this was, in fact, the problem.
Faced with moving the whole party off Cloudflare to, say, AWS, or leaving just the request forwarding service in place, I opted for the latter. At this point I was done. It had taken longer than it should have — due to my actions / decisions — and I needed to move onto other things.
I had calculated that my meagre usage of AWS was going to either cost me nothing, or very little, so it was an easy decision to make.
The final solution, then, was:
- Fetch, merge, transform and compress the Fuel Finder data, on Cloudflare’s Worker platform, on a CRON schedule, where the requests are sent via an AWS Lambda function (in London).
- Upload the data to Cloudflare’s R2 service, and serve the data via their CDN.
Phew
The best part isn’t that it costs nothing, or that there’s no server to maintain. It’s that the user opens the app and the stations are already there. No spinner. No “finding stations near you”. No coordinates leaving their device.
The whole detour: the PostGIS clusters I never built, the tile server I almost built, the dread about scale, came from architecting around assumptions I’d never bothered to check. The dataset was tiny. It was always going to be tiny. I just hadn’t looked.
Early on, I said I’d build the app by feel rather than design. I didn’t expect that to apply to the backend too, but the system I ended up with is the system I would have designed on day one, if I’d looked at the data before worrying about the load.
I took the scenic route. But I’m where I wanted to be, and the app is the part that matters now.