schmelczer.dev

Turning linear search into the most-performant data processing engine

Describe your ideal area, and a wealth of national statistics helps you find the hidden gems: 85+ filters, a live heatmap, and sub-100 ms responses.

6 min read
The Perfect Postcode dashboard with active filters on property type, price, transit time, and crime, showing a Manchester map with matching properties highlighted as a heatmap.

The pitch

There’s an immense amount of publicly available, highly granular data of the UK, especially England and Wales: the price of every property transaction, floor areas and construction years, all with full addresses; street-level crime; noise levels at a 10m by 10m granularity; the full schedule of every public transit option; the location of all trees and woodlands; conservation areas, council houses, listed buildings, schools; and detailed demographics on education, renting, and so on. When I was looking to buy property, these came in really handy for deciding where to move and knowing what to expect.

There are numerous websites, like StreetCheck1 or CrystalRoof2, which give you some of this data for a given postcode. My problem was that I only moved to London four years ago, and London has way too many areas for me to have the full picture of all its neighbourhoods. So I wanted to flip the lookup: instead of going from a postcode to its attributes, I wanted to go from the description of my ideal area to the postcode. That’s why I created perfect-postcode.co.uk3.

Perfect Postcode has found me multiple hidden-gem areas I hadn’t even heard about, but fell in love with when visiting in person. It became an easy way to leave the comfort zone of well-known places and get a holistic picture of London instead. On top of this, the site gave a grounded reality check on how much my expectations would cost, and how the equation changes with a compromise or two.

In short, users can filter properties by setting minimum and maximum allowed values for each attribute: only show me properties at most a 10-minute walk from a station, with a noise level of less than 56 dB, and 85 more filters like these. The result is a heatmap of which areas have the most matching properties (based on past sales), suggesting where the criteria are most likely to be satisfied. From there, the next step is to narrow the search to those areas, contact estate agents, get access to off-market properties, and keep an eye on Rightmove and Zoopla.

Rough architecture

The filtering happens on a per-property level: all the public data is projected onto the 25M or so properties, and the results are shown as a heatmap of H3 hexagons of varying granularity, then as postcode boundaries at the highest zoom.

Given the spec, it’s easy to come up with a simple architecture: download all the data, normalise, join, add a filtering UI, and write a backend that applies the filters and returns the matching areas. But then we have to consider one big non-functional requirement: the filtering has to be snappy, so that it’s easy to get an intuitive sense of the cause and effect of changing filters.

The backend has to filter the 25M-row dataset for every query. Given that this is actually a fairly small number, and it’s not expected to grow exponentially over time, I figured we can just keep the full dataframe in memory and see how far brute-force linear search takes us. Effective brute-forcing requires some preparation. So most attributes are quantised to 16 bits, as things like the number of crimes committed or the number of rooms don’t justify anything larger. There’s only a single index: a spatial grid for pruning all but the addresses roughly overlapping the user’s viewport. The data is stored in a row-major format, as queries often filter on a few dozen attributes at once. And the rows are sorted by their spatial position, so the linear search touches contiguous chunks, which makes the scanning incredibly cache-friendly, especially when each chunk is iterated over in parallel.

The actual evaluation logic is as simple as:

let base = row * num_features;
filters.iter().all(|f| {
    let raw = feature_data[base + f.feat_idx];
    raw != NAN_U16 && raw >= f.min_u16 && raw <= f.max_u16
})

In the end, it takes around 12 GB of memory to run the server, which is very much justified by the P99 latency it enables: less than 100 ms.

Still, 100 ms plus network latency is too long for proper real-time feedback on the UI. Fortunately, the filters always apply in an AND fashion, and users can only adjust one at a time by changing an attribute’s min/max bounds. This leaves room for a simple trick to make the map more responsive: when a user starts adjusting a filter, the frontend requests that attribute’s min/max values for every visible hexagon or postcode, so the colour coding and filtering can happen on the frontend without a round trip. You can see both the backend and frontend optimisations in action here:

Of course, there’s a bit more to it, especially around the coarser H3 hexagons: they’re the most expensive to filter on, so they get cached. A lot more data is kept outside the hot path too, such as the history of individual properties, or the travel times, pre-rendered for every combination of postcode and destination under different constraints, complete with the full itinerary. But the short version really is that brute force can sometimes be the best solution when applied in a smart way.

Derived data

Besides the open data, the data pipelines powering the app create interesting derived values, like:

  • Tree canopy density percentile: based on the dataset of all trees and woodlands within the UK
  • Price growth percentile: to help understand the expanding and shrinking parts of the market
  • Public transit travel time: calculated for every single station and neighbourhood using R54

What’s next?

Going to viewings and scouting areas in person for six months gave the perfect opportunity to find which feature was missing whenever the numbers and my impression of an area differed. Now that the product has been perfected for me, it’s time to iterate on user feedback.

Originally, the app’s main target audience was my partner and me. However, as we added more features, it became apparent that there’s no similar offering on the market, and others could get value out of it too. That’s why I’ve started focusing on Perfect Postcode’s user experience: to appeal to users, it needs convenient features, like saving and sharing filters, explanations, an actually usable mobile experience, a payment flow, and some marketing.

  1. StreetCheck: https://www.streetcheck.co.uk↩
  2. CrystalRoof: https://crystalroof.co.uk↩
  3. https://perfect-postcode.co.uk↩
  4. R5: https://conveyal.com↩