Turning Linear Search into a Surprisingly Performant Data Processing Engine

The pitch

There’s an immense amount of public, highly granular data about the UK, especially England and Wales: the price of every property transaction¹; floor areas and construction years², all with full addresses; street-level crime³; noise levels at 10 m by 10 m granularity⁴; public transport timetables⁵; mapped woodlands⁶; conservation areas, council houses, listed buildings⁷, and schools⁸; plus detailed demographics⁹ on education, renting, and more. When I was looking to buy a property, these datasets helped me narrow down where to move and what to expect.

Sites such as HouseMetric¹⁰ and CrystalRoof¹¹ already provide some of this data for a given postcode. I had been living in London for only four years, so I didn’t know its many neighbourhoods well enough. That’s why I wanted to flip the lookup: instead of going from a postcode to its attributes, I wanted to go from the description of my ideal area to matching postcodes. This is how perfect-postcode.co.uk¹² began.

Perfect Postcode introduced me to several hidden gems I’d never heard of but fell in love with after visiting. It pushed me beyond familiar areas and gave me a broader picture of London. It also showed me what my expectations would cost and how the equation changed with a compromise or two.

In short, users can set minimum and maximum values for each property and area attribute. For example, a user can ask to see only areas within a 10-minute walk of a station, with noise below 56 dB, with a 2-bed costing less than £600k, plus any of 83 other criteria. The result is a heatmap showing which areas contain the highest number of matching past sales, and therefore where the criteria are most likely to be met. From there, users can narrow their search, contact estate agents, get access to off-market properties, and keep an eye on Rightmove and Zoopla.

Rough architecture

Filtering happens per property: all the public data is joined onto roughly 25 million property records. Matches appear as H3¹³ hexagons at lower zoom levels and postcode boundaries at the highest zoom.

The basic architecture is straightforward: download the data, normalise and join it, add a filtering UI, then write a backend that returns matching areas. The difficult part is the main non-functional requirement: filtering must feel instant so users can see how each change affects the results.

The full dataset contains 25 million rows. That’s really not that much data for today’s servers; it’s not too large to keep in memory, and it also won’t grow exponentially. That’s why I decided to see how far a brute-force linear scan could take me with it. Making brute force fast requires some preparation. Most attributes are quantised to 16 bits because values such as crime counts and room counts need no greater range or precision. There is only one index: a spatial grid that prunes addresses outside the user’s viewport. The data uses a row-major layout because queries often filter on dozens of attributes at once. Rows are also sorted by spatial position, so each scan touches contiguous chunks. This keeps scans cache-friendly and easy to process in parallel too.

The actual evaluation logic is as simple as:

let base = row * num_features;
filters.iter().all(|f| {
    let raw = feature_data[base + f.feat_idx];
    raw != NAN_U16 && raw >= f.min_u16 && raw <= f.max_u16
})

For me, the server’s roughly 12 GB memory footprint is justified by its sub-100 ms p99 query latency.

Still, 100 ms plus network latency is too slow for instantaneous feedback in the UI. Fortunately, filters are always combined with AND, and users can adjust only one at a time by changing an attribute’s bounds. This allows a simple trick: when a user starts adjusting a filter, the frontend requests that attribute’s minimum and maximum values for every visible hexagon or postcode. It can then update the colours and filtering without another round trip by applying a single attribute’s filter client-side. You can see both the backend and frontend optimisations in action here:

Of course, there’s a bit more to it, especially at the coarser H3 levels, where expensive queries benefit most from caching. Other data stays outside the hot path, including individual property histories and travel times. The latter are precomputed for every postcode and destination combination under several constraints, together with full itineraries. But the short version is that brute force can be the best solution when applied in a smart way.

Derived data

Besides the open datasets, the pipelines powering the app create interesting derived values, for example:

• Tree canopy density percentile: based on mapped tree-canopy and woodland coverage
• Price growth percentile: showing where prices are rising or falling fastest
• Public transport travel time: calculated between stations and neighbourhoods using R5¹⁴

What’s next?

Six months of viewings and scouting areas in person gave me the perfect feedback loop. Whenever the numbers and my impression of an area differed, it usually pointed to a missing feature. For example, one Saturday we visited an area of Loughton that had looked perfect based on the numbers. However, we counted more than a dozen English flags hanging from properties, which prompted me to add a Reform UK voter share filter. Now that the product works well for me, it’s time to iterate on user feedback.

Originally, the app’s target audience was my partner and me. As we added features, it became clear that other house hunters could benefit from it too. That means improving the user experience with saved and shareable filters, clearer explanations, and a genuinely usable mobile site, then adding a payment flow and doing some marketing.

1. price of every property transaction: https://www.gov.uk/guidance/about-the-price-paid-data↩
2. floor areas and construction years: https://epc.opendatacommunities.org/↩
3. street-level crime: https://data.police.uk/↩
4. noise levels at 10 m by 10 m granularity: https://environment.data.gov.uk/dataset/562c9d56-7c2d-4d42-83bb-578d6e97a517↩
5. public transport timetables: https://www.bus-data.dft.gov.uk/↩
6. woodlands: https://www.forestresearch.gov.uk/tools-and-resources/national-forest-inventory/↩
7. listed buildings: https://opendata-historicengland.hub.arcgis.com/↩
8. schools: https://get-information-schools.service.gov.uk/↩
9. detailed demographics: https://www.ons.gov.uk/census↩
10. HouseMetric: https://housemetric.co.uk/↩
11. CrystalRoof: https://crystalroof.co.uk/↩
12. https://perfect-postcode.co.uk/↩
13. H3: https://h3geo.org/↩
14. R5: https://conveyal.com/↩