What we learned rebuilding search for a 40,000-page catalogue

The site had around forty thousand indexable URLs and roughly nine templates. That ratio is the whole story of large-catalogue search, and it is why the standard audit deliverable — a spreadsheet of per-page issues — is close to useless at this scale.

Fix templates, not pages

Every issue we found existed in one of nine places and manifested in thousands. A missing canonical was not a missing canonical, it was a template that omitted one. This reframing changes the shape of the work: the deliverable stops being a list of fixes and becomes a set of pull requests.

It also changes who does the work. A per-page audit gets handed to a marketing team who cannot act on it. A template diff gets handed to the engineers who own the template, which is where it was always going to have to land.

The crawl budget conversation is usually a red herring

Teams at this scale reach for crawl budget as an explanation early. It is occasionally correct. Far more often the problem is that the internal link graph is flat: every page is three clicks from the homepage, no page is more important than any other, and the site has provided no signal about what matters.

A flat link graph tells a crawler that nothing on your site is more important than anything else. It will believe you.

We spent more time on internal linking than on any other workstream, and it produced more than any other workstream. Hub pages that actually aggregate, breadcrumbs that reflect a real hierarchy, and related-item modules driven by the content model rather than by a recommendation widget.

Faceted navigation: pick a rule and hold it

• One facet combination indexable, everything beyond it noindexed and nofollowed at the link level.
• Which combination? The one with genuine search demand, established by looking rather than by guessing.
• Write the rule down. Put it in the codebase as a test. Facet rules erode within two releases otherwise.

Rendering

The site rendered its primary content client-side and had been doing so for three years. Google handled it, mostly, on a delay of several days to several weeks. Every other crawler — including the ones now feeding answer engines — handled it considerably less well. Server-rendering the primary content was the single largest change we made, and the one that took the most convincing.