Faceted navigation: how filters quietly eat your crawl budget

Faceted navigation is the filter and sort controls on a category page: size, colour, brand, price range, rating, availability. They are good for users and they are one of the largest technical SEO problems in ecommerce, because of how the maths works.

The maths, briefly

Suppose a category page has five filters, each with an average of six options, and they can be combined in any order.

The number of possible filtered URLs is not thirty. It is every combination of every subset of those filters, multiplied by the orderings your URL structure allows, multiplied again by sort options and pagination.

That is comfortably in the millions from one category. Add a few hundred categories and the crawlable URL space on your site is effectively infinite, while the number of pages you actually want indexed might be five thousand.

Every one of those URLs is a page a crawler can find and might request. The consequences are all downstream of that:

• Crawl budget goes to URLs you do not care about. New products and updated pages get discovered slowly, because the crawler is busy elsewhere.
• Ranking signals get diluted. Internal links spread across thousands of near-identical URLs instead of concentrating on the category page.
• Near-duplicates compete. Several filtered variants can rank for the same query, and search engines pick between them arbitrarily.
• Index bloat. Thin, filtered pages get indexed and drag on how the site is assessed overall.

Three checks that tell you if this is your problem

You can establish this in about an hour.

1. Compare discovered URLs to real pages. In Search Console, look at the total URLs known versus the number of pages you intend to have. If you have 4,000 products and Search Console knows about 900,000 URLs, you have found it.

2. Read your server logs, or the crawl stats report. Look at what crawlers actually requested over the last month. If a large share of requests carry filter parameters, that is crawl budget you are paying for and getting nothing from.

3. Search your own site. Use a site: search for your domain with a filter parameter in it. If filtered URLs are showing up in the index, they are being indexed, whatever you thought your configuration said.

If all three come back clean, this guide is not your priority. Plenty of smaller catalogues never hit it.

Decide which facets deserve a page

Before touching any configuration, make a deliberate decision per facet. There are only three categories, and the test is search demand, not technical convenience.

Indexable, with a real page. Facets that match how people genuinely search. “Waterproof hiking boots” or “size 12 running shoes” are things people type. If there is real volume behind a combination, it deserves a proper page: its own title, its own description, its own intro copy, and a stable URL. Not a filtered view that happens to be crawlable, but a page you chose to make.

Crawlable but not indexable. Facets that are useful to users and produce reasonable pages, but that nobody searches for as a phrase. Let crawlers through so links are followed, keep them out of the index.

Neither. Sort orders, pagination beyond the first few pages, price sliders, availability toggles, and any combination of three or more filters. These produce no search demand and unlimited URLs. They should not be crawlable.

Most sites end up with a small handful in the first bucket, a moderate number in the second, and the overwhelming majority in the third. Getting that split written down is most of the work.

Implementing it without breaking the filters

Once the decision is made, the mechanics are straightforward, but the tools do different jobs and get confused with each other constantly.

noindex on the page. Allows crawling, prevents indexing. Links on the page are still followed, so value flows through. Use this for the crawlable-but-not- indexable bucket, and for any pattern that already has inbound links or traffic you do not want to sever.

robots.txt disallow. Prevents crawling entirely, which is the only thing that genuinely saves crawl budget. The catch is important: a blocked URL is never fetched, so a noindex tag on it is never seen. If a URL is already indexed, blocking it in robots.txt can leave it stuck in the index. Deindex first with noindex, then block once it has dropped out.

Canonical tags. Point filtered URLs at the unfiltered category to consolidate ranking signals. Useful, and worth doing, but it does not stop crawling. Canonicals are a consolidation tool, not a crawl-control tool. Treating them as one is the most common mistake here.

rel="nofollow" on filter links. Reduces how readily crawlers discover the combinations in the first place. Worth doing on the third-bucket facets as a supporting measure rather than a primary defence.

URL design. If you are in a position to change it, keeping filter state in a consistent parameter order, or out of the URL entirely for non-indexable facets, prevents the ordering explosion at source. This is the cheapest fix if you are building, and usually the most expensive if you are retrofitting.

A sensible order of work

1. Measure first with the three checks above, so you know the scale and can tell later whether the fix worked.
2. Write down the facet decision for every facet, in the three buckets. Get whoever owns the catalogue to agree with it.
3. Build real pages for the small set of combinations with genuine demand.
4. Apply noindex to the crawlable-but-not-indexable set.
5. Wait for those to drop out of the index. This takes weeks. Check before moving on.
6. Then apply robots.txt rules to the never-crawl set.
7. Re-measure discovered URLs and crawl distribution after a month.

Step five is the one people skip, and skipping it is what leaves URLs stranded in the index with no way to remove them.

What good looks like afterwards

You should see discovered URLs fall substantially, crawl requests shift toward product and category pages, new products getting indexed faster, and category pages consolidating rather than competing with their own filtered variants.

The last of those is worth watching specifically. If a category page and its filtered views were splitting a query between them, consolidating usually shows up as one URL improving rather than the total staying flat. Measured properly, on the page and query that changed rather than sitewide, that is a clean before-and-after result: exactly the kind of change worth measuring per page rather than by traffic chart.