AI Visibility Snapshot

Glossary · Search fundamentals

Crawling and indexing

Definition: Crawling is how search engines discover and fetch pages; indexing is how they store them so they can appear in results. A page that is not indexed cannot rank.

Crawling and indexing, explained

Search engines work in stages. Crawlers such as Googlebot discover pages by following links and reading XML sitemaps, then fetch and render them. Pages judged useful and allowed to be indexed are stored in the search index. Only indexed pages can appear in results, and only pages that can be crawled and read are available to the AI systems that retrieve through search.

Many things quietly interrupt the process: a stray noindex tag or robots rule, pages reachable only through scripts or search forms, endless parameter and filter URLs, redirect chains, duplicate versions without canonical tags, and thin pages a search engine decides are not worth keeping. Search Console shows which pages are indexed and why others are not.

Crawl budget is the number of URLs a search engine is willing and able to crawl on a site in a given period. Google notes that most sites do not need to worry about it; it matters for very large sites, sites that generate many URLs automatically, and sites with slow servers. For everyone else, the aim is simpler: make the important pages easy to find, and keep the index free of pages that should not be there.

AI Visibility Snapshot

Know exactly where you stand.

Request an AI Visibility Snapshot. We test the questions your clients ask across ChatGPT, Gemini, Perplexity and Google AI Overviews, and show you where you stand. Every inquiry is read by a person and answered within one business day.

  • ChatGPT
  • Gemini
  • Perplexity
  • Google AI Overviews