What is Crawling?
Definition
Crawling is the process by which search engine bots (Googlebot, Bingbot, etc.) systematically browse the web, following links from page to page to discover content. Crawled pages are then processed and potentially added to the search index. Crawling frequency depends on page authority, freshness, and site crawl budget.
Why Crawling matters for Shopify stores
For Shopify stores, crawling determines how quickly new products and content are discovered. Stores with strong internal linking, clean sitemaps, and high authority get crawled more frequently — meaning new products appear in search results faster. Poor site structure delays discovery by days or weeks.
Shopify example
A store publishes 50 new products. With poor internal linking, Googlebot discovers them only through the sitemap, taking 3 weeks to crawl all 50. After adding a 'New Arrivals' section to the homepage linking to all new products, Googlebot crawls them within 2 days.
How Sauron handles Crawling
Sauron's GSC integration shows crawl statistics and highlights pages that haven't been crawled recently — a signal that Googlebot is deprioritising them and that internal linking or content quality improvements may be needed.
See Product SEO on Autopilot feature →Common mistakes
- ✕ Blocking important pages in robots.txt, preventing them from being crawled
- ✕ Not providing internal links to new pages — new pages without links take weeks to be discovered
- ✕ Assuming all pages are crawled equally — low-authority pages are crawled less frequently
- ✕ Confusing crawling with indexing — a crawled page is not guaranteed to be indexed
Related terms
Automate your Shopify SEO
Understanding Crawling is step one. Sauron handles the implementation — automatically, for your entire Shopify store.