Ecommerce SEO
Log file analysis for large catalogs
For a catalog of tens of thousands of URLs, server logs are the only record of what Googlebot fetches. How to get the logs, verify the bot and act by template.
By CartKernel · Published
Search Console tells you what Google indexed and how often it crawled in total. Only the server logs tell you which URLs Googlebot requested, on which day, and what response it got. On a catalog of a few hundred pages the difference does not matter. On a catalog of fifty thousand, the gap between the pages that should be crawled and the pages that are crawled is where rankings quietly leak, and the logs are the only place you can see it.
Where the logs come from
The source depends on the stack:
- Self-hosted WooCommerce, Magento or custom. The web server’s access log, or the hosting provider’s log export.
- Behind a CDN. The CDN’s log delivery, which is usually richer and easier to ship to storage. If the origin is behind a CDN, the origin log sees only cache misses, so take the CDN’s.
- BigCommerce and other hosted platforms. Whatever log access the platform or its CDN exposes; check before planning the analysis.
- Shopify. Raw access logs are not available to merchants. The substitute is the Crawl Stats report in Search Console, which breaks requests down by response, file type, purpose and crawler, plus URL inspection on samples. It answers the first three questions below at lower resolution.
- Headless. The edge or hosting platform’s request logs, which are typically the easiest of all to query.
The fields you need on every row: timestamp, requested URL with query string, status code, user agent, client IP, bytes returned and response time. Referrer is useful when present.
Verify the crawler before you trust a row
A user agent string is a claim, not proof, and a meaningful share of traffic claiming to be Googlebot is not. Google documents two ways to verify: a reverse DNS lookup on the IP that resolves to a googlebot.com or google.com host, confirmed by a forward lookup, or a match against the published IP range lists. Run the verification once over the distinct IPs, cache the result, and drop the rows that fail.
Then separate the crawlers, because they answer different questions. Googlebot smartphone is the one that matters for ranking. Googlebot-Image explains crawl on product images. Storebot-Google is Merchant Center checking landing pages. AdsBot checks ad destinations. Google-InspectionTool is you, testing in Search Console. Treat each as its own series.
Classify every URL by template
The analysis lives or dies on this step. Write a set of patterns that assign every URL to a template: product, collection, collection paginated, collection filtered, internal search, cart and account, blog, static assets, sitemaps and robots, API and ajax endpoints, and everything else. Test the patterns until “everything else” is small. Then count verified Googlebot requests by template over the last thirty days.
An illustrative shape from a store with a filter problem: 31 percent of requests to filtered collection URLs, 22 percent to static assets, 18 percent to products, 11 percent to paginated collections, 8 percent to internal search, and the remainder spread across the rest. The products, which are the pages that earn revenue, are getting less than a fifth of the crawl. That single table is usually enough to get the filter work approved.
Six questions the logs answer
1. How much crawl goes to URLs that should not be crawled? Filtered collections, internal search, session parameters, tracking parameters, legacy paths. This share is your crawl budget going to pages you never wanted indexed, and it is the direct evidence behind facet URLs being indexed and index bloat.
2. Which money pages have not been crawled in thirty, sixty or ninety days? Join the product and collection export with the last verified Googlebot request per URL. The list with no request at all is tomorrow’s product pages not indexed report. Sort it by revenue or margin and you have a priority list for internal linking.
3. What status codes is the crawler receiving? Redirect chains from old migrations still being fetched, 404s from retired products still linked somewhere, 5xx responses under load, and 429 responses if rate limiting is catching the crawler. Google’s documentation on status codes explains how each one changes crawling; a run of server errors reduces crawl rate until the errors stop.
4. How fast are responses by template? Crawl capacity is bounded by how quickly the server answers. If product pages respond slowly, they are crawled less, which is one more reason site speed affects rankings on large stores even before Core Web Vitals come into it.
5. How quickly are new products crawled? Measure the days between a product’s first appearance in the sitemap and its first verified Googlebot request. On a healthy store it is days; on a store with a crawl problem it is weeks, and new arrivals miss their launch window.
6. What happens to retired pages? Count requests to URLs returning 410 or 404. A large number that does not decline over months means those URLs are still linked from somewhere, internally or externally, and the crawler keeps finding them.
Tools that match the scale
- A week of logs on a mid-sized store. Command-line tools are enough: filter to verified Googlebot, extract the URL and status, count by template pattern.
- A month on a large store. A dedicated log analysis tool, or the filtered Googlebot rows loaded into a spreadsheet or a notebook.
- Ongoing on a very large store. Ship CDN logs to a query engine such as BigQuery on a schedule, keep at least ninety days of crawler rows, and build the six questions as saved queries so the review is a report rather than a project.
Whatever the tool, keep the raw crawler rows. Aggregates cannot answer the next question you think of.
Turning findings into changes
| Finding | The fix |
|---|---|
| Large share of crawl on filtered or search URLs | Facet rules: canonicals, fragments or robots rules, no crawlable links to the deep combinations; see faceted navigation SEO |
| Money pages never crawled | Internal links from collections and guides, sitemap freshness, click depth reduction |
| Redirect chains still fetched | Update internal links to the final URL, collapse chains to one hop |
| 404s on retired products | Remove internal links, correct the sitemap, redirect where a successor exists |
| Slow product template | Site speed work on the template, caching at the edge |
| 5xx under load | Hosting capacity, or a rate limit that treats verified crawlers differently from unknown bots |
| Paginated pages absorbing crawl | Fewer pages per collection through sub-collections, correct pagination links, a check that infinite scroll has a paginated fallback |
| Duplicate URLs both crawled | One canonical form in every internal link, confirmed by the canonical tag |
The monthly review
Run the six questions on the same day each month and keep the numbers in one sheet. Watch the share of crawl on money templates, the count of uncrawled products, the error rate, and the median response time per template. After a change such as a facet rule set or a speed release, the logs show the effect within a couple of weeks, well before Search Console’s indexing counts move. That early read is the main reason to do this at all.
For stores that cannot get logs, the Crawl Stats report plus a comparison of sitemap URLs against indexed URLs gets part of the way. For stores that can, an afternoon with the logs tends to reset the ecommerce SEO priorities more sharply than any audit tool, because it shows what the crawler did rather than what the site looks like.