HTTP Headers for SEO: What Crawlers Actually Read

Search Console's URL Inspection tool tells you what Google decided about a page, but the HTTP response that produced that decision sits a few clicks deeper, and few people ever open it. Long before any algorithm reads your title tag or your schema markup, Googlebot's fetcher is reading response headers: Cache-Control, Last-Modified, ETag, Vary, Content-Type, and sometimes a Link header carrying a canonical signal. Get these wrong and the symptoms look like ranking problems — pages that get crawled too rarely, updates that take weeks to register, duplicate versions of a page competing against each other — when the real cause is a header nobody has looked at since the server was configured.

This is not about X-Robots-Tag, which is a single, well-documented switch for indexing that a different article on this site already covers in depth. This is about the rest of the response: the headers that decide how often Google re-fetches a URL, whether it trusts a 304 instead of downloading the page again, and which representation of a page it treats as authoritative when your CDN, your CMS and your HTML disagree with each other.

How to See the Response Google Actually Gets

A browser's developer tools show you a negotiated response — one shaped by your browser's Accept-Encoding, your cookies, and your IP address. That is not necessarily what Googlebot received. The most direct way to inspect raw headers is curl, run against the live URL with the body discarded so only the header block prints.

The gap between your own curl test and what Search Console shows is itself diagnostic. A persistent mismatch usually means a CDN edge, a WAF, or a bot-detection layer is treating Googlebot's request differently from a plain request sent from your terminal, and that difference is worth chasing down before touching anything else.

• curl -sD - -o /dev/null https://example.com/page — fetches the page, prints the response headers to standard output, and throws the body away
• Avoid testing with curl -I (a HEAD request) as your only check — some servers and CDNs answer HEAD differently from GET, so a HEAD-only audit can hide a header that only appears on the real response
• In Search Console, open URL Inspection, then View Crawled Page, to see the HTTP response Google actually received on its last crawl — the closest thing to ground truth without server log access
• Server or CDN access logs filtered to verified Googlebot IP ranges are the real ground truth: they show exactly which headers shipped to the crawler on a specific request, not what a manual curl test produces a minute later

Cache-Control and How Often Google Comes Back

Cache-Control tells any cache — a browser, a CDN edge, a proxy — how long a response can be reused before asking the origin again. max-age=3600 means an hour of freshness; no-cache means it can be stored but must be revalidated before every reuse; no-store means it must never be stored at all. Googlebot's own crawl scheduling is not a direct function of max-age — it learns how often a URL actually changes and adjusts its own fetch cadence independently of the cache header. Where Cache-Control does matter is indirect but real: it governs what a CDN serves from its edge, and a CDN serving stale content because of an overly long max-age means Google can fetch a URL promptly and still receive an outdated version, because the staleness lives one layer below the crawler.

A blanket no-store across an entire site removes conditional caching everywhere, which increases origin load and slows response times for every request, Googlebot included. Slower responses eat into the finite time Google allocates to a site during a crawl session — the opposite of what the no-store was probably meant to protect.

• Static, versioned assets (CSS, JS, hashed image filenames) — a long max-age, often a year, because the filename itself changes when the content does
• Pages that update occasionally (an article, a product page) — a short max-age or no-cache with a validator, so a CDN revalidates instead of blindly serving a stale copy
• Pages that must never leak between sessions (an account page, a cart) — no-store or private, so a shared cache never mixes one visitor's response with another's
• Never set no-store sitewide as a blanket precaution — it disables conditional caching for pages that never needed the protection and gains nothing for the ones that did

Last-Modified, ETag, and the Value of a 304

A conditional request is how a crawler avoids re-downloading a page that has not changed. The server sends a Last-Modified date and, optionally, an ETag — an opaque identifier for the exact version of the response. On the next visit, the client sends that value back as If-Modified-Since or If-None-Match. If the server confirms nothing has changed, it replies with a 304 Not Modified and an empty body instead of resending the page.

This matters because a reliable Last-Modified or ETag is one of the cheapest signals a site can give a crawler about whether content actually changed, without forcing a full re-download and re-render to find out. At the scale of millions of URLs, a working conditional-request setup is what lets a crawler spend its finite fetch and render budget on pages that changed, instead of re-processing pages that did not.

• Last-Modified generated from the current timestamp on every request, rather than the content's actual last edit, tells every crawler that everything changed every time — it defeats the purpose and invites more frequent, wasted re-fetching, not less
• An ETag that differs by server node behind a load balancer, because each node computes it slightly differently, makes every request look changed even when the content is byte-identical, so 304 responses never fire and the conditional request accomplishes nothing
• Last-Modified missing entirely on templated pages, even though the CMS already knows the real content-updated timestamp and could send it for free
• A weak ETag (prefixed W/) is normal and often correct for compressed responses — gzip and brotli produce different byte streams from the same content, so treating a weak ETag as a bug and forcing it to be strong can break the comparison it was designed to make

Vary: The Header That Quietly Breaks Caching

Vary tells any cache which request headers change the response, so the cache knows it needs a separate stored copy per value of that header. Vary: Accept-Encoding is normal and expected — a gzip-compressed response and an uncompressed one are legitimately different bytes for the same content. Problems start with headers that fragment the cache far more than the situation calls for.

Vary: User-Agent shows up when a server sends meaningfully different HTML to different user agent strings — desktop versus mobile, or a bot-specific version. Sending Googlebot content that differs substantially from what a regular visitor sees is exactly the ambiguity that makes user-agent-based serving risky; a responsive layout that sends the same HTML to every user agent avoids the question entirely. Vary: Cookie tends to do the most damage in practice: if a tracking or session cookie is present on nearly every request, a CDN honoring that header effectively cannot cache the page for anyone, because every visitor becomes a unique cache key. The result is slower responses sitewide, crawlers included.

• Audit which cookies your site sets before a page renders, and check whether Vary lists them — one always-present tracking or consent cookie behind Vary: Cookie can silently disable caching across the entire site
• Vary: * tells every cache the response is never reusable under any condition — this is almost always a misconfiguration rather than an intended state, and worth removing
• If desktop and mobile genuinely receive different HTML at the same URL, Vary: User-Agent at least keeps caches from mixing the two versions, but treat it as a stopgap on the way to a single responsive template, not a permanent fix

The Link Header Version of rel=canonical

Most canonical signals live in an HTML link tag inside the head of the document. That option does not exist for a response that has no HTML head at all — a CSV export, an API response, or any downloadable file. For those, the same signal travels as a Link response header instead, naming the canonical URL and the rel=canonical relation directly in the header value. Google reads this header on any resource type, which makes it the only mechanism available when there is no document structure to put a tag inside.

The risk shows up when a page has both a header canonical and a tag canonical and they disagree — after a migration, a redirect chain, or a template change that updated one but not the other. Treat the header and the tag as the same signal expressed through two different channels, not two independent choices; when both are present, they should point at exactly the same URL.

• Syntax: Link, followed by the absolute target URL wrapped in angle brackets, then a semicolon and rel="canonical" in quotes — the format is strict and a missing bracket or quote is typically ignored rather than flagged
• A single response can carry several Link headers for unrelated purposes, such as preloading a resource or pointing to an alternate version; only the one carrying the canonical relation affects this signal
• Verify it with curl, not a browser extension built to read the HTML tag — most canonical-checking tools never look at the response headers at all, so a header-only canonical can go unaudited indefinitely

Content-Type: The Header That Decides How Everything Else Gets Parsed

A Content-Type of text/html with a UTF-8 charset tells a crawler how to turn raw bytes into characters and markup before it can read anything else in the response — the title, the canonical tag, the structured data. A missing or wrong charset causes the classic mojibake symptom: apostrophes and accented characters rendering as garbled sequences in a search result snippet, even though the page looks correct in a browser that happened to guess the right encoding anyway.

A wrong Content-Type category is a different, larger problem. An HTML page mistakenly served as plain text, or a JSON API response served as HTML, changes whether the body gets parsed as a document at all. A parser treating a response as plain text has no reason to look for a canonical tag, a set of links, or any embedded markup, because as far as that Content-Type is concerned none of it exists — it is just characters.

• Confirm the charset in Content-Type matches the document's actual encoding — a header claiming UTF-8 over content that is actually a different encoding, or the reverse, is the standard cause of encoding-related mojibake
• Check non-HTML deliverables individually: an XML sitemap should be served with an XML content type, an RSS or Atom feed with its own matching type, robots.txt as plain text — a mismatch here does not always break parsing outright, but it strays from what each format's own readers expect
• A redirect's own Content-Type rarely matters, since a redirect response carries no body meant to be parsed — the header that matters is on the final successful response at the end of the chain

A Practical Header Audit, Start to Finish

Run this as a pass across a representative sample of templates — the homepage, a listing or category page, a content page, and any non-HTML endpoint the site serves — rather than checking a single URL and assuming the rest match.

None of these headers will move a page up the results on their own, and none are a substitute for content worth ranking. What they control is whether Google's fetcher can crawl and re-crawl a site efficiently, and whether the version of a page it caches and serves in search results is the one the site meant to publish.

• Pull raw headers on each template, including one URL served fresh from origin and one served from a warmed CDN edge, and compare them side by side
• Check that Cache-Control and Vary make sense per template — an hourly-updated page should not carry a static asset's cache lifetime, and a page with no personalization should not carry Vary: Cookie
• Test the conditional request directly by replaying the previous ETag as If-None-Match and confirming the second request comes back as a 304, not another full response with a freshly generated ETag
• Cross-check canonical signals: does the HTML tag agree with any Link header canonical, and does either point somewhere unintended after a past migration or redirect
• Compare it against Search Console's View Crawled Page for the same URL — a persistent difference from your own test points to a CDN, WAF, or bot-detection layer treating Googlebot differently from a normal request