Sitemap.xml: The Complete 2026 Guide for SEO and AI Crawlers

sitemap.xml looks like a problem the industry solved long ago. Every CMS generates one, every SEO plugin has a checkbox for it, and most teams stop thinking about it.

That's usually fine, until it isn't. A sitemap full of redirecting URLs, a lastmod date that changes on every deploy, or a file nobody noticed was over 50,000 URLs can quietly make crawling less efficient.

Companion tool: Lumina's Sitemap Validator parses your sitemap, checks the listed URLs for dead links, noindex pages and canonical mismatches, and warns about stale or uniform lastmod dates.

What a sitemap is and why it still matters

sitemap.xml is an XML file, usually at https://example.com/sitemap.xml, that lists the URLs you want search engines to crawl, with optional metadata such as the last modification date. The format is defined by the sitemaps.org protocol (version 0.9), and Google adds extensions for images, videos and news.

A sitemap helps discovery. Google calls it a hint: submitting one doesn't guarantee that every URL gets crawled or indexed. Google's documentation says sitemaps are most useful for large sites, new sites with few external links, and sites with lots of rich media or news content. For a small, well-linked site, Google may find everything without one.

It still earns its place for three reasons. It gives crawlers a clean list of canonical URLs, which helps on sites where internal linking is uneven. The image, video and news extensions carry information that plain links don't. And an accurate lastmod tells crawlers which pages changed, which matters most for sites whose content changes often.

Sitemap XML structure: every field that matters

The minimum valid sitemap is one URL inside a urlset element:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-05-21</lastmod>
  </url>
</urlset>

Each URL entry can have four fields. Two of them matter.

• <loc> (required): the canonical URL, absolute and including the protocol.
• <lastmod> (optional): when the page last changed significantly, in W3C Datetime format such as 2026-05-21 or 2026-05-21T11:00:00+02:00. This is the one optional field Google uses, under conditions covered below.
• <changefreq>: how often the page changes. Google says it ignores this field, and Bing said in July 2025 that it ignores it too.
• <priority>: a value from 0.0 to 1.0. Google and Bing both ignore it.

The file must be UTF-8 encoded, and the protocol requires entity escaping for special characters. Most important in practice: an ampersand inside a URL has to be written as &, otherwise the XML is invalid and can't be parsed.

Size limits: one sitemap may contain at most 50,000 URLs and at most 50 MB uncompressed. Above either limit, split into several files and use a sitemap index. Gzip (sitemap.xml.gz) is allowed, but the 50 MB limit applies to the uncompressed file.

The 5 sitemap variants

Most sites only need the standard sitemap. The others solve specific problems.

1. Standard URL sitemap

The default. Lists pages with loc and usually lastmod. Use it for HTML pages, PDFs and any other URL you want crawled.

2. Sitemap index

A file that points to other sitemaps. You need one above 50,000 URLs or 50 MB per file, and many sites use one earlier to split by content type.

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemap-pages.xml</loc>
    <lastmod>2026-05-21</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-products.xml</loc>
    <lastmod>2026-05-20</lastmod>
  </sitemap>
</sitemapindex>

The root element is sitemapindex and each entry is a sitemap. You can submit only the index in Search Console, and Google reads the child sitemaps from it.

3. Image sitemap

Adds image:image elements to URL entries so Google can find images it might otherwise miss, for example images loaded in ways the crawler doesn't pick up.

<url>
  <loc>https://example.com/product/widget</loc>
  <image:image>
    <image:loc>https://example.com/photos/widget-1.jpg</image:loc>
  </image:image>
  <image:image>
    <image:loc>https://example.com/photos/widget-2.jpg</image:loc>
  </image:image>
</url>

Add the namespace xmlns:image="http://www.google.com/schemas/sitemap-image/1.1" to the urlset. Each URL entry can contain up to 1,000 images. Google deprecated the other image tags (caption, title, license and so on) in 2022, so only image:loc remains.

4. Video sitemap

Describes videos on your pages. Google recommends video sitemaps but also accepts mRSS feeds and VideoObject structured data, so it's one of several ways to get video found, not a requirement.

<url>
  <loc>https://example.com/tutorials/setup</loc>
  <video:video>
    <video:thumbnail_loc>https://example.com/thumbs/setup.jpg</video:thumbnail_loc>
    <video:title>Setup Tutorial</video:title>
    <video:description>5-minute setup walkthrough</video:description>
    <video:content_loc>https://example.com/videos/setup.mp4</video:content_loc>
  </video:video>
</url>

Namespace: xmlns:video="http://www.google.com/schemas/sitemap-video/1.1". Required: thumbnail_loc, title, description, and either content_loc or player_loc.

5. News sitemap

For news publishers. It lists articles published in the last two days.

<url>
  <loc>https://example.com/news/headline</loc>
  <news:news>
    <news:publication>
      <news:name>Example News</news:name>
      <news:language>en</news:language>
    </news:publication>
    <news:publication_date>2026-05-21T08:00:00Z</news:publication_date>
    <news:title>Headline of the article</news:title>
  </news:news>
</url>

Namespace: xmlns:news="http://www.google.com/schemas/sitemap-news/0.9". Older articles don't drop out on their own: Google asks you to remove those URLs from the news sitemap, or remove their news:news element. A news sitemap can contain up to 1,000 news:news tags.

The lastmod field everyone misuses

Almost every guide says "set lastmod to the date the page changed." That's right, but it leaves out the part that decides whether Google uses the field at all.

Google's sitemap documentation says: "Google uses the <lastmod> value if it's consistently and verifiably (for example by comparing to the last modification of the page) accurate." It also defines what counts: "an update to the main content, the structured data, or links on the page is generally considered significant, however an update to the copyright date is not."

Google doesn't publish how it decides that a site's lastmod isn't reliable. The practical takeaway is simple, though: if lastmod changes whenever a file is saved, it stops telling Google anything. I learned that on this site. In April 2026, 50 of 82 sitemap URLs carried the same date because a sweep had touched them, while the visible content of almost all of them hadn't changed. Whether Google discounted the field is unknowable from outside, but a date that changes for no reason is exactly what Google's wording describes as not verifiably accurate.

The rule I use:

Update lastmod when something on the page changes that a reader or a search engine would care about: body copy, headings, FAQ answers, links in the content, or structured data that now says something different.

Don't update lastmod for changes that leave the page's content and data the same:

• CSS-only changes such as colors or refactored styles
• JavaScript fixes that don't change what users see
• Whitespace, indentation and HTML comments
• Re-generating structured data that describes the same content as before
• Adding width and height to images that already rendered correctly
• Copyright year updates in the footer

Navigation or footer link changes are a gray area. Google counts link changes as significant, but a sitewide menu tweak doesn't change what any single page is about. I don't bump lastmod on every page for that.

The opposite mistake counts too: not updating lastmod when content did change withholds a signal that could get the page recrawled sooner.

Submission methods compared

There are five ways to tell search engines about your sitemap or your changed URLs. One of them no longer works.

1. Sitemap line in robots.txt

The simplest method. Add a line anywhere in /robots.txt:

Sitemap: https://example.com/sitemap.xml

Google, Bing and other crawlers that follow the sitemaps protocol read it. It doesn't belong to any User-agent group, and its position in the file doesn't matter. You can list several sitemaps with several lines.

2. Google Search Console

In Search Console, open Sitemaps and submit the URL. The Sitemaps report shows when Google last read the file, whether it could be processed, and how many pages it discovered. To see which of those URLs are indexed, filter the Page indexing report by that sitemap. For a specific URL, use URL Inspection.

Submitting doesn't make Google crawl faster than the robots.txt line would. The value is the feedback: fetch status, parse errors and a view of indexing per sitemap.

3. Bing Webmaster Tools

The equivalent for Bing, also under Sitemaps. Bing uses sitemaps and lastmod for discovery, and its reporting shows fetch status and errors.

Bing matters beyond Bing search. Microsoft Copilot answers draw on it, and OpenAI lists Bing among the third-party search providers ChatGPT search works with, alongside its own OAI-SearchBot. How much each source contributes isn't published, so treat Bing submission as sensible hygiene, not a direct ChatGPT lever.

4. Ping URLs (no longer work)

Google announced in June 2023 that its sitemap ping endpoint (google.com/ping?sitemap=...) would stop working, and it was switched off at the end of 2023. Bing removed its anonymous ping endpoint in May 2022. Don't use either.

5. IndexNow

An open protocol for pushing changed URLs to participating search engines, instead of waiting for a recrawl. It's covered in the next section.

IndexNow: push instead of waiting

IndexNow launched on October 18, 2021, backed by Microsoft Bing and Yandex. Seznam joined early, Naver followed in 2023, and Yep joined in 2023 as well. The current list published at indexnow.org also includes the Internet Archive and Amazon. Google doesn't support IndexNow.

The idea is simple. Instead of search engines rechecking your pages to find changes, you send the changed URLs to an IndexNow endpoint, and participating engines share the submission. IndexNow is independent of your sitemap: you submit URLs directly, and the protocol's own FAQ recommends using it alongside a sitemap, not instead of one.

Setup takes a few minutes. Generate a key of 8 to 128 characters (letters, digits and dashes), host it as a text file containing only the key, usually at https://yoursite.com/<your-key>.txt, and send changed URLs to the API. If the key file lives in a subdirectory, pass keyLocation, which limits the key to URLs under that path.

POST https://api.indexnow.org/IndexNow
Content-Type: application/json

{
  "host": "example.com",
  "key": "your-32-char-key-here",
  "keyLocation": "https://example.com/your-32-char-key-here.txt",
  "urlList": [
    "https://example.com/new-page/",
    "https://example.com/updated-page/"
  ]
}

A 200 response means the submission was received, and 202 means it was accepted while the key is still being validated. Neither guarantees that the URL gets crawled or indexed at a particular time. IndexNow's FAQ is explicit that it doesn't guarantee immediate indexing.

If you use Cloudflare, you don't need code. Enable Crawler Hints under Caching → Configuration. Cloudflare then notifies IndexNow when it sees a cache miss that suggests content has likely changed. On other platforms, most CMS plugins or a small script after publishing can call the API. For Fastly, that means custom code on Fastly Compute.

Where it helps most: sites that publish or change many URLs, like news sites or shops with frequent price updates, and that care about Bing and the other participating engines. For a small blog that publishes weekly, the effect is modest.

6 common sitemap mistakes

Most of these don't break anything visibly, they just make the sitemap less useful.

• Blocking the sitemap in robots.txt. A broad rule like Disallow: /sitemap also blocks /sitemap.xml. curl can't prove anything here, because it ignores robots.txt. Check your robots.txt rules directly, or look at the fetch status in the Search Console Sitemaps report.
• Including non-canonical URLs. Parameter URLs, URLs that redirect, pages whose canonical points elsewhere. List canonical URLs only. Paginated pages with their own self-referencing canonical are canonical, so they can stay. Sitemap inclusion is a weak canonical signal, so a redirecting URL in the sitemap mostly sends a mixed message and wastes a fetch.
• lastmod that changes on every save. Covered above.
• Not checking size limits on custom setups. WordPress core, Yoast and Rank Math split large sitemaps automatically. Custom generators often don't. If you build the sitemap yourself, make sure it switches to a sitemap index before 50,000 URLs or 50 MB.
• Encoding errors. An unescaped ampersand in a URL (?a=1&b=2 instead of ?a=1&b=2) makes the XML invalid. The Search Console Sitemaps report shows the file as not readable, and none of its URLs are used until you fix it.
• Including noindex URLs. A sitemap lists URLs you want indexed. A page with noindex contradicts that, so leave it out.

Sitemap and AI search: do GPTBot, ClaudeBot, Perplexity use it?

Nobody documents it. OpenAI's, Anthropic's and Perplexity's crawler pages describe what each bot is for and how to control it in robots.txt, but none says whether their crawlers read the Sitemap line or fetch sitemap.xml.

Google-Extended isn't a crawler at all. Google Search, including AI Overviews, uses Googlebot, which does read sitemaps.

There's also no documentation saying a sitemap improves how often a site is cited in AI answers. What a sitemap does is make your URLs easier to discover for any crawler that uses it, Bing included (see above).

For how AI crawlers work and how to control them, read the AI Crawlers Guide.

How to test your sitemap

Four checks, from quickest to most thorough:

1. Fetch the file. Confirm it exists, returns 200 and contains XML. Useful right after every deploy.

curl -I https://yoursite.com/sitemap.xml
curl https://yoursite.com/sitemap.xml | head -50

2. Lumina Sitemap Validator. Free, no signup. The tool parses the sitemap, follows sitemap indexes, checks up to 2,000 listed URLs for their status code, flags noindex pages and canonical mismatches, and warns when every URL carries the same lastmod date.

3. Google Search Console → Sitemaps. Shows when Google last read your sitemap, whether it could be processed and how many pages it discovered. Filter the Page indexing report by the sitemap to see which URLs are indexed.

4. Bing Webmaster Tools → Sitemaps. The same for Bing. If you use IndexNow, Bing Webmaster Tools also shows your submissions.

FAQ

Where to start

Five steps: