Details
After these changes, validation is enforced both where sitemap-derived HTTP requests are produced (sitemap and robots.txt loaders) and where they are consumed (HTTP clients). A regression at either layer is caught by the other.
Behaviour change for upgraders
SitemapRequestLoader and RobotsTxtFile.get_sitemaps() now default to enqueue_strategy='same-hostname'. Deployers that legitimately relied on cross-host sitemap entries (e.g., a sitemap index on sitemaps.example.com that points to content on www.example.com) must opt in explicitly with enqueue_strategy='same-domain' or enqueue_strategy='all'.
Finder credits
- @r0otsu
- @Yuremin (Zhengmin Yu)
- @FORIMOC
- @invoke1442 (Ethan Carter)
- @Arturo0x90 (Arturo Melgarejo)