Advisory

GHSA-23j4-mw76-5v7h

Also known as: None

Summary

Scrapy allows redirect following in protocols other than HTTP

Details

Impact

Scrapy was following redirects regardless of the URL protocol, so redirects were working for data://, file://, ftp://, s3://, and any other scheme defined in the DOWNLOAD_HANDLERS setting.

However, HTTP redirects should only work between URLs that use the http:// or https:// schemes.

A malicious actor, given write access to the start requests (e.g. ability to define start_urls) of a spider and read access to the spider output, could exploit this vulnerability to:

  • Redirect to any local file using the file:// scheme to read its contents.
  • Redirect to an ftp:// URL of a malicious FTP server to obtain the FTP username and password configured in the spider or project.
  • Redirect to any s3:// URL to read its content using the S3 credentials configured in the spider or project.

For file:// and s3://, how the spider implements its parsing of input data into an output item determines what data would be vulnerable. A spider that always outputs the entire contents of a response would be completely vulnerable, while a spider that extracted only fragments from the response could significantly limit vulnerable data.

Patches

Upgrade to Scrapy 2.11.2.

Workarounds

Replace the built-in retry middlewares (RedirectMiddleware and MetaRefreshMiddleware) with custom ones that implement the fix from Scrapy 2.11.2, and verify that they work as intended.

References

This security issue was reported by @mvsantos at https://github.com/scrapy/scrapy/issues/457.

Severity

Medium Basis: The source's own band: MODERATE

Dates

Published
2024-05-14 20:14 UTC
Modified
2024-11-28 05:42 UTC

Affected packages and fixes

scrapy

PyPI · All systems

Affected

  • introduced 0, fixed 2.11.2 (ecosystem)

Fixed versions

2.11.2

Fix

  • Fixed version 2.11.2
    pip install --upgrade scrapy==2.11.2

Fix commands are the export's own generic commands. Ojuju never runs them. Check them against your setup first.

Check your own computer: guard packages check

View this record on OSV.dev

Source

Source: GitHub Advisory Database (reviewed advisories) · CC BY 4.0

Sources & licences