Details
Run:
cd /Users/rexliu/Documents/GA\ code/REDit\ Deployment/stack/deploy
env PRAISONAI_SPIDER_TOOLS_FILE=/path/to/PraisonAI/src/praisonai-agents/praisonaiagents/tools/spider_tools.py \
uv run --with requests --with beautifulsoup4 --with lxml --python 3.11 \
poc_spider_tools_redirect_ssrf.py
Observed on current main:
DIRECT_CONTROL: {'error': 'Invalid or potentially dangerous URL: http://127.0.0.1:<port>/secret'}
REDIRECT_RESULT: {'url': 'http://attacker.test:<port>/go', 'status_code': 200, ... 'content': 'SPIDER-INTERNAL-SECRET', ...}
REDIRECT_SERVER_HIT: True
INTERNAL_SERVER_HIT: True
NO_REDIRECT_CONTROL: {'url': 'http://attacker.test:<port>/go', 'status_code': 302, ... 'Location': 'http://127.0.0.1:<port>/secret', ...}
NO_REDIRECT_SERVER_HIT: True
NO_REDIRECT_INTERNAL_HIT: False
PRAI-CAND-004 CONFIRMED: SpiderTools follows a redirect to loopback
The direct control proves direct loopback is blocked. The redirect result proves
the same blocked destination is reached through a public-looking initial URL.
The no-redirect control proves that disabling automatic redirects prevents the
internal request while still receiving the external redirect response.
Why this is not intended behavior
The Spider Tools documentation says scrape_page, extract_links, crawl, andextract_text refuse dangerous URLs before network requests. The documented
blocked classes include loopback, private/reserved IPs, link-local/cloud
metadata endpoints, internal TLDs, non-HTTP(S) schemes, and parser-smuggling
forms. The same page states the validation is always on for bundled spider tools
and does not require enable_security().
The current code also documents _validate_url() as URL validation "to prevent
SSRF attacks." A redirect to a loopback target bypasses that documented
protection.
Impact
An attacker who can influence a URL passed to scrape_page(),extract_links(), crawl(), or extract_text() can cause the PraisonAI process
to request destinations that SpiderTools is designed to block.
Potential impact includes:
- reading loopback-only HTTP services;
- probing or reading private network services reachable from the PraisonAI host;
- reading link-local/cloud metadata endpoints if reachable in the deployment environment.
The PoV demonstrates returned response-body disclosure from a loopback-only
service. This report does not claim arbitrary code execution or live cloud
credential theft without deployment-specific evidence.
Severity
Suggested default severity: Moderate.
High severity may be appropriate for deployments where untrusted users can
directly invoke SpiderTools through a network-facing agent, bot, API, or MCP
service and sensitive internal or metadata services are reachable.
Suggested fix
Disable automatic redirects in scrape_page():
response = session.get(
url,
timeout=timeout,
verify=verify_ssl,
allow_redirects=False,
)
If redirects should remain supported, follow them manually and validate everyLocation target before each hop using the same SSRF guard:
- require
httporhttps; - resolve and validate every redirect hostname;
- reject loopback, private, link-local, reserved, multicast, unspecified, internal, and metadata destinations;
- cap redirect count;
- apply the same safe fetch path to
scrape_page(),extract_links(),crawl(), andextract_text().
Regression tests should cover direct loopback rejection, public-to-loopback
redirect rejection, public-to-public redirects if supported, and allscrape_page() callers.