Open actionable advisories and recorded vendor fixes. This public package view has no inventory attached and cannot establish whether your deployment is affected. A vendor fix never resolves your finding; verify your installed version and the source guidance.
Open advisories
12 actionable advisories, newest first. Withdrawn advisories and news are excluded. Coverage is partial.
Impact Since version 1.4.0, Scrapy respects the Referrer-Policy response header to decide whether and how to set a Referer header on follow-up requests. If the header value looked like a valid Python import path, Scrapy would import the referenced object and call it, assuming it referred to a referrer policy class (for example, scrapy.spidermiddlewares.referer.DefaultReferrerPolicy) and attempting to instantiate it to handle the Referer header. A malicious site could exploit this by setting Referrer-Policy to a path such as sys.exit, causing Scrapy to import and execute it and potentially terminate the process. Patches Upgrade to Scrapy 2.14.2 (or later). Workarounds If you cannot upgrade to Scrapy 2.14.2, consider the following mitigations. - Disable the middleware: If you don't need the Referer header on follow-up requests, set REFERERENABLED to False. - Set headers manually: If you do need a Referer, disable the middleware and set the header explicitly on the requests that require…
Recommended action: Upgrade to 2.14.2 or later, then redeploy and verify.A vendor fix does not resolve your finding until you verify it in your own systems.
Scrapy versions up to 2.13.3 are vulnerable to a denial of service (DoS) attack due to a flaw in its brotli decompression implementation. The protection mechanism against decompression bombs fails to mitigate the brotli variant, allowing remote servers to crash clients with less than 80GB of available memory. This occurs because brotli can achieve extremely high compression ratios for zero-filled data, leading to excessive memory consumption during decompression. Mitigation for this vulnerability needs security enhancement added in brotli v1.2.0.
Recommended action: Upgrade to 2.13.4 or later, then redeploy and verify.A vendor fix does not resolve your finding until you verify it in your own systems.
Impact Scrapy was following redirects regardless of the URL protocol, so redirects were working for data://, file://, ftp://, s3://, and any other scheme defined in the DOWNLOADHANDLERS setting. However, HTTP redirects should only work between URLs that use the http:// or https:// schemes. A malicious actor, given write access to the start requests (e.g. ability to define starturls) of a spider and read access to the spider output, could exploit this vulnerability to: - Redirect to any local file using the file:// scheme to read its contents. - Redirect to an ftp:// URL of a malicious FTP server to obtain the FTP username and password configured in the spider or project. - Redirect to any s3:// URL to read its content using the S3 credentials configured in the spider or project. For file:// and s3://, how the spider implements its parsing of input data into an output item determines what data would be vulnerable. A spider that always outputs the entire contents of a response would be…
Recommended action: Upgrade to 2.11.2 or later, then redeploy and verify.A vendor fix does not resolve your finding until you verify it in your own systems.
Impact When using system proxy settings, which are scheme-specific (i.e. specific to http:// or https:// URLs), Scrapy was not accounting for scheme changes during redirects. For example, an HTTP request would use the proxy configured for HTTP and, when redirected to an HTTPS URL, the new HTTPS request would still use the proxy configured for HTTP instead of switching to the proxy configured for HTTPS. Same the other way around. If you have different proxy configurations for HTTP and HTTPS in your system for security reasons (e.g., maybe you don’t want one of your proxy providers to be aware of the URLs that you visit with the other one), this would be a security issue. Patches Upgrade to Scrapy 2.11.2. Workarounds Replace the built-in retry middlewares (RedirectMiddleware and MetaRefreshMiddleware) and the HttpProxyMiddleware middleware with custom ones that implement the fix from Scrapy 2.11.2, and verify that they work as intended. References This security issue was reported by…
Recommended action: Upgrade to 2.11.2 or later, then redeploy and verify.A vendor fix does not resolve your finding until you verify it in your own systems.
Impact Since version 2.11.1, Scrapy drops the Authorization header when a request is redirected to a different domain. However, it keeps the header if the domain remains the same but the scheme (http/https) or the port change, all scenarios where the header should also be dropped. In the context of a man-in-the-middle attack, this could be used to get access to the value of that Authorization header Patches Upgrade to Scrapy 2.11.2. Workarounds There is no easy workaround for unpatched versions of Scrapy. You can replace the built-in redirect middlewares with custom ones patched for this issue, but you have to patch them yourself, manually. References This security issue was reported and fixed by @szarny at https://huntr.com/bounties/27f6a021-a891-446a-ada5-0226d619dd1a/.
Recommended action: Upgrade to 2.11.2 or later, then redeploy and verify.A vendor fix does not resolve your finding until you verify it in your own systems.
Impact Scrapy limits allowed response sizes by default through the DOWNLOADMAXSIZE and DOWNLOADWARNSIZE settings. However, those limits were only being enforced during the download of the raw, usually-compressed response bodies, and not during decompression, making Scrapy vulnerable to decompression bombs. A malicious website being scraped could send a small response that, on decompression, could exhaust the memory available to the Scrapy process, potentially affecting any other process sharing that memory, and affecting disk usage in case of uncompressed response caching. Patches Upgrade to Scrapy 2.11.1. If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.11.1 is not an option, you may upgrade to Scrapy 1.8.4 instead. Workarounds There is no easy workaround. Disabling HTTP decompression altogether is impractical, as HTTP compression is a rather common practice. However, it is technically possible to manually backport the 2.11.1 or 1.8.4 fix, replacing the…
Recommended action: Upgrade to 2.11.1 or 1.8.4 by release line, then redeploy and verify.A vendor fix does not resolve your finding until you verify it in your own systems.
Impact When you send a request with the Authorization header to one domain, and the response asks to redirect to a different domain, Scrapy’s built-in redirect middleware creates a follow-up redirect request that keeps the original Authorization header, leaking its content to that second domain. The right behavior would be to drop the Authorization header instead, in this scenario. Patches Upgrade to Scrapy 2.11.1. If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.11.1 is not an option, you may upgrade to Scrapy 1.8.4 instead. Workarounds If you cannot upgrade, make sure that you are not using the Authentication header, either directly or through some third-party plugin. If you need to use that header in some requests, add "dontredirect": True to the request.meta dictionary of those requests to disable following redirects for them. If you need to keep (same domain) redirect support on those requests, make sure you trust the target website not to redirect your…
Recommended action: Upgrade to 2.11.1 or 1.8.4 by release line, then redeploy and verify.A vendor fix does not resolve your finding until you verify it in your own systems.
Impact The following parts of the Scrapy API were found to be vulnerable to a ReDoS attack: - The XMLFeedSpider class or any subclass that uses the default node iterator: iternodes, as well as direct uses of the scrapy.utils.iterators.xmliter function. - Scrapy 2.6.0 to 2.11.0: The openinbrowser function for a response without a base tag. Handling a malicious response could cause extreme CPU and memory usage during the parsing of its content, due to the use of vulnerable regular expressions for that parsing. Patches Upgrade to Scrapy 2.11.1. If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.11.1 is not an option, you may upgrade to Scrapy 1.8.4 instead. Workarounds For XMLFeedSpider, switch the node iterator to xml or html. For openinbrowser, before using the function, either manually review the response content to discard a ReDos attack or manually define the base tag to avoid its automatic definition by openinbrowser later. Acknowledgements This security…
Recommended action: Upgrade to 2.11.1 or 1.8.4 by release line, then redeploy and verify.A vendor fix does not resolve your finding until you verify it in your own systems.
Impact When the built-in HTTP proxy downloader middleware processes a request with proxy metadata, and that proxy metadata includes proxy credentials, the built-in HTTP proxy downloader middleware sets the Proxy-Authentication header, but only if that header is not already set. There are third-party proxy-rotation downloader middlewares that set different proxy metadata every time they process a request. Because of request retries and redirects, the same request can be processed by downloader middlewares more than once, including both the built-in HTTP proxy downloader middleware and any third-party proxy-rotation downloader middleware. These third-party proxy-rotation downloader middlewares could change the proxy metadata of a request to a new value, but fail to remove the Proxy-Authentication header from the previous value of the proxy metadata, causing the credentials of one proxy to be leaked to a different proxy. If you rotate proxies from different proxy providers, and any of…
Recommended action: Upgrade to 2.6.2 or 1.8.3 by release line, then redeploy and verify.A vendor fix does not resolve your finding until you verify it in your own systems.
Impact Responses from domain names whose public domain name suffix contains 1 or more periods (e.g. responses from example.co.uk, given its public domain name suffix is co.uk) are able to set cookies that are included in requests to any other domain sharing the same domain name suffix. Patches Upgrade to Scrapy 2.6.0, which restricts cookies with their domain set to any of those in the public suffix list. If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.6.0 is not an option, you may upgrade to Scrapy 1.8.2 instead. Workarounds The only workaround for unpatched versions of Scrapy is to disable cookies altogether, or limit target domains to a subset that does not include domain names with one of the public domain suffixes affected (those with 1 or more periods). References https://publicsuffix.org/ For more information If you have any questions or comments about this advisory: Open an issue Email us
Recommended action: Upgrade to 2.6.0 or 1.8.2 by release line, then redeploy and verify.A vendor fix does not resolve your finding until you verify it in your own systems.
Impact If you manually define cookies on a Request object, and that Request object gets a redirect response, the new Request object scheduled to follow the redirect keeps those user-defined cookies, regardless of the target domain. Patches Upgrade to Scrapy 2.6.0, which resets cookies when creating Request objects to follow redirects¹, and drops the Cookie header if manually-defined if the redirect target URL domain name does not match the source URL domain name². If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.6.0 is not an option, you may upgrade to Scrapy 1.8.2 instead. ¹ At that point the original, user-set cookies have been processed by the cookie middleware into the global or request-specific cookiejar, with their domain restricted to the domain of the original URL, so when the cookie middleware processes the new (redirect) request it will incorporate those cookies into the new request as long as the domain of the new request matches the domain of the…
Recommended action: Upgrade to 2.6.0 or 1.8.2 by release line, then redeploy and verify.A vendor fix does not resolve your finding until you verify it in your own systems.
Impact If you use HttpAuthMiddleware (i.e. the httpuser and httppass spider attributes) for HTTP authentication, all requests will expose your credentials to the request target. This includes requests generated by Scrapy components, such as robots.txt requests sent by Scrapy when the ROBOTSTXTOBEY setting is set to True, or as requests reached through redirects. Patches Upgrade to Scrapy 2.5.1 and use the new httpauthdomain spider attribute to control which domains are allowed to receive the configured HTTP authentication credentials. If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.5.1 is not an option, you may upgrade to Scrapy 1.8.1 instead. Workarounds If you cannot upgrade, set your HTTP authentication credentials on a per-request basis, using for example the w3lib.http.basicauthheader function to convert your credentials into a value that you can assign to the Authorization header of your request, instead of defining your credentials globally using…
Recommended action: Upgrade to 2.5.1 or 1.8.1 by release line, then redeploy and verify.A vendor fix does not resolve your finding until you verify it in your own systems.