In Scrapy you set a proxy per request with meta={"proxy": "http://user:pass@host:port"}, or globally via the http_proxy / https_proxy environment variables. The built-in HttpProxyMiddleware (enabled by default) reads those sources, strips credentials from the proxy URL in meta, and sends them as a Proxy-Authorization header. For rotation, assign meta["proxy"] in a small custom downloader middleware that runs before HttpProxyMiddleware, or point every request at your provider's rotating gateway hostname.
This tutorial targets scrapy proxy, scrapy proxy middleware, scrapy rotating proxies, and related searches. It complements our multi-framework guide on integrating proxies with popular scraping frameworks. Browse the proxy directory, compare providers, or see Webshare as one example of a scraper-friendly datacenter entry tier. For blocking, proxy types, and pricing roundups without vendor benchmarks here, see avoiding blocks while scraping, residential vs datacenter, best residential proxies for scraping, cheapest datacenter prices, cheapest residential prices, and static residential proxies.
Responsible scraping: Enable ROBOTSTXT_OBEY = True where appropriate, use AUTOTHROTTLE or conservative DOWNLOAD_DELAY, respect site terms of service and data-protection rules, and treat proxies as routing infrastructure—not a license to ignore rate limits or personal data laws.
Last verified: November 19, 2026 against Scrapy 2.20.0 HttpProxyMiddleware docs, RetryMiddleware docs, and PyPI Scrapy 2.20.0 (released November 6, 2026; requires Python >= 3.11). Code patterns below were checked against Scrapy 2.20.0; ProxyCorner does not benchmark proxy providers.
How do I set a proxy for a single Scrapy request?
Pass the proxy URL in the request meta dict. Per-request meta["proxy"] overrides environment variables and is not limited by no_proxy.
import scrapy
class IpSpider(scrapy.Spider):
name = "ip_check"
async def start(self):
proxy = "http://USERNAME:PASSWORD@proxy.example.com:8000"
yield scrapy.Request(
"https://httpbin.org/ip",
meta={"proxy": proxy},
)
In Scrapy 2.13+, spiders may implement async def start(self) instead of start_requests(); the latter still works. Set meta["proxy"] = None on a request to disable proxying for that URL even when global env vars are set.
How do I set a proxy for the whole Scrapy project?
HttpProxyMiddleware follows the same conventions as urllib.request: export http_proxy, https_proxy, and optionally no_proxy before starting the crawl. Both http:// and https:// proxy URL schemes work for HTTP and HTTPS destination URLs; documentation usually shows http:// for the proxy endpoint.
# Linux / macOS
export http_proxy="http://USERNAME:PASSWORD@proxy.example.com:8000"
export https_proxy="http://USERNAME:PASSWORD@proxy.example.com:8000"
scrapy crawl myspider
Keep credentials in environment variables or Scrapy settings—not committed to git. You can load them in settings.py:
import os
HTTP_PROXY = os.environ.get("SCRAPY_HTTP_PROXY")
HTTPS_PROXY = os.environ.get("SCRAPY_HTTPS_PROXY")
HttpProxyMiddleware is on by default (HTTPPROXY_ENABLED = True). Proxy meta on an individual request still wins over env vars.
How do I authenticate with a username and password?
Embed credentials in the proxy URL: http://username:password@host:port. HttpProxyMiddleware removes them from request.meta["proxy"] before the request leaves Scrapy and adds Proxy-Authorization: Basic … with encoding HTTPPROXY_AUTH_ENCODING (default latin-1). If a later middleware changes the proxy to a different host, Scrapy drops the old Proxy-Authorization header so credentials are not sent to the wrong proxy.
Wrong credentials on an HTTPS target typically surface as TunnelError: Could not open CONNECT tunnel with proxy … 407 Proxy Authentication Required. Fix the username/password or IP whitelist on the provider dashboard—do not disable TLS verification to work around auth failures.
How do I rotate proxies in Scrapy?
Custom middleware (recommended): Set request.meta["proxy"] when no proxy is already set. Register the middleware with priority below 750 so it runs before HttpProxyMiddleware (750) and can rely on built-in credential handling. RetryMiddleware defaults to 550; anything in the 350–543 range is a common choice.
# myproject/middlewares.py
import os
import random
def _proxy_list_from_env():
raw = os.environ.get("SCRAPY_PROXY_LIST", "")
return [p.strip() for p in raw.split(",") if p.strip()]
class RotatingProxyMiddleware:
def process_request(self, request, spider=None):
if "proxy" not in request.meta:
proxies = _proxy_list_from_env()
if proxies:
request.meta["proxy"] = random.choice(proxies)
# settings.py
DOWNLOADER_MIDDLEWARES = {
"myproject.middlewares.RotatingProxyMiddleware": 350,
}
After HttpProxyMiddleware runs, response.meta["proxy"] shows the proxy URL without embedded credentials—useful when logging which endpoint handled a request.
Rotating gateway: Many providers expose one hostname that rotates exit IPs per request or session. Point all traffic at that gateway (env var or fixed meta["proxy"]) without maintaining a local list. Sticky sessions are usually controlled via provider-specific username flags or port/session parameters documented in their dashboard—not Scrapy core.
Third-party package: scrapy-rotating-proxies (latest 0.6.2, May 25, 2019) is an older community option with no release since 2019—test carefully on Scrapy 2.20 before relying on it in production. A ~15-line custom middleware is usually simpler to own and debug.
Should I use a rotating gateway or my own proxy list?
| Approach | Best when | Trade-offs |
|---|---|---|
| Provider rotating gateway | Large pools, geo targeting, minimal ops | Per-GB or per-request billing; less per-IP control unless sticky sessions are supported |
| Your own proxy list + middleware | Dedicated datacenter IPs, fixed budgets, custom ban logic | You refresh dead IPs, handle auth, and monitor success rates |
| Hybrid | Datacenter for easy targets, residential gateway for hard pages | Two billing models; route by spider or by domain in middleware |
Compare product types in our directory and comparison pages rather than assuming one model fits every target.
How do I retry banned or failed proxy requests?
RetryMiddleware defaults to RETRY_TIMES = 2 (initial attempt plus two retries) and RETRY_HTTP_CODES including 408, 429, 500, 502, 503, 504, 522, and 524—not 403 by default. Set per-request meta["max_retry_times"] to override. Failed CONNECT tunnels (e.g. proxy 407) raise TunnelError, which is in RETRY_EXCEPTIONS by default.
For 403/429 from the origin (not the proxy), combine retries with rotation: on retry, clear or replace meta["proxy"] in a custom middleware or use get_retry_request() from a spider callback. Pair with AUTOTHROTTLE_ENABLED to back off under load. Raising RETRY_HTTP_CODES to include 403 can amplify blocks—use only with rotation and strict concurrency limits.
# settings.py (example tweaks)
RETRY_TIMES = 3
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1
AUTOTHROTTLE_MAX_DELAY = 10
Does Scrapy support SOCKS5 proxies?
SOCKS URLs such as socks5://user:pass@host:port work only when the active download handler supports SOCKS. The default HTTP11DownloadHandler does not. HTTP proxy URLs in meta["proxy"] are not handled when the download handler is H2DownloadHandler—proxy meta is unsupported there.
HttpxDownloadHandler was added in Scrapy 2.15. Install Scrapy with the httpx extra, enable asyncio support (the asyncio reactor is the default in current Scrapy releases), and register the handler for both http and https in DOWNLOAD_HANDLERS. It supports HTTP proxies and SOCKS5 proxies through the same meta["proxy"] flow as HttpProxyMiddleware. The Scrapy 2.20 download handler docs describe it as experimental and not yet recommended for production. It also maintains a separate connection pool per proxy URL, which can increase memory and socket use when you rotate many distinct endpoints.
# pip install "scrapy[httpx]"
# settings.py
DOWNLOAD_HANDLERS = {
"http": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
"https": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
}
Most residential and datacenter plans expose HTTP CONNECT endpoints; SOCKS5 is optional. Confirm the protocol your provider documents before switching handlers.
Which proxy type should I use with Scrapy?
| Type | Typical use with Scrapy | Notes |
|---|---|---|
| Datacenter | Fast crawls on open APIs, catalog sites, internal tools | Often blocked on strict anti-bot stacks; see datacenter price comparison |
| Residential (rotating) | Geo-sensitive or higher-friction public data | Gateway + per-GB billing common; see residential scraping guide and price comparison |
| ISP / static residential | Long-lived logged-in sessions, stable allowlists | One IP per port; rotation is your inventory, not the gateway |
How do I check that my proxy is working?
Scrape a echo service through the same meta["proxy"] or env configuration you use in production:
yield scrapy.Request("https://httpbin.org/ip", meta={"proxy": proxy_url}, callback=self.parse_ip)
def parse_ip(self, response):
self.logger.info("Exit IP payload: %s", response.text)
Run twice with different list entries or gateway session modes and confirm the JSON origin field changes when you expect rotation. Log response.meta.get("proxy") (credential-stripped) alongside status codes to correlate bans with endpoints.
Scrapy proxy troubleshooting
| Symptom | Likely cause | What to try |
|---|---|---|
| 407 / TunnelError on HTTPS | Bad proxy credentials or IP not allowlisted | Verify URL encoding for special characters in passwords; test with curl through the same proxy |
| 403 or 429 from target site | Origin blocking, not proxy auth | Throttle, rotate IP or proxy type, review anti-block practices; retries alone rarely help |
| Timeouts | Slow proxy, wrong port, or overloaded pool | Increase DOWNLOAD_TIMEOUT; drop dead IPs from SCRAPY_PROXY_LIST |
| Env proxy ignored | meta["proxy"] set, or HTTPPROXY_ENABLED = False |
Unset per-request meta; confirm middleware order |
| Rotation middleware never runs | Priority >= 750 (after HttpProxyMiddleware) | Set custom middleware to 350–543; only set meta["proxy"] when absent |
| SOCKS URL fails | Default HTTP11 handler (no SOCKS) | Set both http and https in DOWNLOAD_HANDLERS to scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler after pip install "scrapy[httpx]", or use an HTTP CONNECT proxy URL instead |
Frequently Asked Questions
What is the correct Scrapy proxy meta key?
Use the string key proxy on the request meta dict, for example meta={"proxy": "http://host:port"}. HttpProxyMiddleware reads that value before environment variables. Setting meta["proxy"] = None disables the proxy for that single request.
Does HttpProxyMiddleware strip passwords from logs?
Yes. Credentials in the proxy URL are removed from request.meta["proxy"] before the request is sent and are transmitted only via the Proxy-Authorization header. You should still avoid logging full proxy URLs that contain passwords in your own spider code.
What downloader middleware order does Scrapy use for proxies?
Built-in ordering places RetryMiddleware at 550, UserAgentMiddleware at 500, DownloadTimeoutMiddleware at 350, and HttpProxyMiddleware at 750. A custom rotation middleware should use a number lower than 750 so HttpProxyMiddleware can normalize auth headers afterward.
Can I use scrapy-rotating-proxies in 2026?
The PyPI package scrapy-rotating-proxies last shipped version 0.6.2 in May 2019. It may still work on some projects, but it is unmaintained relative to Scrapy 2.20. Most teams implement a short custom middleware or a provider gateway instead.
Why does Scrapy retry TunnelError but not every 403?
RetryMiddleware includes TunnelError in RETRY_EXCEPTIONS because CONNECT failures are often transient proxy issues. HTTP 403 is excluded from default RETRY_HTTP_CODES because blind retries without changing IP or headers can worsen rate limits. Add 403 to RETRY_HTTP_CODES only with rotation and throttling.
Do I need a separate scrapy https proxy setting?
Usually you set both http_proxy and https_proxy environment variables to the same HTTP proxy URL. Per-request meta uses one proxy value for the scheme being downloaded. Destination HTTPS still uses CONNECT through the proxy; you rarely need a distinct scrapy https proxy setting beyond mirroring env vars.
Where can I compare proxy providers for a Scrapy spider?
Use the ProxyCorner directory and compare tools, plus pricing guides such as cheapest residential proxies and cheapest datacenter proxies, to shortlist products—then validate on your own targets with httpbin.org/ip or equivalent.
Tags
Proxy & Web Scraping Research Team
The ProxyCorner editorial team researches and reviews residential, datacenter, mobile, and ISP proxy providers using each vendor’s published pricing pages, documentation, and terms. Performance figures in our content reflect provider-stated claims unless a third-party source is cited; ProxyCorner does not run its own proxy benchmarks.
Reviews follow our published review methodology, including affiliate disclosure and editorial independence standards.
Ready to Choose a Proxy Provider?
Explore our comprehensive directory of residential proxy providers and find the perfect match for your web scraping needs.
Browse Proxy Providers