Proxy servers are middlemen who stand between clients and servers. Security, load balancing, and anonymity are just a few of the uses they can be put to. This article will examine the use of proxies with the requests library for Python, a well-liked module for sending HTTP requests.
Using Proxies with Python Requests
To use a proxy with Python requests, you will need to specify the proxy’s address and port number when making the request. This can be done using the proxies parameter in the requests.get() function.
import requests proxies = { "http": "http://10.10.1.10:3128", "https": "http://10.10.1.10:1080", } response = requests.get("http://example.com", proxies=proxies)
In the example above, we are using a dictionary to specify the proxy for HTTP and HTTPS requests. The key is the protocol (http or https) and the value is the proxy’s address and port number.
Authenticating with a Proxy
If the proxy requires authentication, you can use the auth parameter to provide a username and password.
import requests proxies = { "http": "http://user:pass@10.10.1.10:3128", "https": "http://user:pass@10.10.1.10:1080", } response = requests.get("http://example.com", proxies=proxies)
In the example above, we are providing a username and password in the proxy’s URL. This will authenticate the request with the proxy server.
Using a SOCKS Proxy
In addition to HTTP and HTTPS proxies, Python requests also support the use of SOCKS proxies. To use a SOCKS proxy, you will need to install the socks library and specify the proxy using the socks.SOCKSProxy class.
import requests import socks import socket socks.set_default_proxy(socks.SOCKS5, "10.10.1.10", 1080) socket.socket = socks.socksocket response = requests.get("http://example.com")
In the example given above, we are using the socks.set_default_proxy() function to specify the SOCKS proxy and the socket.socket function to set the default socket to a SOCKS socket.
Rotating Proxies
If you need to make a large number of requests through a proxy, you may want to use a rotating proxy to avoid being detected and blocked. A rotating proxy is a pool of proxies that are rotated on each request to provide a level of anonymity.
To use a rotating proxy with Python requests, you can use a library such as rotating-proxies or proxymesh.
import requests from rotating_proxies import RotatingProxyMiddleware proxy_middleware = RotatingProxyMiddleware(num_proxies=10) response = requests.get("http://example.com", middleware=proxy_middleware)
In this example, we are using the “RotatingProxyMiddleware” class from the “rotating-proxies” library to rotate through a pool of 10 proxies on each request.
Using a Proxy Pool
Another option for using rotating proxies with Python requests is to use a proxy pool. A proxy pool is a collection of proxy servers that can be accessed through a single endpoint. To use a proxy pool with requests, you will need to make a request to the proxy pool’s endpoint to retrieve a new proxy for each request.
import requests def get_proxy(): response = requests.get("http://proxy-pool-endpoint.com/new") if response.status_code == 200: return response.text return None def use_proxy(proxy): proxies = { "http": "http://" + proxy, "https": "http://" + proxy, } return proxies proxy = get_proxy() if proxy: proxies = use_proxy(proxy) response = requests.get("http://example.com", proxies=proxies)
Above, we are using the get_proxy() function to retrieve a new proxy from the proxy pool’s endpoint and the use_proxy() function to create a dictionary of proxies for the requests.get() function.
Conclusion
In this article, we have learned how to use proxies with Python’s requests library. We covered the basic usage of proxies, authenticating with a proxy, using a SOCKS proxy, and rotating proxies using a proxy middleware or proxy pool. With this knowledge, you should be able to use proxies to make HTTP requests in Python with greater flexibility and control. On our website, you can find the best proxy reviews about the top providers that will help you easily select the right proxy type and provider for your Python requests projects.
