Problem: When I am conducting a pentest, I commonly write python scripts to use the requests module and need to proxy them through Burp. I have been using the "Easy way out," but there are problems with doing this and there is a much more efficient way in handling this.
Easy way out: I can proxy requests through Burp Suite fairly easily by creating a proxies dictionary and assigning that dictionary to the proxies argument. I then have to set the verify argument to False because Burp's certificate is not trusted by the requests library's certificate bundle. Example code:
import requests
proxies = {"http": "http://127.0.0.1:8080", "https": "http://127.0.0.1:8080"}
r = requests.get("https://www.google.com/", proxies=proxies, verify=False)
Problem with easy way out: What happens if you have many calls to the requests library and you don't want to set the proxies and verify arguments for each request. Or possibly you have been given a test harness that utilizes the requests library and you don't want to modify each and every call to the library. I have always searched for this answer and only found that I can export two environment variables HTTP_PROXY and HTTPS_PROXY. However, this does not fix the fact that I have to set the verify argument to False on every single request.
Solution: In addition to the HTTP_PROXY and HTTPS_PROXY environment variables, there is also a REQUESTS_CA_BUNDLE which can be set to specify the location of a certificate. However, the documentation is not very clear about the certificate format required. After some basic troubleshooting, I was able to determine the encoding needed for the REQUESTS_CA_BUNDLE file is PEM.
After you have downloaded your certificate from Burp (either through the browser or directly from the application's GUI), it is DER formatted. In order to convert it to the needed PEM encoded format, run the following command:
openssl x509 -inform der -in certificate.cer -out certificate.pem
You are now ready to export your environment variables and use requests with Burp.
export REQUESTS_CA_BUNDLE="/path/to/pem/encoded/cert"
export HTTP_PROXY="http://127.0.0.1:8080"
export HTTPS_PROXY="http://127.0.0.1:8080"
Now all of your HTTP requests made through the requests library without the proxies argument configured will be routed through Burp. In order to remove these environment variables, run the following commands:
unset REQUESTS_CA_BUNDLE
unset HTTP_PROXY
unset HTTPS_PROXY