I have been doing a lot of http retrieval lately and the most efficient way to do that is with gzip compression enabled. Fortunately Python makes that really easy. All you have to do is derive from urllib2.HTTPHandler and override http_open().
import httplib, urllib, urllib2
def http_open(self, req):
r = self.do_open(httplib.HTTPConnection, req)
if 'Content-Encoding'in r.headers and \
r.headers['Content-Encoding'] == 'gzip':
fp = gzip.GzipFile(fileobj=StringIO(r.read()))
fp = r
resp = urllib.addinfourl(fp, r.headers, r.url, r.code)
resp.msg = r.msg
The Accept-encoding header tells the server that this client supports gzip compression and if the Content-Encoding header is set to gzip the server returned an compressed response. Now you just need to build your opener.
request = urllib2.Request(url)
opener = urllib2.build_opener(GzipHandler)
For more information see:
Download the source
My home on the web. Uses the Django
web framework, uwsgi
as the WSGI server,
as the media server and load
for the syntax highlighting.
Author and designer.
streams trending videos to
your browser like a never ending tv show. This was
awesome idea. I was responsible for the backend development.
to find the music relationships. Author and designer.
is a photo sharing and
tagging website. Facial detection was run on the photos so the bounding boxes were
already drawn around the faces eliminating a step in a tedious process. I authored
views and the process to run facial detection on the photos and upload them to S3.
first photo site. It was
event based and slideshows would be created from the albums. I authored the views
and process to generate the slideshows from the photos on EC2.
Below is a snippet of my resume. Click here to view
the full version in pdf, its proper format.