Appengine Files Api – part 1: Storing/Fetching remote images in Blobstore using Django

Posted by blainegarrett on Apr 2nd, 2011 | View Comments

Here is a quick tutorial on how to fetch remote images with the new Files API on Google Appengine 1.4.3 released March 30th. These examples use Django.

Before we begin, read up on how to write files to blobstore using the Files API. Prior to the 1.4.3 release of the SDK, the only way to get files into the blobstore (and thus leverage things like BlobInfo) was to upload them via a POST request. To do what I am attempting below required a really confusing multistep process to fetch the file data and then generate a post request to the blobstore upload url. This was very error prone and awkward, but it worked. With 1.4.3, the solution is fairly straight forward. Also, note, the below example works with any file data, not just images.

First lets set up a simple url to handle the fetching and displaying.

# Put this in your /urls.py
from django.conf.urls.defaults import *
urlpatterns = patterns('',
...
(r'^remote_fetch_image/$', 'cdn.views.remote_fetch_image'),
(r'^remote_display_image/(?P[^/]+)/$', 'cdn.views.remote_display_image'),
...
)

Next create a folder in your main application called “cdn”. Inside of it, add and empty __init__.py file as well as a views.py.
You may also need to add to add “cdn” to your list of installed apps in settings.py

INSTALLED_APPS = (
...
'cdn',
...
)

Next, open up cdn/views.py and add the following code:

from __future__ import with_statement # Note this MUST go at the top of your views.py
from google.appengine.ext import blobstore
from django import http
from google.appengine.ext import db
def remote_fetch_image(request):
    from google.appengine.api import files, urlfetch
    from django.core import urlresolvers
    from django import http

    # Fetch image content
    image_url = request.GET.get('image_url', None)
    if not image_url:
        return http.HttpResponse('Please provide a query argument named "image_url" that is the full url to an image. ')
    fetch_response = urlfetch.fetch(image_url)

    file_data = fetch_response.content
    content_type = fetch_response.headers.get('content-type', None)

    # Create the file
    file_name = files.blobstore.create(mime_type=content_type)

    # Open the file and write to it
    with files.open(file_name, 'a') as f:
      f.write(file_data)

    # Finalize the file. Do this before attempting to read it.
    files.finalize(file_name)

    # Get the file's blob key
    blob_key = files.blobstore.get_blob_key(file_name)
    # We're technically done, but lets redirect and display the image
    return http.HttpResponseRedirect(urlresolvers.reverse(remote_display_image, args=[blob_key]))

def remote_display_image(request, blob_key):
    # Fetch blob by key from blobstore
    blob_info = blobstore.BlobInfo.get(blob_key)
    if not blob_info:
        raise Exception('Blob Key does not exist')

    blob_file_size = blob_info.size
    blob_content_type = blob_info.content_type

    # Attempt to fetch the blob in one or more chunks depending on size and api limits
    blob_concat = ""
    start = 0
    end = blobstore.MAX_BLOB_FETCH_SIZE - 1
    step = blobstore.MAX_BLOB_FETCH_SIZE - 1

    while(start < blob_file_size):
        blob_concat += blobstore.fetch_data(blob_key, start, end)
        temp_end = end
        start = temp_end + 1
        end = temp_end + step
    return http.HttpResponse(blob_concat, mimetype=blob_content_type)

Finally, open your app in the browser with the url to a file. For me, my sdk is mapped to the domain garrettclan.org:
example: http://garrettclan.org/remote_fetch_image/?image_url=http://blainegarrett.com/files/2010/01/birthmarks2.jpg

You should see the image at the url and it is now in the blobstore. This is an example usage of the new files api. Please note that this is experimental and may change as Appengine progresses. Also, not that the above processes is subject to the urlfetch quota, blobstore quota, and cpu quotas. See more details about the files API here.

Also, if interested, the image I used is a painting my collaborative art crew did called “Nuroma Flare”, part of the Birthmarks series.

The next steps:

1. Part 2: Fetching the Facebook Profile Image and attaching it to a user
2. Part 3: Scale/crop down large uploaded images to a reasonable size before storing in blobstore to save on disk space quotas
3. Part 4: Utilizing existing blobstore features to dynamically generate thumbnails and sized images