CS6320:  SW Engineering of Web Based Systems
Instructor Outline Syllabus Projects Links  

 

Google App Engine Task Queue

 

  • Task Queue API (both Java and Python)

  • perform work initiated by a user request, outside of that request.

  • If an app needs to execute some background work, it may use the Task Queue API to organize that work into small, discrete units, called tasks.

  • The app then inserts these tasks into one or more queues based on the queue's configuration.

 

Why?

  • 30 second response needed
  • Good for tasks don't need to wait on --- asynchronous --- possibly URLFetches, or DataStore transactions (update)??

 

Step 1: Declare a Task Queue in queue.xml (Java based)

You can define any number of individual queues by providing a queue <name>. You can control the rate at which tasks are processed in each queue by defining other directives, such as <rate>, <bucket-size>, and <max-concurrent-requests>. You can read more about these directives in the Queue Definitions section.

Configuring the Default Queue

All apps have a push queue named default. This queue has a preset rate of 5 tasks per second, but you can change this rate by defining a default queue in queue.xml

If you do not configure a default queue in queue.xml, the default queue doesn't display in the Administration Console until the first time it is used.


<queue-entries>
<!--Change the refresh rate of the default queue to 1/s-->
  <queue>
    <name>default</name>
    <rate>1/s</rate>
  </queue>
</queue-entries>

 

Configuring the Maximum Number of Concurrent Requests

You can further control the processing rate by setting <max-concurrent-requests>, which limits the number of tasks that can execute simultaneously.

<queue-entries>
<!--Set the number of max concurrent requests to 10-->
  <queue>
    <name>optimize-queue</name>
    <rate>20/s</rate>
    <bucket-size>40</bucket-size>
    <max-concurrent-requests>10</max-concurrent-requests>
  </queue>
</queue-entries>

 

Setting the Storage Limit for All Queues

You can use queue.xml to define the total amount of storage that task data can consume over all queues.
To define the total storage limit, include an element named <total-storage-limit> at the first level under <queue-entries>:

<!--Change the total storage limit for all queues to 50MB-->
<queue-entries>
  <total-storage-limit>50M</total-storage-limit>
  <queue>
    ...
  </queue>
</queue-entries>

 

 

 

Step 2: Grab a Queue defined in queue.xml in your code (you can have more than one defined)

    • You can get a named queue specified in the queue.xml file using the getQueue()method of the factor

    • or you can get the default queue using getDefaultQueue()

 

Step 3: Create a task and add it to your queue

OPTION 1 -- create a task with default options for queue

    • queueObject.add()  call it with no arguments to create a task with the default options for the queue.

OPTION 2 - use TaskOptions instance to specify task options

  • queueObject.add() method. . You can call the Queue's add() method with a TaskOptions instance (produced byTaskOptions.Builder), see below for example

 

//STEPS 1 - 3: Tasks added to this queue will execute by calling the request handler at the URL /worker with the parameter key. They will execute at the rate set in the queue.xml file, or the default rate of 5 tasks per second.

import com.google.appengine.api.taskqueue.Queue;
import com.google.appengine.api.taskqueue.QueueFactory;
import static com.google.appengine.api.taskqueue.TaskOptions.Builder.*;

// ...
    Queue queue = QueueFactory.getDefaultQueue();
    queue.add(withUrl("/worker").param("key", key));

 

 

 

 

Step 4: Setup Task Execution --- Request Handler

  • GAE executes push tasks by calling application-specific URLs with request handlers for those tasks. YOU DONT WRITE EXECUTION CODE (for push queues --there is something called pull queues you can controll when, how often,etc tasks are run) --GAE selects tasks on the task queue and executes if for you!!! REMEMBER ASYNCHRONOUS from the applicaiton that put the task on the queue!!

 

Why use a Task for DataStore updates????

  • Enqueueing a task is fast, about three times faster than writing to the datastore.

     

    SCENARIO

    • For anything whose success or failure doesn’t need to be reported to the user

    Example, an app can write a value to the memcache, then enqueue a task to persist that value to the datastore. This saves time during the user request, and allows the task to do bookkeeping or make other time-consuming updates based on the change (assuming it meets the application’s needs that the bookkeeping happens later than the initial update).

     

     

Ok so it is asynchronous -- how do I get (later) any data/results created --- the task must store the data in the datastore and/or memcache!!!!

  • URLs must be local to your application root directory and specified as a relative URL.

  • system then processes them asynchronously (by invoking the HTTP request).

  • How LONG??? A task must finish executing and send an HTTP response value between 200–299 within 10 minutes of the original request

  • In above code the Handler will be mapped to the URL /worker and passed the parameter key

 

What if you need more than 10 minutes -->Read more about time limits and different kinds of services
© Lynne Grewe