My Problem

My initial problem was that I was working inside a transaction and needed to fetch all the entities of a one to many relationship. In my case, I designed my data to be in the same entity group, thus I figured it was transaction safe. However, I was getting the error: BadRequestError: Only ancestor queries are allowed inside transactions.. This had me a bit stumped since I was not actually doing any queries inside of the code. However, Appengine itself was under the hood.

The Issue

With a bit of discussion with my coworkers, I traced the problematic query back to the underlying query behind Appengine’s handy named reference property.
I had a kind (we’ll call it Hand) that had a one-to-many relationship on a kind (we’ll call Finger). The issue is that hand.fingers does not run an ancestor query. In my specific case, this did not make sense, since all the fingers were in the same entity group as their parent hand. However, since reference properties do not have to be in the same entity group, the query resembles Finger.all().filter(‘hand =’, hand) which is obviously not a ancestor query.

The solution

The solution is to avoid using the reference property and manually query fingers = Finger.all().ancestor(hand). This will fetch all of the finger entities that have an ancestor (parent) of hand. Note: that this is not the same as a reference property query and that the links between parent/child entities among an entity group have a different purpose than reference properties even though they often coincidentally have an overlap in the data. This solution leverages this coincidence.

Example Code

Note: I didn’t actually run this specific code. Use at your own risk.

from google.appengine.ext import db
from base.datastore.util import transaction

class Hand(db.Model):
    is_right = db.BooleanProperty()
    is_dominant = db.BooleanProperty()

class Finger(db.Model):
	position = db.IntegerProperty(default=0)
	hand = db.ReferenceProperty(Hand, collection_name='fingers')

	def trim_nail(self):
		#... do work
		pass

def trim_my_nails(hand):
	@transaction
	def txn(hand):
		#fingers = hand.fingers # NOT TRANSACTION SAFE
		fingers = Finger.all().ancestor(hand) # Transaction safe
		for finger in fingers:
			finger.trim_nail()
		#... do whatever transaction friendly tasks
		return hand
	trimmed_hand = txn(hand)
    return hand

Quick Notes and Pitfalls

* Transactions only allow ancestor queries
* Ancestor queries only work when the data is in the same entity group
* Given the above, transactions can only query data in the same entity group.
* Ancestor queries are far faster than regular queries because of the way that GAE stores their data for entity groups.
* Entity groups are recommended to only be “about the size of one user’s data” and should not get too big. In the example above, you probably don’t want “everyone’s hands and their fingers” in the same entity group just to be able to work inside of transactions. Model your workflow better to avoid this.
* If you are on high-replication datastore (HRD), then this works even faster.
* If you have a long chain of parent relationships and the model in the ancestor query is the parent of the parent, you may get more entities back than you expect. The hand/finger example is a bad analogy in this case. However, if you think of Forums with Sub-Forums and you try to get all the posts with the ancestor some forum entity, you might be getting back all the posts of that forum as well as any of its child forums. This is ideal for some situations and is unexpected behavior for others.

Further Reading

• Datastore Indexes and Queries Google App Engine Docs
• Transactions Google App Engine Docs

Here is the skinny:
I was running into an unexpected issue with the AppEngine users service on a folder locked down by app.yaml. I had a folder /cms/ that I wanted the contents of to only be viewed by admin users to my Google AppEngine project. All pages inside of the directory displayed the email address of the user as well as a log out link. I was using users.GetCurrentUser() and users.create_logout_url to display this. The folder itself was being locked down at the app.yaml level. Aside from this, there was no other security (@require_admin decorators, etc). After testing, I discovered I was able to return to pages within /cms/ while being logged out via the users.create_logout_url link. users.GetCurrentUser() was returning None as expected, but I figured I would be redirected to a Google login page.

I figured this might be a bug with the current sdk, but I wasn’t willing to assume that until I had support. After submitting to the python mailing list with no reply, some googling, I started debugging the app.yaml. In the end I discovered it to be the url matching condition to the app.yaml file – essentially a logical “typo”. For what it is worth, I didn’t check here first because I am making changes to an existing set of code that already had the app.yaml condition in place. However, if you are in the same boat ever, hopefully this info will help.

So, if you think you have a folder locked down with app.yaml and users are still able to visit the page while logged out, check the url conditions. Examples below:

My app.yaml has

#- url: /cms # BAD VERSION - only protects a file named cms in the root, not the cms folder</del>
- url: /cms.* # GOOD VERSION - protects any url matching the regexp "starts with '/cms' followed by any number of characters (including '/') "
  script: main.py
  login: admin
  secure: always

Hopefully, that helps someone out. I know I wasted lunch break figuring it out.

On the page, I display the value of users.GetCurrentUser() and have a logout link generated by users.create_logout_url(‘/home/’).
When I click logout, I am redirected to /home/ as expected.

The odd behavior is that I can then go to /cms/ again without having to reauthenticate. The value of users.GetCurrentUser() is None (as expected).
However, I would expect the app.yaml to cause me to be redirected to the Google login screen when trying to revisit logged out.

I have not tested on production yet. Is this correct behavior and/or a known issue on dev sdk?

Edit 2009-02-02 - Added note about inequality query complexity. Spoiler – inequalities are 2 queries under the hood.

In my further adventures in Google AppEngine land, I wanted to ensure uniqueness of an email field while still allowing the user to change their email address. Since, Google App Engine’s BigTable doesn’t allow you to custom define unique indexes, this poses a problem. Instead of relying on database defined indexes, you have to set up your own rules, which is a wee bit tricky for GAE beginners such as me.
This can be done in general by overloading the put() method on an entity, but I wanted to be able to handle the errors nicely for the form UI.

The old Django Method:

#account = account entity you are editing
email = form.cleaned_data['email'].lower()
users_with_this_email = User.objects.filter(email=email).exclude(pk=account.pk).count()
if (users_with_this_email > 0):
# do something...

The new Google AppEngine Big Table GQL method:

#account = account entity you are editing
...
email = form.cleaned_data['email'].lower()
users_with_this_email = Account.all().filter('email = ' , email).filter('__key__ != ' , account.key()).count(1)
if (users_with_this_email > 0):
# do something...

The key here is making sure some OTHER entity does not also have the value for their email field.
In regular Django, an entity will evaluate to its primary key when used in a query and can be filtered on ‘pk’ .
GQL, however, the primary key needs to be explicitly fetched via entity.key() and filtered on __key__

A couple other notes:

• GQL does have an email property. I have not run tests yet to tell if this is case sensitive.
• GQL does not allow case-insensitive searches like regular Django does. Thus, I am making sure all email addresses are lowercased when inserting and querying against the db. Even if this is not needed for EmailProperties, you will need it for ensuring the uniqueness of other types.
• != filters are actually ran as two queries – one where the != is replace with a less than inequality filter, and one where it is replaced with a greater than inequality  filter. No word at the moment if this applies to __key__ searches.
• A query with a != cannot have inequality filters because of the above rule nor other != filters.

Related Links

• Add A Unique Constraint to Google App Engine via Squeeville
• Case Insensitive Queries via GAE Java Persistence
• Appengine Types and Properties via Google Code
• Queries and Index via Google Code – discussion on inequalities and != filters