Amazon SimpleDB - Is there a way to list all the attributes in a domain?
I am using C # and the AWSSDK library at Amazon to test a few things in SimpleDB. Everything is going well so far.
However, I am trying to find a neat way to retrieve all the attributes applicable to the Domain. It turns out to be tricky without having to fetch the element, and obviously I can get a list of attributes. But what if I have 100,000 items in the domain. Let's say the first 70,000 items in the Human domain:
FirstName, LastName, Address
And then I hit the item with
FirstName, LastName, address, phone
And then I hit another element around the 80,000 mark, which has:
FirstName, LastName, Email, Phone
In the above example for the Person domain, how would I get a list containing:
FirstName, LastName, Address, Email, Phone
... without executing a ridiculous number of select statements?
Many thanks!
a source to share
You should be able to get a very accurate list of attributes using random sampling for domains with many elements. Here's some C # pseudo code:
int domainCount = "select count(*) from Person";
int avgSkipCount = domainCount/2500;
int processedCount = 0;
string nextToken = null;
Set attributeNames;
do
{
int nextSkipCount = Random.Next(0, avgSkipCount*2);
string nextToken = "select count(*) from Person limit " + nextSkipCount;
var countRequest = new SelectRequest
{
NextToken = nextToken,
SelectExpression = "select count(*) from Person limit " + nextSkipCount
};
var countResponse = SimpleDb.Select(countRequest);
nextToken = countResponse.NextToken;
processedCount += countResponse.Count;
var getRequest = new SelectRequest
{
NextToken = nextToken,
SelectExpression = "select * from Person limit 1"
};
var getResponse = SimpleDb.Select(getRequest);
nextToken = getResponse.NextToken;
processedCount++;
attributeNames.Add(getResponse.AttributeNames);
} while (domainCount > processedCount);
It depends that you can use the NextToken returned from the select count (*) query to skip records in SimpleDB. Mocky has a great explanation on how to do this . And I explained how to perform efficient paging like this with Simple Savant .
This will give you 99% accuracy with most datasets, which should be good enough for most real world applications. Statistical theory says that a sample size of 2500 gives you the same precision for any size dataset, so this method scales even for millions of items.
This is obviously not ideal as it still requires a lot of queries, but you should be able to do the same thing with a much smaller sample size if your dataset has a relatively limited number of attribute variations.
a source to share