An unexpected issue with the Azure StorageClient and continuation tokens.

Normally, it should work.

I've been experimenting with the Azure Diagnostics feature the last few days, and had come to the point where I had to retrieve the performance counter data from Azure Table Storage. I remember having heard in a PDC presentation (probably Windows Azure Tables and Queues Deep Dive by Jai Haridas, he talks about continuation tokens, but I didn't listen to the session again to find an exact quote) that the new StorageClient for Windows Azure Storage would automagically take care of continuation tokens. So I expected that the following code would work:

    var results = from s in context.PerformanceCounters
                  where s.EventTickCount > fromTimeTickCount && s.EventTickCount < toTimeTickCount
                  select s;
    return Convert(results);
    ...
    static List<PerformanceDataSeries> Convert(IEnumerable<PerformanceCounterSampleEntity> entities)
    {
        foreach(var entity in entities)  //This is line where the request to Azure occurs.
        {
            // Fill in list
            ...
    

But apparently it doesn't.

However, it didn't. I only saw 1 request to Azure Table Storage in the Fiddler trace, and no entities were returned, even though I could clearly see that the entities were there with Cloud Storage Studio. After a bit of Googling I found an explicit statement that the StorageClient will automatically deal with continuation tokens. However, re-reading the statement it occurred to me that it only mentioned the CloudTableQuery.Execute() method. Could it be that the Execute() method would have to be called explicitly? I quickly tried out the following code:

    var results = from s in context.PerformanceCounters
                  where s.EventTickCount > fromTimeTickCount && s.EventTickCount < toTimeTickCount
                  select s;
    var query = results.AsTableServiceQuery();
    var entities = query.Execute();
    return Convert(entities);                    
    ...
    static List<PerformanceDataSeries> Convert(IEnumerable<PerformanceCounterSampleEntity> entities)
    {

        foreach(var entity in entities)
        {
            // Fill in list
            ...
    

Calling AsTableServiceQuery() followed by Execute() solved the problem. In Fiddler I saw multiple requests (probably more on this in a follow-up post), and the query returned the expected results.

Conclusion.

Apparently the Azure StorageClient properly supports continuation tokens, but only if you explicity use the TableServiceQuery object. If you let the Linq lazy evaluation do it's work, only one request will be done to Azure Table Storage. This may work if there is little data in the table, but will return no or insufficient data if the amount of data in the table grows. No error will be thrown, you will just get no or insufficient data. This is a very unexpected result, and will probably be the source of many errors with Azure Table Storage in the future. Actually, Steve Marx himself had a similar bug with continuation tokens that caused his blog to disappear, although in his case it was with the sample storage client code, while this issue is with the official StorageClient code.