We have tables on the Azure service that we use the RESTful API to interface with.
I've noticed that as our tables have grown, the larger tables (4500+ records and 7000+ records) are returning more query responses that require continuation tokens. According to what I've read, supposedly when your query exceeds 1000 records
or the query crosses a partition boundary, continuation tokens are returned.
When the continuation tokens are presented for a query that crosses a partition boundary, the second response, i.e. the response retrieved by using the continuation tokens 'forgets' the original query and seemingly returns random records.
For example, when present this query:
Last query = 'ObjectID eq '73a781fb-4657-467d-b893-f8110635b3bc' and UserID eq 'f70eb1a0-b221-4577-9a8d-c1fb0f6ebd79'' from Table 'Permissions'.
using this code:
Dim resource As String = [String].Format(_tableName & "()?$filter=" & Uri.EscapeDataString(_filter))
request = CreateRESTRequest("GET", resource, Nothing, Nothing)
request.Accept = "application/atom+xml,application/xml"
where CreateRESTRequest is:
Public Function CreateRESTRequest(ByVal method As String, _
ByVal resource As String, _
Optional ByVal requestBody As String = Nothing, _
Optional ByVal headers As SortedList(Of String, String) = Nothing, _
Optional ByVal contentType As String = "", _
Optional ByVal ifMatch As String = "", _
Optional ByVal md5 As String = "") As HttpWebRequest
Dim strTrace As String = "General Fault."
Dim strRoutine As String = _rootClass & ":CreateRESTRequest"
Try
Dim byteArray As Byte() = Nothing
Dim now As DateTime = DateTime.UtcNow
Dim uri As String = Endpoint & resource
Dim request As HttpWebRequest = TryCast(HttpWebRequest.Create(uri), HttpWebRequest)
request.Method = method
request.ContentLength = 0
request.Headers.Add("x-ms-date", now.ToString("R", System.Globalization.CultureInfo.InvariantCulture))
request.Headers.Add("x-ms-version", "2009-09-19")
If IsTableStorage Then
request.ContentType = "application/atom+xml"
request.Headers.Add("DataServiceVersion", "1.0;NetFx")
request.Headers.Add("MaxDataServiceVersion", "2.0;NetFx") ' Changed 2012-09-24
End If
If Not String.IsNullOrEmpty(contentType) Then request.ContentType = contentType ' Supports the batch call
If headers IsNot Nothing Then
For Each header As KeyValuePair(Of String, String) In headers
request.Headers.Add(header.Key, header.Value)
Next
End If
If Not [String].IsNullOrEmpty(requestBody) Then
request.Headers.Add("Accept-Charset", "UTF-8")
byteArray = Encoding.UTF8.GetBytes(requestBody)
request.ContentLength = byteArray.Length
End If
request.Headers.Add("Authorization", AuthorizationHeader(method, now, request, ifMatch, md5))
If Not [String].IsNullOrEmpty(requestBody) Then
request.GetRequestStream().Write(byteArray, 0, byteArray.Length)
End If
strTrace = "The default timeout is: " & request.Timeout.ToString
TraceLogger(strTrace, strRoutine)
request.Timeout = ThisAddIn.AddinConfig.FocusSyncTimeout
'' Add Proxy if necessary - if a proxy is used, by default .Net knows the proxy settings and will use
'' the default credentials. This code is only useful if Windows credentials do not match what is
'' expected by the proxy
strTrace = "Establishing a proxy if required."
Dim myProxy As System.Net.WebProxy = GetMyProxy()
If Not IsNothing(myProxy) Then
request.Proxy = myProxy
End If
' Trace
strTrace = "Method: '" & method & "' Resource: '" & resource & "' Request Body: '" & requestBody & "'"
TraceLogger(strTrace, strRoutine)
Return request
Catch ex As Exception
TraceLogger(strTrace & " " & ex.StackTrace, strRoutine)
ErrorLogger(ex.Message & " " & strTrace, Err.Number, strRoutine)
Return Nothing
End Try
End Function
In the first response - we get the requested record (since there is only one corresponding to that query).
<feed xml:base="https://myName.table.core.windows.net/" xmlns="http://www.w3.org/2005/Atom" xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
<id>https://myName.table.core.windows.net/Permissions</id>
<title type="text">Permissions</title>
<updated>2015-09-03T06:22:57Z</updated>
<link rel="self" title="Permissions" href="Permissions" />
<entry m:etag="W/"datetime'2014-05-12T15%3A49%3A04.7327577Z'"">
<id>https://myName.table.core.windows.net/Permissions(PartitionKey='hyekJUQxeLrYZHtUu6GGBg%3D%3D',RowKey='fd22ba37-543f-40b7-8a31-075e044f8cbf')</id>
<category term="myName.Permissions" scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme" />
<link rel="edit" title="Permissions" href="Permissions(PartitionKey='hyekJUQxeLrYZHtUu6GGBg%3D%3D',RowKey='fd22ba37-543f-40b7-8a31-075e044f8cbf')" />
<title />
<updated>2015-09-03T06:22:57Z</updated>
<author>
<name />
</author>
<content type="application/xml">
<m:properties>
<d:PartitionKey>hyekJUQxeLrYZHtUu6GGBg==</d:PartitionKey>
<d:RowKey>fd22ba37-543f-40b7-8a31-075e044f8cbf</d:RowKey>
<d:Timestamp>2014-05-12T15:49:04.7327577Z</d:Timestamp>
<d:ID>fd22ba37-543f-40b7-8a31-075e044f8cbf</d:ID>
<d:UserID>f70eb1a0-b221-4577-9a8d-c1fb0f6ebd79</d:UserID>
<d:ObjectID>73a781fb-4657-467d-b893-f8110635b3bc</d:ObjectID>
<d:OwnerID>f70eb1a0-b221-4577-9a8d-c1fb0f6ebd79</d:OwnerID>
<d:Permission>7</d:Permission>
<d:TableName>Devices</d:TableName>
<d:ModifiedDate>2014-05-12T15:49:04.2488952Z</d:ModifiedDate>
<d:CreatedDate>2014-05-12T15:49:04.2488952Z</d:CreatedDate>
</m:properties>
</content>
</entry>
</feed>
We would expect that response without any additional records, however, we get continuation tokens - then when we use the continuation tokens to make subsequent request, we get a larger response
set (over 500 records, from four different partitions) where the data doesn't match the query.
Here's a subset of that response:
<feed xml:base="https://myName.table.core.windows.net/" xmlns="http://www.w3.org/2005/Atom" xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices"
xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
<id>https://myName.table.core.windows.net/Permissions</id>
<title type="text">Permissions</title>
<updated>2015-09-03T06:22:58Z</updated>
<link rel="self" title="Permissions" href="Permissions" />
<entry m:etag="W/"datetime'2014-10-27T00%3A01%3A02.521521Z'"">
<id>https://myName.table.core.windows.net/Permissions(PartitionKey='tAKu1Tgk4Y7TOp3zY2BrXA%3D%3D',RowKey='f6c83bfd-5bbd-4dcb-92a8-ddd94df762bc')</id>
<category term="myName.Permissions" scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme" />
<link rel="edit" title="Permissions" href="Permissions(PartitionKey='tAKu1Tgk4Y7TOp3zY2BrXA%3D%3D',RowKey='f6c83bfd-5bbd-4dcb-92a8-ddd94df762bc')" />
<title />
<updated>2015-09-03T06:22:58Z</updated>
<author>
<name />
</author>
<content type="application/xml">
<m:properties>
<d:PartitionKey>tAKu1Tgk4Y7TOp3zY2BrXA==</d:PartitionKey>
<d:RowKey>f6c83bfd-5bbd-4dcb-92a8-ddd94df762bc</d:RowKey>
<d:Timestamp>2014-10-27T00:01:02.521521Z</d:Timestamp>
<d:ID>f6c83bfd-5bbd-4dcb-92a8-ddd94df762bc</d:ID>
<d:UserID>b3b2a2e9-9111-4709-92ed-71e2b3360622</d:UserID>
<d:ObjectID>1fc6ba0e-304a-47b3-81d8-2201cdac9e49</d:ObjectID>
<d:OwnerID>b3b2a2e9-9111-4709-92ed-71e2b3360622</d:OwnerID>
<d:Permission>7</d:Permission>
<d:TableName>Focuses</d:TableName>
<d:ModifiedDate>2014-10-27T00:01:02.1738624Z</d:ModifiedDate>
<d:CreatedDate>2014-10-27T00:01:02.1738624Z</d:CreatedDate>
</m:properties>
</content>
</entry>
<!...>
<entry m:etag="W/"datetime'2015-09-03T03%3A48%3A55.4824602Z'"">
<id>https://myName.table.core.windows.net/Permissions(PartitionKey='yRL5TND2w4mVaqkvkQABAQ%3D%3D',RowKey='0c710b16-3f34-47e6-bba4-5a849100f393')</id>
<category term="myName.Permissions" scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme" />
<link rel="edit" title="Permissions" href="Permissions(PartitionKey='yRL5TND2w4mVaqkvkQABAQ%3D%3D',RowKey='0c710b16-3f34-47e6-bba4-5a849100f393')" />
<title />
<updated>2015-09-03T06:22:58Z</updated>
<author>
<name />
</author>
<content type="application/xml">
<m:properties>
<d:PartitionKey>yRL5TND2w4mVaqkvkQABAQ==</d:PartitionKey>
<d:RowKey>0c710b16-3f34-47e6-bba4-5a849100f393</d:RowKey>
<d:Timestamp>2015-09-03T03:48:55.4824602Z</d:Timestamp>
<d:ID>0c710b16-3f34-47e6-bba4-5a849100f393</d:ID>
<d:UserID>c56dfd79-850e-4eb8-9144-c0c9cb1334ee</d:UserID>
<d:ObjectID>19d495c3-dd17-44b2-970a-c9f06b136c8c</d:ObjectID>
<d:OwnerID>c56dfd79-850e-4eb8-9144-c0c9cb1334ee</d:OwnerID>
<d:Permission>7</d:Permission>
<d:TableName>Devices</d:TableName>
<d:ModifiedDate>2015-09-03T03:48:57.3381567Z</d:ModifiedDate>
<d:CreatedDate>2015-09-03T03:48:57.3381567Z</d:CreatedDate>
</m:properties>
</content>
</entry>
<!...>
<entry m:etag="W/"datetime'2015-01-11T11%3A41%3A47.8576124Z'"">
<id>https://myName.table.core.windows.net/Permissions(PartitionKey='yjj3m2tJR1ubuHomKn3Sg%3D%3D',RowKey='ce58baa0-e6a8-4df7-9917-8f32cd135126')</id>
<category term="myName.Permissions" scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme" />
<link rel="edit" title="Permissions" href="Permissions(PartitionKey='yjj3m2tJR1ubuHomKn3Sg%3D%3D',RowKey='ce58baa0-e6a8-4df7-9917-8f32cd135126')" />
<title />
<updated>2015-09-03T06:22:58Z</updated>
<author>
<name />
</author>
<content type="application/xml">
<m:properties>
<d:PartitionKey>yjj3m2tJR1ubuHomKn3Sg==</d:PartitionKey>
<d:RowKey>ce58baa0-e6a8-4df7-9917-8f32cd135126</d:RowKey>
<d:Timestamp>2015-01-11T11:41:47.8576124Z</d:Timestamp>
<d:ID>ce58baa0-e6a8-4df7-9917-8f32cd135126</d:ID>
<d:UserID>23903b0b-4a83-41dd-bf31-7effbb606bce</d:UserID>
<d:ObjectID>2156cf4f-6a75-4925-b858-f92cefe9b2f4</d:ObjectID>
<d:OwnerID>23903b0b-4a83-41dd-bf31-7effbb606bce</d:OwnerID>
<d:Permission>7</d:Permission>
<d:TableName>Devices</d:TableName>
<d:ModifiedDate>2015-01-11T11:41:48.2049915Z</d:ModifiedDate>
<d:CreatedDate>2015-01-11T11:41:48.2049915Z</d:CreatedDate>
</m:properties>
</content>
</entry>
<!...>
<entry m:etag="W/"datetime'2014-07-30T10%3A15%3A43.1214265Z'"">
<id>https://myName.table.core.windows.net/Permissions(PartitionKey='zgl8%2BvItzHtLq73fEC%2BbAw%3D%3D',RowKey='ff309c48-cde3-412c-b1c9-be9514ed7ec0')</id>
<category term="myName.Permissions" scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme" />
<link rel="edit" title="Permissions" href="Permissions(PartitionKey='zgl8%2BvItzHtLq73fEC%2BbAw%3D%3D',RowKey='ff309c48-cde3-412c-b1c9-be9514ed7ec0')" />
<title />
<updated>2015-09-03T06:22:58Z</updated>
<author>
<name />
</author>
<content type="application/xml">
<m:properties>
<d:PartitionKey>zgl8+vItzHtLq73fEC+bAw==</d:PartitionKey>
<d:RowKey>ff309c48-cde3-412c-b1c9-be9514ed7ec0</d:RowKey>
<d:Timestamp>2014-07-30T10:15:43.1214265Z</d:Timestamp>
<d:ID>ff309c48-cde3-412c-b1c9-be9514ed7ec0</d:ID>
<d:UserID>cd80b091-d344-4ba8-8d20-1be79ff7d22d</d:UserID>
<d:ObjectID>1f5fade0-cf15-4110-97ee-df75ab6a0dd2</d:ObjectID>
<d:OwnerID>cd80b091-d344-4ba8-8d20-1be79ff7d22d</d:OwnerID>
<d:Permission>7</d:Permission>
<d:TableName>Projects</d:TableName>
<d:ModifiedDate>2014-07-30T10:15:44.7787126Z</d:ModifiedDate>
<d:CreatedDate>2014-07-30T10:15:44.7787126Z</d:CreatedDate>
</m:properties>
</content>
</entry>
</feed>
As a reminder, this is from one of our larger tables (7000+ records) - although larger than many of our other tables, 7000 records is not a large table.
Can anyone let me know if Azure is having any issues with continuation token requests?
thanks
Chris
- Edited by clindstrom Thursday, September 03, 2015 1:35 PM