WoW API giving possibly incorrect 429 errors

API Bug Report
#1

Hello,

Since the patch on Aug 3, I have noticed a large amount of HTTP 429 errors being returned by the WoW API calls. I slowed down my services to roughly half of what is allowed, and yet the number of 429 errors did not seem to decrease at all. This makes me think that something else is going on, and that these 429 errors are incorrect.

It feels similar to recent problems where the WoW APIs were giving lots of 404 errors, even though data existed. See WoW Services return 404 altough there's data

Since it seemed to help last time, here is a log of Trace IDs for the most recent 429 errors. Thank you for any help you can provide.

2021-08-04T10:57:10.270-04:00	'x-trace-traceid': [ 'e31b307a-1fbb-3961-b755-42eeeb8f9cac' ],
2021-08-04T10:57:10.270-04:00	'x-trace-spanid': [ '1697d146-caa1-0270-f53e-fa164c73d3b8' ],
2021-08-04T10:57:10.270-04:00	'x-trace-parentspanid': [ '1697d146-caa1-0220-f53e-fa164c73d3b8' ],

2021-08-04T10:57:20.212-04:00	'x-trace-traceid': [ '54b45f33-04a1-3646-90a4-9e14a5e89584' ],
2021-08-04T10:57:20.212-04:00	'x-trace-spanid': [ '1697d146-caa6-15e0-f53e-fa164c73d3b8' ],
2021-08-04T10:57:20.212-04:00	'x-trace-parentspanid': [ '1697d146-caa6-1590-f53e-fa164c73d3b8' ],

2021-08-04T10:57:27.362-04:00	'x-trace-traceid': [ 'e73f032c-2cdc-344b-bbf9-e0747911ee8e' ],
2021-08-04T10:57:27.362-04:00	'x-trace-spanid': [ '1697d146-caa9-9c00-f53e-fa164c73d3b8' ],
2021-08-04T10:57:27.362-04:00	'x-trace-parentspanid': [ '1697d146-caa9-9bb0-f53e-fa164c73d3b8' ],

2021-08-04T10:57:39.324-04:00	'x-trace-traceid': [ '94bd5ef8-618d-3d6d-a922-d90a38832c1e' ],
2021-08-04T10:57:39.324-04:00	'x-trace-spanid': [ '1697d146-caaf-1cc0-f53e-fa164c73d3b8' ],
2021-08-04T10:57:39.324-04:00	'x-trace-parentspanid': [ '1697d146-caaf-1c70-f53e-fa164c73d3b8' ],

2021-08-04T10:57:44.318-04:00	'x-trace-traceid': [ '3a041137-8bc4-35fd-858c-427a7fee9a1a' ],
2021-08-04T10:57:44.318-04:00	'x-trace-spanid': [ '1697d146-cab1-ea30-f53e-fa164c73d3b8' ],
2021-08-04T10:57:44.318-04:00	'x-trace-parentspanid': [ '1697d146-cab1-e9e0-f53e-fa164c73d3b8' ],
4 Likes
#2

Can confirm that since August 3, instead of being allowed to do 100 api calls per second (as per the developer portal documentation), I have been getting 429s after 5 api calls per second consistently.

3 Likes
#3

The problem seems to have gotten much worse with today’s patch (Aug 10). I am trying to slow my services down, but I seem to get 429 errors no matter what I do. Currently I have it set to roughly 2 requests per second, which equals 7,200 requests per hour. Which should be well under the rate limit of 36,000/hour with 100/second burst. Would appreciate any help on this issue, thanks!

#4

Yep, still ongoing issue :frowning_face: waiting for response…

Other threads with this problem:

2 Likes
#5

I got some support, posting it here for anybody else it might help:

The main problem appeared to be that I am using HTTP Keep-Alives during my API calls. This means that connections were staying open to the same server for a long time. However, Blizzard APIs try to load balance requests across many servers. Since my connections were holding on to a single server, I was receiving only a small portion of the overall rate limit, and therefore seeing lots of 429 errors.

I modified my code settings so that HTTP Keep-Alives will only last 5 minutes maximum. This hopefully gives most of the benefits of Keep-Alives while still ensuring connections cycle enough to get load balanced. Other suggestions were also to use more concurrent connections (which I was already doing) and to close out a connection after seeing a 429 error (which I could not easily accomplish).

Since making these changes, I have seen a large improvement, though I have still had a few 429 errors. I will continue to keep an eye out but so far it appears the issue can be considered resolved.

I still am not quite sure why things were working fine for so long, and then suddenly this problem started happening. I do think something changed. However, the solution 100% makes sense and am very thankful for the assistance. Thank you!

#6

I’ve changed my HTTP client settings, it’s odd that I don’t get any 429 on US but I continue getting 429 on EU servers. US APIs sometimes don’t respond within 30 seconds which cause exception but we may look into this later.

Sample headers that I just recorded with 429 response on EU region:
x-trace-traceid: 005ddb48-2735-3b75-8aac-8208569120f3
x-trace-spanid: 1697d284-2ec7-4440-ab0a-fa165b5d6d4e
x-trace-parentspanid: 1697d284-2ec7-43f0-ab0a-fa165b5d6d4e

x-trace-traceid: 59ae8b7c-2275-370b-9b28-f33cc03ac700
x-trace-spanid: 1697d284-2efd-1550-ab0a-fa165b5d6d4e
x-trace-parentspanid: 1697d284-2efd-1500-ab0a-fa165b5d6d4e

Thx…

#7

Thank you for your continued patience and those that sent sample traces to aid investigation. We looked into this issue more closely last week and were able to identify a configuration change that inadvertently reduced the per-node throttle in place. As called out, this would have impacted consumers that were relying on persistent connections or keep-alive more heavily.

The problem has been corrected so quota limits should be restored to the previous behavior. Please reach out if you continue to see issues.

3 Likes
API Access Client Quotas
API Access Throttled
#8

Still seems to be getting the issue on US endpoints. Getting throttled just after 35ish requests

'x-trace-traceid': '7155d9d2-bb53-37d1-a47a-ab5595a7ecdd', 
'x-trace-spanid': '1697d14c-8d6e-d500-2611-fa164c46c167', 
'x-trace-parentspanid': '1697d14c-8d6e-d4b0-2611-fa164c46c167'
#9

Same here, the error rate has decreased significantly, but the problem remains. In my case(StarCraft2 API), only the EU endpoints are affected now(previously all regions were affected)

#10

Hello! I experienced this 429 issue at approximately 11 PM Pacific today, so it appears to still be happening. The call was to /oauth/token. I tried again immediately and it worked on the 2nd attempt.

Further, when I went to log in to this site, I got a 500 Internal Server Error, so it looks like some shenanigans are still happening.

#11

Adding some trace ids

x-trace-traceid: 9d4d24af-da48-334d-9791-7faf368ac152
x-trace-spanid: 1697d281-3ad6-a1e0-477a-fa165aed92fd
x-trace-parentspanid: 1697d281-3ad6-a190-477a-fa165aed92fd

x-trace-traceid: b869d9ae-253a-315b-a501-ef503d1a6f85
x-trace-spanid: 1697d286-b2d2-ebd0-d9bb-fa165ba37419
x-trace-parentspanid: 1697d286-b2d2-eb80-d9bb-fa165ba37419

x-trace-traceid: 5d442857-0f14-3c43-9caf-cdb2363374f9
x-trace-spanid: 1697d2fc-c089-7b30-e228-fa165ae0da3f
x-trace-parentspanid: 1697d2fc-c089-7ae0-e228-fa165ae0da3f

x-trace-traceid: 3d121068-ea32-3eaf-bb42-4bf639a4c59a
x-trace-spanid: 1697d291-277f-fe40-6363-fa165b3fe0b2
x-trace-parentspanid: 1697d291-277f-fdf0-6363-fa165b3fe0b2

x-trace-traceid: 339cf7b8-ad7f-3170-a23e-aab231377c1f
x-trace-spanid: 1697d281-3aed-c7b0-477a-fa165aed92fd
x-trace-parentspanid: 1697d281-3aed-c760-477a-fa165aed92fd

x-trace-traceid: 9074167d-993b-355f-b5b0-85d99278cb24
x-trace-spanid: 1697d283-7e73-6e80-aba8-fa165a322291
x-trace-parentspanid: 1697d283-7e73-6e30-aba8-fa165a322291

x-trace-traceid: 9f7d6ed2-41ca-3463-b898-4f9bc728033e
x-trace-spanid: 1697d287-3cfe-f4f0-ab0a-fa165b5d6d4e
x-trace-parentspanid: 1697d287-3cfe-f4a0-ab0a-fa165b5d6d4e

x-trace-traceid: 2067c926-d86c-37e0-9eee-11ff49cacf45
x-trace-spanid: 1697d284-5776-9ce0-81e3-fa165a854915
x-trace-parentspanid: 1697d284-5776-9c90-81e3-fa165a854915

x-trace-traceid: 393d102d-ddc4-3de2-a177-7b782a59cb94
x-trace-spanid: 1697d285-25cd-7200-3f97-fa165bdb965a
x-trace-parentspanid: 1697d285-25cd-71b0-3f97-fa165bdb965a

x-trace-traceid: f94a6fd1-f3bf-3d8e-951e-2b84533a2ab2
x-trace-spanid: 1697d28a-4e88-8f70-0156-fa165b360c58
x-trace-parentspanid: 1697d28a-4e88-8f20-0156-fa165b360c58