Personally the biggest issue with this for me, is the database(RDS).
It does not like me pushing all the AH’s processed data for too many AH’s at once . I’m running serverless everywhere else, so the compute is not an issue for me. It auto-scales.
It would be something like 123 212 unique items with bonuses, spesicesId, etc * 242 = 29 817 304 per hour, within a 10-minute timespan. So this is probably something that could be solved, by paying more for a more powerful database. But that is not tempting when I earn nothing from having my app running
But the raw auction house data itself, I personally have moved out to a CDN (AWS S3) per region. So I just download the AH data, dump it to a file, and trigger a Lambda function to process it etc. Then the users can just get the data ASAP as it’s ready in the S3 bucket…
Like AlexZeDim, I fetch AH data in bulks of 20 but spread out over a minute and repeat this until it’s done.
I do prioritize updates on when users last requested them etc. So if someone, is currently using the app for a specific realm. Then they should get it as soon as the dump is available from Blizzard.