That would work quite well. Unfortunately, I assume the new endpoint will not be available on December 16th so my service will be down until it is 
If a hashed ID were provided in place of the owner field it may satisfy the data protection requirement and allow third party devs to group auctions by seller.
It would not satisfy the data protection requirement because you should have no identifier for a person, including hashed/anonymized identifiers.
For example, you want to see Jaina’s auctions. You know Jaina is selling a rare Ring of Frost for 10g by observing that in-game. You use the API to find all Ring of Frost auctions, and find one being sold for 10g, by ZXCVBNM. You find all the auctions posted by ZXCVBNM and we’re back where we started: you can find Jaina’s auctions and know they’ll always be under ZXCVBNM.
“But what if we changed the hashed name with every snapshot?” The auction IDs (or, at the very least, the details including the item and price) will remain the same between snapshots. So you’ll just need to update your aliases when reading the new snapshot.
1 Like
Thank you for your hard work and your time.
On removing AH seller names
Be sure to acknowledge that this reduces the possibilities to build useful auction comparison (competitive) tools around this, or possibilities to make an app to detect if we have been undercut and how much gold we’re missing out potentially. You were mentioning an opt-in solution. If players can opt-in to allow their names to be retrievable(including our own names) then it would solve almost all issues I can think of.
This also removes the possibility to compare the realms markets sizes between each other. Counting the amount of unique items or number of items on each realm is not representative of a realm AH size/activity. To provide enjoyable tools we need to be able to compare the realms auction house sizes between each others. This can be the number of unique AH users (sellers) seen within the last X days, 14, 30 or 60 for instance. Just a number updated per realms. We could want to be able to compare the size of each “markets”. The prices we use for our auctions can depend on market size, a possibility to determine a level of demand before listing is required.
On AH dump delay
We’ve been used to updates every ~30 minutes for the Auction house snapshots. Will we be able to access the refreshed data with a similar frequency? Since the AH can change very fast, the timing is important. While 1h is decent, 30 mins would greatly improve the quality of the data and service provided, and 15 mins would be awesome. I’m referring to market notifications. Even with a 15 min timing, players that are in front of the actual AH looking for items will always have the edge, as expected I suppose.
Improving the service further: game stats
Let’s talk about new kinds of fun data that Blizzard could provide.
Remembering the old game stats page:
h ttps://web.archive.org/web/20070220131111/h ttp://www.wow-europe.com/en/info/gamestats/gamestats.html
Stats have been appreciated by the community, I often see players positively mentioning the game stats we had in the old website. Such stats can greatly improve the current auction house API. Here’s some self explanatory ideas for each ItemIds or (ItemIds+ilvl pairs):
“monthly_sold_count”
“monthly_auctioned_count”
“monthly_looted_count”
“monthly_gathered_count”
“monthly_crafted_count”
“monthly_sold_price_average”
“monthly_sold_price_median”
This would work well on a region basis, if we have a way to determine the size of each realm’s AH (see above). details per realms would be better obviously but I can imagine this would be quite more work.
Such feedback is popular enough to be featured in places like weekly wow economy recaps that takes place officially on wowhead: h ttps://www.wowhead.com/wow-economy-dungeon-xp-hotfixes-auction-api-tokens-bfa-wishes#hopes-for-battle-for-azeroth
We love your api to play around, and I think we should always look forward into ways to enhance and improve the service into something that simply makes us enjoy the game even more.
5 Likes
This may not be exactly what your project needs but if we had some sort of anonymous counts of things that were provided by the guild API endpoint, we could still create statistical projects. For example, if I could lookup my guild and see there are x amount of 120s, and y amount are Warlocks, that would be a fun start. This of course still leads to the issue of knowing the guild names however. My original idea was to mimic realmpop but go even deeper, but only for my realm/connected realms. Like, it would be fun to compare anonymous achievement data. How many characters actually have the BFA Pathfinding Achievement for flying. Or how many characters have stepped foot into LFR, Normal, Heroic, or Mythic raiding?
I’m fairly certain that Blizzard knows this, but we as a community do not. And I think it would be fun to see what my realm is actually doing as a whole. I don’t want character named, or even guild names, just comparative data for the game.
I feel that adding some of this statistical data to some either guild or realm endpoint, could be a lot of fun.
1 Like
I certainly understand not wanting to return players that have opted-out for privacy reasons, and I also understand removing players that have been inactive for years and years.
But with the new APIs, I have noticed that even level 120 players that have been inactive for a very short amount of time seem to be removed from the APIs. Obviously, I cannot 100% confirm this, since I don’t know who has opted out or not.
My app is a leaderboards site, and I believe that my visitors want to see how they rank against all other players, regardless of if they’ve been inactive recently. I would ask Blizzard to please consider: At a minimum, perhaps continue to return max-level characters, unless they have explicitly opted out.
I don’t even want to imagine the heartache that sites like WoWProgress and RaiderIO will have as they have to expunge historical data about past winners of the World 1st Race due to the new API terms.
2 Likes
This is intentional. If they haven’t logged in since some of these processes were implemented, then they haven’t “opted-in” to the new process, so they are hidden by default. For privacy reasons - this is ideal. They should always opt-in to being shown or it defeats the whole purpose of implementing data privacy measures.
That’s fair, I guess my comment was thinking more long-term, and whether players are going to keep being removed after a short amount of inactivity.
I think its just transition pains at this moment. In a year, it may not be as big of a deal and the active players are re-established.
The only thing that I’d like to say was already mentioned here, by multiple devs which solutions used by thousands of players on day-to-day basis.
As for me, this changes is a huge step back, and it will “kill” or heavily affects multiple sites like realmpop/TUJ/TSM/checkpvp/etc.
I doubles Aada-a-million-gold-mailbox about the market changes and I also pleased to see more API endpoints that allows us to receive more information from the auction house and do it much quicker then every hour as it is realised for now. For the other hand, Erorus is right about hashed-IDs which is could be compared between each auction house snapshots. He’s already paid for this with “realmpop”, and enabling anonymous lots-info on TUJ.
Also, I guess the whole community will agrees with this, that we need an endpoint which allows us to receive information about realm’s population and so on.
Kryo is right, that we still have guild-roster endpoint, which allows us to receive such data, but there are a lot of players which are not associated with any guild. And these players will become literally invisible, especially if the player has been inactive for a while.
For example, I have found many of my old in-game buddies, just because I could trace their characters via armory. And also we are unable to analyse data about “how much % of players have X”. So it will become a challenge to all of us.
2 Likes
While I agree it will be a step back for those apps lets also face the truth here: those endpoints were never meant to track realm population, as this kind of information is not publicly released by Blizzard itself since 2010 or around that year.
Personally I think it would be really great to have accurate realm population data, but I wish that information could be available for the realms endpoint or a specific endpoint.
Even if the AH endpoint still returned character names with the possibility of a character “hiding” their profile some statistics for realm population like class/race or even profession related data would be missing.
It would be really nice - and I know it is a huge thing to expect - to have a realm statistics endpoint. Something that would periodically return the amount of players, but also useful information like amount of characters that learned certain profession, amount of characters of certain race/class, amount of characters created per day over last X days, amount of characters above X item level, etc
That kind of information could really help players find the best realm to match their expectations.
3 Likes
And what is known about AH API for the classics, will access be introduced?
Blizzard read …
And now the delay reaches up to two hours on US realms.
I’m curious if it would be properly anonymized (and useful enough) to attach a “Session” hash to auction data
Specifically - I’m trying to track how item values change over time and the popularity of specific item values. It’s beneficial to know that two auctions were initiated by the same person
If, every day, a user was generated a new, random “Session ID” - and that Session ID was attached to all the auctions they posted that day - It would allow us to know the difference between 10 people posting 1 auction VS 1 person posting 10 auctions.
To “de-anonymize” a day, someone would already have to know a handful of auctions made by a specific user on a specific day. And then that’d only get them a day of information on that user.
Don’t know if it’s worth the technical hassle, on Blizzard’s side. But thought I’d pitch the idea!
1 Like
I’m not sure a full migration is even possible - some of the endpoints have vastly different (and missing) data.
For example: Community API item lookup shows the source type of the item (crafted, world drop, etc) along with the recipe ID if crafted, but Game Data API doesn’t have any of this information at all.
Until that data exists at the new location, it doesn’t seem like a migration would be possible.
In my opinion, to achieve both user privacy and still have those kind of statistics a solution would be to remove seller names completely (even from in-game) and replace it with a seller ID. With something like that it would be even possible to allow users to share that ID with 3rd party tools to map their own auctions and even allow the user to regenerate a new ID whenever they need (similar to our API secrets).
I don’t spend much time on the AH so I might be missing something, but I never felt the need to know who is selling what except for the same reason people reported here: Are those 200 Netherweave Bags being sold by the same char or not ? At least for this purpose an ID could easily replace char names.
Anyway, I understand something like that might be even impossible to achieve, same as Melol just throwing some ideas.
The issue with using long-lived IDs for anonymization is this -
- Let’s say you tied your Auction History to a random ID
- Let’s say that I knew you sold a Netherweave Bag on 12/01/2019
- Let’s say that I knew you sold Akunda’s Bite on 12/10/2019
There are VERY few people who would have done that exact set of trades at those exact times. Each small piece of data narrows down the possible set of ID dramatically. It’s an extremely high-dimensional space
I would then be able to map every auction you’ve ever made, back to you through that ID. Also every auction you made in the future - unless you knew to hit a “Regenerate ID” button in which case I’d just repeat the process.
It’s an issue that Netflix and some other “anonymized” datasets have faced after being released to the public
But if you had a new, random generated ID for every small chunk of time then it would be much harder to de-anonymize (less data points) and you’d have to have insider information for every single random ID that was generated, to fully know someone’s history. You also theoretically couldn’t use information from the past to easily track a person through the present.
This could be done in two ways -
-
Every day or time the user logs in, automatically hit the “Regenerate ID” button that you proposed
-
Make the Session ID something like SHA(User ID + item.id + “12-01-2019”)
You could help the “Sharing of auction history” problem by literally making an OAuth endpoint that lists all the auctions the user has done
^ This information would not likely teach the system how to identify your Session ID moving forward - so wherever you authed wouldn’t have an indefinite amount of time to track you
Note:
I add the disclaimer for “probably” not allowing folks to track you in the future. Because if you’re the biggest supplier of Netherweave Bags in the game then folks can probably make some assumptions about the future when a bunch of bags appear all with the same Session ID. But that’s in no way an easy to automate nor is it relevant for 99.99% of all players
Regarding the AH API: I think most points have been made already, but let me provide some more background information on how I actually used the AH API until 2019-12-17.
I used the API for two quite different purposes:
- I play many different characters and several of them post auctions. I don’t want to spend too much time in-game on auctioning, so it was convenient to use the AH API to get a list of which of my own auctions had ended, and therefore know which alts should check their mailbox within a month. Since the owner names are no longer present I cannot see any longer which auctions are mine.
- Trying to estimate for how much gold products were actually selling, by comparing subsequent auction scans and determining which auctions disappeared. The problem is in the simple way that cannot distinguish between sold and canceled auctions (expired auctions can be estimated by checking the auction duration). My heuristic for distinguishing canceled vs. sold auctions revolved on checking if the same owner added new auctions for the same item that just disappeared. Without access to some kind of owner ID that test is no longer possible, polluting the “sold” data heavily with “canceled” data.
Solving these 2 quite different problems could be done in many different ways, but in all cases would require support from Blizzard.
The first problem is compounded by the problem that (last time I checked) there is no way to get the auction IDs that appear in the auction listings in-game. One option could be based on exposing the raw auction ID in the in-game AH API (in the list of pending auctions), so an addon could save that information for your own auctions to its LUA saved variables, where an external app can pick it up and correlate it with the auction scans.
An alternative could be a new web API that would return the list of pending auctions for “myself” (parametrized by my character name or battle.net id)
The second problem is harder to solve without leaking identifiers for all auction owners (which I guess was the core issue in the eyes of Blizzard). Several people already suggested introducing new character identifiers instead of actual user names. As far as I am concerned, what would be enough for this second problem is to have an identifier that is unique for each (owner, item ID) combination, but can not be reversed to the owner ID. To put it mathematically, something like HASH([server secret],[owner id], [item id]) could work.
A very different solution would be to realize what we are actually trying to do: get a listing of all auctions and their final fate. The list of current auctions is merely a way of approaching that. If there were an API that retrieves actual sales information (in a form suitable for statistical analysis), that would be perfect!
There is a third solution that would solve the problem of distinguishing cancellations from actual sales. I suspect that not every auction player will be happy when I suggest it but here it goes: remove the ability to cancel auctions …
1 Like
My use-cases are very similar to Yoco - Love the detailed breakdown!
I was also in the process of trying to automatically identify what I was calling “Price fixers” - Folks who randomly put over-priced items into the Auction House, messing with all estimations of what the actual worth of items are
^ Not to call them out or anything but to remove their messy data from my calculations of item popularity/value
But with the removal of Owner ID I can’t identify whether or not items are actually worth X gold or if the same “Price Fixer” has flooded the Auction House 100 times with obscene prices to try to mess with folks
Yep, I also used to check for those profiles trying to overprice everything.
That is why I thought a Seller ID would be a good solution. You still know the same person posted the same items but you don’t actually know who is this person. Because unless that person shared the ID with a 3rd party app, there is no easy way to associate the ID to a character or bnet account.
Something similar is used in some e-commerce platforms where you don’t expose a partner’s personal information to end customers (except for legal issues) and the whole transaction is managed by the platform.
But again, this only works if seller names were also removed from the in-game AH, otherwise it is very easy to map the IDs.
1 Like
- Love it!
Sounds like a solution that’d help us all for these use cases while still maintaining a large level of privacy for sellers is either:
Session ID = Hash(Secret + User ID + Item ID)
OR
Session ID = Hash(Secret + User ID + Item ID + (week date?, month date?, day date?, login time?))
Then an OAuth endpoint that gives access to the auctions lists for the given user
You could also theoretically use that ID in the UI (like Schiller recommended) of the Auction House (though some effort would have to be made to make it easier for the user to understand visually). That should fully anonymize the AH data while allowing folks to still better understand trends and market manipulation!
Totally possible but it’d depend on if Blizzard finds it useful and secure enough to implement!