Option to Compress SavedVariables

Support Technical Support
1

Hi there.

I’m a developer that maintains the AllTheThings addon for all versions of WoW and this is something that’s been itching at me for the longest time.

SavedVariables for WoW are encoded to be human readable. They have comments for table positions and indexes and tabulation and new lines literally everywhere. This makes for a data document containing several hundred thousand indexes in a table to be massive.

Not only does this make the file large, but since Lua is a text-based language, it also makes for loading the SavedVariables take a lot longer than they need to. Expand that by 100 addons with a lot of saved variables and you’ve got yourself a mountain of load time attributed to data that no normal user needs to see.

I propose an option in the TOC document for an addon to enable human readable SavedVariables, but to have the default behaviour be to compress the file as much necessary. (bare in mind that there is a hard limit on the number of characters you can have on a single line, make sure not to exceed this limit, you can add a random newline character somewhere and it’ll prevent it from crashing the interpreter)

With an option in the TOC doc, it makes it so that addon developers can still enable this option to makes their documents readable, but also provide compression so that we can reclaim a lot of wasted time and energy that is currently being taken by these huge files.

I personally have a tool that compresses all of my SavedVariables when I exit the game and it simply removes all newlines, all tabs, all spaces, and all comments and my load time was cut down by about 10-15 seconds. Imagine the gains for people with an excessive number of addons!

Thanks!

3 Likes
2

That’s a good suggestion, but the developers don’t receive suggestions posted in the Support forum group (including Technical Support). You’ll want to post in another forum outside the Support group (like General Discussion or UI and Macro, which is topical for discussion about addon support). Or use the in-game suggestion feedback tool.

1 Like
3

Okay, I reposted here: Option to Compress SavedVariables

2 Likes
4

It should stay the way it is because LUA is a scripting language. At the end user level, the saved data should be human readable. Can’t tell you how many times I’ve manually changed broken variables through the saved data.

Also, you’d need to throw in extra exes to the mix like a portable 7zip, which then causes a bunch more headaches. Oh and not to forget the potential security risks. LUA is sandboxed, sure, but you can still run some semi-malicious scripts that can do harmful things to your character. There have been things like WA attacks in the past that could do harmful things like mail all your gold to someone else. WA scripts are copy/pasted in a compressed format, so even if you were an experienced coder, you’d have no way of knowing if the WA was going to be safe or not, without going straight to the source of it on sites like WAGO.

It’s an extremely trivial amount of extra overhead on even a low end modern computer. Now if it were being done on tick, sure, you might want to optimize a little bit, but it’s not.

I really don’t see this being fully true unless you’ve intentionally made some ultra massive fake padded data table. While I don’t use many addons these days, back in the day, I did. And even with older school PCs on mechanical HDDs, the difference with and without addons enabled was pretty negligible in load times.

That 10-15 seconds you’re talking about was likely just the client hanging on the load screen waiting for the server to spawn you into the world; due to being busy. The next time you logged in for testing, it might not have had you in that little virtual queue.

Here’s how you actually test the difference:
Once you are logged into the world and you can verify the world server isn’t acting up or lagging out, type /reload and stopwatch the time it took. Then, disable all addons and run /reload again and time it again. That’s how you’d obtain the real performance metrics.

The other thing too is that loading of the data can be highly dependent on the addon doing the loading. Not all of them will load it all in one tick and some will stagger things out over multiple frames or will perform complex operations on each thing being loaded. In terms of actually loading the data though, those same addons wouldn’t care if it were 1kb or 10kb of data, since it’s being bottlenecked by operations instead and not by storage transfer speeds and seek times.

5

I’d gladly trade all of the points you made for faster reload times.

With what I proposed, you could re-enable it in the TOC doc. 99.9% of the time you don’t need to be able to read any of it.

Also, you can take a minified Lua document and “beautify” it in an online script beautifier to make it readable. I’m not asking for minification, just for the removal of extra whitespace and filler characters.

1 Like
6

Again, the difference is usually negligible and if anything, you’d see far higher reductions in load times by having the addons coded better.

A really good example of this was GTA5 a few years back. It turned out they were doing a really dumb recursive load loop, but wasn’t that apparent when the game launched. It wasn’t until GTAO had thousands of added items, that it became apparent. This was leading to extremely long load times in the minutes range until a modder figured it out and solved it. RS eventually implemented a fix on their end.

Take an addon like TSM, for example, they do all their database refresh stuff over the course seconds to minutes, rather than trying in one frame(would crash anyways). They don’t do it on the initial load of the addon either, otherwise, you’d have multiminute loadscreens when it decided it needed to redo the database.

Long story short: Code the addons better. The plaintext storage and wasted whitespace is the least of concerns.

7

It would help addons with large databases that need to be saved and restored like DataStore, AllTheThings, CanIMogIt, to name a few.

And your comment “Code the addons better.” comes off as really rude. I dare you to try better.

8

Yeah, if you want to take it as such. Using the Rockstar example, one of the largest AAA companies known for extremely optimized games, just goes to show there is always room for improvement.

There are ways to optimize things like this and like I said, you can easily do it after the initial load during “runtime” once the character is already in the world. From there, you can stagger out the loading over multiple frames where maybe it loads/processes a 1000 entries per frame or something that only costs like 1ms per frame. Like how TSM does it when they have to do their database stuff and theirs are in the 100+ megabyte range. Though they do a lot more work per frame and even warn you of such, you might go from 150fps to like 25fps while it’s doing its thing.

The point is, there are far better ways to mitigate the issue and the save file size isn’t even a drop in the bucket, in terms of how you’d streamline the issue.

9

We do this already. What I’m talking about and the only I have been talking about is the SavedVariables which are raw text files with extraneous characters. I’m simply asking for those extraneous characters and comments to not be included when writing the files back to the hard drive.

With absolutely 0 code changes and having a minifier run on the SavedVariables for just AllTheThings, which if you don’t know it, is the single largest database addon that WoW has ever seen and I personally help maintain, I’m telling you that I have observed about 4 seconds of less loading time.

If I minify the contents of the WTF folder, I get about 10 seconds of loading time removed.

This means by doing nothing to the addon code and having Blizzard simply change how they save SavedVariables, we get an insane amount of improvement. I’m not trying to have a discussion about what AAA studios do. That’s irrelevant to this discussion.

1 Like
10

Yeah, sounds like your parsing could be optimized then and that there’s still a lot of room for improvement. But I’m not going to grill you about it, just know that similar to an artist and a painting, the painting is never actually done.

No, I’m giving you an almost 1:1 example of how bad parsing can affect things like load times. It’s industry agnostic because it’s just about the code loop itself. The fact that a studio like them had this issue, shows that ANYONE can have an issue like this.

Reminds me of a couple years back when I had to write a raymarching algorithm for a game and it turned out I was doing it extremely inefficiently, no joke, it was 1/10 the performance of what it was once I figured out the dumb problem. I kept trying to point the finger at other things because I thought I was managing it correctly, but I definitely wasn’t. It wasn’t until a friend spotted the problem that I smacked myself in the forehead for being so bullheaded about it. It can happen to anyone.

11

There’s literally no parsing involved with SavedVariables on the side of the Addon Developers. This is a process that is 100% handled by Blizzard Entertainment.

My suggestion is simply to reduce the size of the files (that they generate) by removing extraneous characters and comments. With that done, I have observed insane improvements to loading times across the board.

This is what they are exporting for a table called “ATTSources” with 42 values stored in an array:

ATTSources = {
	nil, -- [1]
	nil, -- [2]
	nil, -- [3]
	nil, -- [4]
	nil, -- [5]
	nil, -- [6]
	nil, -- [7]
	nil, -- [8]
	nil, -- [9]
	nil, -- [10]
	nil, -- [11]
	nil, -- [12]
	nil, -- [13]
	nil, -- [14]
	nil, -- [15]
	nil, -- [16]
	1, -- [17]
	1, -- [18]
	1, -- [19]
	1, -- [20]
	1, -- [21]
	1, -- [22]
	1, -- [23]
	1, -- [24]
	1, -- [25]
	1, -- [26]
	nil, -- [27]
	1, -- [28]
	1, -- [29]
	1, -- [30]
	1, -- [31]
	1, -- [32]
	1, -- [33]
	1, -- [34]
	nil, -- [35]
	nil, -- [36]
	nil, -- [37]
	nil, -- [38]
	nil, -- [39]
	nil, -- [40]
	nil, -- [41]
	nil, -- [42]
}

And this is what it could be:

ATTSources={nil,nil,nil,nil,nil,nil,nil,nil,nil,nil,nil,nil,nil,nil,nil,nil,1,1,1,1,1,1,1,1,1,1,nil,1,1,1,1,1,1,1,nil,nil,nil,nil,nil,nil,nil,nil}

Now imagine that table, but with 45k elements in it. For a text-based language like Lua, that significantly increases the load time where it doesn’t need to. Mind you, that’s not a great example and yeah, it could probably be adjusted to use 0 instead of nil, but that’s not the point. The point is that the file exported by Blizzard is too big and cluttered with nonsense that they don’t need in the file.

1 Like
12

So your addon isn’t parsing/looping on what needs to be saved to file? You can also do some hacky things to store multiple values in a single value. There are lots of ways to do this. And your addon isn’t then having to read through all of the information that you wrote to be saved, to then be used to populate variables in memory for things like UI elements showing things like quest completions in lists?

I still think you’d be better of writing a loop for after the player is loaded in, that then finishes populating what it needs to. Likely wouldn’t even take a handful of seconds with you staggering it out over frames. You could load some small chunk of critical data during the load screen, but that way, it wouldn’t have to load all the other data at the same time. As far as I can tell, it’s not vital for it all to be loaded before the loadscreen is over.

13

Dude, you have no idea how the code base looks for either game. How one AAA studio handles things doesn’t show that another is doing the same thing. Rockstars handling of their internal data for loads and how lua reads string files are two different things.

Also, please put up the addon you maintain so we can see how well your save routines are. You must be a maintainer on some large addon with your talk of absolutes.

I’ve run TSM in the past, it still affects your LOAD time, not the time the addon spends doing maintenance once it’s loaded. Addons can’t do much of anything on a load. Even with TSM being my ONLY addon it can affect your load time by 5-10 seconds.

This change only asks for the way that Blizzard saves files to change. It has to be an internal change as the addon authors have no say in all the extra comments and CRLF’s that get added to the files.

My WeakAura’s saved file is 17mb It’s the largest I have and affects my load times quite a bit. Are you saying the WeakAura’s dev’s are just as bad at writing save routines?

14

I don’t know, maybe you guys have something else weird going on with your client or with saved data or something. To test this out, I installed a fresh copy of AllTheThings and made sure there were no saved files in the WTF folder associated with the addon. I tested with a character in Bastion, each time, logging out to the character select screen and then logging back in.

From character select screen with addon enabled:
9.22s
8.88s
8.68s

From character select screen with addon disabled:
7.65s
7.44s
7.53s

If I’m just doing a /reload they both take under two seconds. With the addon, it’s like an extra 600-800ms or so. This PC isn’t some extreme performance PC either. It’s a non overclocked 13600kf, ddr4 3200 ram, and the game is installed on a cheap gen4 nvme. So I don’t know how or why you guys are getting hit with these 5-10 second increased load times, unless there’s something messed up with your client or unless you’ve got some weird issues happening with your Windows security that’s slowing down memory and IO operations by a lot.

15

The “It doesn’t affect me so it must not be a thing” defense. Many years in IT have taught me that doesn’t fly. Also, assuming everyone has the latest generation of Intels processor and Gen4 PCIE motherboards is a stretch, not everyone upgrades their PC’s often. I’m still on a I7 9800K and a z390 motherboard. Gen 4 PCIE being almost 2x as fast with read/writes.

Many folks in my guild who run heavy addons talk of 45 second+ load times. So a minor change to how things get saved, and one that doesn’t affect you at all. Not sure why you are arguing so hard against this and blaming the addon authors and other users for their issues.

16

The access latencies are still likely on par with a regular sata3 ssd. Want to say nvmes are in the 50-100μs range and sata3s are in the 100-150μs range. For reference, μ=one millionth of a second. 1000μs=1ms and 1000ms=1s. The data speed is negligible between the two when it comes to loading such small amounts of data.

Things that can interfere with that are things like security features, drivers, programs running, etc.

So dated hardware from like 2017 that was slammed with tons of microcode updates and workarounds for exploits over the years?

If they are experiencing that long of loading times, it likely has little to do with the actual data loading times and is more likely due to the initialization code being ran by the addons(bottlenecking since ALL lua code is ran on the main game thread and is only single threaded). That and they also are likely still on mechanical hard drives, which are no longer recommended or supported. The minimum requirement is an SSD now.

There are also a lot of zones that have longer loading times, but that’s due to server delays giving the okay to stop displaying the loading screen and show the world behind it. During Legion, this was a really annoying issue in Dalaran where you’d hear everything going on in the world around you, but the load screen would still be multiple minutes sometimes.

If you really want to test things, find a stable and mostly barren world server. Go to somewhere like SL or WoD zones. That’s why it’s easier and more reliable to test using /reload because you don’t have to wait for server delays to spawn you into the world. Also make sure to not perform any testing in cities or around hubs with tons of people.

1 Like
17

You make way too many assumptions about things. Have a good day my friend, this isn’t productive at all.

1 Like
18

No, I’m just making an informed and educated guess. Even before I upgraded PCs, I was on an i3-10100 with WoW on a very cheap sata3 SSD. I ran my share of addons during that time, with addons that had databases to them like TSM, and I don’t think I had ever seen a loading screen longer than 15 seconds unless the server was taking its sweet time. /reloads were still in the seconds range then.

An old laptop that I used to play on was still on a mechanical drive (during Legion/BfA) and I still rarely saw loading times more than 1-1.5 minutes long unless it was Dalaran. That was with a i7-2630QM CPU from 2011…

I’m sorry that the numbers and evidence aren’t supporting your claims, but good luck.

19

One data point does not make a data set. Again, just because it doesn’t affect you does not mean it’s not a thing.

Everyone in my guild meets minimum specs and beats them in most cases, including the SSD/nvme requirement. There are tons of posts here and on reddit about long load times. The fix from Legion doesn’t work anymore. My load times range from 30-60 seconds, and I have an nvme. Reloads are 10-20. I’m not upset about my load times, I’d love to see them lower though without getting rid of addons that add QOL additions.

So, again, really not sure what your stance is here besides:

  • Addon authors suck and should fix their stuff
  • People need to update their computers to WAY beyond RECOMMENDED (not minimum) requirements.

So, really not helpful in a tech support forum. But you proved me wrong, so you have that.

20

Alright I just did some more testing.

  • I renamed my interface and WTF folders
  • I created hard symlinks (mklink /j “link” “target” in cmd prompt) to folders stored on a cheap 35 dollar sata3 SSD because I don’t feel like moving 100gb to it and wasting drive life. This means the rest of WoW will load with the nvme speeds, but the addons and wtf files will be limited to sata3 ssd levels
  • I went on curseforge and downloaded a ton of the most popular addons, the list is as follows: https://imgur.com/cTNqdMw I don’t most of these, just figured I’d clog it up with some garbage though. I suppose I could also throw on ElvUI as well, but it wouldn’t make much difference
  • Installed and updated the external TSM app and helper addon
  • Opened the WoW client and redid my graphics settings, everything is on max
  • Logged in, waited for TSM to finish database work
  • I closed all the popup windows for every other addon that had some first launch welcome thing
  • /Reload
  • Closed the client
  • Reopened the client and logged into my character

Three timed tests from the character select screen to seeing my character in the world:

  • 12.44s
  • 12.98s
  • 12.62s

Three timed /reloads, once already in game:

  • 5.2s
  • 5.7s
  • 5.6s

Again, these are tests done where the addons and their saved data are having to be loaded off of a cheap sata3 SSD (silicon power a55). I suppose I could go into my bios and disable e-cores, along with XMP for my ram to drop it to 2666. In Windows, I could use throttlestop to kill hyperthreading and then gimp my CPU’s clock speeds to mimic older gen CPUs. Though not much I can do about the L1/2/3 cache speeds and sizes. Still, even if I did all that, those numbers would likely only rise by 30% or so.

So this leads me to believe that you’ve got some kind of other systemwide issue going on. There have been other people with weird thread or IO related issues like this in the past, mostly to do with the suttering issues and audio crackle related threads. You likely share some things in common with those that experience those kinds of issues.

1 Like