Option to Compress SavedVariables

Community General Discussion
1

Hi there.

I’m a developer that maintains the AllTheThings addon for all versions of WoW and this is something that’s been itching at me for the longest time.

SavedVariables for WoW are encoded to be human readable. They have comments for table positions and indexes and tabulation and new lines literally everywhere. This makes for a data document containing several hundred thousand indexes in a table to be massive.

Not only does this make the file large, but since Lua is a text-based language, it also makes for loading the SavedVariables take a lot longer than they need to. Expand that by 100 addons with a lot of saved variables and you’ve got yourself a mountain of load time attributed to data that no normal user needs to see.

I propose an option in the TOC document for an addon to enable human readable SavedVariables, but to have the default behaviour be to compress the file as much necessary. (bare in mind that there is a hard limit on the number of characters you can have on a single line, make sure not to exceed this limit, you can add a random newline character somewhere and it’ll prevent it from crashing the interpreter)

With an option in the TOC doc, it makes it so that addon developers can still enable this option to makes their documents readable, but also provide compression so that we can reclaim a lot of wasted time and energy that is currently being taken by these huge files. I’m sure if this option is provided in the TOC that addon distributors such as CurseForge or WAGO.io can export addons with this option disabled so that end users always get production quality compression when targeting Release builds.

I personally have a tool that compresses all of my SavedVariables when I exit the game and it simply removes all newlines, all tabs, all spaces, and all comments and my load time was cut down by about 10-15 seconds. Imagine the gains for people with an excessive number of addons!

Thanks!

EDIT: To give you a figure, ATT’s SavedVariable file as is is about 4.5 MB. With just removing the whitespace/comments, it goes down to about 2.1 MB. While that doesn’t sound like a whole lot, bare in mind that’s just extraneous characters. The raw data itself isn’t compressed at all.

151 Likes
2

My lord my savior has arrived on the forums.

21 Likes
3

I’ve been wanting a change like this for a bit now, literally hours of my life could be saved each week for more play time if I saved around 30 seconds for every dungeon, portal or character swap I did in a day.

4 Likes
4

I have often wondered if addons ever reviewed by Blizzard, they could avoid a lot of the bloat, and save a ton of resources. I am not a developer and never knew these components of an addon, but this sounds amazing. Please Blizzard find anyway to incorporate resource savings in our game! Let the Upvote party begin!

2 Likes
5

It’d be nice if Blizzard allowed some ATT devs into the community council to help with technical points like this Crieve.

10 Likes
6

I’m a developer that maintains things (not All The Things, just some things). SavedVariable handling has many areas where it could potentially improve. One big pain point is reaching the constants limit and having all of your SavedVariables thrown into the shadow realm if the user doesn’t immediately notice what happened. There’s another suggestion here on GitHub regarding early-load of SavedVariables, with some other nice suggestions in the comments.

8 Likes
7

This should absolutely be addressed. I have experienced this for quite a while with a few bigger var mods, and it’s a bit of a ‘meme’ how long my load times are for a top-end PC.

4 Likes
8

This is a great idea, I was actually planning on writing a suggestion for this as well. SavedVariable management is such a pain. Another thought I had was to add the ability to addon-scope the variables the same way that we do for source files currently. Each source file receives the addon reference as the second argument like this:

local appName, app = ...;
app.MySavedVariable = 23;

That way you could theoretically scope SavedVariables and pollute the global scope less.

5 Likes
9

Just out of curiosity.

I use exclusively NVME drives.

Could using the builtin Windows drive compression tool make a difference?

edit: My game is installed on a separate 1TB NVME all by itself.

1 Like
10

Storage compression isn’t what’s being asked for here.

In high-level terms, Crieve is asking for a form of compression that reduces unnecessary characters/whitespace/etc. that the LUA interpreter is currently wasting time and resources on reading.

5 Likes
11

So two entirely different things then…
Ok, just curious, seems the drive compression would just add overhead while reading to me, so was asking to clarify.

1 Like
12

Exactly this. To give you a figure, ATT’s SavedVariable file as is is about 4.5 MB. With just removing the whitespace/comments, it goes down to about 2.1 MB. While that doesn’t sound like a whole lot, bare in mind that’s just extraneous characters. The raw data itself isn’t compressed at all.

6 Likes
13

Correct. It would be negligible, but it would add some sort of overhead.

Using imaginary numbers here, let’s say a LUA file is 6KB uncompressed, and let’s say storage compression pulls that down to 3KB.

While it’d be stored in a compressed format at 3KB, when the call is made to open the file, it’d be expanded back to its 6KB original state prior to being read by the interpreter.

Whereas, the request here is to straight-up trim the fat and knock that 6KB down to 3KB, so the interpreter would only handle 3KB of data instead of 6KB.

These are made-up numbers but hopefully that helps clarify.

(Or see Crieve’s post before this one that clarifies it using real numbers).

2 Likes
14

Yeah that’s a massive difference. Then add it up across all the addons running at any given time, and yeah…as you said, load times become a total meme.

2 Likes
15

Oh please please please yes please. My load/reload times are atrocious. I tolerate them well enough when it’s just me, but feel real bad when I have to ask my group to wait for a reload.

1 Like
16

Essentially, minification (or as some call it, uglification). Standard practice in web development to sandwich all the JavaScript files into a single file, remove all comments and white space, and replace long variable/functions/etc with non-intelligent (but short) characters instead.

2 Likes
17

Minification is a step beyond what I’m asking for, but I’d love it if they did go that route as well!

Anything that makes it so that we can get into the game faster the better. My team and I have done our best to keep things small, but this is something beyond our control.

5 Likes
18

This forum isn’t for asking for things that are actionable and real. This forum is for complaining that my axe didn’t drop. The real devs read Blizz forum.

2 Likes
19

A lot of us users have the experience of less-well-maintained addons we rely on needing to be fixed by the smartest/most-LUA-literate person in our guild. *cough* Altoholic *cough* Bagnon *cough* Would this change make that harder, or nah?

3 Likes
20

Nah, if you modified the addon’s TOC file to re-enable human readable output, it would export as human readable on the next reload. It helps technical people still do their tech support, but makes it so that the non-technical people don’t need to deal with the extra excess data.

2 Likes