Authors CurseForge

i am trying to figure out the best method to take my gyp trade skill scanning code and turn it into a library.

the issue is that it's a data scanning function that takes a few minutes after each patch to collect up-to-date info. this means it's got saved variables and isn't exactly designed to be run from different mods at the same time.

so what's the best solution for sharing this code with other authors?

my thinking has me going down a callback registration and data sharing system.


i'm thinking what i would do is have mods register a callback for when the data is ready and also provide a table of the data they currently have (if any). when the scan is complete, the callbacks could be fired off passing the new data back. if any mod registers with data that is already up-to-date, then the scan would be skipped and that data would be passed back to each mod via the same "scan complete" callback.

similarly, the lib itself could be packaged with current data to avoid the need for scans. the scan code is really more of a fallback for when there's a new patch and the lib doesn't get updated right away.

however, this method means there will be duplicate tables out there for each mod that uses the data, which isn't ideal.


another option might be to designate a master mod in some way that owns the data -- like maybe each mod registers with its name and the first one (alphabetically) is the one that saves its data. the other mods would be given the table reference owned by the master mod and would not save their data... this seems like it might be complicated, but would certainly reduce duplicate data storage...

any advice/thoughts?

I'm not sure if that matches your needs but have you looked at the Datastore collection of addons by Thaoky (on curseforge), that Altoholic uses as backend?

i'll give that a closer look, but i wonder if it could be accomplished without having to create "fake" addons at the top level of the addon folder.

i kinda started thinking of a lib something libStub that would be used for shared data. the idea would be to load tables of data instead of library code. maybe use callbacks to tell mods that the data has changed (by other mods loading new data).... hmm...

Libs really aren't great for storing data... and you need to ask yourself "will anyone else even use the lib?" If you're storing data from the game (not mining from an external source) the answer is very likely "no". Users either aren't going to run multiple addons that use the same data store, or each addon will be different enough that they'll want their own means of storing data.

And why the hell are you re-caching things after a patch? That's nuts, you shouldn't need to do that.

the whole point of the code is the rescan on patch. the only other option is to self-disable on a new patch until the data can be verified as being up-to-date. it's certainly a viable option and solves the whole question of how to make it a lib -- in fact, that lib already exists as libtradelinks. it does its job very well, but it kinda sucked having my mod die every time there's a new patch -- particularly since i'm kinda lazy about repackaging things for places like wowinterface. so i put together a scanner to mine the data automatically.

this has only really come up as an issue because i've had other mod authors ask to use my scanning code. i suppose i could simply place the requirement that gyp be installed and that they piggyback on it, but i'm intrigued by the challenge of sorting this out.

i know libs are really not good at storing data, but if a lib could be created as a co-operative data manager that might actually open up some nice possibilities for mods to co-exist in some nice ways. but maybe that's more than needs to be done here. it'd probably suffice just to work with this particular case and not try to make something too generic. if this works well, maybe it'd open some avenues to explore later.

Quote from OrionShock

As Tekkub said, libs arn't good at storing data.

Being one that uses your auto updater code in Guild Craft is is a very real problem as 90% of the usefullness of Guild Craft comes from being able to decod the links and the need of that auto generated data.

However a top level, non-embedable, Addon/Lib may be required in this case here.

well the easiest solution for me is to simply say that gyp IS that top-level non-embedable Addon/lib. :) honestly, i'm not sure i really like the idea of splitting gyp into two mods.

how are you managing to avoid stepping on gyp's scan with you gc scan?

If adding GYP ended up being the solution, I'd only have 2 issues:

1) I already programmed GuildAds to use the LibTradeLinks API but if GYP is similar (haven't checked) that would be easily fixed.

2) GYP is 500KB in size whereas LibTradeLinks (14KB code+73KB table)+GYP scanning code (19KB) is approx. 106KB in size. I find this to be a significant difference in size.

Due to size alone, I prefer to be able to include only the scanning code (as modified by OrionShock).

What would be the problem if the scanning code would be placed in a library with instructions that it cannot be embedded (due to need of saving the data)?

Or what happens if multiple addons all have the same variable they want to save? I assume saving would be working (except for multiple copies on disk) and loading would load all saved copies but only the last one loaded would be in memory (which shouldn't matter, right?).

Quote from galmok

What would be the problem if the scanning code would be placed in a library with instructions that it cannot be embedded (due to need of saving the data)?

i don't know how the packaging system here works with libs like that, but i personally wouldn't want to split gyp into two mods. i could see maybe having an "external" mod that is the scanning code alone that disables itself if gyp is around, but then you run into potential issues with gyp coming to the game after the other mods have already esablished a database.

Or what happens if multiple addons all have the same variable they want to save? I assume saving would be working (except for multiple copies on disk) and loading would load all saved copies but only the last one loaded would be in memory (which shouldn't matter, right?).

dunno how well that would work. when you use a saved variable, you have to still have to declare it in your lua code. if you install one mod after the others have already done the scan, that new one won't have any data and could step on the current data with its empty table, depending on load order and conditions.

i think the best solution is to have each mod save its own data by itself. those mods would register with the scanning code as being interested in the results. if there is already current data (either embedded in the lib or supplied by another mod that's already been around for a scan) then that data is sent to the other mods and will be saved by them. it makes for some redundant data, but it in gyp's case, it's actually important to keep my links and my spell data specific to them together so that i can update the links when a new patch comes out (instead of simply invalidating them as being potentially inaccurate/broken i decode each link with the saved data and re-encode with the new data).

Quote from OrionShock

i check for your addon and then steal it from the main addon table.

which works if there are only two mods using the code, but once you get into 3 or 4 or 5...

I am not following you...

Just don't make any attempts at libraries with SavedVariables. It does not work properly at all.

I would have no objections to having a seperate addon as a requirement to GuildAds, but GYP is a tad too big and the tradeskill grinding code is much smaller (about 20KB, a bit more with additional support functions).

Reg. the scanning in the background, that would probably not work as the TradeSkill window is needed to perform the scan (as I understand it). However, if the data is in the local cache, I guess the scan could be reasonably fast. I take it you hint at simply not storing the gathered data at all?

yeah, once the scan has gone thru and cached everything, a new scan is only a min or two. would probably be faster without the progress bar/feedback system.

but it uses the tradeskill api so it locks out the user interactions with the tradeskill frame until it has completed. also (at the moment) it has issues if you hit escape (which fires a trade_skill_close i think -- probably some i could fix). not sure what happens if you zone while the scan is in progress or generally try to play the game.

it COULD be reworked to be less obtrussive, perhaps. maybe trade some speed for some hardiness but then any dependent mods would not function for the first few minutes after login... might not be a problem tho.


galmok - how are you determining the size of gyp vs the tradeskill code?

Quote from lilsparky

galmok - how are you determining the size of gyp vs the tradeskill code?

I am simply summing up the file-sizes... GYP sums to about 500KB (including images) while scan-code alone is much smaller (but needs support functions like LibTradeSkill (or similar interface from GYP) + a preloaded scan (the largest part, ca. 70KB).

okay, so i'm working a bit with the code. i *think* i can get the scan down to under 5 seconds if all the data is already cached. it's a little iffy, tho, because of timing issues.

the way it currently works, i have an OnUpdate script that handles closing the tradeskill frame once i have detected that good data has come thru. this means there's a throttle on the caching of at least 1 screen refresh per recipe scanned. depending on your fps, it'd take somewhere arounda 1 minute to scan everything.

but if i hardwire a close right into the logic, i don't have to wait for the OnUpdate script to fire, which means there's no throttle. i need to make sure that this technique doesn't cause any problems, tho. if i get events slightly out of order, it could corrupt the data which would be very bad.

if it all works out, tho, this would mean that the scan could be nearly free after the cache has been populated. i'm kinda curious just how much i could actually scan with this. certainly i could dig a little deeper and add recipe to itemid conversions and probably reagent requirements (tho i'm a little leary of cases where unique reagents aren't cached on the server).... more investigating to be done...