okay, i think i've got a version now that scans in about 4 seconds if the cache is populated already and if it's not, then it takes however long it needs to (typically something in the realm of 5 minutes or so). i need to run some tests to make sure it's getting good scans every time still.
i'm seriously debating going ahead and making a library for this that does a bit more than simply providing info for trade link decoding. i believe it would be possible to turn this into a live-scanning tradeskillinfo type database.... i need to look into the memory cost of going down that road...
scanning in the spell list tables ends up taking about 94k of data.
scanning in the item and reagent data (including backwards references for where items come from and where reagents are used) takes up about 1.5megs, but doesn't seem to take any additional processing time... it might be possible to avoid storing this data and instead use a version of the scanning code to access the data as needed, but that means doing a lot of unregistering/re-registering events on the fly which seems kinda sketchy...
You could compress the data similar to how PeriodicTables does it and unpack on demand.
You could also try to see if you can get a list of all item ids not in cache yet in order to cache them without having to use the tradeskill api for triggering the caching so you would only block that api to a) check all items are cached b) generate your data
yeah, i thought about storing as encoded strings, but i'm not sure if it's really worth it. 1.5megs to have quick access to the information seems like a reasonable trade off vs having to decode stuff.
in regards to caching, i'm trying my best to avoid hardcoded tables. right now the only data supplied is the list of trade skill id's and a guess at how many skills they each have. but i'll poke around with that technique to see if it speeds up the initial post-patch caching. it would certainly free up the tradeskill api, but i think it's also possible that the server might not know about some rare reagents in that case -- at least, until somebody with a recipe that uses that reagent browses that trade skill. probably a rarity, but still possible. if i can get a speed increase out of it, i'll give some thought to how it might fold into the process...
Would it be possible to get the enchant: link as well as the item: link? I use LibPeriodicTables for that currently. I also use it to get the list craftable items when mouse-over a reagent item. Everything can be gotten from the tradeskill window somehow. It sounds like you have already gathered that information (the 1.5MB tables), right?
Would it be possible to get the enchant: link as well as the item: link? I use LibPeriodicTables for that currently. I also use it to get the list craftable items when mouse-over a reagent item. Everything can be gotten from the tradeskill window somehow. It sounds like you have already gathered that information (the 1.5MB tables), right?
yeah, the recipes are stored by spellid so it's easy to get the itemid from the spellid (direct lookup). i stored the reverse lookup separately so it's also a direct lookup to go from the itemid back to the spellid (if it's a crafted item). there are some crafted items that have multiple recipe sources, so that is taken into consideration.
i also store a reagent usage table so that each reagent has a list of spellid's associated with it.
the info i can't get from the api is things like who sells it, what recipe item teaches the spell, what enchant scroll is created by a particular enchant, etc. that stuff still needs to be mined ahead of time.
honestly, i'm a little vague about how libpt works with the compressed data. when you access the data, doesn't it build the resulting table? doesn't that take additional memory?
i suppose going with a "flat" system where the data is recorded in lots of different tables for the different records might be more memory conscious than going with a nested table system, so i can look into that...
I think the problem here is that you're REmining this data over and over again. Yes, reagents can change, but you don't have to remine shit all the time due to that. When you see the recipe again, update the info on it. I don't see why you need to force recaching of all this information... just use whatever info is cached when you come across it.
TLDR version: You shouldn't ever be forcing caching unless a user opens a tooltip, which will do that caching for you on its own.
the caching is a side-effect of the method used to determine the spell list order.
the way the trade links work depends on knowing which spellids match up with which bits. the order is only ever known to be fixed for each client -- that is, a patch might change the order (if a recipe is added or removed). it can't be assumed to one thing until proven to be something else. the list has to be re-mined on every patch to avoid errors.
mining the spell list ends up caching all the item/reagent data so it becomes a freebie. now, it's possible (and perhaps likely) that people interested in trade skill info might not be the same people interested in trade link encoding/decoding, so that's certainly something to consider...
the way gyp currently functions is that i package the current spell list data (the only thing it mines) and only when that data has become obsolete does a scan take place. this means that when i'm in an updating mood, most people will never see the scan until they hit a new patch (which unfortunately is the very thing that makes the scan take a long time).
it's possible the same technique could be applied here -- up-to-date info included in the lib with a scanning fallback for when a patch hits and the mod hasn't been repackaged with that data.
as it relates to gyp, i've haven't updated in a while because really the scan is only a few minutes and probably only takes a moment longer than finding the latest version of gyp and extracting it...
i got the code working, still some questions to figure out...
should this be simply a data source or should it include common operations for the data -- link encode/decode/etc, item/recipe queries, etc? i'm leaning towards the latter so that the internal data doesn't need to be nailed down right away.
should the lib supply hardwired data and only scan when that data is obsoleted by a new patch? i think this would be the best way to handle things -- perhaps opening the door to add offline data (like vendor locations or skill levels) if that becomes something interesting.
If it can fully replace LibTradeSkill (while being a seperate addon), that would be fine. Then I for one don't have to load LibTradeSkill anymore (and it is currently unsupported) :)
There is only the problem that the users installing apps with this new dependency. If multiple apps depend on it and include it, this could cause problems if a new version was overwritten by an old version. :-/ Make sure the savedvariables file holds version information... and can warn the user if a downgrade has happened.
the way i'm going to set it up, the lib will come with its own data. that data will have a patch # so it will know if it's old (and therefore needs to scan). the whole point of the lib is to never have a case of downgrading because the data will always be up-to-date.
old versions of the lib will also deactivate via libStub if there's a new version also installed so everything should be okay.
my aim is to provide tradeLink management (encode/decode/transcode, bit logic, is-this-spellid-in-this-bitmap, etc) as well as traditional api stuff (reagent info for a particular spellid, crafted itemid/count, what spells use a reagent, what spell crafts which item, etc.)
i'm not sure about providing tool info. any thoughts?
also, when giving reagent data, does anybody think order is important? right now i'm just doing a simple table like this:
reagentTable[reagentID] = numNeeded
this means you can't get a # of different reagents for a particular spell, but then who ever needs that? any time you really want to do anything with reagents, you need to iterate over them. but this also means the order the reagents come is random. the alternative is to provide something like:
basically an extra table for each reagent entry, which seems costly and also unwarranted.
yeah, stick with the original thought.
as for downgrading issues, LibTradeLinks allows you to provide data for a patch that is not current, this allows for decoding a link from a previous patch so that you can re-encode it. If you keep this architecture, then downgrading shouldn't be a problem.
as for downgrading issues, LibTradeLinks allows you to provide data for a patch that is not current, this allows for decoding a link from a previous patch so that you can re-encode it. If you keep this architecture, then downgrading shouldn't be a problem.
yeah, gyp has the same functionality. i update all links when a new patch comes out so that the links are still functional.
i'll probably expose a method that allows updates based on a supplied spell id list like:
which means i'll also need to have a means of providing the mined data directly to any addon that wants it (so that it could be recorded in that addons saved variables for upgrading or whatever else).
I am not sure I follow it correctly: Will it be possible to get the item-links and enchant-links that use certain reagent?
yes. you will be able to look up which spells use a particular reagent. from there, you can determine which items are created from those spells.
i could probably also add a support function to tell you which items use particular reagents as a short cut.
I agree that the order in which the reagents are returned is of no use to most (if any).
LibPeriodicTable would store this in a string like such:
18344x3,1533x1,17463x6
Very compact yet still easy to access (LPT provides iterators and can return tables as well).
my plan right now is to simply store the table directly in lua code and not monkey with string encodings. i don't know how much memory is really saved by encoding and decoding strings.
Will it be possible to lookup the enchant links belonging to items links (and vica versa)?
Tables take a *lot* of time to initialize. You're better off encoding it into a string and only decoding the string when you actually need the data. At that time you can cache it in a table, which make subsequent queries faster.
Tables take a *lot* of time to initialize. You're better off encoding it into a string and only decoding the string when you actually need the data. At that time you can cache it in a table, which make subsequent queries faster.
hmm... i'll have to look into this. is warmup a good way to judge the different techniques?
Rollback Post to RevisionRollBack
To post a comment, please login or register a new account.
i'm seriously debating going ahead and making a library for this that does a bit more than simply providing info for trade link decoding. i believe it would be possible to turn this into a live-scanning tradeskillinfo type database.... i need to look into the memory cost of going down that road...
scanning in the item and reagent data (including backwards references for where items come from and where reagents are used) takes up about 1.5megs, but doesn't seem to take any additional processing time... it might be possible to avoid storing this data and instead use a version of the scanning code to access the data as needed, but that means doing a lot of unregistering/re-registering events on the fly which seems kinda sketchy...
-
View User Profile
-
View Posts
-
Send Message
Forum ModeratorYou could also try to see if you can get a list of all item ids not in cache yet in order to cache them without having to use the tradeskill api for triggering the caching so you would only block that api to a) check all items are cached b) generate your data
in regards to caching, i'm trying my best to avoid hardcoded tables. right now the only data supplied is the list of trade skill id's and a guess at how many skills they each have. but i'll poke around with that technique to see if it speeds up the initial post-patch caching. it would certainly free up the tradeskill api, but i think it's also possible that the server might not know about some rare reagents in that case -- at least, until somebody with a recipe that uses that reagent browses that trade skill. probably a rarity, but still possible. if i can get a speed increase out of it, i'll give some thought to how it might fold into the process...
Would it be possible to get the enchant: link as well as the item: link? I use LibPeriodicTables for that currently. I also use it to get the list craftable items when mouse-over a reagent item. Everything can be gotten from the tradeskill window somehow. It sounds like you have already gathered that information (the 1.5MB tables), right?
yeah, the recipes are stored by spellid so it's easy to get the itemid from the spellid (direct lookup). i stored the reverse lookup separately so it's also a direct lookup to go from the itemid back to the spellid (if it's a crafted item). there are some crafted items that have multiple recipe sources, so that is taken into consideration.
i also store a reagent usage table so that each reagent has a list of spellid's associated with it.
the info i can't get from the api is things like who sells it, what recipe item teaches the spell, what enchant scroll is created by a particular enchant, etc. that stuff still needs to be mined ahead of time.
honestly, i'm a little vague about how libpt works with the compressed data. when you access the data, doesn't it build the resulting table? doesn't that take additional memory?
i suppose going with a "flat" system where the data is recorded in lots of different tables for the different records might be more memory conscious than going with a nested table system, so i can look into that...
TLDR version: You shouldn't ever be forcing caching unless a user opens a tooltip, which will do that caching for you on its own.
the way the trade links work depends on knowing which spellids match up with which bits. the order is only ever known to be fixed for each client -- that is, a patch might change the order (if a recipe is added or removed). it can't be assumed to one thing until proven to be something else. the list has to be re-mined on every patch to avoid errors.
mining the spell list ends up caching all the item/reagent data so it becomes a freebie. now, it's possible (and perhaps likely) that people interested in trade skill info might not be the same people interested in trade link encoding/decoding, so that's certainly something to consider...
the way gyp currently functions is that i package the current spell list data (the only thing it mines) and only when that data has become obsolete does a scan take place. this means that when i'm in an updating mood, most people will never see the scan until they hit a new patch (which unfortunately is the very thing that makes the scan take a long time).
it's possible the same technique could be applied here -- up-to-date info included in the lib with a scanning fallback for when a patch hits and the mod hasn't been repackaged with that data.
as it relates to gyp, i've haven't updated in a while because really the scan is only a few minutes and probably only takes a moment longer than finding the latest version of gyp and extracting it...
-
View User Profile
-
View Posts
-
Send Message
Curse Premiumshould this be simply a data source or should it include common operations for the data -- link encode/decode/etc, item/recipe queries, etc? i'm leaning towards the latter so that the internal data doesn't need to be nailed down right away.
should the lib supply hardwired data and only scan when that data is obsoleted by a new patch? i think this would be the best way to handle things -- perhaps opening the door to add offline data (like vendor locations or skill levels) if that becomes something interesting.
old versions of the lib will also deactivate via libStub if there's a new version also installed so everything should be okay.
my aim is to provide tradeLink management (encode/decode/transcode, bit logic, is-this-spellid-in-this-bitmap, etc) as well as traditional api stuff (reagent info for a particular spellid, crafted itemid/count, what spells use a reagent, what spell crafts which item, etc.)
i'm not sure about providing tool info. any thoughts?
also, when giving reagent data, does anybody think order is important? right now i'm just doing a simple table like this:
reagentTable[reagentID] = numNeeded
this means you can't get a # of different reagents for a particular spell, but then who ever needs that? any time you really want to do anything with reagents, you need to iterate over them. but this also means the order the reagents come is random. the alternative is to provide something like:
reagentTable[reagentIndex] = { reagentID = id, numNeeded = count }
basically an extra table for each reagent entry, which seems costly and also unwarranted.
thoughts?
-
View User Profile
-
View Posts
-
Send Message
Curse Premiumyeah, stick with the original thought.
as for downgrading issues, LibTradeLinks allows you to provide data for a patch that is not current, this allows for decoding a link from a previous patch so that you can re-encode it. If you keep this architecture, then downgrading shouldn't be a problem.
I agree that the order in which the reagents are returned is of no use to most (if any).
LibPeriodicTable would store this in a string like such:
18344x3,1533x1,17463x6
Very compact yet still easy to access (LPT provides iterators and can return tables as well).
Will it be possible to lookup the enchant links belonging to items links (and vica versa)?
yeah, gyp has the same functionality. i update all links when a new patch comes out so that the links are still functional.
i'll probably expose a method that allows updates based on a supplied spell id list like:
newLink = lib:TradeLinkUpdate(oldLink, oldSpellList)
which means i'll also need to have a means of providing the mined data directly to any addon that wants it (so that it could be recorded in that addons saved variables for upgrading or whatever else).
yes. you will be able to look up which spells use a particular reagent. from there, you can determine which items are created from those spells.
i could probably also add a support function to tell you which items use particular reagents as a short cut.
my plan right now is to simply store the table directly in lua code and not monkey with string encodings. i don't know how much memory is really saved by encoding and decoding strings.
yes.
hmm... i'll have to look into this. is warmup a good way to judge the different techniques?