ItemPrice-1.0 has been desupported in favor of ItemPrice-1.1
ItemPrice-1.0 is a library containing vendor sell prices for items. Using the library is as simple as it can be; there is one method called GetPrice that takes either an itemId or an itemString as argument and returns the price as a number (amount of copper).
The prices are mined/scraped from www.wowhead.com. How accurate and complete they are is difficult to say for sure. I think Wowhead's data are usually considered fairly accurate and up-to-date. I did some comparisons with other addons that include price data (e.g. Informant) and there were very few discrepancies.
The library currently holds 15,589 items. Note that items without a sell price are not included in the library (e.g. most quest items). The size of the Lua file itself is about 200KB and my measurements have shown that the in-game memory required to hold the data is about 400KB. It would be possible to use compression techniques on the data similar to what PeriodicTable-3.0 does. However, I am not sure that this library would benefit greatly from using compression. The current representation allows for constant-time lookup and produces no additional garbage when used. Thus, the only concern is the "static" amount of memory required to hold the price table. But I am open for suggestions and opinions on this matter.
Compression is implemented and only uses about 100kb memory. Due to the compression technique used, there is practically no overhead whatsoever in terms of both memory and CPU usage when retrieving prices. It is pretty much as small and fast as it can possibly be.
/Bam
P.S. As this is my first contribution to Ace2, I would be thankful if somebody with more experience would check that I have used both AceLibrary and SVN in the recommended way. :)
Sounds great. At this moment I use sellfish to show sellvalues. Is there any sellvalue addon that uses this library yet?
I'd be surprised if there were as I released the library yesterday. :)
However, I have been using it myself in the addon I am currently working on. That addon is not yet on the SVN though, and still a good bit away from being releasable. Before I made the library, I hooked into other addons like Informant to get sell prices. But it always gave me a bad feeling that I would have to require users to install another addon that they may or may not like. That's why I figured that a simple, load-on-demand Ace library would be the answer. It's simple and takes care of only one thing: item prices. From there your addon can do whatever else is needed.
For example, your addon might consider checking if the user has a price-addon installed that you can interface with. Only if they don't, you would load ItemPrice-1.0. Or you could make it up to the user to decide which to use.
An Addon like ColaLight using ItemPrice-1.0 + TipHookerLib would be cool :)
Addons like that are definitely good candidates for using this library. Many people have several similar addons installed that all carry price data with them. It would be nice to reduce some of this duplicated data which can take up quite a lot of your available memory. (However, I am not under the illusion that one day all addons will use just one library for that.)
Quote from Gandharva »
What's the difference between [10004]=5738 and 5t9,3fi (like Tuller uses in SellFish)? Some sort of compression?
Yes, that is the compression technique that also PeriodicTable-3.0 uses. As I mentioned in my first post, I don't think the nature of the data in ItemPrice-1.0 will benefit as much from such a compression as for example PT3 does. But I am sure much debate can be had on that subject. :D
ItemPrice-1.0 now uses base-36 compression in a similar way as PeriodicTable-3.0 and SellFish do - though with some differenses. It decreases initial memory usage quite a lot (something like a factor 4-5). But obviously lookup time is no longer constant. There is still room for improvements such as caching and perhaps a faster lookup-method using binary searching (right now it's a na?ve linear search).
* Use your initial find to capture the price data as well, instead of using both a find and a match.
I did that at first. But I found that using a plain string.find was substantially faster in this case. I think it is because regular expressions can be quite time consuming on very long strings.
Quote from Nymbia »
* Maybe split the price data into a few different strings.. maybe like this:
1-5000
5001-10000
10001-15000
15001-20000
20001-25000
25001-30000
30001-35000
35001-40000
You'd be able to choose which string to search before starting, which ought to speed it up a decent amount.
Yes! :) I have considered that too. Right now it takes longer the higher the item-id you search for. And searching for non-existing items are the worst as they have to run through all of the string. That's not a very nice property for a data structure. So splitting them up would probably be a good trade-off.
I am also considering a simple caching. The problem I can see, though, is that the usage pattern could affect the advantages and disadvantage of caching a lot. If it is an addon that just needs to show a few items in tooltips, caching is probably not that important. But if you need a fairly large amount of items - fx for sorting items according to price - caching makes a lot of sense. Also, instead of na?vely caching any item accessed and store them in a weak table, it might make sense to use a cyclic buffer using an LRU-strategy.
I've implemented a new compression technique that has fast, constant-time lookup. Thus, no caching should be needed at all. Sounds like magic? Well, it isn't. It is a bit of a dirty trick that requires knowing the size of the data domain. I realized that for item prices it is unlikely that they ever get over 36^4-1 = 1679614 copper. Using that assumption we can now place prices in fixed size "fields" of length 4 and and use a simple string.sub to lookup prices. Basically it means we are treating the compressed string as an array. The size of the string is a bit larger since we need to use padding. But we save having the "keys" in the string and we save all delimiters. In the end the string has only increased by about 20% in size.
I don't think this technique is applicable to all problems needing compression. It relies on some assumptions about the data domain that may not hold true for other problems. In this case the assumptions could break if Blizzard introduces some items with extreme sell values or if they start using much higher item-id's. I consider both to be fairly unlikely. And if it's only a few items that break the assumptions, a special case can be made for those.
I've implemented a new compression technique that has fast, constant-time lookup.
Suggestion :
local byte = string.byte
local function get(id)
if id and id > 0 then
local index = id * 3
local a, b, c = byte(prices, index - 2, index)
return a * 65536 + b * 256 + c
end
end
Static data size is reduced by one quarter and string.byte() is much faster than tonumber(). I haven't tested but using bitlib might make it even faster.
That looks interesting, Jerry. If I understand it right, you suggest going to a base 256 using the entire range of an 8-bit character? But is that not problematic considering that the data need to be represented as a literal string in a file that is expected to be a text-file?
However, I do think we can safely go higher than base 36 if we are certain that string.byte returns consistent values on all platforms (or rather all encodings). I'm no expert on character encodings, but I think we can only be reasonably sure of that if we stay within the 7-bit ASCII range. Even then we likely cannot use the entire range from 0-127 safely.
Let me know if I have understood your idea correctly. It does sound promising even if we can only use, say, base 64.
What I used to convert your dataset to my suggested format :
local s = {}
for i = 1, 32883 do
local p = ip:GetPrice(i) or 0
local c, r = fmod(p, 256), modf(p / 256)
local b, a = fmod(r, 256), modf(r / 256)
s[#s+1] = char(a, b, c)
end
s = table.concat(s)
f = io.open("output.lua", "w")
f:write("prices = \"")
for i = 1, string.len(s) do
f:write(string.format("\\%d", string.byte(s, i)))
end
f:write("\"\n")
f:close()
The data is written on file using escape sequences, so that it stays encoded in ASCII.
I have implemented the compression that Jerry suggested. It works very well. Big thanks to Jerry. :)
One issue may be left, though. By representing the literal string using escape sequences, the size of the Lua file has nearly doubled. The main reason is that a large number of the bytes are 0 (zero). These now have to be written with two characters. I don't really know if the file size matters that much. It is about 234Kb now versus about 130Kb before.
I do think there is a way to solve this issue if we really want to squeeze the last bits. We could "roll" each byte such that 0 would end up being within the range of printable ASCII chars. I've never been too good with these bit operations, so..... Jerry? :D
My personnal feeling is that the size of the file does not matter much. Remember that it only affects the size of the uncompressed data file on disk. It will affect compressed size marginally (dictionnary based compression will nicely "eat up" the escaping) and parsing time is also extremely lightly affected.
That's it. I find the penalty of "rolling" the byte representation (a small runtime cost) not worth trying to reduce the file size.
That's it. I find the penalty of "rolling" the byte representation (a small runtime cost) not worth trying to reduce the file size.
Yea, That sounds right. I suppose one could make a simple, dirty trick by just swapping zero and some other value when generating data and swapping back when retrieving. But even that might be overkill.
I did that because it contains a fairly large amount of uncompressed data that I didn't think made sense users should have to download. Perhaps 'branches' is not the most logical place to put it. But I couldn't think of a better place.
Thanks for the link. I had not seen that before. It sounds like a great idea if addon authors can agree to use it.
However, ItemPrice-1.0 is a library and not an addon. Adding such a global function will not directly do anything good. Only when the library is actually used by an addon, would it have any effect. To my knowledge only my own addon ItemPriceTooltip uses the library. So the correct thing is probably that I add the global GetSellValue to that addon. But I'm not really sure what would be best in this case.
I'd say it'd be fine to implement GetSellValue globally in the library, as long as you make sure to upgrade it if/when the lib is upgraded.
I've decided to do that. But it gave me some headache figuring out a safe way to ensure that it can both be upgraded and that I don't hook more than once. The problem is that once you expose a global, "hookable" function, there is really no way back. You cannot simply redefine it later. The only way I think I can achieve safety, is to use another global variable as a "marker". Basically, my idea looks like this:
local get -- The existing function that looks up price by id.
-- Support for [url]http://www.wowwiki.com/API_GetSellValue[/url]
function Lib:GetSellValue(item, hook)
local id
-- Find valid id from argument item. Return if not found.
return get(id) or hook and hook(id)
end
local HookDefinedName = MAJOR_VERSION .. " has defined GetSellValue"
if not _G[HookDefinedName] then
local origGetSellValue = _G.GetSellValue
function _G.GetSellValue(item)
return Lib:GetSellValue(item, origGetSellValue)
end
_G[HookDefinedName] = MINOR_VERSION
end
AceLibrary:Register(Lib, MAJOR_VERSION, MINOR_VERSION)
Lib = AceLibrary(MAJOR_VERSION)
Thus, I introduce a new library function :GetSellValue that works as Tekkub suggests on API_GetSellValue with the main differense that it accepts the originally hooked GetSellValue (if any) as argument. A global variable (with a funny name) is tested and the global GetSellValue is only hooked if this variable is not already set. This new global GetSellValue redirects to Lib:GetSellValue and passes the hooked function. This should take advantage of AceLibrary's normal upgrade functionality.
Does it look ok? Can you think of a way to do it without introducing another global variable? I considered if I could use the activate/deactivate functions. But I could not see how.
A (non-Ace) addon wanting to take advantage of GetSellValue defined by ItemPrice-1.0 should only need to include ItemPrice-1.0 as OptionalDeps in the TOC file and the user must have ItemPrice-1.0 installed as stand-alone. If the user already has and addon using ItemPrice-1.0, it should work without any further actions.
Rollback Post to RevisionRollBack
To post a comment, please login or register a new account.
ItemPrice-1.0 is a library containing vendor sell prices for items. Using the library is as simple as it can be; there is one method called GetPrice that takes either an itemId or an itemString as argument and returns the price as a number (amount of copper).
The prices are mined/scraped from www.wowhead.com. How accurate and complete they are is difficult to say for sure. I think Wowhead's data are usually considered fairly accurate and up-to-date. I did some comparisons with other addons that include price data (e.g. Informant) and there were very few discrepancies.
The library currently holds 15,589 items. Note that items without a sell price are not included in the library (e.g. most quest items). The size of the Lua file itself is about 200KB and my measurements have shown that the in-game memory required to hold the data is about 400KB. It would be possible to use compression techniques on the data similar to what PeriodicTable-3.0 does. However, I am not sure that this library would benefit greatly from using compression. The current representation allows for constant-time lookup and produces no additional garbage when used. Thus, the only concern is the "static" amount of memory required to hold the price table. But I am open for suggestions and opinions on this matter.
Compression is implemented and only uses about 100kb memory. Due to the compression technique used, there is practically no overhead whatsoever in terms of both memory and CPU usage when retrieving prices. It is pretty much as small and fast as it can possibly be.
/Bam
P.S. As this is my first contribution to Ace2, I would be thankful if somebody with more experience would check that I have used both AceLibrary and SVN in the recommended way. :)
I'd be surprised if there were as I released the library yesterday. :)
However, I have been using it myself in the addon I am currently working on. That addon is not yet on the SVN though, and still a good bit away from being releasable. Before I made the library, I hooked into other addons like Informant to get sell prices. But it always gave me a bad feeling that I would have to require users to install another addon that they may or may not like. That's why I figured that a simple, load-on-demand Ace library would be the answer. It's simple and takes care of only one thing: item prices. From there your addon can do whatever else is needed.
For example, your addon might consider checking if the user has a price-addon installed that you can interface with. Only if they don't, you would load ItemPrice-1.0. Or you could make it up to the user to decide which to use.
What's the difference between [10004]=5738 and 5t9,3fi (like Tuller uses in SellFish)? Some sort of compression?
Addons like that are definitely good candidates for using this library. Many people have several similar addons installed that all carry price data with them. It would be nice to reduce some of this duplicated data which can take up quite a lot of your available memory. (However, I am not under the illusion that one day all addons will use just one library for that.)
Yes, that is the compression technique that also PeriodicTable-3.0 uses. As I mentioned in my first post, I don't think the nature of the data in ItemPrice-1.0 will benefit as much from such a compression as for example PT3 does. But I am sure much debate can be had on that subject. :D
* Use your initial find to capture the price data as well, instead of using both a find and a match.
* Maybe split the price data into a few different strings.. maybe like this:
- 1-5000
- 5001-10000
- 10001-15000
- 15001-20000
- 20001-25000
- 25001-30000
- 30001-35000
- 35001-40000
You'd be able to choose which string to search before starting, which ought to speed it up a decent amount.I did that at first. But I found that using a plain string.find was substantially faster in this case. I think it is because regular expressions can be quite time consuming on very long strings.
Yes! :) I have considered that too. Right now it takes longer the higher the item-id you search for. And searching for non-existing items are the worst as they have to run through all of the string. That's not a very nice property for a data structure. So splitting them up would probably be a good trade-off.
I am also considering a simple caching. The problem I can see, though, is that the usage pattern could affect the advantages and disadvantage of caching a lot. If it is an addon that just needs to show a few items in tooltips, caching is probably not that important. But if you need a fairly large amount of items - fx for sorting items according to price - caching makes a lot of sense. Also, instead of na?vely caching any item accessed and store them in a weak table, it might make sense to use a cyclic buffer using an LRU-strategy.
Thanks for the suggesions. :)
I don't think this technique is applicable to all problems needing compression. It relies on some assumptions about the data domain that may not hold true for other problems. In this case the assumptions could break if Blizzard introduces some items with extreme sell values or if they start using much higher item-id's. I consider both to be fairly unlikely. And if it's only a few items that break the assumptions, a special case can be made for those.
Suggestion :
Static data size is reduced by one quarter and string.byte() is much faster than tonumber(). I haven't tested but using bitlib might make it even faster.
However, I do think we can safely go higher than base 36 if we are certain that string.byte returns consistent values on all platforms (or rather all encodings). I'm no expert on character encodings, but I think we can only be reasonably sure of that if we stay within the 7-bit ASCII range. Even then we likely cannot use the entire range from 0-127 safely.
Let me know if I have understood your idea correctly. It does sound promising even if we can only use, say, base 64.
local s = {} for i = 1, 32883 do local p = ip:GetPrice(i) or 0 local c, r = fmod(p, 256), modf(p / 256) local b, a = fmod(r, 256), modf(r / 256) s[#s+1] = char(a, b, c) end s = table.concat(s) f = io.open("output.lua", "w") f:write("prices = \"") for i = 1, string.len(s) do f:write(string.format("\\%d", string.byte(s, i))) end f:write("\"\n") f:close()The data is written on file using escape sequences, so that it stays encoded in ASCII.
One issue may be left, though. By representing the literal string using escape sequences, the size of the Lua file has nearly doubled. The main reason is that a large number of the bytes are 0 (zero). These now have to be written with two characters. I don't really know if the file size matters that much. It is about 234Kb now versus about 130Kb before.
I do think there is a way to solve this issue if we really want to squeeze the last bits. We could "roll" each byte such that 0 would end up being within the range of printable ASCII chars. I've never been too good with these bit operations, so..... Jerry? :D
That's it. I find the penalty of "rolling" the byte representation (a small runtime cost) not worth trying to reduce the file size.
Yea, That sounds right. I suppose one could make a simple, dirty trick by just swapping zero and some other value when generating data and swapping back when retrieving. But even that might be overkill.
I did that because it contains a fairly large amount of uncompressed data that I didn't think made sense users should have to download. Perhaps 'branches' is not the most logical place to put it. But I couldn't think of a better place.
It needs the global function GetSellValue to work. http://www.wowwiki.com/API_GetSellValue
Thanks for the link. I had not seen that before. It sounds like a great idea if addon authors can agree to use it.
However, ItemPrice-1.0 is a library and not an addon. Adding such a global function will not directly do anything good. Only when the library is actually used by an addon, would it have any effect. To my knowledge only my own addon ItemPriceTooltip uses the library. So the correct thing is probably that I add the global GetSellValue to that addon. But I'm not really sure what would be best in this case.
I've decided to do that. But it gave me some headache figuring out a safe way to ensure that it can both be upgraded and that I don't hook more than once. The problem is that once you expose a global, "hookable" function, there is really no way back. You cannot simply redefine it later. The only way I think I can achieve safety, is to use another global variable as a "marker". Basically, my idea looks like this:
Thus, I introduce a new library function :GetSellValue that works as Tekkub suggests on API_GetSellValue with the main differense that it accepts the originally hooked GetSellValue (if any) as argument. A global variable (with a funny name) is tested and the global GetSellValue is only hooked if this variable is not already set. This new global GetSellValue redirects to Lib:GetSellValue and passes the hooked function. This should take advantage of AceLibrary's normal upgrade functionality.
Does it look ok? Can you think of a way to do it without introducing another global variable? I considered if I could use the activate/deactivate functions. But I could not see how.
A (non-Ace) addon wanting to take advantage of GetSellValue defined by ItemPrice-1.0 should only need to include ItemPrice-1.0 as OptionalDeps in the TOC file and the user must have ItemPrice-1.0 installed as stand-alone. If the user already has and addon using ItemPrice-1.0, it should work without any further actions.