• 0

    posted a message on LibCompress
    Quote from Allara »

    I like it. Alternatively (or in addition), Compress could take one or more additional arguments which specify encoding type (or no encoding if none specified). Decompress could decode transparently.


    That would mean the encoding type must be included in the header (additional header byte). This hardly seems necessary as the receiver probably always will know how the data was encoded, but can't always know how it was compressed.
    Posted in: Libraries
  • 0

    posted a message on LibCompress
    Quote from Allara »

    If AceComm-2.0 handles it's own encoding, I see no reason to also handle it here.

    Look at the AceComm-2.0 encoding. It is buggy and will not provide safe encoding (we tested it).

    I would think anyone using this library would also be using Ace to some degree.

    I do not think so, but that is not really the issue as AceComm-2.0 is buggy and large.

    I am a fan of AceComm-3.0, which is a lot more minimal (but doesn't support channels and such IIRC, so it wouldn't work for GuildAds). I personally see 2 uses for compression -- either direct addon-to-addon communication, or stored in saved variables. Since AceComm-3.0 doesn't encode \000 characters (I don't believe), it needs to happen somewhere else.

    AceComm-3.0 cannot be used as it only supports SendAddonMessage and doesn't encode \000. External encoding is needed.

    If you're really going to get into other types of encoding, then maybe this should be made into a separate library. If you're only going to encode for \000, I think it should just be built in, since that character is disallowed everywhere (or could be if it's not).

    \000 is not allowed for SendAddonMessage nor SendChatMessage. I do not know if \000 is allowed in storage.

    However, the argument for inclusion of all encoding methods in LibCompress is this: LibCompress will only be used in WoW. WoW requires some kind of encoding for all uses of strings, period. Therefore, in order to provide an end-to-end solution, LibCompress should handle encoding. Otherwise it will always be useless without some other library included.

    True. And this was my thought when I wrote that I probably would end up including the encoding in LibCompress directly. This leaves it up to the user to use the encoding after the compressed, e.g. like this:

    -- encode
    c = LibStub:GetLibrary("LibCompress")
    chat_data = c:Encode(c:Compress(raw_data))

    -- decode
    raw_data = c:Decode(c:Decompress(chat_data))

    The Encode and Decode functions may take names after which channel to encode for and will also also allow custom addition bytes to be encoded (used in e.g. a transmission/encapsulating protocol).

    Or:

    encode_func = c:GetChatEncodeFunction(optional_escape_byte_to_use, list_of_optional_reserved_byte_values)
    chat_data = encode_func(c:Compress(raw_data))

    Posted in: Libraries
  • 0

    posted a message on LibCompress
    Quote from OrionShock »

    Check out AceComm-2.0 's code wrt chat channel comms, it covers just about everything iirc.


    We did look at it to use in GuildAds, but AceComm-2.0 didn't cover all problems as far as I can remember. Also, AceComm-2.0 is huge and does much more than is required for LibCompress... (I know the space is only used once for all addons using AceComm-2.0).

    The major problem with AceComm-2.0 is that when it is encoding values above 128, it may risk using the bytes "s" and "S" which will lead to corruption if the player is drunk. This is fixed in GuildAds (based on AceComm-2.0).

    I'll probably include the GuildAds encoding in LibCompress (if I do not make this a separate library, LibEncode), but seeing as encoding and decoding functions aren't that big and probably will get used with LibCompress anyway, I'll probably add them to LibCompress.
    Posted in: Libraries
  • 0

    posted a message on LibCompress
    Quote from Yssaril »

    for chat messages the itemlinks have to be valid for addon messages it just has not contain illegal characters i think i wouldn't worry about chat messages since all the data should always be sent via the addon channels


    Well, for example a cross-guild tool, communication is forced to happen on chat channels... GuildAds is one such tool (but has its own encoding/decoding) and GuildEventManager as well.

    Ideally, Blizzard should provide addon-channels on the same principle as normal chat channels. Maybe even just enable addon messages to happen on the chat channels you are on (except perhaps the general/trade/etc channels). But until this is possible, normal chat is the only way.

    Anyway, addon channel allows every but \000. Chat channels is something like:

    -- Following byte values are not allowed/unwanted:
    -- \000, s, S, \010, \124, %
    -- s and S triggers the drunken "corruption"
    -- \010 = newline and \124 = | (pipe)
    -- Because SendChatMessage will error if an UTF8 multibyte character is incomplete,
    -- all character values above 127 have to be encoded to avoid this. This costs quite a bit of bandwidth (about 13-14%)
    -- Also, because drunken status is unknown for the received, strings used with SendChatMessage should be terminated with
    -- an identifying byte value, after which the server MAY add "...hic!" or as much as it can fit(!).
    -- Pass the identifying byte as a reserved character to this function to ensure the encoding doesn't contain that value.

    This can be done. Requires more than 1 escape character or deeper nesting. But doable.

    But I have no information about what characters are safe/unsafe when data is stored in the saved variables files.
    Posted in: Libraries
  • 0

    posted a message on LibCompress
    Does anyone actually know which bytes are allowed to be sent on the different channels? I know there are rules regarding item links that have to be valid, possibly even using SendAddonMessage, but I can't really find any information.
    Posted in: Libraries
  • 0

    posted a message on LibCompress
    Because some addons may not want that kind of encoding.

    For instance, I do not want any kind of encoding in GuildAds as it handles encoding by itself.

    Also, if several compression schemes are implemented, having them all encode (or support) would cost performance and/or space.

    Different encodings for different channels, e.g. SendChatMessage requires one kind of encoding, SendAddonMessage another, and storage a third kind. And I do not know what kind of byte values are allowed for all 3 types. But it is up to the addon to decide which encoding is wanted (if any).

    This could be handled by "LibEncode" that would have :EncodeChat, :EncodeAddon, and :EncodeStorage (and their 3 counterparts).

    Update: Also, the addon in question may want to reserve certain byte values for its own protocol and the :Encode functions have to allow that as well.
    Posted in: Libraries
  • 0

    posted a message on LibCompress
    You'll have to encode/decode manually. If you just want \000 to "disappear", then try something like this:
    compressed="\000\001\254\255"
    
    -- encode compressed_string (using \255 as escape character)
    local encode_translate = {
     [255] = "\255\001",
     [0] = "\255\002"
    }
    local function encode_helper(char)
     return encode_translate[char:byte()]
    end
    encoded = compressed:gsub("([\255%z])", encode_helper)
    print("ENCODED")
    for i=1,strlen(encoded) do
     print(string.byte(encoded:sub(i,i)))
    end
    
    -- decode
    local decode_translate = {
     ["\001"] = "\255",
     ["\002"] = "\000"
    }
    local function decode_helper(text)
     return decode_translate[text]
    end
    decoded = encoded:gsub("\255([\001\002])", decode_helper)
    print("DECODED")
    for i=1,strlen(decoded) do
     print(string.byte(decoded:sub(i,i)))
    end
    Posted in: Libraries
  • 0

    posted a message on LibCompress
    Committed my changes. Fixed a bug with :Decompress being unable to decompress uncompressed data. Slight speed increase in both :CompressHuffman and :DecompressHuffman. Significant less memory used. Some table reuse (cleared after 15 seconds).
    Posted in: Libraries
  • 0

    posted a message on LibCompress
    Ok, I think I get it now (after having decoded LibStub).

    local LibCompress,oldminor = LibStub:NewLibrary(MAJOR, MINOR)


    This actually returns LibCompress as either {} OR the old LibCompress table if it exists.

    Then I fully understand your suggestion:

    lib.frame = lib.frame or CreateFrame("LibCompress", nil, UIParent)


    "lib" is my "LibCompress". (and the "LibCompress" in CreateFrame is supposed to be "frame", my error).
    Posted in: Libraries
  • 0

    posted a message on LibCompress
    Hmm, I am not quite sure what the "lib" part is.

    I create the library like this:

    local LibCompress,oldminor = LibStub:NewLibrary(MAJOR, MINOR)


    This does not give me a globally accessible "LibCompress" table and I can't reference the old LibCompress without using LibStub:GetLibrary (which only gives an "instance" of the library as far as I understand). I am a bit confused...
    Posted in: Libraries
  • 0

    posted a message on LibCompress
    But does that make sense when the variables to cleanup aren't attached to the library?

    As long as lua is single threaded, there doesn't seem to be a reason to attach frame, Huffman_uncompressed, or Huffman_large_uncompressed (or other variables in that scope) to the library directly.

    Anyway, what I was thinking about was the situation when this revision is loaded at the same time as a newer revision, both of them running the CreateFrame and SetScript calls (first the oldest is loaded, then the newer, and possibly even newer versions). Shouldn't the newer version delete the old frame? If so, a way to find that old frame should be provided (global variable?).
    Posted in: Libraries
  • 0

    posted a message on LibCompress
    I was going to use this:

    local timeout = 30
    
    -- tables that may be erased
    local function cleanup()
    	Huffman_uncompressed = {}
    	Huffman_large_uncompressed = {}
    end
    
    local function onEvent(frame, elapsed)
    	timeout = timeout - elapsed
    	if timeout < 0 then
    		frame:Hide()
    		cleanup()
    	end
    end
    
    local function setCleanupTimeout()
    	local timeout = 30 -- empty tables after 30 seconds
    	if not frame:IsShown() then
    		frame:Show()
    	else
    end
    
    local frame = CreateFrame("LibCompress", nil, UIParent)
    frame:SetScript("OnEvent", onEvent)


    but what if there are different versions of the library? The last two lines would create more than 1 window, each calling their own cleanup function. How do I handle the library upgrade properly?
    Posted in: Libraries
  • 0

    posted a message on LibCompress
    Quote from Yssaril »

    id say just have a timer and if the table hasnt been used in say 10 sec just toss it to the GC table = {} :) just make sure nothing refenrences it anymore for large tables tossing it to the GC is alot more efficient than to delete each value (which dosnt get rid of all the overhead anyways) check this for more info on when to erase a table or to toss it to the GC

    Ok. I'll find a simple timer solution and just erase the table(s). The tables are only used inside a function and not across calls.

    most efficient is to keep it and overwrite its values remember memory shouldn't be an issue CPU is :)

    That is what I do now, but if 1 addon has compressed e.g. 1MB data, then that space is permanently lost unless I do something about it. :-P
    Posted in: Libraries
  • 0

    posted a message on LibCompress
    It seems I cannot use weak table for my purpose which means I have to implement a garbage collection manually. My table holds integer keys and string values. My code can already handle with the table not being empty (it just overwrites the values and remebers which are valid). So clearing the table is not necessary for the algorithm, but it does take up space when not in use. And it cannot be garbage collected due to its content.

    Would it be ok to use the OnUpdate frame handler to just delete keys from the table, 1 key every frame perhaps? The overhead wont be large, but it will be there for every frame. Or should I provide a :Cleanup function that can clear out the tables?
    Posted in: Libraries
  • 0

    posted a message on LibCompress
    I am working on making the DecompressHuffman use less table keys and even reuse table. So far it looks very good. There is even a performance gain in it (though small). Also, I found a bug in the general :Decompress function.

    But, I would like to know how I can flag a table so the garbage collection may erase all keys/values as long as the :Decompress isn't active. I believe this is possible?
    Posted in: Libraries
  • To post a comment, please or register a new account.