As some of you may already know, DataStore is a series of libraries which, unlike most existing libraries, handle their own SV files.
More information can be found here : http://wow.curseforge.com/addons/datastore/
For those of you who do not know what they do, their goal is to be data repositories for client addons, so that new developpers don't have to care about which events should be registered to get specific data, or even how to save them in the most efficient way.
A Brief history
I started coding Altoholic a little bit more than 2 years ago, and although I'm a programmer, I did not know anything about Lua, the WoW API, or even the Ace Framework, so I made things work before I made them better.
This meant having to manage a large database full of information, with the annoying consequence that one false move could hurt the DB and result in having to clean the add-on's SV file. That, and the lack of a clean interface to access my data. So after numerous iterations, I thought that adopting a layered/client-server approach could be a solution to my problem, and if it could help other authors at the same time, that would be the cherry on the cake.
How it works
DataStore is the main module, it contains character/guild keys to address the information stored by the other modules.
I have been using DataStore since July 2009, and the biggest advantage I found was that having a single interface to access data is pure bliss from a coder's perspective. There is less need for integrity checks since DS does them for you, and more than anything, data is shared for client addons. I currently use them in two projects, Altoholic & Odyssey, but I'm confident that more addons could benefit from this work.
Another benefit I see is that having a limited number of addons registering the same events avoid conflicts/deadlocks, since client addons basically get rid of this issue. Bag addons, for instance, no longer have to care about which event to register to get bag updates, and especially about when is the right time to do that.
On the other hand, the biggest inconvenient I have faced so far is packaging issues.
At this point, DataStore and its modules are all separate projects, so the svn repository cannot copy them into an addon's package since they would have to be at the same level as the client addon's folder.
Ex: Under Interface\Addons\
- Altoholic\
- DataStore\
- ...
This means that release packages have to be assembled manually. It's a pain, but I live with it until I find a better solution...
About the future
.. which is where this post comes into play. As I mentioned above, I have been using DS since last summer, and I intentionally waited until I felt certain that the approach was right before making it a bit more public.
What I need now is a peer review, and suggestions on what is right/wrong. The approach is right, but it's perfectible, and seeing how my backlog is growing, I would gladly welcome more hands on the project :)
Just to give an idea, here are some targets I have for the libs:
Add support for data export (to xml, or whichever format).
Review packaging options. Mikk suggested me to have turn the libs into simple addons that only contain the SV file, and to put all the code into real libs, that's a valid alternative.
On the other hand, the biggest inconvenient I have faced so far is packaging issues.
At this point, DataStore and its modules are all separate projects, so the svn repository cannot copy them into an addon's package since they would have to be at the same level as the client addon's folder.
I think you could have the packager to layout the folders as need using .pkgmeta:
externals:
ds/DataStore: svn://svn.curseforge.net/wow/datastore/mainline/trunk
ds/DataStore_Achievements: svn://svn.curseforge.net/wow/datastore_achievements/mainline/trunk
# more externals there
move-folders:
ds/DataStore: DataStore
ds/DataStore_Achievements: DataStore_Achievements
# more move-folders there
There are two issues though :
- the "nolib" package may cause Datastore not to be included (though you could disable this kind of packages),
- the Curse Client may have issue identifying which addons installed DataStore if several of them are installted at the same time (the DataStore version of each addon erases the other one).
IMO the whole datastore modules could have been made a single project.
Another solution would be to defined "required dependencies" but I'm not sure how the Curse Client really handle them ; plus it is not handled at all when downloading directly from the website.
IMO the whole datastore modules could have been made a single project.
If you're talking about "a single project" in the same manner as the Ace3 modules are "a single project", I agree. Otherwise, it would be a huge monolithic library that would include a bunch of unnecessary fluff - someone wanting to handle auctions wouldn't need the stuff for quests, for example.
I agree with Torhal, we had that discussion on IRC back when I finished coding the first versions, and it made more sense to keep them separated as they are now. The only drawback is the one I mentioned: the packaging issue.
I do not know if the method you suggested works, but if it does, it would be the answer to my problem.
The -nolib issue is not too much of a problem for me, since I rely on the presence of libraries into DataStore (ex: ace libs are not in Altoholic, they're in DS), so I could easily disable them.
The Curse Client issue bothers me a bit more, since I'd really like to make sure that an update to either a DS module or to a client addon doesn't lead to unexpected behavior. Some users have told me that some alpha pushes of Altoholic did cause the CC to delete DS modules. If I could get rid of that, the biggest problem would be out of the way :)
Edit: I won't have time to test that this evening, but since Odyssey is still alpha, I'll make packaging tests tomorrow morning. It won't disturb anyone.
I agree with Torhal, we had that discussion on IRC back when I finished coding the first versions, and it made more sense to keep them separated as they are now. The only drawback is the one I mentioned: the packaging issue.
What Adirelle was suggesting is the Ace3 approach - the suite is broken up into several modules (AceHook-3.0, AceEvent-3.0, etc) but they're all contained in the same project. AddOns which use them pull the modules they need directly from the project's trunk via the .pkgmeta file.
Well, I'm not against the idea of putting all modules under the same project. If Adirelle's suggestion works as expected, then there's no reason to keep projects separated.
One project including the top level directory of another project in its own top level is not allowed. It breaks both the Curse Client and the WoWI Minion.
This is one of the main reasons libraries like this are a bad idea.
This is one of the main reasons libraries like this are a bad idea.
Actually the library itself isn't a bad idea. What is hard to achieve is handling the SavedVariables :
- using it a as standard library would require to duplicate the SV for each addon using it, which would negate the purpose of a centralized datastore.
- requiring the including addons to have the same SV for the library won't work either, because if several addons use it, they will erase each other variables, with funny things happening if any of them is disabled.
I'm wondering how the Curse Client honors the "required dependencies" of the .pkgmeta. It could be the solution though it doesn't help with manual installation.
With all due respect for the efforts that have been put into the packagers, I do not agree with the fact that such libs are a bad idea. There is no technical reason on the game's side against the approach, and based on my own experience with the libs (and real life exp with layered dev), it's rather the opposite.
To me, it's the game that should drive the design of an addon/library, not the packaging options/abilities/design choices of external tools.
While I agree with the fact that I must be the only one using this approach, is there a chance to see this supported in the future ? Maybe under the form of a "copy-folders:" tag ?
i'm a bit clueless about packaging stuff, but i'll chime in...
seems to me that these are not libraries and are instead addons that would be considered required dependencies.
if datastore gets embedded by the packager, since it's a top-level addon, how do you make sure that it's always the latest version? does the packager need to re-run on every addon that uses includes datastore each time datastore is updated by another addon?
seems like the curse client and wow minion could (and should?) read dependencies and install dependent addons automatically (or at least identify them as being needed).
They're indeed "addons" and not libraries (due to the SV file issue).
There's a .pgkmeta tag where the list of required dependencies are mentioned, so it would be technically possible to let the updaters handle that automatically.
As for the updates, well, I would say that client addons could be repackaged when DS is updated, but then it really depends on how popular it becomes. If it means packaging 10 addons, it's ok, if it's 500, the server load has to be taken into account.
We had automagic repacking with the old packager, and it was decided to not implement in the new one. There have been many issues where a library update broke addons and the author didnt even know because it was repacked silently. Now the author is aware which version is in his addon. Stuff may still break due to onthefly lib upgrading, however the standalone zip is guaranteed to be what the author designed it to be.
On your packaging issue: the biggest issue with it have the update clients. They use file signatures to recognize the addons you installed. If such a "library" is part of multiple addons, it would need some extra special logic or metadata to handle that. If you install a second addon using that, which happens to be sbhipped with an older version of it, do you blindly overwrite?
There is a ton of problems with this, thats why most auhtors try to stay away from that design. The best solution for updaters is the use of real dependencys, so that the client can manage the library as a seperate addon. However, that will require people that download it from the website to manually download the library, as its nop longer packaged. However they would always have a recent version and not something that came with the zip.
Another design would be to force addon authors using your lib to provide the SV for it. But that opens another can of worms.
Implementing in the future. Well. The packager is opensource and if we can come up with a proper way to handle it, there is no reason not to implement it and send a patch to kaelten. However, that would need additional client support, which is really the hard part. And you still screw over website-downloaders, as they would still blindly overwrite no matter what version it is.
As a final thought, it doesnt matter if you package manually, or let the packager do it. The problems remain the same ;)
At worst, automatic repacking is not the most important for me. I'd be perfectly fine to tell authors to tag their addons every once in a while to make sure they have an updated version of the lib.
Now, on the client side, managing versions could be solved using Mikk's suggestion: each client addons hard-embeds the required modules which would only contain a .toc for the SV file, while all the code could go into a real lib. That way, they could overwrite one another and it wouldn't matter. That's an approach that I intend to test in the future. If that works as I expect, it would solve the problems for updaters as well.
I don't see any easy solution, but since versioning is bound to become an issue in the long run, it's very likely that I'll switch to that design at some point. Thanks for the input btw :)
Well the problem of the client identifying the wrong addon because multiple addons include the same directory with the same content still exists if multiple addons are just including a directory with nothing but a .toc listing a saved var. It is a serious issue that has been around since updater clients have existed.
To make it perfectly clear what I'm talking about I'll give a real example of a past situation. There was some guide addon similar to TourGuide or some of the paid leveling guides (but it was rather bad so nobody really remembers the name of it) that was using TomTom for the direction arrow. The addon included the top level TomTom directory in its zip file and sundenly a lot of users were reporting that some crappy guide addon got installed somehow by the client but they never installed it and didnt want it. Whenever they deleted the directory the client would just reinstall it. It was discovered that the client was seeing the TomTom directory and thought it was part of the guide addon so it should update that guide addon instead of seeing it as the TomTom addon. The solution was to purge all records of the zip files from the database and physically delete the files from the server. That was a very drastic solution when you consider that Curse never deletes anything, when you mark a file as deleted it still exists but its just hidden from normal users, only the project owner and the admins can see it.
What's wrong with having DataStore as a require dependency & let the packager / clients work accordingly? The only real problem is the standalone ZIP doesn't include the required dependency, a simple note on the description page should handle that nicely.
Using the Required Dependency IMO is the better of solutions here than trying to redesign the curseforge system.
What's wrong with having DataStore as a require dependency & let the packager / clients work accordingly? The only real problem is the standalone ZIP doesn't include the required dependency, a simple note on the description page should handle that nicely.
Using the Required Dependency IMO is the better of solutions here than trying to redesign the curseforge system.
i agree. the manual downloads could also be helped along with direct d/l links for dependencies near to the d/l link for the addon itself. users need not be left in the dark here.
What's wrong with having DataStore as a require dependency & let the packager / clients work accordingly? The only real problem is the standalone ZIP doesn't include the required dependency, a simple note on the description page should handle that nicely.
Using the Required Dependency IMO is the better of solutions here than trying to redesign the curseforge system.
You covered it all. Either you break the client or standalone users have to download two addons to make it work. Personally I think it might be a little easier on the users if the stub addon was marked in .pkgmeta as a required dependency but only as an optional dependency in the toc so that the addon would still load but could popup a clean message box informing the user that the stub addon is missing and where they can download it. Also marking the stub as LoD would let the main addons enable it and load it even if it was disabled instead of telling the user to download it when they already have it.
Rollback Post to RevisionRollBack
To post a comment, please login or register a new account.
As some of you may already know, DataStore is a series of libraries which, unlike most existing libraries, handle their own SV files.
More information can be found here : http://wow.curseforge.com/addons/datastore/
For those of you who do not know what they do, their goal is to be data repositories for client addons, so that new developpers don't have to care about which events should be registered to get specific data, or even how to save them in the most efficient way.
A Brief history
I started coding Altoholic a little bit more than 2 years ago, and although I'm a programmer, I did not know anything about Lua, the WoW API, or even the Ace Framework, so I made things work before I made them better.
This meant having to manage a large database full of information, with the annoying consequence that one false move could hurt the DB and result in having to clean the add-on's SV file. That, and the lack of a clean interface to access my data. So after numerous iterations, I thought that adopting a layered/client-server approach could be a solution to my problem, and if it could help other authors at the same time, that would be the cherry on the cake.
How it works
DataStore is the main module, it contains character/guild keys to address the information stored by the other modules.
Samples can be found here : http://wow.curseforge.com/addons/datastore/pages/api/
Pros & cons
I have been using DataStore since July 2009, and the biggest advantage I found was that having a single interface to access data is pure bliss from a coder's perspective. There is less need for integrity checks since DS does them for you, and more than anything, data is shared for client addons. I currently use them in two projects, Altoholic & Odyssey, but I'm confident that more addons could benefit from this work.
Another benefit I see is that having a limited number of addons registering the same events avoid conflicts/deadlocks, since client addons basically get rid of this issue. Bag addons, for instance, no longer have to care about which event to register to get bag updates, and especially about when is the right time to do that.
On the other hand, the biggest inconvenient I have faced so far is packaging issues.
At this point, DataStore and its modules are all separate projects, so the svn repository cannot copy them into an addon's package since they would have to be at the same level as the client addon's folder.
Ex: Under Interface\Addons\
- Altoholic\
- DataStore\
- ...
This means that release packages have to be assembled manually. It's a pain, but I live with it until I find a better solution...
About the future
.. which is where this post comes into play. As I mentioned above, I have been using DS since last summer, and I intentionally waited until I felt certain that the approach was right before making it a bit more public.
What I need now is a peer review, and suggestions on what is right/wrong. The approach is right, but it's perfectible, and seeing how my backlog is growing, I would gladly welcome more hands on the project :)
Just to give an idea, here are some targets I have for the libs:
- Add support for data export (to xml, or whichever format).
- Review packaging options. Mikk suggested me to have turn the libs into simple addons that only contain the SV file, and to put all the code into real libs, that's a valid alternative.
- Add support for LDB feeds.
- Add more methods/options to each modules.
- etc...
Feedback would be greatly appreciated :)-
View User Profile
-
View Posts
-
Send Message
Curse PremiumI think you could have the packager to layout the folders as need using .pkgmeta:
There are two issues though :
- the "nolib" package may cause Datastore not to be included (though you could disable this kind of packages),
- the Curse Client may have issue identifying which addons installed DataStore if several of them are installted at the same time (the DataStore version of each addon erases the other one).
IMO the whole datastore modules could have been made a single project.
Another solution would be to defined "required dependencies" but I'm not sure how the Curse Client really handle them ; plus it is not handled at all when downloading directly from the website.
-
View User Profile
-
View Posts
-
Send Message
Project AuthorIf you're talking about "a single project" in the same manner as the Ace3 modules are "a single project", I agree. Otherwise, it would be a huge monolithic library that would include a bunch of unnecessary fluff - someone wanting to handle auctions wouldn't need the stuff for quests, for example.
-
View User Profile
-
View Posts
-
Send Message
Curse PremiumThis is what I had in mind.
I do not know if the method you suggested works, but if it does, it would be the answer to my problem.
The -nolib issue is not too much of a problem for me, since I rely on the presence of libraries into DataStore (ex: ace libs are not in Altoholic, they're in DS), so I could easily disable them.
The Curse Client issue bothers me a bit more, since I'd really like to make sure that an update to either a DS module or to a client addon doesn't lead to unexpected behavior. Some users have told me that some alpha pushes of Altoholic did cause the CC to delete DS modules. If I could get rid of that, the biggest problem would be out of the way :)
Edit: I won't have time to test that this evening, but since Odyssey is still alpha, I'll make packaging tests tomorrow morning. It won't disturb anyone.
-
View User Profile
-
View Posts
-
Send Message
Project AuthorWhat Adirelle was suggesting is the Ace3 approach - the suite is broken up into several modules (AceHook-3.0, AceEvent-3.0, etc) but they're all contained in the same project. AddOns which use them pull the modules they need directly from the project's trunk via the .pkgmeta file.
-
View User Profile
-
View Posts
-
Send Message
Project Authorexternals: Data/Core: url: svn://svn.curseforge.net/wow/datastore/mainline/trunk tag: latest Data/Currencies: url: svn://svn.curseforge.net/wow/datastore/mainline/trunk/DataStore_Currencies tag: latest Data/Inventory: url: svn://svn.curseforge.net/wow/datastore/mainline/trunk/DataStore_Inventory tag: latest move-folders: Altoholic/Data/Core: DataStore Altoholic/Data/Currencies: DataStore_Currencies Altoholic/Data/Inventory: DataStore_InventoryThis could also allow you to clean up your ToC, depending on how you set things up, so all you would need is
This is one of the main reasons libraries like this are a bad idea.
-
View User Profile
-
View Posts
-
Send Message
Curse PremiumActually the library itself isn't a bad idea. What is hard to achieve is handling the SavedVariables :
- using it a as standard library would require to duplicate the SV for each addon using it, which would negate the purpose of a centralized datastore.
- requiring the including addons to have the same SV for the library won't work either, because if several addons use it, they will erase each other variables, with funny things happening if any of them is disabled.
I'm wondering how the Curse Client honors the "required dependencies" of the .pkgmeta. It could be the solution though it doesn't help with manual installation.
To me, it's the game that should drive the design of an addon/library, not the packaging options/abilities/design choices of external tools.
While I agree with the fact that I must be the only one using this approach, is there a chance to see this supported in the future ? Maybe under the form of a "copy-folders:" tag ?
seems to me that these are not libraries and are instead addons that would be considered required dependencies.
if datastore gets embedded by the packager, since it's a top-level addon, how do you make sure that it's always the latest version? does the packager need to re-run on every addon that uses includes datastore each time datastore is updated by another addon?
seems like the curse client and wow minion could (and should?) read dependencies and install dependent addons automatically (or at least identify them as being needed).
There's a .pgkmeta tag where the list of required dependencies are mentioned, so it would be technically possible to let the updaters handle that automatically.
As for the updates, well, I would say that client addons could be repackaged when DS is updated, but then it really depends on how popular it becomes. If it means packaging 10 addons, it's ok, if it's 500, the server load has to be taken into account.
-
View User Profile
-
View Posts
-
Send Message
Forum ModeratorOn your packaging issue: the biggest issue with it have the update clients. They use file signatures to recognize the addons you installed. If such a "library" is part of multiple addons, it would need some extra special logic or metadata to handle that. If you install a second addon using that, which happens to be sbhipped with an older version of it, do you blindly overwrite?
There is a ton of problems with this, thats why most auhtors try to stay away from that design. The best solution for updaters is the use of real dependencys, so that the client can manage the library as a seperate addon. However, that will require people that download it from the website to manually download the library, as its nop longer packaged. However they would always have a recent version and not something that came with the zip.
Another design would be to force addon authors using your lib to provide the SV for it. But that opens another can of worms.
Implementing in the future. Well. The packager is opensource and if we can come up with a proper way to handle it, there is no reason not to implement it and send a patch to kaelten. However, that would need additional client support, which is really the hard part. And you still screw over website-downloaders, as they would still blindly overwrite no matter what version it is.
As a final thought, it doesnt matter if you package manually, or let the packager do it. The problems remain the same ;)
Now, on the client side, managing versions could be solved using Mikk's suggestion: each client addons hard-embeds the required modules which would only contain a .toc for the SV file, while all the code could go into a real lib. That way, they could overwrite one another and it wouldn't matter. That's an approach that I intend to test in the future. If that works as I expect, it would solve the problems for updaters as well.
I don't see any easy solution, but since versioning is bound to become an issue in the long run, it's very likely that I'll switch to that design at some point. Thanks for the input btw :)
-
View User Profile
-
View Posts
-
Send Message
Forum ModeratorTo make it perfectly clear what I'm talking about I'll give a real example of a past situation. There was some guide addon similar to TourGuide or some of the paid leveling guides (but it was rather bad so nobody really remembers the name of it) that was using TomTom for the direction arrow. The addon included the top level TomTom directory in its zip file and sundenly a lot of users were reporting that some crappy guide addon got installed somehow by the client but they never installed it and didnt want it. Whenever they deleted the directory the client would just reinstall it. It was discovered that the client was seeing the TomTom directory and thought it was part of the guide addon so it should update that guide addon instead of seeing it as the TomTom addon. The solution was to purge all records of the zip files from the database and physically delete the files from the server. That was a very drastic solution when you consider that Curse never deletes anything, when you mark a file as deleted it still exists but its just hidden from normal users, only the project owner and the admins can see it.
-
View User Profile
-
View Posts
-
Send Message
Curse PremiumUsing the Required Dependency IMO is the better of solutions here than trying to redesign the curseforge system.
i agree. the manual downloads could also be helped along with direct d/l links for dependencies near to the d/l link for the addon itself. users need not be left in the dark here.
You covered it all. Either you break the client or standalone users have to download two addons to make it work. Personally I think it might be a little easier on the users if the stub addon was marked in .pkgmeta as a required dependency but only as an optional dependency in the toc so that the addon would still load but could popup a clean message box informing the user that the stub addon is missing and where they can download it. Also marking the stub as LoD would let the main addons enable it and load it even if it was disabled instead of telling the user to download it when they already have it.