It’s been a long time since I first started working on TKMap’s repositories. The repository backbone manages various repositories which allow to use certain locations as resource source. This would potentially allow people to offer online repositories which game designers can use for their games while they’re still able to use their own local repositories.
TKMap progressed slowly in the recent months. I started working on repositories sometime around November 2014. Back then I tried using the .Net’s FileSystemWatcher which turned out to be useless. Lots of file interactions got lost and it was and is still important to me that users can interact with the repository in a natural way. This means that you can add, modify, rename, move and delete files in the repository folder. With no detour via TKMap. Unlike Unity which saves metadata in files which lie next to the actual resource file TKMap uses a database driven approach. This makes it a lot more difficult to handle file interactions, because everything is asynchronous.
In order to get a reliable system, I gave CallbackFilter a shot. And it turned out to be perfect for this situation. Though I actually expected a little bit more of it, it’s easy to use and with some effort it detects each single file interaction. It troubled me a lot since Windows handles files different than you would expect it. So it was required to implement some sort of state management which tracked various interactions with files and later interprets it to certain logical actions. For example “Open, MoveOrRename, Close” would end up in either a move, rename or even a delete. Why a delete? Well, “deleting” files in the explorer does not delete them but move them to the recycle bin. Thus a move can also be a deletion. Vice versa a move from the bin must be interpreted as creation. This makes it really hard to keep track of all the changes.
Another huge challenge was the interaction of applications like Photoshop or Visual Studio. They don’t open the file, write and close to trigger a change. They actually copy (and thus create) the original file. When saved they delete the original and rename the copy. Thus the repository must interpret these actions as change rather than deletion and rename. In order to get this working it was necessary to make deletion asynchronous. This means that the resource of the original (triggered by for example Photoshop) is just flagged as “to delete”. A task then waits for ~3-4 seconds until it actually deletes the resource. But only if the flag is still set to “delete”. Otherwise it will be cancelled and a rename is being treated as change. Pretty complex, but after hours of experimenting and analyzing I can happily say that it is finally working (for any case). It also synchronizes any changes with the database (one-way, luckily as two-way would be another difficulty).
It’s incredible how difficult it is to synchronize the file system with an in-memory tree (and that one with a database). On the contrary it’s been really interesting learning and experimenting with Windows file system. Nonetheless I’m very happy that I can finally move on to something new. The next task will be implementing the actual resource loading functionality into the repository and the backbone. After that I’ll do some testing. If everything works fine, I’ll integrate the asset browser and soon to follow the first map editing tools. I’m very excited about that. I think development will progress a lot faster from now on. I doubt that these things will be as challenging and time-consuming as the repository.
Last but not least I hope I can finally show off some actual stuff soon. I expect first results in 3-6 months (still a lot of time but studies and work take up most of my time).