Jad Engine Blog

Jade Virtual File System (VFS)

The 13 September 2006 I posted in this blog:

"Nothing more I think, probably the next post will be about the new virtual file system of Jade."

And then the nexts posts were not about the VFS :p They were more reports about how the project is going. I'll do a quickly one and move on: 1.1 is really close, most features are there (collada is there, animation is there, new input is there, scripting is there, VFS is there,...). But polishing takes a lot of time :( But you can download the latest check-ins and play with all the new things ;)

And now about the VFS. Back to when I started thinking about this subject (severals months ago), I asked in Stratos if someone had already done it (just to avoid common mistakes and to get some guidelines). I got several very useful answers, and a link to a very good article in old Flipcode (here). I also got several good answers from Jade forums.

Most of my objectives for the VFS were pretty similar to those of the article:

- it should be able to handle archives (a big file with lots of files inside, like a .zip file)
- it should allow plugins to read or write in archives (encryption, compression,...)
- accessing normal files and archives should be transparent to the user
- it should allow several file paths

The idea was to write the new VFS and replace the old class of the engine that managed that (JPath). I had a more or less idea of what JPath was doing and I though switching from one thing to the other wouldn't be very hard (BIG mistake, more on this later) .

So, with that in mind, I started to code. I took the decision to leave the plugins for the end (not a great idea, but not too bad either) because I wanted to have first a bare minimum functionality. The main code was written pretty fast (it was quite easy), and were I spent most time was writing the tool to create the "archives" (a small winforms application). I was lucky that Fernando (delahermosa) was not too busy during that time and he helped me a lot in several parts of the tool.

The structure for the VFS at the start was like this:

- there was a JVFS class that represented the file system. The engine or the user would ask for files to this class. This class would hold internally a dictionary of FileSources (places where you can find files) and pass the request to them. One of the features from JPath that the engine used a lot was that you could ask for a file without knowing where it is exactly and then JPath would search for it.

- I wrote to FileSources implementations: HardDiskSources and StorageSources. HardDiskSources is normal IO, and StorageSources are the so called archives. A StorageSource holds virtual directories and files inside it.

- The engine used to read files the .NET methods called FromFile(string path,...), but now it would have to use the FromStream(Stream stream,...), because there´s not a real file when reading from an archive file. To allow this a new type of stream class is created: a VirtualStream. A virtual stream works over a file stream (the stream for the archive file) and controls that the reading is done the right way (you read the file you want to, and not more or less than you should). This way, the archive is only open once really and close at the end (so you avoid lot of costly IO operations).

And that was it more or less. It sounded great, but when I started merging the VFS with the engine, things weren't so great ;)

First, I decided to do the plugins support. And well, it was hell to write that. Plugins are small classes that modify how a file is written or read, and you can apply several of them to a file. This is their final interface:

Stream ApplyForRead(Stream input);
Stream ApplyForWrite(Stream input);

In theory it is pretty easy: most times filters will be streams wrapping streams. In practice it was quite complicated. Why? Things go more or less like this: we are writing our new archive (remember, lots of files together like a .zip), so we have one stream open and a binary writer writing on it. But now, when writing files with filters applied, we have to modify the stream the writer uses to write (filters modify the stream). So:

- we cache the position where we are in the stream
- we close our stream
- we reopen it and apply the filters to write
- we write the file
- we get the new position in the stream. Now, we can calculate the number of bytes we wrote :) We can't use the Length property of a file stream to know the number of bytes we wrote because we can be reading a compressed file or writing a compressed file, and compressed streams don't support the Length property (System.IO.Compression).
- we close the stream
- we reopen it again to continue doing work with it (file header, and other things)

Sure you loved it :p But then things get better. After finishing that part (I can tell you it took me a while to write those 40 lines of code), now I needed to give support for the filters in the visual tool. It involved writting a custom attribute so filters can tell the type that edits them in a visual environment (like VS visual designers) and a lot of strange errors with reflection and assembly loading.

And the last headache I got was when testing the encryption filters, because for some strange reason, when using AES, DES or TDES symmetric cyphers with PaddingMode.ISO10126 the output ended truncated. It took me a while to realize it was the algorithm padding, so well, I set it to PaddingMode.Zeros and at last, filters were working :)

And things got worse :p

Now I started replacing JPath for JVFS. I deprecated all the JPath code and started searching the warnings around to replace it. I had to change also some configuration stuff, but nothing too serious.

After looking how JPath was working I found the harsh reality: my JVFS interface wasn't prepared to manage it. JPath allowed the user to get all files from a directory, write a file (I didn't allow to write files! You can't save your game, you have to finish it at once :p), set special directories (like for example, Shaders directories where all shaders would be), ignore extensions,... Well, a lot of stuff I haven't thought. So I had to stop the merge and rework the VFS.

This rework involved mostly the FileSources interface. Before this change, the VFS had methods to ask for a file to a specific FileSource (all of them have an unique name) or to all of them at the same time. But now, the VFS class was reworked to be also a FileSource. So, I took out the methods to ask for a specific file or to a generic file. Now the method was something like this:

public abstract Stream GetFile(string path, string fileName, bool recurse, System.IO.FileAccess access);

So, if you do VFS.GetFile, the VFS implementation would search in all its FileSources, but if you do VFS["FileSourceName"].GetFile, it will be use whatever that specific FileSource implementation is. Everything is much clearer like that. FileSources also got some new methods (all files from one directory and minor things needed).

Next step was to allow for defined directories. The old way of putting directories for Jade was something like this:

X:\Whatever\InHouse
X:\Whatever\Base

InHouse is for internal engine files. Base is for your game files. But the engine assumes some directories always exist. For example, it loads all shaders in the Shaders defined directory, that is that the engine searches for InHouse\Shaders and Base\Shaders. So I had to allow a way to get all files from a specific path from all the FileSources. And the defined directories where born.

A defined directory is a pair of key (name, for example Shaders) and value (path, for example MyWork\Shaders). What the engine does when it finds a request to a defined directory is to ask every FileSource if the defined directory key exists, and if it does exist, it performs a normal search using the path associated to that key. So every user can put the Shaders wherever he likes the most as long as the key is Shaders.

All of this is defined in the configuration file, like this:

[VFS]
[FilesSource Type="HardDisk" Name="Base" Path="../../../../Base"]
[DefinedPath Name="Materials" Path="Materials"/]
[DefinedPath Name="Models" Path="Models"/]
[DefinedPath Name="Particles" Path="Particles "/]
[DefinedPath Name="Scenes" Path="Scenes"/]
[DefinedPath Name="Textures" Path="Textures"/]
[/FilesSource]
[FilesSource Type="HardDisk" Name="InHouse" Path="../../../../InHouse"]
[DefinedPath Name="Effects" Path="Effects"/]
[DefinedPath Name="PostProcess" Path="Effects/PostProcess"/]
[DefinedPath Name="Shaders" Path="Shaders"/]
[DefinedPath Name="Geometry" Path="Shaders/Geometry"/]
[DefinedPath Name="Internal" Path="Shaders/Internal"/]
[DefinedPath Name="Lighting" Path="Shaders/Lighting"/]
[DefinedPath Name="Materials" Path="Materials"/]
[DefinedPath Name="Textures" Path="Textures"/]
[DefinedPath Name="Gizmos" Path="Textures/Gizmos"/]
[/FilesSource]
[/VFS]

(Note: it's a normal XML File, I just replaced it with brackets [] to avoid some formatting problems here in the blog)

Quite easy: you define a FilesSource, with its type (HardDisk is normal files, Storage is an archive), its name and its path (relative to the configuration.xml file or absolute, as you wish). And then inside it you can add defined paths with their name and their real path.

With everything done, I went back to replace JPath for the VFS again: now things were fitting much better as there weren't strange things around that I hadn't thought. But, when I finished, the engine didn't work at all :( It would load and start, but it would not render correctly, probably related to a shaders loading problem, but I was unable to find it, so in the end I had to go back and undo all the changes in the engine (I backed up everything first). See changesets 14554, 14572 and 14574 comments to see what I mean (I broke the build compilation on the first check-in).

And so, there I'm at the moment, trying (for the 3rd time) to replace JPath for JVFS, but this time, instead of changing all at the same time, I'm going to have both systems living together and then I'll remove JPath little by little.

Let's hope this time it works ;)

Edit: I forgot to say (and I shouldn't have) that all of this work was possible thanks to the support, advice and comments from Jader, Reed, Quimbo and delahermosa. They helped me a lot to catch bugs, write some parts of the code, test, ideas,... Thks team! ;)