Dateilager is a content-addressed, networked filesystem for keeping large, similar directories among a wide variety of hosts. It has a few key properties:
- file contents and versions are stored in by a central server, with readers and writers as clients
- files and trees of files are stored only once in a content-addressed store by the server for efficiency, but are still served securely to only clients with access
- readers and writers keep a copy of the filesystem synced locally for very fast local access (and trade off consistency or atomicity)
- incremental updates by readers are O(changes) and optimized to be very fast
- one Dateilager server supports storing many different independent filesystems with secured access to each (multi-tenancy)
Dateilager is used in production to power https://gadget.dev to sync the filesystems for all the apps on the Gadget platform. Dateilager shines at syncing the (often very large) node_modules
folder shared between the many node.js applications, as well as the comparatively small .js
files comprising the actual business logic of one particular app.
You can read more about DateiLager's design in the docs/
directory. A good place to start is the core concepts file.
- Go 1.19
- Postgresql
- Node v16
- mkcert (https://github.com/FiloSottile/mkcert)
- Protobuf Compiler (https://grpc.io/docs/protoc-installation/)
Create a Postgres database named dl
. The default Postgres host is 127.0.0.1
you can override it by
exporting DB_HOST
.
$ export DB_HOST=10.0.0.1
Ensure that you have a working Go development environment and that you are running at least Go 1.19.
You will also require npm
.
$ make install
We recommend using VSCode for development, and there's an example settings file at .vscode/settings.example.json
to get started with:
cp .vscode/settings.example.json .vscode/settings.json
This will build the server and client executables along with the *.proto
files.
$ make build
Ensure there is a Postgres database named dl_tests
. These tests will write to a real database instance
but all writes will be rolled back as every test runs within its own transaction.
$ make test
Set up a local development environment by resetting the local database, and building an example dataset
within the input/
directory.
$ make setup-local
Then launch the server process, by default it will run on port 5051.
$ make server
Our input directory contains 3 folders, all meant as different versions of the same project. There are also diff files listing which files have changed between them.
We can now load these into the database.
$ make client-update
And then use the client app to read all files within the latest version.
$ make client-get
You can also filter the results with a prefix search.
$ make client-get prefix=n1
Or filter for a specific version.
$ make client-get to_version=1
If you want to rebuild an entire project's directory locally, use the rebuild
command.
$ make client-rebuild to_version=3 dir=./rebuild
$ ls -lah ./rebuild
Ensure a server is running with make server
.
Import the DateiLagerGrpcClient
from the module in js/
and use it to query objects:
const client = new DateiLagerGrpcClient("localhost", 5051);
// Get a single object
const object = await client.getObject(1n, "a");
console.log("[getObject] path: " + object.path);
console.log("[getObject] content:\n" + object.content);
// List all objects
for await (const object of client.listObjects(1n, "")) {
console.log("[listObjects] path: " + object.path);
console.log("[listObjects] content:\n" + object.content);
}
Update objects and await the successful commit of a new version:
const stream = client.updateObjects(1n);
stream.send({
path: "a",
mode: 0o755,
content: "foo bar",
});
const version = await stream.complete();
console.log("[updateObject] version: " + version);
When you're ready to release a new version, perform the following steps:
- Update the version in
default.nix
- Update the version in
js/package.json
- Update the version in
js/package-lock.json
(runcd js && npm install
) - Commit the changes (e.g.
git commit -am "Bump version to 0.0.x"
)
New versions are released and hosted on Github. (https://github.com/gadget-inc/dateilager/releases)
Create a new tag and push it to GitHub, GoReleaser will handle building it.
$ git tag v0.0.x
$ git push origin v0.0.x
We also need to build the server docker image and push it to Gadget's container registry.
$ make upload-container-image version=0.0.x