[Contribution] qubes-incremental-backup-poc OR Wyng backup #858
Comments
marmarek
added
T: bug
This comment was marked as outdated.
This comment was marked as outdated.
marmarek
added
C: core
and removed
C: artwork
marmarek
added
T: enhancement
|
Comment by joanna on 14 May 2014 10:42 UTC https://groups.google.com/d/msg/qubes-devel/Gcrb7KQVcMk/CK-saQU_1HYJ |
andrewdavidwong
added
the
help wanted
It looks like Duplicity supports making an incremental backup even when part of a file was changed (include diff of file, not full changed files). So indeed it may be good idea to somehow use it. But mounting VM filesystem in dom0 is a big NO-GO. On the other hand, it may be good idea to simply run duplicity in the VM, and collect its output. Take a look at the linked discussion for an idea how to handle powered off VMs (in short: launch minimal "stubdomain" like system with access to its disk for the backup purpose). Of course all this requires evaluation whether duplicity correctly handle encryption / validation. To not make things worse than they currently are... |
|
On 06/12/2016 07:04 PM, Marek Marczykowski-Górecki wrote:
No need to mount any VM filesystem in dom0. That is also not how The way it would work with Duplicity, is a backend would need to be
See this: https://bazaar.launchpad.net/~duplicity-team/duplicity/0.7-series/files/head:/duplicity/backends/ Later today, or perhaps tomorrow, I will use my Qubes bombshell-client |
|
Better still: Remarkably much like my Qubes Ansible plugin. |
|
Can you explain more how exactly that would work?
Generally dom0 shouldn't be exposed to VM filesystem in any way (regardless of the form: mounting directly, accessing via sftp-like service etc). If incremental data needs to be computed at VM-file level, it should be done in separate VM and dom0 should treat the result as opaque blob. Also during restore. |
|
On 06/12/2016 08:15 PM, Marek Marczykowski-Górecki wrote:
Duplicity + hypotheticalplugin (from now on, HP) would never retrieve
Incremental data is computed in dom0 using both the rdifflib database
That cannot work because that involves copying the IMG files into the Duplicity's backends are designed to make the storage opaque to |
Not necessary - it can be attached with qvm-block-like mechanism. Anyway, IIUC you want to backup |
|
Well, look at what I have got here: I have a (shit) complete Qubes VM backend for duplicity which you can add to your dom0's duplicity backends directory, and then run something like this: duplicity /var/lib/qubes qubesvm://backupvm/Backupsmountpoint/mydom0/qubesvms Very early working prototype and building block. Enjoy! |
|
I have some experience with Duplicity:
I would prefer using Duplicity per-VM, so one could exclude caches etc. from the backup. My idea is something like the following process invoked from dom0:
As you can see, this would require some data stored in dom0 to be also backed up, but this could be handled by some very similar mechanism. (It would probably run directly in dom0 rather than in dom0.) Some implementation and security notes on those points:
|
# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
#
# This file is NOT part of duplicity. It is an extension to Duplicity
# that allows the Qubes OS dom0 to back up to a Qubes OS VM.
#
# Duplicity is free software, and so is this file. This file is
# under the same license as the one distributed by Duplicity.
import os
import pipes
import subprocess
import duplicity.backend
from duplicity.errors import *
BLOCKSIZE = 1048576 # for doing file transfers by chunk
MAX_LIST_SIZE = 10485760 # limited to 10 MB directory listings to avoid problems
class QubesVMBackend(duplicity.backend.Backend):
"""This backend accesses files stored on a Qubes OS VM. It is intended to
work as a backed within a Qubes OS dom0 (TCB) for the purposes of using
Duplicity to back up to a VM. No special tools are required other than
this backend file itself installed to your Duplicity backends directory.
Missing directories on the remote (VM) side will NOT be created. It is
an error to try and back up to a VM when the target directory does
not already exist.
module URL: qubesvm://vmname/path/to/backup/directory
"""
def __init__(self, parsed_url):
duplicity.backend._ensure_urlparser_initialized()
duplicity.backend.urlparser.uses_netloc.append("qubesvm")
duplicity.backend.urlparser.clear_cache()
properly_parsed_url = duplicity.backend.ParsedUrl(parsed_url.geturl())
duplicity.backend.Backend.__init__(self, properly_parsed_url)
if properly_parsed_url.path:
self.remote_dir = properly_parsed_url.path
else:
self.remote_dir = '.'
self.hostname = properly_parsed_url.hostname
def _validate_remote_filename(self, op, remote_filename):
if os.path.sep in remote_filename or "\0" in remote_filename:
raise BackendException(
("Qubes VM %s failed: path separators "
"or nulls in destination file name %s") % (
op, remote_filename))
def _dd(self, iff=None, off=None):
cmd = ["dd", "status=none", "bs=%s" % BLOCKSIZE]
if iff:
cmd.append("if=%s" % iff)
if off:
cmd.append("of=%s" % off)
return cmd
def _execute_qvmrun(self, cmd, stdin, stdout):
subcmd = " ".join(pipes.quote(s) for s in cmd)
cmd = ["qvm-run", "--pass-io", "--", self.hostname, subcmd]
return subprocess.Popen(
cmd,
stdin=stdin,
stdout=stdout,
bufsize=MAX_LIST_SIZE,
close_fds=True
)
def put(self, source_path, remote_filename=None):
"""Transfers a single file to the remote side."""
if not remote_filename:
remote_filename = source_path.get_filename()
self._validate_remote_filename("put", remote_filename)
rempath = os.path.join(self.remote_dir, remote_filename)
cmd = self._dd(off=rempath)
fobject = open(source_path.name, "rb")
try:
p = self._execute_qvmrun(cmd,
stdin=fobject,
stdout=open(os.devnull))
except Exception, e:
raise BackendException(
"Qubes VM put of %s (as %s) failed: (%s) %s" % (
source_path.name, remote_filename, type(e), e))
finally:
fobject.close()
err = p.wait()
if err != 0:
raise BackendException(
("Qubes VM put of %s (as %s) failed: writing the "
"destination path exited with nonzero status %s") % (
source_path.name, remote_filename, err))
def get(self, remote_filename, local_path):
"""Retrieves a single file from the remote side."""
self._validate_remote_filename("get", remote_filename)
rempath = os.path.join(self.remote_dir, remote_filename)
cmd = self._dd(iff=rempath)
fobject = open(local_path.name, "wb")
try:
p = self._execute_qvmrun(cmd,
stdin=open(os.devnull),
stdout=fobject)
except Exception, e:
raise BackendException(
"Qubes VM get of %s (as %s) failed: (%s) %s" % (
remote_filename.name, local_path, type(e), e))
finally:
fobject.close()
err = p.wait()
if err != 0:
raise BackendException(
("Qubes VM get of %s (as %s) failed: writing the "
"destination path exited with nonzero status %s") % (
remote_filename.name, local_path, err))
def _list(self):
"""Lists the contents of the one duplicity dir on the remote side."""
cmd = ["find", self.remote_dir, "-maxdepth", "1", "-print0"]
try:
p = self._execute_qvmrun(cmd,
stdin=open(os.devnull, "rb"),
stdout=subprocess.PIPE)
except Exception, e:
raise BackendException(
"Qubes VM list of %s failed: %s" % (self.remote_dir, e))
data = p.stdout.read(MAX_LIST_SIZE)
p.stdout.close()
err = p.wait()
if err != 0:
raise BackendException(
("Qubes VM list of %s failed: list command finished "
"with nonzero status %s" % (self.remote_dir, err)))
if not data:
raise BackendException(
("Qubes VM list of %s failed: list command returned "
"empty" % (self.remote_dir,)))
filename_list = data.split("\0")
if filename_list[0] != self.remote_dir:
raise BackendException(
("Qubes VM list of %s failed: list command returned a "
"filename_list for a path different from the remote folder") % (
self.remote_dir,))
filename_list.pop(0)
if filename_list[-1]:
raise BackendException(
("Qubes VM list of %s failed: list command returned "
"wrongly-terminated data or listing was too long") % (
self.remote_dir,))
filename_list.pop()
filename_list = [ p[len(self.remote_dir) + 1:] for p in filename_list ]
if any(os.path.sep in p for p in filename_list):
raise BackendException(
("Qubes VM list of %s failed: list command returned "
"a path separator in the listing") % (
self.remote_dir,))
return filename_list
def delete(self, filename_list):
"""Deletes all files in the list on the remote side."""
if any(os.path.sep in p or "\0" in p for p in filename_list):
raise BackendException(
("Qubes VM delete of files in %s failed: delete "
"command asked to delete a file with a path separator "
"or a null character in the listing") % (
self.remote_dir,))
pathlist = [os.path.join(self.remote_dir, p) for p in filename_list]
cmd = ["rm", "-f", "--"] + pathlist
try:
p = self._execute_qvmrun(cmd,
stdin=open(os.devnull, "rb"),
stdout=open(os.devnull, "wb"))
except Exception, e:
raise BackendException(
"Qubes VM delete of files in %s failed: %s" % (
self.remote_dir, e))
err = p.wait()
if err != 0:
raise BackendException(
("Qubes VM delete of files in %s failed: delete "
"command finished with nonzero status %s") % (
self.remote_dir, err))
duplicity.backend.register_backend("qubesvm", QubesVMBackend)Finally: I disagree with the direction of implementation which suggests we need to play Towers of Hanoi with a backup VM and attaching disk images to it. That's entirely unnecessary complication, and it also demands VMs be off for backup purposes. Entirely unnecessary and extremely discouraging of backup procedures. The three step process is all that is necessary:
That is all that is needed. Of course, the Qubes setting of which VMs to back up, plus options like |
|
Update: Duplicity is no good for the process, because differential backups slow to a crawl. We need to research other stuff, like Attic. |
|
I have not experienced such issue with Duplicity. I have managed to perform incremental backup on roughly 15GiB (after exclusions) of various data (many small files and few large files) in ~2 minutes even on HDD + dm-crypt. (Of course, this depends on size of changes.) Encryption and compression were enabled. Maybe it skips many files just because of timestamps and diffs just few files. So, I feel you must be doing something wrong. (No offense.) What was your scenario? What files did you try to backup? (E.g., dom0's ~ with 1GiB of small files.) What was your drive setup? (E.g., 7200 RPM HDD with dm-crypt with ext4.) How much time did it take? Where do you store the backup? (Other VM using your storage backend? Well, theoretically, this should not matter so much for performance of scanning, as Duplicity caches metadata locally, AFAIK somewhere in ~/.cache. But I am curious.) Did you have the metadata cached? How much data did you add to the backup? (Well, if you backup ~ and you don't exclude ~/.cache, you might add Duplicity metadata to the backup, which could explain both some time and space penalty. I am not sure if Duplicity is smart enough to exclude this automagically.) |
|
On 07/19/2016 05:45 PM, Vít Šesták wrote:
My backup of a 60 GB disk image progressed essentially nothing over the |
|
Wyng now has a format specification draft: https://github.com/tasket/wyng-backup/tree/04alpha3/doc/Wyng_Archive_Format_V3.md |
|
Once |
|
@tasket maybe a quick update is needed here? Or @marmarek, I see you removed assignment to yourself. Maybe you want to say what is missing to go forward under tasket/wyng-backup#102 instead? |
That's not specific to this issue. It's just a general change in how issue assignments are used. Before, issues were assigned to people who work in certain areas, even if there was no current work being done (including just "someday, maybe"). Now, by contrast, issues are only assigned to devs while those devs are actively working on the issues. You can read more about the new policy here. |
|
@tlaurion @marmarek Wyng is now in final beta for v0.8 and all features are frozen. Its exhibiting good stability overall, but I have added a caveat to the Readme about lvmthin's need for metadata space since adding Wyng's snapshots on top of Qubes' snapshots will naturally consume more and LVM does not have good defaults or out-of-space handling. The wyng-util-qubes wrapper for Qubes integration has just gone to v0.9 beta and pushed to main branch, as I recommend using this version now. The new version includes support for both reflink and lvmthin pools; the wrapper can now alias volume names as necessary between the two pool types during This is what a typical backup session from a Btrfs Qubes system looks like: This is a restore from a backup session containing two VMs: Currently, the Qubes default pool for newly created VMs, or whichever one an existing VM resides, is used for restore but you can also specify |
|
Also, if wyng is to be considered in newer versions of QubesOS, I would recommend thinking about seperating dom0 data from qubes hypervisor. That is a whole separated topic here, but on my side i'm used wyng against a lot of cloned templates and cloned qubes which results in the following: To talk about moving to brtfs+ bees dedup here is out of point of course and would be #6476, not here. But since the caveat of TLVM metadata is touched, I would love to take the opportunity to remind devs that having separated dom0 from vm-pool was one step into the direction of limiting impacts of using TLVM in the first place under Qubes considering that QubesOS uses a lot of sometimes stalling snapshots for back+x and volatile volumes which could have locked the user out, which impacts pool metadata, a lot, also to be discussed under #6476 not here. All of those could of course be dodged altogether if TLVM was reconsidered as first QubesOS candidate to envvision going into something more practical fitting specialized cloned templates, salting qubes and massive qubes private disk data preventing quick shutdown and all that jazz. But that would be #6476 not here. So I would just want to remind that using wyng consumes dom0 space for meta-dir (chunks mapping downloaded from the archive qube) and that dom0 LVM is 20GB considering past decisions (it could be within same pool if not TLVM), and that extending TLVM pool metadata requires to steal some of the swap space (manually), while having 20Gb today is quite limitating (cannot instlal multiple templates, be cautious and check dom0 usage) because dom0 should actually be quite static, if dom0 was just dom0 without dom0 keeping external states that should not exist in dom0 in the first place. @tasket and @marmarek @DemiMarie : amazing work outside of those critical, but constructive points, as usual :) Currently ongoing: users hacking qubes-backup encryption/authentication to use incremental backup tools: ... What about writing a grant application to achieve that pressing goal? Yes? |
|
@tlaurion Just FYI, Wyng has an issue open (109) for reducing metadata footprint. Currently it keeps both a compressed and uncompressed copy of manifests in /var. It does set the compression bit on the uncompressed files, so if /var is on Btrfs or other compressing fs then used space will be reduced somewhat. To be completely honest, the main issue with that issue is the low level of interest in it, otherwise I think it would have been done already. BTW, you can keep deleting all of the 'manifest' files, which should reduce the footprint by up to 2/3.
Its not something I'm familiar with, and the goal needs to be better-defined. AFAIC, Wyng is fully functional now and the only things a typical user would really miss would be a GUI and perhaps a way to mount archived volumes. |
@tasket Can you contact me over Matrix /mastodon/QubesOS forum? |
|
@marmarek @tasket @rapenne-s should we team up for a grant application? Interest? |
I'm be happy to work on this |
Random thoughts for grant application. What I envision to be done as of now :
@rapenne-s I think you are amazing candidate for documentation and insights on infra optimization. @marmarek @marmarta @DemiMarie UX and GUI integration is needed. Fill us in with high level requirements? @tasket of course to tackle wyng-backup wyng-util-qubes work and input other features envisioned missing to reach massive adoption with clear use cases. @tlaurion : heads and general plumbing, facing NLnet, grant application writeup, validation of proof of work (PoW, normally PR) prior of request for payment (RfP) and project management. Heads integration, possibly using SecureDrop workstation use case to drive this. @deeplow interest from FPF' SD and interfacing with them with requirements? Goal here would be to have restorable states from network booted environement to pull states from the network and prepare drive content to be ready to use under minutes, hosted on FPF infra. Maybe we should create discussion on QubesOS forum if there is interest or in a seperate issue? Whether you see fit best. Notes on grant application process. Grant work is paid upon PoW for scoped tasks. So for teaming up here, we need first acceptation (consent) of engagement into doing the work within a year after grant application is accepted. Scoping of general tasks, by whom the work will be done and required approximate funds to be budgetized do such high level tasks, to be then broken down in smaller tasks upon project approval in terms of deliverables paid upon PoW. @rapenne-s @marmarek @marmarta @deeplow @DemiMarie @tasket : would you be willing to engage into teaming up to expend on the needs of such integration and documentation, and agree on the first step, which is to consent into accomplishing such integration as a goal if such grant application was accepted to fund the work needed? |
|
Aside from organizational plans, I've decided that there will have to be a beta5 now. The metadata caching issue @tlaurion pointed out needs to be addressed before v0.8 rc, along with a unicode-handling issue I identified. Fixes are now in the 08wip branch and I expect beta5 to be available within a week. Session metadata will now be aged-out from /var/wyng before 3 days, by default. This can be controlled using an option like |
|
@tlaurion here is what I'd like from a backup solution, to be considered a replacement for the current (non-incremental) one:
Those are the main ones on the functional side. Some are optional (marked "nice to have"). Some (if not most) are already satisfied by wyng, but I've written them down anyway. Some of those apply to Wyng itself, some to its integration with Qubes OS. Lets discuss how we can make this happen :) |
@tasket yhis needs a thorough project status update from you here! |
|
@marmarek Thanks for taking time to post your queries/requirements. My answers follow:
For backing up the backup, simple CC @tlaurion |
|
One more nice-to-have thing I would like to see: possibility to exclude a directory/file (not sure if possible since wyng works on LVMs) or at least backup just the VM metadata (name, color, netvm, template name, etc.) without backing up its contents. Example use case: You have a "media" qube where you watch videos offline. You download a playlist from youtube with |
|
@marmarek @tasket #858 (comment): ping |
|
@tasket @marmarek @marmarta @rapenne-s QubesOS mini-summit talk published https://youtu.be/It13u9UASs4?list=PLuISieMwVBpL5S7kPUHKenoFj_YJ8Y0_d @marmarek said wyng-backups was in the pipelines in opening talks. Do we try to organize for funding? Safe disks state as a firmware service had some push-backs in Q&A of the talk. I understand there is reluctance in having the firmware be network-ready (some reluctance stating that machines using QubeOS seperate this with sys-net, sys-firewall etc), but again I have to remind people that on-demand network access from firmware (IPXE->DTS under Dasharo to produce HCL, upgrade firmware etc) is common, where Heads rely on kernel modules loaded when asked per user and where measurements are extended and wouldn't permit unsealing any secret from that boot session. I also want to remind that the promoted way of doing this under Heads is through CDC tethering over phone, so the Phone itself is exposing itself of network risks, where phone can use VPN or whatever is needed per threat model to mitigate network risks. Heads only gets an IP unless Ethernet drivers are used to connect directly to network, and where SSH would be used to connect to wyng-backup server defined in config. So provisioning of ssh private key (with/without passphrase) would be fused in CBFS alongside of provisioned wyng-backup location, and where those states maintained by IT dpt would not require encryption, where authentication and integrity of backups would be maintained by IT dpt where storage would be accessible only in read only for pre-authorized ssh public key for that server. If the above is accepted (on-demand network access from ethernet/CDC tethering/IPXE pointing to kernel+initrd), where Heads would need to include new curl dependency to pull those initrd+kernel images, verify them and then boot into them (just like DTS today), then Heads could pass credentials and wyng backup location and be able to pull trusted states from the network and apply them to disk. This is my dream, but this is disconnected of the first steps needed. First step is to integrate wyng-backups into QubesOS, define proper UX and GUI and revise @marmarek requirements being filled by @tasket anwers above. And then look how we could merge efforts into a grant application or other funding sources to wor torward the goal of having differential backups into QubesOS today, and then see how we could easily integrate this better so that those states could be restored outside of QubesOS, from external disks or network. Please ping me back when you are ready to organize toward that goal and what would be the next steps. |
That does somewhat (though not entirely) mitigate the risks. Thank you. A better solution is for the firmware itself to include an isolated network and USB stack that runs in a micro-VM. However, this is significantly more work, not least because of flash size constraints. Requiring it might be a case of “the perfect is the enemy of the good.”
That is an interesting approach. I’m not sure if the Linux kernel CDC drivers are secure, but compromising a phone with strong verified boot is not easy. It’s all about tradeoffs here. |
|
My comment is about the "networked backup tool in the firmware" part and not wyng backup specific. It might make sense to move that part of the discussion elsewhere.
Yeah I was one of those commenters. There are two main aspects here:
And there's the minor aspect that I haven't quite seen yet why this would much better than having a little backup system that you load and update normally, for example on a USB drive, a separate partition, etc. (depending on the exact use case).
Yes, that's Dasharo's decission. Dasharo is not limited to being a Qubes OS loader. It has it's own goals and I see how it's useful for them. Doesn't changes my opinion that I don't think that that exposure is a good fit for Qubes.
But that system is intended to restore a backup, no? So unless you have a very fancy setup with pre-sealed secrets (or very locked down verified boot) for the state you are restoring that networked system has full control about the OS install it's restoring and can get access to the secrets after a reboot into the restored system.
If the users considers the used phone to be highly secure then this works as a strange sys-net, so addresses the isolation concern. But I'm not sure if that's a common case. If you don't consider the phone highly secure I consider this setup worse to a wired ethernet connection, since now you are also exposing the USB stack of the Linux kernel.
I'm not sure what you mean with accepted here. Heads is it's own project and doesn't need approval from the Qubes side. As stated I'm not convinced that the proposed implementation (whether backup tool directly in firmware or via network boot) is a great fit for Qubes' security model. But this doesn't stop Heads from implementing something. Since you already compared to Dasharo: They have some great features from Qubes' perspective, like the ability to disable network and USB stack of the firmware, but also features like the IPXE boot that I wouldn't recommend to be used together with Qubes. But since they are an independent open source project that targets to be a general purpose boot firmware, that makes a lot of sense. How big is Heads allowed to get? Maybe it's practical to squeeze some isolated network setup in there (but that surely isn't as simple as packaging curl/wyng/... into it). |
|
@marmarek @HW42 @DemiMarie @tasket @marmarta : please don't let my personal plan for Heads providing trustable disk states as a firmware service get in the way of joint drafting grant application (I feel it's another case where people agreeing onvdesirable outcome will happen only once made: please neglect for now while letting it be in a joint grant application). To be clear once for all on that firmware/OS made available disk state as a service:
Anyway! Above as said is irrelevant here outside of joint drafting grant application, maybe @mfc would want to jump in? So let's focus first on current blocking points
Making sure, first, that:
|
Community Devs: @v6ak, @tasket
@v6ak's PoC: https://github.com/v6ak/qubes-incremental-backup-poc
@tasket's PoC: https://github.com/tasket/wyng-backup | Status update as of 2022-08-16: #858 (comment)
Reported by joanna on 14 May 2014 10:38 UTC
None
Migrated-From: https://wiki.qubes-os.org/ticket/858
Note to any contributors who wish to work on this issue: Please either ask for details or propose a design before starting serious work on this.
The text was updated successfully, but these errors were encountered: