VirtioFS support for an alternative to 9P for sharing folder #16
Comments
|
Or support an option for disabling shared folder mounting for kernel without 9p virtio support. |
|
Disabling shared directory support might make sense for an image target, but IIUC almost all distros ship module support for 9pfs. For the kernel target, we need some way to map userspace into the guest. I'd need to take a closer look at virtiofs to see if it'd work, but I'm not sure we can disable shared directory support altogether for kernel targets. |
|
I think that actually virtiofs support is less likely to be present by default. I haven't done a comprehensive survey or anything of that sort, but the Arch kernel built as part of That being said, I am looking at using it to address #83. Wondering if we should just switch over or make virtiofs vs. 9p configurable? |
|
But don't we control the guest kernel? We already have instructions for users to build their guest kernels with a few configs set to |
I mean yes, we can always tell users what is necessary. But at the same time there is big benefit in being able to just plug in pretty much any distro kernel floating around, in my opinion. Wider compatibility is just a nice-to-have. Edit: But if we are solely focused on the host kernel, then yes, it may be a win. That being said, I'd be totally onboard with ditching 9P completely, assuming VirtioFS provides some tangible benefits (hopefully without any downsides other than the above, hopefully) |
|
Don't most distros already build 9p support as =m? So virtiofs as =m wouldn't really regress anything. Have you found any distro kernels that can plug and play with vmtest? |
|
Virtiofs is also said to be faster. Could be nice if true. |
I don't know. But looking at the configuration that you use for $ cargo run -- -k bzImage-v6.6-archlinux 'sh -c "zcat /proc/config.gz | grep NET_9P"'
CONFIG_NET_9P=y
CONFIG_NET_9P_FD=m
CONFIG_NET_9P_VIRTIO=y
CONFIG_NET_9P_XEN=m
CONFIG_NET_9P_RDMA=m
# CONFIG_NET_9P_DEBUG is not setOn the other hand, it has I don't have (or, personally, particularly care about) other kernels, because I always build my own anyway. |
And does that kernel work with Edit: Just to clarify, I was under the impression that most distro kernels would work with |
|
Nope, failed: |
Haha, well I had the exact opposite impression. |
Ah okay. Are the provided Arch and Fedora kernels not based on the respective stock distro config then? |
|
I think I took the stock configs and changed the required ones from =m -> =y |
danielocfb
pushed a commit
to d-e-s-o/vmtest
that referenced
this issue
Aug 29, 2024
Switch over to using virtiofsd for sharing file system data with the host. virtiofs is a file system designed for the needs of virtual machines and environments. That is in contrast to 9P fs, which we currently use for sharing data with the host, which is first and foremost a network file system. 9P is problematic if for no other reason that it lacks proper support for usage of the "open-unlink-fstat idiom", in which files are unlinked and later referenced via file descriptor (see danobi#83). virtiofs does not have this problem. This change replaces usage of 9P with that of virtiofs. In order to work, virtiofsd needs a user space server. Contrary to what is the case for 9P, it is not currently integrated into Qemu itself and so we have to manage it separately (and require the user to install it). I benchmarked both the current master as well as this version with a bare-bones custom kernel: Benchmark 1: target/release/vmtest -k bzImage-9p 'echo test' Time (mean ± σ): 1.316 s ± 0.087 s [User: 0.462 s, System: 1.104 s] Range (min … max): 1.232 s … 1.463 s 10 runs Benchmark 1: target/release/vmtest -k bzImage-virtiofsd 'echo test' Time (mean ± σ): 1.244 s ± 0.011 s [User: 0.307 s, System: 0.358 s] Range (min … max): 1.227 s … 1.260 s 10 runs So it seems there is a ~0.7s speed up, on average (and significantly less system time being used). This is great, but I suspect a more pronounced speed advantage will be visible when working with large files, in which virtiofs is said to significantly outperform 9P (typically >2x from what I understand, but I have not done any benchmarks of that nature). A few other notes: - we solely rely on guest level read-only mounts to enforce read-only state. The virtiofsd recommended way is to use read-only bind mounts [0], but doing so would require root. - we are not using DAX, because it still is still incomplete and apparently requires building Qemu (?) from source. In any event, it should not change anything functionally and be solely a performance improvement. - interestingly, there may be the option of just consuming the virtiofsd crate as a library and not require any shelling out. That would be *much* nicer, but the current APIs make this somewhat cumbersome. I'd think we'd pretty much have to reimplement their entire main() functionality [1]. I consider this way out of scope for this first version. I have adjusted the configs, but because I don't have Docker handy I can't really create those kernel. CI seems incapable of producing the artifacts without doing a fully-blown release dance. No idea what empty is about, really. I suspect the test failures we see are because it lacks support? Some additional resources worth keeping around: - https://virtio-fs.gitlab.io/howto-boot.html - https://virtio-fs.gitlab.io/howto-qemu.html [0] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/README.md?ref_type=heads#faq [1] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/src/main.rs?ref_type=heads#L1242 Closes: danobi#16 Closes: danobi#83 Signed-off-by: Daniel Müller <[email protected]>
danielocfb
pushed a commit
to d-e-s-o/vmtest
that referenced
this issue
Aug 29, 2024
Switch over to using virtiofsd for sharing file system data with the host. virtiofs is a file system designed for the needs of virtual machines and environments. That is in contrast to 9P fs, which we currently use for sharing data with the host, which is first and foremost a network file system. 9P is problematic if for no other reason that it lacks proper support for usage of the "open-unlink-fstat idiom", in which files are unlinked and later referenced via file descriptor (see danobi#83). virtiofs does not have this problem. This change replaces usage of 9P with that of virtiofs. In order to work, virtiofsd needs a user space server. Contrary to what is the case for 9P, it is not currently integrated into Qemu itself and so we have to manage it separately (and require the user to install it). I benchmarked both the current master as well as this version with a bare-bones custom kernel: Benchmark 1: target/release/vmtest -k bzImage-9p 'echo test' Time (mean ± σ): 1.316 s ± 0.087 s [User: 0.462 s, System: 1.104 s] Range (min … max): 1.232 s … 1.463 s 10 runs Benchmark 1: target/release/vmtest -k bzImage-virtiofsd 'echo test' Time (mean ± σ): 1.244 s ± 0.011 s [User: 0.307 s, System: 0.358 s] Range (min … max): 1.227 s … 1.260 s 10 runs So it seems there is a ~0.7s speed up, on average (and significantly less system time being used). This is great, but I suspect a more pronounced speed advantage will be visible when working with large files, in which virtiofs is said to significantly outperform 9P (typically >2x from what I understand, but I have not done any benchmarks of that nature). A few other notes: - we solely rely on guest level read-only mounts to enforce read-only state. The virtiofsd recommended way is to use read-only bind mounts [0], but doing so would require root. - we are not using DAX, because it still is still incomplete and apparently requires building Qemu (?) from source. In any event, it should not change anything functionally and be solely a performance improvement. - interestingly, there may be the option of just consuming the virtiofsd crate as a library and not require any shelling out. That would be *much* nicer, but the current APIs make this somewhat cumbersome. I'd think we'd pretty much have to reimplement their entire main() functionality [1]. I consider this way out of scope for this first version. I have adjusted the configs, but because I don't have Docker handy I can't really create those kernel. CI seems incapable of producing the artifacts without doing a fully-blown release dance. No idea what empty is about, really. I suspect the test failures we see are because it lacks support? Some additional resources worth keeping around: - https://virtio-fs.gitlab.io/howto-boot.html - https://virtio-fs.gitlab.io/howto-qemu.html [0] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/README.md?ref_type=heads#faq [1] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/src/main.rs?ref_type=heads#L1242 Closes: danobi#16 Closes: danobi#83 Signed-off-by: Daniel Müller <[email protected]>
danielocfb
pushed a commit
to d-e-s-o/vmtest
that referenced
this issue
Aug 29, 2024
Switch over to using virtiofsd for sharing file system data with the host. virtiofs is a file system designed for the needs of virtual machines and environments. That is in contrast to 9P fs, which we currently use for sharing data with the host, which is first and foremost a network file system. 9P is problematic if for no other reason that it lacks proper support for usage of the "open-unlink-fstat idiom", in which files are unlinked and later referenced via file descriptor (see danobi#83). virtiofs does not have this problem. This change replaces usage of 9P with that of virtiofs. In order to work, virtiofsd needs a user space server. Contrary to what is the case for 9P, it is not currently integrated into Qemu itself and so we have to manage it separately (and require the user to install it). I benchmarked both the current master as well as this version with a bare-bones custom kernel: Benchmark 1: target/release/vmtest -k bzImage-9p 'echo test' Time (mean ± σ): 1.316 s ± 0.087 s [User: 0.462 s, System: 1.104 s] Range (min … max): 1.232 s … 1.463 s 10 runs Benchmark 1: target/release/vmtest -k bzImage-virtiofsd 'echo test' Time (mean ± σ): 1.244 s ± 0.011 s [User: 0.307 s, System: 0.358 s] Range (min … max): 1.227 s … 1.260 s 10 runs So it seems there is a ~0.7s speed up, on average (and significantly less system time being used). This is great, but I suspect a more pronounced speed advantage will be visible when working with large files, in which virtiofs is said to significantly outperform 9P (typically >2x from what I understand, but I have not done any benchmarks of that nature). A few other notes: - we solely rely on guest level read-only mounts to enforce read-only state. The virtiofsd recommended way is to use read-only bind mounts [0], but doing so would require root. - we are not using DAX, because it still is still incomplete and apparently requires building Qemu (?) from source. In any event, it should not change anything functionally and be solely a performance improvement. - interestingly, there may be the option of just consuming the virtiofsd crate as a library and not require any shelling out. That would be *much* nicer, but the current APIs make this somewhat cumbersome. I'd think we'd pretty much have to reimplement their entire main() functionality [1]. I consider this way out of scope for this first version. I have adjusted the configs, but because I don't have Docker handy I can't really create those kernel. CI seems incapable of producing the artifacts without doing a fully-blown release dance. No idea what empty is about, really. I suspect the test failures we see are because it lacks support? Some additional resources worth keeping around: - https://virtio-fs.gitlab.io/howto-boot.html - https://virtio-fs.gitlab.io/howto-qemu.html [0] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/README.md?ref_type=heads#faq [1] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/src/main.rs?ref_type=heads#L1242 Closes: danobi#16 Closes: danobi#83 Signed-off-by: Daniel Müller <[email protected]>
danielocfb
pushed a commit
to d-e-s-o/vmtest
that referenced
this issue
Aug 29, 2024
Switch over to using virtiofsd for sharing file system data with the host. virtiofs is a file system designed for the needs of virtual machines and environments. That is in contrast to 9P fs, which we currently use for sharing data with the host, which is first and foremost a network file system. 9P is problematic if for no other reason that it lacks proper support for usage of the "open-unlink-fstat idiom", in which files are unlinked and later referenced via file descriptor (see danobi#83). virtiofs does not have this problem. This change replaces usage of 9P with that of virtiofs. In order to work, virtiofs needs a user space server. The current state-of-the-art implementation (virtiofsd) is implemented in Rust and so we interface directly with the library. Most of this code is extracted straight from virtiofsd, as it's a lot of boilerplate. An alternative approach is to install the binary via distribution packages or from crates.io, but availability (and discovery) can be a bit of a challenge. Note that this now means that both libcap-ng as well as libseccomp need to be installed. I benchmarked both the current master as well as this version with a bare-bones custom kernel: Benchmark 1: target/release/vmtest -k bzImage-9p 'echo test' Time (mean ± σ): 1.316 s ± 0.087 s [User: 0.462 s, System: 1.104 s] Range (min … max): 1.232 s … 1.463 s 10 runs Benchmark 1: target/release/vmtest -k bzImage-virtiofsd 'echo test' Time (mean ± σ): 1.244 s ± 0.011 s [User: 0.307 s, System: 0.358 s] Range (min … max): 1.227 s … 1.260 s 10 runs So it seems there is a ~0.7s speed up, on average (and significantly less system time being used). This is great, but I suspect a more pronounced speed advantage will be visible when working with large files, in which virtiofs is said to significantly outperform 9P (typically >2x from what I understand, but I have not done any benchmarks of that nature). A few other notes: - we solely rely on guest level read-only mounts to enforce read-only state. The virtiofsd recommended way is to use read-only bind mounts [0], but doing so would require root. - we are not using DAX, because it still is still incomplete and apparently requires building Qemu (?) from source. In any event, it should not change anything functionally and be solely a performance improvement. I have adjusted the configs, but because I don't have Docker handy I can't really create those kernel. CI seems incapable of producing the artifacts without doing a fully-blown release dance. No idea what empty is about, really. I suspect the test failures we see are because it lacks support? Some additional resources worth keeping around: - https://virtio-fs.gitlab.io/howto-boot.html - https://virtio-fs.gitlab.io/howto-qemu.html [0] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/README.md?ref_type=heads#faq [1] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/src/main.rs?ref_type=heads#L1242 Closes: danobi#16 Closes: danobi#83 Signed-off-by: Daniel Müller <[email protected]>
danielocfb
pushed a commit
to d-e-s-o/vmtest
that referenced
this issue
Aug 29, 2024
Switch over to using virtiofsd for sharing file system data with the host. virtiofs is a file system designed for the needs of virtual machines and environments. That is in contrast to 9P fs, which we currently use for sharing data with the host, which is first and foremost a network file system. 9P is problematic if for no other reason that it lacks proper support for usage of the "open-unlink-fstat idiom", in which files are unlinked and later referenced via file descriptor (see danobi#83). virtiofs does not have this problem. This change replaces usage of 9P with that of virtiofs. In order to work, virtiofs needs a user space server. The current state-of-the-art implementation (virtiofsd) is implemented in Rust and so we interface directly with the library. Most of this code is extracted straight from virtiofsd, as it's a lot of boilerplate. An alternative approach is to install the binary via distribution packages or from crates.io, but availability (and discovery) can be a bit of a challenge. Note that this now means that both libcap-ng as well as libseccomp need to be installed. I benchmarked both the current master as well as this version with a bare-bones custom kernel: Benchmark 1: target/release/vmtest -k bzImage-9p 'echo test' Time (mean ± σ): 1.316 s ± 0.087 s [User: 0.462 s, System: 1.104 s] Range (min … max): 1.232 s … 1.463 s 10 runs Benchmark 1: target/release/vmtest -k bzImage-virtiofsd 'echo test' Time (mean ± σ): 1.244 s ± 0.011 s [User: 0.307 s, System: 0.358 s] Range (min … max): 1.227 s … 1.260 s 10 runs So it seems there is a ~0.7s speed up, on average (and significantly less system time being used). This is great, but I suspect a more pronounced speed advantage will be visible when working with large files, in which virtiofs is said to significantly outperform 9P (typically >2x from what I understand, but I have not done any benchmarks of that nature). A few other notes: - we solely rely on guest level read-only mounts to enforce read-only state. The virtiofsd recommended way is to use read-only bind mounts [0], but doing so would require root. - we are not using DAX, because it still is still incomplete and apparently requires building Qemu (?) from source. In any event, it should not change anything functionally and be solely a performance improvement. I have adjusted the configs, but because I don't have Docker handy I can't really create those kernel. CI seems incapable of producing the artifacts without doing a fully-blown release dance. No idea what empty is about, really. I suspect the test failures we see are because it lacks support? Some additional resources worth keeping around: - https://virtio-fs.gitlab.io/howto-boot.html - https://virtio-fs.gitlab.io/howto-qemu.html [0] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/README.md?ref_type=heads#faq [1] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/src/main.rs?ref_type=heads#L1242 Closes: danobi#16 Closes: danobi#83 Signed-off-by: Daniel Müller <[email protected]>
danielocfb
pushed a commit
to d-e-s-o/vmtest
that referenced
this issue
Aug 29, 2024
Switch over to using virtiofsd for sharing file system data with the host. virtiofs is a file system designed for the needs of virtual machines and environments. That is in contrast to 9P fs, which we currently use for sharing data with the host, which is first and foremost a network file system. 9P is problematic if for no other reason that it lacks proper support for usage of the "open-unlink-fstat idiom", in which files are unlinked and later referenced via file descriptor (see danobi#83). virtiofs does not have this problem. This change replaces usage of 9P with that of virtiofs. In order to work, virtiofs needs a user space server. The current state-of-the-art implementation (virtiofsd) is implemented in Rust and so we interface directly with the library. Most of this code is extracted straight from virtiofsd, as it's a lot of boilerplate. An alternative approach is to install the binary via distribution packages or from crates.io, but availability (and discovery) can be a bit of a challenge. Note that this now means that both libcap-ng as well as libseccomp need to be installed. I benchmarked both the current master as well as this version with a bare-bones custom kernel: Benchmark 1: target/release/vmtest -k bzImage-9p 'echo test' Time (mean ± σ): 1.316 s ± 0.087 s [User: 0.462 s, System: 1.104 s] Range (min … max): 1.232 s … 1.463 s 10 runs Benchmark 1: target/release/vmtest -k bzImage-virtiofsd 'echo test' Time (mean ± σ): 1.244 s ± 0.011 s [User: 0.307 s, System: 0.358 s] Range (min … max): 1.227 s … 1.260 s 10 runs So it seems there is a ~0.7s speed up, on average (and significantly less system time being used). This is great, but I suspect a more pronounced speed advantage will be visible when working with large files, in which virtiofs is said to significantly outperform 9P (typically >2x from what I understand, but I have not done any benchmarks of that nature). A few other notes: - we solely rely on guest level read-only mounts to enforce read-only state. The virtiofsd recommended way is to use read-only bind mounts [0], but doing so would require root. - we are not using DAX, because it still is still incomplete and apparently requires building Qemu (?) from source. In any event, it should not change anything functionally and be solely a performance improvement. I have adjusted the configs, but because I don't have Docker handy I can't really create those kernel. CI seems incapable of producing the artifacts without doing a fully-blown release dance. No idea what empty is about, really. I suspect the test failures we see are because it lacks support? Some additional resources worth keeping around: - https://virtio-fs.gitlab.io/howto-boot.html - https://virtio-fs.gitlab.io/howto-qemu.html [0] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/README.md?ref_type=heads#faq [1] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/src/main.rs?ref_type=heads#L1242 Closes: danobi#16 Closes: danobi#83 Signed-off-by: Daniel Müller <[email protected]>
danielocfb
pushed a commit
to d-e-s-o/vmtest
that referenced
this issue
Aug 29, 2024
Switch over to using virtiofsd for sharing file system data with the host. virtiofs is a file system designed for the needs of virtual machines and environments. That is in contrast to 9P fs, which we currently use for sharing data with the host, which is first and foremost a network file system. 9P is problematic if for no other reason that it lacks proper support for usage of the "open-unlink-fstat idiom", in which files are unlinked and later referenced via file descriptor (see danobi#83). virtiofs does not have this problem. This change replaces usage of 9P with that of virtiofs. In order to work, virtiofs needs a user space server. The current state-of-the-art implementation (virtiofsd) is implemented in Rust and so we interface directly with the library. Most of this code is extracted straight from virtiofsd, as it's a lot of boilerplate. An alternative approach is to install the binary via distribution packages or from crates.io, but availability (and discovery) can be a bit of a challenge. Note that this now means that both libcap-ng as well as libseccomp need to be installed. I benchmarked both the current master as well as this version with a bare-bones custom kernel: Benchmark 1: target/release/vmtest -k bzImage-9p 'echo test' Time (mean ± σ): 1.316 s ± 0.087 s [User: 0.462 s, System: 1.104 s] Range (min … max): 1.232 s … 1.463 s 10 runs Benchmark 1: target/release/vmtest -k bzImage-virtiofsd 'echo test' Time (mean ± σ): 1.244 s ± 0.011 s [User: 0.307 s, System: 0.358 s] Range (min … max): 1.227 s … 1.260 s 10 runs So it seems there is a ~0.7s speed up, on average (and significantly less system time being used). This is great, but I suspect a more pronounced speed advantage will be visible when working with large files, in which virtiofs is said to significantly outperform 9P (typically >2x from what I understand, but I have not done any benchmarks of that nature). A few other notes: - we solely rely on guest level read-only mounts to enforce read-only state. The virtiofsd recommended way is to use read-only bind mounts [0], but doing so would require root. - we are not using DAX, because it still is still incomplete and apparently requires building Qemu (?) from source. In any event, it should not change anything functionally and be solely a performance improvement. I have adjusted the configs, but because I don't have Docker handy I can't really create those kernel. CI seems incapable of producing the artifacts without doing a fully-blown release dance. No idea what empty is about, really. I suspect the test failures we see are because it lacks support? Some additional resources worth keeping around: - https://virtio-fs.gitlab.io/howto-boot.html - https://virtio-fs.gitlab.io/howto-qemu.html [0] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/README.md?ref_type=heads#faq [1] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/src/main.rs?ref_type=heads#L1242 Closes: danobi#16 Closes: danobi#83 Signed-off-by: Daniel Müller <[email protected]>
danielocfb
pushed a commit
to d-e-s-o/vmtest
that referenced
this issue
Aug 29, 2024
Switch over to using virtiofsd for sharing file system data with the host. virtiofs is a file system designed for the needs of virtual machines and environments. That is in contrast to 9P fs, which we currently use for sharing data with the host, which is first and foremost a network file system. 9P is problematic if for no other reason that it lacks proper support for usage of the "open-unlink-fstat idiom", in which files are unlinked and later referenced via file descriptor (see danobi#83). virtiofs does not have this problem. This change replaces usage of 9P with that of virtiofs. In order to work, virtiofs needs a user space server. The current state-of-the-art implementation (virtiofsd) is implemented in Rust and so we interface directly with the library. Most of this code is extracted straight from virtiofsd, as it's a lot of boilerplate. An alternative approach is to install the binary via distribution packages or from crates.io, but availability (and discovery) can be a bit of a challenge. Note that this now means that both libcap-ng as well as libseccomp need to be installed. I benchmarked both the current master as well as this version with a bare-bones custom kernel: Benchmark 1: target/release/vmtest -k bzImage-9p 'echo test' Time (mean ± σ): 1.316 s ± 0.087 s [User: 0.462 s, System: 1.104 s] Range (min … max): 1.232 s … 1.463 s 10 runs Benchmark 1: target/release/vmtest -k bzImage-virtiofsd 'echo test' Time (mean ± σ): 1.244 s ± 0.011 s [User: 0.307 s, System: 0.358 s] Range (min … max): 1.227 s … 1.260 s 10 runs So it seems there is a ~0.7s speed up, on average (and significantly less system time being used). This is great, but I suspect a more pronounced speed advantage will be visible when working with large files, in which virtiofs is said to significantly outperform 9P (typically >2x from what I understand, but I have not done any benchmarks of that nature). A few other notes: - we solely rely on guest level read-only mounts to enforce read-only state. The virtiofsd recommended way is to use read-only bind mounts [0], but doing so would require root. - we are not using DAX, because it still is still incomplete and apparently requires building Qemu (?) from source. In any event, it should not change anything functionally and be solely a performance improvement. I have adjusted the configs, but because I don't have Docker handy I can't really create those kernel. CI seems incapable of producing the artifacts without doing a fully-blown release dance. No idea what empty is about, really. I suspect the test failures we see are because it lacks support? Some additional resources worth keeping around: - https://virtio-fs.gitlab.io/howto-boot.html - https://virtio-fs.gitlab.io/howto-qemu.html [0] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/README.md?ref_type=heads#faq [1] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/src/main.rs?ref_type=heads#L1242 Closes: danobi#16 Closes: danobi#83 Signed-off-by: Daniel Müller <[email protected]>
danielocfb
pushed a commit
to d-e-s-o/vmtest
that referenced
this issue
Aug 29, 2024
Switch over to using virtiofsd for sharing file system data with the host. virtiofs is a file system designed for the needs of virtual machines and environments. That is in contrast to 9P fs, which we currently use for sharing data with the host, which is first and foremost a network file system. 9P is problematic if for no other reason that it lacks proper support for usage of the "open-unlink-fstat idiom", in which files are unlinked and later referenced via file descriptor (see danobi#83). virtiofs does not have this problem. This change replaces usage of 9P with that of virtiofs. In order to work, virtiofs needs a user space server. The current state-of-the-art implementation (virtiofsd) is implemented in Rust and so we interface directly with the library. Most of this code is extracted straight from virtiofsd, as it's a lot of boilerplate. An alternative approach is to install the binary via distribution packages or from crates.io, but availability (and discovery) can be a bit of a challenge. Note that this now means that both libcap-ng as well as libseccomp need to be installed. I benchmarked both the current master as well as this version with a bare-bones custom kernel: Benchmark 1: target/release/vmtest -k bzImage-9p 'echo test' Time (mean ± σ): 1.316 s ± 0.087 s [User: 0.462 s, System: 1.104 s] Range (min … max): 1.232 s … 1.463 s 10 runs Benchmark 1: target/release/vmtest -k bzImage-virtiofsd 'echo test' Time (mean ± σ): 1.244 s ± 0.011 s [User: 0.307 s, System: 0.358 s] Range (min … max): 1.227 s … 1.260 s 10 runs So it seems there is a ~0.7s speed up, on average (and significantly less system time being used). This is great, but I suspect a more pronounced speed advantage will be visible when working with large files, in which virtiofs is said to significantly outperform 9P (typically >2x from what I understand, but I have not done any benchmarks of that nature). A few other notes: - we solely rely on guest level read-only mounts to enforce read-only state. The virtiofsd recommended way is to use read-only bind mounts [0], but doing so would require root. - we are not using DAX, because it still is still incomplete and apparently requires building Qemu (?) from source. In any event, it should not change anything functionally and be solely a performance improvement. I have adjusted the configs, but because I don't have Docker handy I can't really create those kernel. CI seems incapable of producing the artifacts without doing a fully-blown release dance. No idea what empty is about, really. I suspect the test failures we see are because it lacks support? Some additional resources worth keeping around: - https://virtio-fs.gitlab.io/howto-boot.html - https://virtio-fs.gitlab.io/howto-qemu.html [0] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/README.md?ref_type=heads#faq [1] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/src/main.rs?ref_type=heads#L1242 Closes: danobi#16 Closes: danobi#83 Signed-off-by: Daniel Müller <[email protected]>
d-e-s-o
added a commit
to d-e-s-o/vmtest
that referenced
this issue
Aug 30, 2024
Switch over to using virtiofsd for sharing file system data with the host. virtiofs is a file system designed for the needs of virtual machines and environments. That is in contrast to 9P fs, which we currently use for sharing data with the host, which is first and foremost a network file system. 9P is problematic if for no other reason that it lacks proper support for usage of the "open-unlink-fstat idiom", in which files are unlinked and later referenced via file descriptor (see danobi#83). virtiofs does not have this problem. This change replaces usage of 9P with that of virtiofs. In order to work, virtiofs needs a user space server. The current state-of-the-art implementation (virtiofsd) is implemented in Rust and so we interface directly with the library. Most of this code is extracted straight from virtiofsd, as it's a lot of boilerplate. An alternative approach is to install the binary via distribution packages or from crates.io, but availability (and discovery) can be a bit of a challenge. Note that this now means that both libcap-ng as well as libseccomp need to be installed. I benchmarked both the current master as well as this version with a bare-bones custom kernel: Benchmark 1: target/release/vmtest -k bzImage-9p 'echo test' Time (mean ± σ): 1.316 s ± 0.087 s [User: 0.462 s, System: 1.104 s] Range (min … max): 1.232 s … 1.463 s 10 runs Benchmark 1: target/release/vmtest -k bzImage-virtiofsd 'echo test' Time (mean ± σ): 1.244 s ± 0.011 s [User: 0.307 s, System: 0.358 s] Range (min … max): 1.227 s … 1.260 s 10 runs So it seems there is a slight speed up, on average (and significantly less system time being used). This is great, but I suspect a more pronounced speed advantage will be visible when working with large files, in which virtiofs is said to significantly outperform 9P (typically >2x from what I understand, but I have not done any benchmarks of that nature). A few other notes: - we solely rely on guest level read-only mounts to enforce read-only state. The virtiofsd recommended way is to use read-only bind mounts [0], but doing so would require root. - we are not using DAX, because it still is still incomplete and apparently requires building Qemu (?) from source. In any event, it should not change anything functionally and be solely a performance improvement. I have adjusted the configs, but because I don't have Docker handy I can't really create those kernel. CI seems incapable of producing the artifacts without doing a fully-blown release dance. No idea what empty is about, really. I suspect the test failures we see are because it lacks support? Some additional resources worth keeping around: - https://virtio-fs.gitlab.io/howto-boot.html - https://virtio-fs.gitlab.io/howto-qemu.html [0] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/README.md?ref_type=heads#faq Closes: danobi#16 Closes: danobi#83 Signed-off-by: Daniel Müller <[email protected]>
d-e-s-o
added a commit
to d-e-s-o/vmtest
that referenced
this issue
Aug 30, 2024
Switch over to using virtiofsd for sharing file system data with the host. virtiofs is a file system designed for the needs of virtual machines and environments. That is in contrast to 9P fs, which we currently use for sharing data with the host, which is first and foremost a network file system. 9P is problematic if for no other reason that it lacks proper support for usage of the "open-unlink-fstat idiom", in which files are unlinked and later referenced via file descriptor (see danobi#83). virtiofs does not have this problem. This change replaces usage of 9P with that of virtiofs. In order to work, virtiofs needs a user space server. The current state-of-the-art implementation (virtiofsd) is implemented in Rust and so we interface directly with the library. Most of this code is extracted straight from virtiofsd, as it's a lot of boilerplate. An alternative approach is to install the binary via distribution packages or from crates.io, but availability (and discovery) can be a bit of a challenge. Note that this now means that both libcap-ng as well as libseccomp need to be installed. I benchmarked both the current master as well as this version with a bare-bones custom kernel: Benchmark 1: target/release/vmtest -k bzImage-9p 'echo test' Time (mean ± σ): 1.316 s ± 0.087 s [User: 0.462 s, System: 1.104 s] Range (min … max): 1.232 s … 1.463 s 10 runs Benchmark 1: target/release/vmtest -k bzImage-virtiofsd 'echo test' Time (mean ± σ): 1.244 s ± 0.011 s [User: 0.307 s, System: 0.358 s] Range (min … max): 1.227 s … 1.260 s 10 runs So it seems there is a slight speed up, on average (and significantly less system time being used). This is great, but I suspect a more pronounced speed advantage will be visible when working with large files, in which virtiofs is said to significantly outperform 9P (typically >2x from what I understand, but I have not done any benchmarks of that nature). A few other notes: - we solely rely on guest level read-only mounts to enforce read-only state. The virtiofsd recommended way is to use read-only bind mounts [0], but doing so would require root. - we are not using DAX, because it still is still incomplete and apparently requires building Qemu (?) from source. In any event, it should not change anything functionally and be solely a performance improvement. I have adjusted the configs, but because I don't have Docker handy I can't really create those kernel. CI seems incapable of producing the artifacts without doing a fully-blown release dance. No idea what empty is about, really. I suspect the test failures we see are because it lacks support? Some additional resources worth keeping around: - https://virtio-fs.gitlab.io/howto-boot.html - https://virtio-fs.gitlab.io/howto-qemu.html [0] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/README.md?ref_type=heads#faq Closes: danobi#16 Closes: danobi#83 Signed-off-by: Daniel Müller <[email protected]>
d-e-s-o
added a commit
to d-e-s-o/vmtest
that referenced
this issue
Aug 30, 2024
Switch over to using virtiofsd for sharing file system data with the host. virtiofs is a file system designed for the needs of virtual machines and environments. That is in contrast to 9P fs, which we currently use for sharing data with the host, which is first and foremost a network file system. 9P is problematic if for no other reason that it lacks proper support for usage of the "open-unlink-fstat idiom", in which files are unlinked and later referenced via file descriptor (see danobi#83). virtiofs does not have this problem. This change replaces usage of 9P with that of virtiofs. In order to work, virtiofs needs a user space server. The current state-of-the-art implementation (virtiofsd) is implemented in Rust and so we interface directly with the library. Most of this code is extracted straight from virtiofsd, as it's a lot of boilerplate. An alternative approach is to install the binary via distribution packages or from crates.io, but availability (and discovery) can be a bit of a challenge. Note that this now means that both libcap-ng as well as libseccomp need to be installed. I benchmarked both the current master as well as this version with a bare-bones custom kernel: Benchmark 1: target/release/vmtest -k bzImage-9p 'echo test' Time (mean ± σ): 1.316 s ± 0.087 s [User: 0.462 s, System: 1.104 s] Range (min … max): 1.232 s … 1.463 s 10 runs Benchmark 1: target/release/vmtest -k bzImage-virtiofsd 'echo test' Time (mean ± σ): 1.244 s ± 0.011 s [User: 0.307 s, System: 0.358 s] Range (min … max): 1.227 s … 1.260 s 10 runs So it seems there is a slight speed up, on average (and significantly less system time being used). This is great, but I suspect a more pronounced speed advantage will be visible when working with large files, in which virtiofs is said to significantly outperform 9P (typically >2x from what I understand, but I have not done any benchmarks of that nature). A few other notes: - we solely rely on guest level read-only mounts to enforce read-only state. The virtiofsd recommended way is to use read-only bind mounts [0], but doing so would require root. - we are not using DAX, because it still is still incomplete and apparently requires building Qemu (?) from source. In any event, it should not change anything functionally and be solely a performance improvement. I have adjusted the configs, but because I don't have Docker handy I can't really create those kernel. CI seems incapable of producing the artifacts without doing a fully-blown release dance. No idea what empty is about, really. I suspect the test failures we see are because it lacks support? Some additional resources worth keeping around: - https://virtio-fs.gitlab.io/howto-boot.html - https://virtio-fs.gitlab.io/howto-qemu.html [0] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/README.md?ref_type=heads#faq Closes: danobi#16 Closes: danobi#83 Signed-off-by: Daniel Müller <[email protected]>
d-e-s-o
added a commit
to d-e-s-o/vmtest
that referenced
this issue
Aug 30, 2024
Switch over to using virtiofsd for sharing file system data with the host. virtiofs is a file system designed for the needs of virtual machines and environments. That is in contrast to 9P fs, which we currently use for sharing data with the host, which is first and foremost a network file system. 9P is problematic if for no other reason that it lacks proper support for usage of the "open-unlink-fstat idiom", in which files are unlinked and later referenced via file descriptor (see danobi#83). virtiofs does not have this problem. This change replaces usage of 9P with that of virtiofs. In order to work, virtiofs needs a user space server. The current state-of-the-art implementation (virtiofsd) is implemented in Rust and so we interface directly with the library. Most of this code is extracted straight from virtiofsd, as it's a lot of boilerplate. An alternative approach is to install the binary via distribution packages or from crates.io, but availability (and discovery) can be a bit of a challenge. Note that this now means that both libcap-ng as well as libseccomp need to be installed. I benchmarked both the current master as well as this version with a bare-bones custom kernel: Benchmark 1: target/release/vmtest -k bzImage-9p 'echo test' Time (mean ± σ): 1.316 s ± 0.087 s [User: 0.462 s, System: 1.104 s] Range (min … max): 1.232 s … 1.463 s 10 runs Benchmark 1: target/release/vmtest -k bzImage-virtiofsd 'echo test' Time (mean ± σ): 1.244 s ± 0.011 s [User: 0.307 s, System: 0.358 s] Range (min … max): 1.227 s … 1.260 s 10 runs So it seems there is a slight speed up, on average (and significantly less system time being used). This is great, but I suspect a more pronounced speed advantage will be visible when working with large files, in which virtiofs is said to significantly outperform 9P (typically >2x from what I understand, but I have not done any benchmarks of that nature). A few other notes: - we solely rely on guest level read-only mounts to enforce read-only state. The virtiofsd recommended way is to use read-only bind mounts [0], but doing so would require root. - we are not using DAX, because it still is still incomplete and apparently requires building Qemu (?) from source. In any event, it should not change anything functionally and be solely a performance improvement. I have adjusted the configs, but because I don't have Docker handy I can't really create those kernel. CI seems incapable of producing the artifacts without doing a fully-blown release dance. No idea what empty is about, really. I suspect the test failures we see are because it lacks support? Some additional resources worth keeping around: - https://virtio-fs.gitlab.io/howto-boot.html - https://virtio-fs.gitlab.io/howto-qemu.html [0] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/README.md?ref_type=heads#faq Closes: danobi#16 Closes: danobi#83 Signed-off-by: Daniel Müller <[email protected]>
danielocfb
pushed a commit
to d-e-s-o/vmtest
that referenced
this issue
Sep 3, 2024
Switch over to using virtiofsd for sharing file system data with the host. virtiofs is a file system designed for the needs of virtual machines and environments. That is in contrast to 9P fs, which we currently use for sharing data with the host, which is first and foremost a network file system. 9P is problematic if for no other reason that it lacks proper support for usage of the "open-unlink-fstat idiom", in which files are unlinked and later referenced via file descriptor (see danobi#83). virtiofs does not have this problem. This change replaces usage of 9P with that of virtiofs. In order to work, virtiofs needs a user space server. The current state-of-the-art implementation (virtiofsd) is implemented in Rust and so we interface directly with the library. Most of this code is extracted straight from virtiofsd, as it's a lot of boilerplate. An alternative approach is to install the binary via distribution packages or from crates.io, but availability (and discovery) can be a bit of a challenge. Note that this now means that both libcap-ng as well as libseccomp need to be installed. I benchmarked both the current master as well as this version with a bare-bones custom kernel: Benchmark 1: target/release/vmtest -k bzImage-9p 'echo test' Time (mean ± σ): 1.316 s ± 0.087 s [User: 0.462 s, System: 1.104 s] Range (min … max): 1.232 s … 1.463 s 10 runs Benchmark 1: target/release/vmtest -k bzImage-virtiofsd 'echo test' Time (mean ± σ): 1.244 s ± 0.011 s [User: 0.307 s, System: 0.358 s] Range (min … max): 1.227 s … 1.260 s 10 runs So it seems there is a slight speed up, on average (and significantly less system time being used). This is great, but I suspect a more pronounced speed advantage will be visible when working with large files, in which virtiofs is said to significantly outperform 9P (typically >2x from what I understand, but I have not done any benchmarks of that nature). A few other notes: - we solely rely on guest level read-only mounts to enforce read-only state. The virtiofsd recommended way is to use read-only bind mounts [0], but doing so would require root. - we are not using DAX, because it still is still incomplete and apparently requires building Qemu (?) from source. In any event, it should not change anything functionally and be solely a performance improvement. I have adjusted the configs, but because I don't have Docker handy I can't really create those kernel. CI seems incapable of producing the artifacts without doing a fully-blown release dance. No idea what empty is about, really. I suspect the test failures we see are because it lacks support? Some additional resources worth keeping around: - https://virtio-fs.gitlab.io/howto-boot.html - https://virtio-fs.gitlab.io/howto-qemu.html [0] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/README.md?ref_type=heads#faq Closes: danobi#16 Closes: danobi#83 Signed-off-by: Daniel Müller <[email protected]>
d-e-s-o
added a commit
to d-e-s-o/vmtest
that referenced
this issue
Sep 5, 2024
Switch over to using virtiofsd for sharing file system data with the host. virtiofs is a file system designed for the needs of virtual machines and environments. That is in contrast to 9P fs, which we currently use for sharing data with the host, which is first and foremost a network file system. 9P is problematic if for no other reason that it lacks proper support for usage of the "open-unlink-fstat idiom", in which files are unlinked and later referenced via file descriptor (see danobi#83). virtiofs does not have this problem. This change replaces usage of 9P with that of virtiofs. In order to work, virtiofs needs a user space server. The current state-of-the-art implementation (virtiofsd) is implemented in Rust and so we interface directly with the library. Most of this code is extracted straight from virtiofsd, as it's a lot of boilerplate. An alternative approach is to install the binary via distribution packages or from crates.io, but availability (and discovery) can be a bit of a challenge. Note that this now means that both libcap-ng as well as libseccomp need to be installed. I benchmarked both the current master as well as this version with a bare-bones custom kernel: Benchmark 1: target/release/vmtest -k bzImage-9p 'echo test' Time (mean ± σ): 1.316 s ± 0.087 s [User: 0.462 s, System: 1.104 s] Range (min … max): 1.232 s … 1.463 s 10 runs Benchmark 1: target/release/vmtest -k bzImage-virtiofsd 'echo test' Time (mean ± σ): 1.244 s ± 0.011 s [User: 0.307 s, System: 0.358 s] Range (min … max): 1.227 s … 1.260 s 10 runs So it seems there is a slight speed up, on average (and significantly less system time being used). This is great, but I suspect a more pronounced speed advantage will be visible when working with large files, in which virtiofs is said to significantly outperform 9P (typically >2x from what I understand, but I have not done any benchmarks of that nature). A few other notes: - we solely rely on guest level read-only mounts to enforce read-only state. The virtiofsd recommended way is to use read-only bind mounts [0], but doing so would require root. - we are not using DAX, because it still is still incomplete and apparently requires building Qemu (?) from source. In any event, it should not change anything functionally and be solely a performance improvement. I have adjusted the configs, but because I don't have Docker handy I can't really create those kernel. CI seems incapable of producing the artifacts without doing a fully-blown release dance. No idea what empty is about, really. I suspect the test failures we see are because it lacks support? Some additional resources worth keeping around: - https://virtio-fs.gitlab.io/howto-boot.html - https://virtio-fs.gitlab.io/howto-qemu.html [0] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/README.md?ref_type=heads#faq Closes: danobi#16 Closes: danobi#83 Signed-off-by: Daniel Müller <[email protected]>
d-e-s-o
added a commit
to d-e-s-o/vmtest
that referenced
this issue
Sep 5, 2024
Switch over to using virtiofsd for sharing file system data with the host. virtiofs is a file system designed for the needs of virtual machines and environments. That is in contrast to 9P fs, which we currently use for sharing data with the host, which is first and foremost a network file system. 9P is problematic if for no other reason that it lacks proper support for usage of the "open-unlink-fstat idiom", in which files are unlinked and later referenced via file descriptor (see danobi#83). virtiofs does not have this problem. This change replaces usage of 9P with that of virtiofs. In order to work, virtiofs needs a user space server. The current state-of-the-art implementation (virtiofsd) is implemented in Rust and so we interface directly with the library. Most of this code is extracted straight from virtiofsd, as it's a lot of boilerplate. An alternative approach is to install the binary via distribution packages or from crates.io, but availability (and discovery) can be a bit of a challenge. Note that this now means that both libcap-ng as well as libseccomp need to be installed. I benchmarked both the current master as well as this version with a bare-bones custom kernel: Benchmark 1: target/release/vmtest -k bzImage-9p 'echo test' Time (mean ± σ): 1.316 s ± 0.087 s [User: 0.462 s, System: 1.104 s] Range (min … max): 1.232 s … 1.463 s 10 runs Benchmark 1: target/release/vmtest -k bzImage-virtiofsd 'echo test' Time (mean ± σ): 1.244 s ± 0.011 s [User: 0.307 s, System: 0.358 s] Range (min … max): 1.227 s … 1.260 s 10 runs So it seems there is a slight speed up, on average (and significantly less system time being used). This is great, but I suspect a more pronounced speed advantage will be visible when working with large files, in which virtiofs is said to significantly outperform 9P (typically >2x from what I understand, but I have not done any benchmarks of that nature). A few other notes: - we solely rely on guest level read-only mounts to enforce read-only state. The virtiofsd recommended way is to use read-only bind mounts [0], but doing so would require root. - we are not using DAX, because it still is still incomplete and apparently requires building Qemu (?) from source. In any event, it should not change anything functionally and be solely a performance improvement. I have adjusted the configs, but because I don't have Docker handy I can't really create those kernel. CI seems incapable of producing the artifacts without doing a fully-blown release dance. No idea what empty is about, really. I suspect the test failures we see are because it lacks support? Some additional resources worth keeping around: - https://virtio-fs.gitlab.io/howto-boot.html - https://virtio-fs.gitlab.io/howto-qemu.html [0] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/README.md?ref_type=heads#faq Closes: danobi#16 Closes: danobi#83 Signed-off-by: Daniel Müller <[email protected]>
d-e-s-o
added a commit
to d-e-s-o/vmtest
that referenced
this issue
Sep 5, 2024
Switch over to using virtiofsd for sharing file system data with the host. virtiofs is a file system designed for the needs of virtual machines and environments. That is in contrast to 9P fs, which we currently use for sharing data with the host, which is first and foremost a network file system. 9P is problematic if for no other reason that it lacks proper support for usage of the "open-unlink-fstat idiom", in which files are unlinked and later referenced via file descriptor (see danobi#83). virtiofs does not have this problem. This change replaces usage of 9P with that of virtiofs. In order to work, virtiofs needs a user space server. The current state-of-the-art implementation (virtiofsd) is implemented in Rust and so we interface directly with the library. Most of this code is extracted straight from virtiofsd, as it's a lot of boilerplate. An alternative approach is to install the binary via distribution packages or from crates.io, but availability (and discovery) can be a bit of a challenge. Note that this now means that both libcap-ng as well as libseccomp need to be installed. I benchmarked both the current master as well as this version with a bare-bones custom kernel: Benchmark 1: target/release/vmtest -k bzImage-9p 'echo test' Time (mean ± σ): 1.316 s ± 0.087 s [User: 0.462 s, System: 1.104 s] Range (min … max): 1.232 s … 1.463 s 10 runs Benchmark 1: target/release/vmtest -k bzImage-virtiofsd 'echo test' Time (mean ± σ): 1.244 s ± 0.011 s [User: 0.307 s, System: 0.358 s] Range (min … max): 1.227 s … 1.260 s 10 runs So it seems there is a slight speed up, on average (and significantly less system time being used). This is great, but I suspect a more pronounced speed advantage will be visible when working with large files, in which virtiofs is said to significantly outperform 9P (typically >2x from what I understand, but I have not done any benchmarks of that nature). A few other notes: - we solely rely on guest level read-only mounts to enforce read-only state. The virtiofsd recommended way is to use read-only bind mounts [0], but doing so would require root. - we are not using DAX, because it still is still incomplete and apparently requires building Qemu (?) from source. In any event, it should not change anything functionally and be solely a performance improvement. Some additional resources worth keeping around: - https://virtio-fs.gitlab.io/howto-boot.html - https://virtio-fs.gitlab.io/howto-qemu.html [0] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/README.md?ref_type=heads#faq Closes: danobi#16 Closes: danobi#83 Signed-off-by: Daniel Müller <[email protected]>
danielocfb
pushed a commit
to d-e-s-o/vmtest
that referenced
this issue
Sep 5, 2024
Switch over to using virtiofsd for sharing file system data with the host. virtiofs is a file system designed for the needs of virtual machines and environments. That is in contrast to 9P fs, which we currently use for sharing data with the host, which is first and foremost a network file system. 9P is problematic if for no other reason that it lacks proper support for usage of the "open-unlink-fstat idiom", in which files are unlinked and later referenced via file descriptor (see danobi#83). virtiofs does not have this problem. This change replaces usage of 9P with that of virtiofs. In order to work, virtiofs needs a user space server. The current state-of-the-art implementation (virtiofsd) is implemented in Rust and so we interface directly with the library. Most of this code is extracted straight from virtiofsd, as it's a lot of boilerplate. An alternative approach is to install the binary via distribution packages or from crates.io, but availability (and discovery) can be a bit of a challenge. Note that this now means that both libcap-ng as well as libseccomp need to be installed. I benchmarked both the current master as well as this version with a bare-bones custom kernel: Benchmark 1: target/release/vmtest -k bzImage-9p 'echo test' Time (mean ± σ): 1.316 s ± 0.087 s [User: 0.462 s, System: 1.104 s] Range (min … max): 1.232 s … 1.463 s 10 runs Benchmark 1: target/release/vmtest -k bzImage-virtiofsd 'echo test' Time (mean ± σ): 1.244 s ± 0.011 s [User: 0.307 s, System: 0.358 s] Range (min … max): 1.227 s … 1.260 s 10 runs So it seems there is a slight speed up, on average (and significantly less system time being used). This is great, but I suspect a more pronounced speed advantage will be visible when working with large files, in which virtiofs is said to significantly outperform 9P (typically >2x from what I understand, but I have not done any benchmarks of that nature). A few other notes: - we solely rely on guest level read-only mounts to enforce read-only state. The virtiofsd recommended way is to use read-only bind mounts [0], but doing so would require root. - we are not using DAX, because it still is still incomplete and apparently requires building Qemu (?) from source. In any event, it should not change anything functionally and be solely a performance improvement. Some additional resources worth keeping around: - https://virtio-fs.gitlab.io/howto-boot.html - https://virtio-fs.gitlab.io/howto-qemu.html [0] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/README.md?ref_type=heads#faq Closes: danobi#16 Closes: danobi#83 Signed-off-by: Daniel Müller <[email protected]>
d-e-s-o
added a commit
to d-e-s-o/vmtest
that referenced
this issue
Sep 23, 2024
Switch over to using virtiofsd for sharing file system data with the host. virtiofs is a file system designed for the needs of virtual machines and environments. That is in contrast to 9P fs, which we currently use for sharing data with the host, which is first and foremost a network file system. 9P is problematic if for no other reason that it lacks proper support for usage of the "open-unlink-fstat idiom", in which files are unlinked and later referenced via file descriptor (see danobi#83). virtiofs does not have this problem. This change replaces usage of 9P with that of virtiofs. In order to work, virtiofs needs a user space server. The current state-of-the-art implementation (virtiofsd) is implemented in Rust and so we interface directly with the library. Most of this code is extracted straight from virtiofsd, as it's a lot of boilerplate. An alternative approach is to install the binary via distribution packages or from crates.io, but availability (and discovery) can be a bit of a challenge. Note that this now means that both libcap-ng as well as libseccomp need to be installed. I benchmarked both the current master as well as this version with a bare-bones custom kernel: Benchmark 1: target/release/vmtest -k bzImage-9p 'echo test' Time (mean ± σ): 1.316 s ± 0.087 s [User: 0.462 s, System: 1.104 s] Range (min … max): 1.232 s … 1.463 s 10 runs Benchmark 1: target/release/vmtest -k bzImage-virtiofsd 'echo test' Time (mean ± σ): 1.244 s ± 0.011 s [User: 0.307 s, System: 0.358 s] Range (min … max): 1.227 s … 1.260 s 10 runs So it seems there is a slight speed up, on average (and significantly less system time being used). This is great, but I suspect a more pronounced speed advantage will be visible when working with large files, in which virtiofs is said to significantly outperform 9P (typically >2x from what I understand, but I have not done any benchmarks of that nature). A few other notes: - we solely rely on guest level read-only mounts to enforce read-only state. The virtiofsd recommended way is to use read-only bind mounts [0], but doing so would require root. - we are not using DAX, because it still is still incomplete and apparently requires building Qemu (?) from source. In any event, it should not change anything functionally and be solely a performance improvement. Given that we are not regressing in terms of performance, this is strictly future work. I have adjusted the configs, but because I don't have Docker handy I can't really create those kernel. CI seems incapable of producing the artifacts without doing a fully-blown release dance. No idea what empty is about, really. I suspect the test failures we see are because it lacks support? Some additional resources worth keeping around: - https://virtio-fs.gitlab.io/howto-boot.html - https://virtio-fs.gitlab.io/howto-qemu.html [0] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/README.md?ref_type=heads#faq Closes: danobi#16 Closes: danobi#83 Signed-off-by: Daniel Müller <[email protected]>
d-e-s-o
added a commit
to d-e-s-o/vmtest
that referenced
this issue
Sep 25, 2024
Switch over to using virtiofsd for sharing file system data with the host. virtiofs is a file system designed for the needs of virtual machines and environments. That is in contrast to 9P fs, which we currently use for sharing data with the host, which is first and foremost a network file system. 9P is problematic if for no other reason that it lacks proper support for usage of the "open-unlink-fstat idiom", in which files are unlinked and later referenced via file descriptor (see danobi#83). virtiofs does not have this problem. This change replaces usage of 9P with that of virtiofs. In order to work, virtiofs needs a user space server. The current state-of-the-art implementation (virtiofsd) is implemented in Rust and so we interface directly with the library. Most of this code is extracted straight from virtiofsd, as it's a lot of boilerplate. An alternative approach is to install the binary via distribution packages or from crates.io, but availability (and discovery) can be a bit of a challenge. I benchmarked both the current master as well as this version with a bare-bones custom kernel: Benchmark 1: target/release/vmtest -k bzImage-9p 'echo test' Time (mean ± σ): 1.316 s ± 0.087 s [User: 0.462 s, System: 1.104 s] Range (min … max): 1.232 s … 1.463 s 10 runs Benchmark 1: target/release/vmtest -k bzImage-virtiofsd 'echo test' Time (mean ± σ): 1.244 s ± 0.011 s [User: 0.307 s, System: 0.358 s] Range (min … max): 1.227 s … 1.260 s 10 runs So it seems there is a slight speed up, on average (and significantly less system time being used). This is great, but I suspect a more pronounced speed advantage will be visible when working with large files, in which virtiofs is said to significantly outperform 9P (typically >2x from what I understand, but I have not done any benchmarks of that nature). A few other notes: - we solely rely on guest level read-only mounts to enforce read-only state. The virtiofsd recommended way is to use read-only bind mounts [0], but doing so would require root. - we are not using DAX, because it still is still incomplete and apparently requires building Qemu (?) from source. In any event, it should not change anything functionally and be solely a performance improvement. Given that we are not regressing in terms of performance, this is strictly future work. I have adjusted the configs, but because I don't have Docker handy I can't really create those kernel. CI seems incapable of producing the artifacts without doing a fully-blown release dance. No idea what empty is about, really. I suspect the test failures we see are because it lacks support? Some additional resources worth keeping around: - https://virtio-fs.gitlab.io/howto-boot.html - https://virtio-fs.gitlab.io/howto-qemu.html [0] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/README.md?ref_type=heads#faq Closes: danobi#16 Closes: danobi#83 Signed-off-by: Daniel Müller <[email protected]>
danielocfb
pushed a commit
to d-e-s-o/vmtest
that referenced
this issue
Oct 1, 2024
Switch over to using virtiofsd for sharing file system data with the host. virtiofs is a file system designed for the needs of virtual machines and environments. That is in contrast to 9P fs, which we currently use for sharing data with the host, which is first and foremost a network file system. 9P is problematic if for no other reason that it lacks proper support for usage of the "open-unlink-fstat idiom", in which files are unlinked and later referenced via file descriptor (see danobi#83). virtiofs does not have this problem. This change replaces usage of 9P with that of virtiofs. In order to work, virtiofs needs a user space server. The current state-of-the-art implementation (virtiofsd) is implemented in Rust and so we interface directly with the library. Most of this code is extracted straight from virtiofsd, as it's a lot of boilerplate. An alternative approach is to install the binary via distribution packages or from crates.io, but availability (and discovery) can be a bit of a challenge. I benchmarked both the current master as well as this version with a bare-bones custom kernel: Benchmark 1: target/release/vmtest -k bzImage-9p 'echo test' Time (mean ± σ): 1.316 s ± 0.087 s [User: 0.462 s, System: 1.104 s] Range (min … max): 1.232 s … 1.463 s 10 runs Benchmark 1: target/release/vmtest -k bzImage-virtiofsd 'echo test' Time (mean ± σ): 1.244 s ± 0.011 s [User: 0.307 s, System: 0.358 s] Range (min … max): 1.227 s … 1.260 s 10 runs So it seems there is a slight speed up, on average (and significantly less system time being used). This is great, but I suspect a more pronounced speed advantage will be visible when working with large files, in which virtiofs is said to significantly outperform 9P (typically >2x from what I understand, but I have not done any benchmarks of that nature). A few other notes: - we solely rely on guest level read-only mounts to enforce read-only state. The virtiofsd recommended way is to use read-only bind mounts [0], but doing so would require root. - we are not using DAX, because it still is still incomplete and apparently requires building Qemu (?) from source. In any event, it should not change anything functionally and be solely a performance improvement. Given that we are not regressing in terms of performance, this is strictly future work. I have adjusted the configs, but because I don't have Docker handy I can't really create those kernel. CI seems incapable of producing the artifacts without doing a fully-blown release dance. No idea what empty is about, really. I suspect the test failures we see are because it lacks support? Some additional resources worth keeping around: - https://virtio-fs.gitlab.io/howto-boot.html - https://virtio-fs.gitlab.io/howto-qemu.html [0] https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/README.md?ref_type=heads#faq Closes: danobi#16 Closes: danobi#83 Signed-off-by: Daniel Müller <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
No description provided.
The text was updated successfully, but these errors were encountered: