Replace the dummy console device by a true console device #978

Fred78290 · 2024-12-18T16:04:51Z

The goal of this pull request is to replace the unusable dummy console device by a useful console device

In fact, the virtio socket is created in the guest machine but the host virtio socket is not created.

Now the console device is renamed tart-agent and the host socket file named tart-agent.sock is create inside the VM directory.

By this way, we can add some agent like incus or LXD to communicate between host and guest.

edigaryev · 2024-12-18T18:41:37Z

Sources/tart/VM.swift


+      if FileManager.default.fileExists(atPath: consoleURL.absoluteURL.path()) == false {
+        FileManager.default.createFile(atPath: consoleURL.absoluteURL.path(), contents: nil)


It seems to me that this won't create a socket, just a regular file 🤔

It's a virtio socket not an unix domain socket.

It's not necessary to set some control flags like unix socket and if it will be necessary it will specified in the apple doc....

How do you use such a file then from the host?

see below an example.

After it's possible to implement some protocoled agent like gRPC, http/https.....
it's a pipe.

it's a pipe.

I'm sorry, but it's not a pipe. Nor a socket, which you call it in #978 (comment) and by giving it a .sock extension.

It's a regular file stored on disk.

Moreover, you're using it for both for reading and writing, which will mess up the file contents badly if you'd ever want to implement any bidirectional communication on top of it.

Moreover, you're using it for both for reading and writing, which will mess up the file contents badly if you'd ever want to implement any bidirectional communication on top of it.

I don't see really the problem because it's a feature enabled by VZ. With a unix domain socket too you have the same problem.

You can write garbage to socket also, the effect is the same.

So it's could be replace by a tty...

edigaryev · 2024-12-18T18:41:42Z

By this way, we can add some agent like incus or LXD to communicate between host and guest.

Would you mind explaining your use-case a bit more in-depth, perhaps with some examples?

Also, it generally feels like this should be an optional feature that is configured through a command-line argument, possibly allowing multiple console devices (or Virtualization.Framework's sockets) to be specified.

Fred78290 · 2024-12-19T08:40:49Z

Would you mind explaining your use-case a bit more in-depth, perhaps with some examples?

For example:

implement a nice shutdown VM thru command shutdown now instead kill app.
Get the VM ip more efficiently without ARP lookup
Do tart shell command without SSH

For concrete example take a look with lima-vm, multipass, lxd, incus...

Also, it generally feels like this should be an optional feature that is configured through a command-line argument, possibly allowing multiple console devices (or Virtualization.Framework's sockets) to be specified.

It's not the same. The vz socket is only to communicate between VM and the hosting program hosting the VM. The socket is not available outside.

With the console device:
inside guest:

ll /dev/virtio-ports/
total 0
drwxr-xr-x  2 root root   80 Dec 19 09:25 ./
drwxr-xr-x 18 root root 3880 Dec 19 09:25 ../
lrwxrwxrwx  1 root root   11 Dec 19 09:25 com.redhat.spice.0 -> ../vport6p0
lrwxrwxrwx  1 root root   11 Dec 19 09:25 tart-agent -> ../vport7p0

ubuntu:~$ sudo cat /dev/virtio-ports/tart-agent 
hello world

on the host:

echo "hello world" > ${TART_HOME}/vms/ubuntu/tart-agent.sock

edigaryev · 2024-12-19T10:17:19Z

For concrete example take a look with lima-vm, multipass, lxd, incus...

Do these solutions use VirtIO console device to communicate with the guest?

Are they also using regular files on the host's side?

implement a nice shutdown VM thru command shutdown now instead kill app.

Do tart shell command without SSH

Why not use SSH for that? 🤔

Get the VM ip more efficiently without ARP lookup

Could you elaborate a bit on the inefficiency of querying the system's ARP table?

But I agree that when using --net-bridged, the IP retrieval process becomes a bit less robust because it expects the VM to advertise itself in the broadcast domain.

However, with the right setup it just works and is simple enough, compared to requiring the user to install the in-VM agent, supporting it, and teaching Tart to communicate with it properly.

Fred78290 · 2024-12-19T10:49:35Z

Do these solutions use VirtIO console device to communicate with the guest?

yes!

Why not use SSH for that? 🤔

Agent is avalable earlier than network. and for macos VM you need to enable session sharing...

However, with the right setup it just works and is simple enough, compared to requiring the user to install the in-VM agent, supporting it, and teaching Tart to communicate with it properly.

This is the big missing part of tart: you dropped cloud-init support for linux machine.... so i developed a tool using tart allowing to boot linux machine with cloud-init support enabled and many other features such as use qcow2 cloud images directly.

Fred78290 · 2024-12-21T14:45:34Z

@edigaryev

I did replaced the regular file with a true socket. Take a look at console-devices-nio

I did also extras implementation:

with a fifo file: console-devices-fifo
with pty: console-devices-pty

But I got some limitations....

Make lint happy

edigaryev · 2024-12-23T13:48:43Z

yes!

Could you please point me to a file/code block in lima-vm/lima that implements agent communication via VirtIO console device?

Agent is avalable earlier than network. and for macos VM you need to enable session sharing...

Well, you still need to prepare the VM, either by:

enabling the Remote Login
or by installing the agent

As for the "earlier", have you measured the delay that the network bring-up causes, compared to communicating with the agent?

This is the big missing part of tart: you dropped cloud-init support for linux machine....

I don't get it. We use cloud-init on a daily basis in cirruslabs/linux-image-templates repository.

In which commit/PR did we drop it?

so i developed a tool using tart allowing to boot linux machine with cloud-init support enabled and many other features such as use qcow2 cloud images directly.

Interesting! Is it published on GitHub?

Fred78290 · 2024-12-23T14:31:25Z

Could you please point me to a file/code block in lima-vm/lima that implements agent communication via VirtIO console device?

daemon_linux.go

console device is special case of socket devices, could also use vsock if present. But on MacOS you don't vsock so the alternative is the console device.

Else I start coded also the usage of VZVirtioSocketDevice with swift-nio. It's work fine, still to catch the remote close of socket by the guest, it miss a delegate method :)

Well, you still need to prepare the VM, either by:
1. enabling the `Remote Login`

2. or by installing the agent

Done thru cloud-init with cloud images but need to wait network fully configured...

Sample user-data to create an Ubuntu desktop with tart and my tool.

#cloud-config
package_update: true
package_upgrade: true
timezone: Europe/Paris
growpart:
  mode: auto
  devices: ["/"]
  ignore_growroot_disabled: false
runcmd:
- hostnamectl set-hostname noble-cloud-image
packages:
- ubuntu-desktop-minimal
- xrdp
- spice-vdagent

Or already installed inside your OCI image. The faster boot

As for the "earlier", have you measured the delay that the network bring-up causes, compared to communicating with the agent?

This is the big missing part of tart: you dropped cloud-init support for linux machine....

I don't get it. We use cloud-init on a daily basis in cirruslabs/linux-image-templates repository.

In which commit/PR did we drop it?

99_cirruslabs.cfg

Is: datasource_list: [ None ]

Should be: datasource_list: [ NoCloud, None ]

so i developed a tool using tart allowing to boot linux machine with cloud-init support enabled and many other features such as use qcow2 cloud images directly.

Interesting! Is it published on GitHub?

On GitHub but private repo until first release is ready

Some usage:

caked help
OVERVIEW:
caked is a tool to wrap tart command and add some features like run as daemon, build VM from cloud image

USAGE: caked <subcommand or tart subcommand>

OPTIONS:
  --version               Show the version.
  -h, --help              Show help information.

SUBCOMMANDS:
  service                 caked as launchctl agent
  certificates            Generate tls certificates for grpc
  build                   Create a linux VM and initialize it with cloud-init
  start                   Run linux VM in background
  stop                    Run linux VM in background
  launch                  Create a linux VM, initialize it with cloud-init and launch in background
  configure               Reconfigure VM
  waitip                  Run linux VM in background
  remote                  Manage simplestream remote
  networks                      List host network devices (physical interfaces, virtual switches, bridges) available
                                to integrate with using the `--bridged` switch to the `launch` command
  purge                   Purge caches or local VMs

  See 'caked help <subcommand>' for detailed help.

caked launch --help
OVERVIEW: Create a linux VM, initialize it with cloud-init and launch in background

USAGE: caked launch [<options>] --name <name> [<url>]

ARGUMENTS:
  <url>                   create a linux VM using a cloud image (default: https://cloud-images.ubuntu.com/releases/noble/release/ubuntu-24.04-server-cloudimg-arm64.img)
        The image could be one of local raw image, qcow2 cloud image, lxc simplestreams image, oci image
        The url image form are:
        - local images: /Users/myhome/disk.img or file:///Users/myhome/disk.img
        - cloud images: https://cloud-images.ubuntu.com/releases/noble/release/ubuntu-24.04-server-cloudimg-arm64.img
        - lxc images: images:ubuntu/noble/cloud, see remote command for detail
        - secure oci images: ocis://ghcr.io/cirruslabs/ubuntu:latest (https)
        - unsecure oci images: oci://unsecure.com/ubuntu:latest (http)

OPTIONS:
  -n, --name <name>       VM name
  -c, --cpu <cpu>         Number of VM CPUs (default: 1)
  -m, --memory <memory>   VM memory size in megabytes (default: 512)
  -d, --disk-size <disk-size>
                          Disk size in GB (default: 10)
  -u, --user <user>       The user to use for the VM (default: admin)
  -w, --password <password>
                          The user password for login, none by default
  -g, --main-group <main-group>
                          The main existing group for the user (default: admin)
  -k, --clear-password    Tell if the user admin allow password for ssh
  -a, --autostart         Tell if the VM must be start at boot
  -t, --nested            Enable nested virtualization if possible
  -i, --ssh-authorized-key <path>
                          Optional ssh-authorized-key file path for linux VM
  --vendor-data <path>    Optional cloud-init vendor-data file path for linux VM
  --user-data, --cloud-init <Path or URL to a user-data cloud-init configuration, or '-' for stdin>
                          Optional cloud-init user-data file path for linux VM
  --network-config <path> Optional cloud-init network-config file path for linux VM
  --display-refit         Whether to automatically reconfigure the VM's display to fit the window
  -p, --publish <host:guest/(tcp|udp|both)>
                          Optional forwarded port for VM, syntax like docker
  -v, --mount <[name:]path[:options]>
                          Additional directory shares with an optional read-only and mount tag options (e.g. --dir="~/src/build" or --dir="~/src/sources:ro")
        See tart help for more infos
  --net-bridged, --bridged <interface name>
                          Use bridged networking instead of the default shared (NAT) networking 
                          (e.g. --net-bridged=en0 or --net-bridged="Wi-Fi")
        See tart help for more infos
  --wait-ip-timeout <wait-ip-timeout>
                          Maximum of seconds to getting IP (default: 180)
  --version               Show the version.
  -h, --help              Show help information.

Fred78290 · 2025-01-04T18:27:49Z

Will be replaced by #985

Fred78290 added 2 commits December 18, 2024 16:53

FEAT: Replace dummy console device by a true console device

696cb5f

TYPO: remove dummy word

9a472ef

Fred78290 requested review from edigaryev and fkorotkov as code owners December 18, 2024 16:04

edigaryev reviewed Dec 18, 2024

View reviewed changes

Fred78290 added 2 commits December 21, 2024 15:48

Update VM.swift

d67d484

Make lint happy

Merge branch 'cirruslabs:main' into console-devices

68c05db

Fred78290 mentioned this pull request Dec 29, 2024

Add socket devices and console device thru unix socket or file descriptor support. Aka qemu #985

Open

Fred78290 closed this Jan 4, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace the dummy console device by a true console device #978

Replace the dummy console device by a true console device #978

Fred78290 commented Dec 18, 2024

edigaryev Dec 18, 2024

Fred78290 Dec 19, 2024

edigaryev Dec 19, 2024

Fred78290 Dec 19, 2024

edigaryev Dec 19, 2024

Fred78290 Dec 19, 2024

edigaryev commented Dec 18, 2024

Fred78290 commented Dec 19, 2024

edigaryev commented Dec 19, 2024

Fred78290 commented Dec 19, 2024

Fred78290 commented Dec 21, 2024

edigaryev commented Dec 23, 2024 •

edited

Loading

Fred78290 commented Dec 23, 2024

Fred78290 commented Jan 4, 2025


		if FileManager.default.fileExists(atPath: consoleURL.absoluteURL.path()) == false {
		FileManager.default.createFile(atPath: consoleURL.absoluteURL.path(), contents: nil)

Replace the dummy console device by a true console device #978

Replace the dummy console device by a true console device #978

Conversation

Fred78290 commented Dec 18, 2024

edigaryev Dec 18, 2024

Choose a reason for hiding this comment

Fred78290 Dec 19, 2024

Choose a reason for hiding this comment

edigaryev Dec 19, 2024

Choose a reason for hiding this comment

Fred78290 Dec 19, 2024

Choose a reason for hiding this comment

edigaryev Dec 19, 2024

Choose a reason for hiding this comment

Fred78290 Dec 19, 2024

Choose a reason for hiding this comment

edigaryev commented Dec 18, 2024

Fred78290 commented Dec 19, 2024

edigaryev commented Dec 19, 2024

Fred78290 commented Dec 19, 2024

Fred78290 commented Dec 21, 2024

edigaryev commented Dec 23, 2024 • edited Loading

Fred78290 commented Dec 23, 2024

Fred78290 commented Jan 4, 2025

edigaryev commented Dec 23, 2024 •

edited

Loading