Automatically detect .so needed

[tailhook]

Motivation

We want to build minimal containers. Currently, to make a minimal container for dynamically linked executable, we need config like this:

containers:
  build-swindon:
    setup:
    # .. build process
    - !Sh |
        mkdir -p /dist/bin /dist/lib64\
          /dist/usr/lib/x86_64-linux-gnu \
          /dist/lib/x86_64-linux-gnu
        cp /lib64/ld-linux-x86-64.so.2 /dist/lib64

        ldd target/release/swindon \
        | awk '/=> \//{print $3}' \
        | xargs -I{} -n1 cp '{}' /dist/usr/lib/x86_64-linux-gnu

        # dynamically loaded stdlib plugins
        cp \
          /lib/x86_64-linux-gnu/libresolv.so.2 \
          /lib/x86_64-linux-gnu/libnss_dns.so.2 \
          /dist/lib/x86_64-linux-gnu

        cp target/release/swindon /dist/bin/swindon
  swindon-deploy:
    setup:
    - !Build
      container: build-swindon
      source: /dist
      path: /

There are basically two things here:

1. Finding libraries by ldd
2. Copying known libraries for name resolution (a/k/a nss, a/k/a DNS)

Option 1

1. We should add rules option for build, like for Copy and Depends
2. We should add a special rule:

  swindon-deploy:
    setup:
    - !Build
      container: build-swindon
      rules:
      - /usr/bin/swindon#so-dependencies

Or:

  swindon-deploy:
    setup:
    - !Copy
      source: target/release/swindon
      rules:
      - /usr/bin/swindon#so-dependencies

Pros:

1. Easily supports multiple binaries
2. Integrates with Depends, Copy and Build

Cons:

1. Complexity of rules
2. Unclear what base paths are for Copy and Depends
3. Unclear now to fix "nss problem" (just document needed rules?)

Option 2a

(by @anti-social )

- !LddCopy
  path: /dist
  binaries:
  - /usr/bin/swindon

This is meant to copy swindon and it's dependencies into /dist folder, so that !Build command picks it up in the subcontainer.

Pros:

1. Do not complicate !Build command
2. May add additional options like nss

Cons:

1. Two commands to do the task in different containers (LddCopy and Build)
2. Two copies of binaries in the build container

Option 2b

- !LddCopy
  libraries-from-container: build
  binaries:
  - target/release/swindon
  # bin-dir: /usr/bin  # default value

Pros:

1. Single command
2. Works both for build artifacts in container and normal builds
3. Additional settings like nss are ok

Cons:

1. Somewhat overlaps functionality with Build
2. Another command which depends on building container
3. If path is absolute (binaries: [ /bin/bash ]) it unclear if binary is from currently building container or the dependency

Option 3

    - !Build
      container: build-swindon
      source-executables: /dist
      copy-nss: true  # detect and copy needed nss files for libc
      path: /

Pros:

1. Quite simple

Cons:

1. Unclear how to copy data files along with executables, so requires several !Build steps

Notes

It looks like possible to use elf crate or static-ldd to find out dependencies

Another motivating implementation by oracle.

Additionally, it's a good idea to make tutorials for common setup with rust, go as major use cases and evaluate scripting languages too.

/cc @anti-social

[anti-social]

I definitely don't like special syntax for rules. Think there should be separate command for that task.

[tailhook]

I definitely don't like special syntax for rules. Think there should be separate command for that task.

I don't get it. What do you propose? "Option 2" from above? I don't see how to make sane semantics for the command.

[anti-social]

Yes, I propose "Option 2":

- !LddCopy
  path: /dist
  binaries:
  - /usr/bin/swindon

Should copy /usr/bin/swindon into /dist/usr/bin/swindon and all the libraries into /dist/usr/lib/x86_64-linux-gnu

Possibly we could make configuration more detailed:

- !LddCopy
  lib_path: /dist
  binaries:
  - 
    source: target/release/swindon
    path: /dist/bin/swindon

[anti-social]

I think !Build and !Copy are already quite complex. I would not like to complicate them even more.

[tailhook]

Okay, added few options. It looks like 2b is my favorite now, but I'm not sure I satisfied with it.