Merge pull request #74 from flatcar/kai/flix-and-flatwrap

flix and flatwrap: Add tools to convert a chroot to a sysext

README.md

@@ -4,6 +4,7 @@
 Flatcar Container Linux as an OS without a package manager is a good fit for extension through systemd-sysext.
 The tools in this repository help you to create your own sysext images bundeling software to extend your base OS.
 The current focus is on Docker and containerd, contributions are welcome for other software.
+See the section at the end on how to bundle any software with the Flix and Flatwrap tools.
 
 ## Systemd-sysext
 
@@ -429,6 +430,73 @@ ls -1
 
 The script supports all vendors and clouds natively supported by Flatcar.
 
+### Flix and Flatwrap
+
+The Flix and Flatwrap tools both convert a given chroot folder into a systemd-sysext image.
+You have to specify which files should be made available to the host.
+
+The Flix tool rewrites specified binaries to use a custom library path.
+You also have to specify needed resource folders and you can specify systemd units, too.
+
+Here examples with Flix:
+
+```
+CMD="apk -U add b3sum" ./oci-rootfs.sh alpine:latest /var/tmp/alpine-b3sum
+./flix.sh /var/tmp/alpine-b3sum/ b3sum /usr/bin/b3sum /bin/busybox:/usr/bin/busybox
+# got b3sum.raw
+
+CMD="apt-get update && apt install -y nginx" ./oci-rootfs.sh debian /var/tmp/debian-nginx
+./flix.sh /var/tmp/debian-nginx/ nginx /usr/sbin/nginx /usr/sbin/start-stop-daemon /usr/lib/systemd/system/nginx.service
+# got nginx.raw
+
+# Note: Enablement of nginx.service with Butane would happen as in the k3s example
+# but you can also pre-enable the service inside the extension.
+# Here a non-production nginx test config if you want to try the above:
+$ cat /etc/nginx/nginx.conf
+user root;
+pid /run/nginx.pid;
+
+events {
+}
+
+http {
+  access_log /dev/null;
+  proxy_temp_path /tmp;
+  client_body_temp_path /tmp;
+  fastcgi_temp_path /tmp;
+  uwsgi_temp_path /tmp;
+  scgi_temp_path /tmp;
+  server {
+        server_name   localhost;
+        listen        127.0.0.1:80;
+  }
+}
+```
+
+The Flatwrap tool generates entry point wrappers for a chroot with `unshare` or `bwrap`.
+You can specify systemd units, too. By default `/etc`, `/var`, and `/home` are mapped from the host but that is configurable (see `--help`).
+
+Here examples with Flatwrap:
+
+```
+CMD="apk -U add b3sum" ./oci-rootfs.sh alpine:latest /var/tmp/alpine-b3sum
+./flatwrap.sh /var/tmp/alpine-b3sum b3sum /usr/bin/b3sum /bin/busybox:/usr/bin/busybox
+# got b3sum.raw
+
+CMD="apk -U add htop" ./oci-rootfs.sh alpine:latest /var/tmp/alpine-htop
+# Use ETCMAP=chroot because alpine's htop needs alpine's /etc/terminfo
+ETCMAP=chroot ./flatwrap.sh /var/tmp/alpine-htop htop /usr/bin/htop
+# got htop.raw
+
+CMD="apt-get update && apt install -y nginx" ./oci-rootfs.sh debian /var/tmp/debian-nginx
+./flatwrap.sh /var/tmp/debian-nginx/ nginx /usr/sbin/nginx /usr/sbin/start-stop-daemon /usr/lib/systemd/system/nginx.service
+# got nginx.raw
+
+# Note: Enablement of nginx.service with Butane would happen as in the k3s example
+# but you can also pre-enable the service inside the extension.
+# (The "non-production" nginx test config above can be used here, too, stored on the host's /etc.)
+```
+
 ### Converting a Torcx image
 
 Torcx was a solution for switching between different Docker versions on Flatcar.

flatwrap.sh

@@ -0,0 +1,137 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+export ARCH="${ARCH-amd64}"
+KEEP="${KEEP-}"
+ETCMAP="${ETCMAP-host}"
+VARMAP="${VARMAP-host}"
+HOMEMAP="${HOMEMAP-host}"
+CHROOT="${CHROOT-/usr /lib /lib64 /bin /sbin}"
+HOST="${HOST-/dev /proc /sys /run /tmp /var/tmp}"
+SCRIPTFOLDER="$(dirname "$(readlink -f "$0")")"
+
+if [ $# -lt 3 ] || [ "$1" = "-h" ] || [ "$1" = "--help" ]; then
+  echo "Usage: $0 FOLDER SYSEXTNAME PATHS..."
+  echo "The script will set up entry points for the specified binary or systemd unit paths (e.g., /usr/bin/nano, /usr/systemd/system/my.service) from FOLDER into a chroot under /usr/local/, and create a systemd-sysext squashfs image with the name SYSEXTNAME.raw in the current folder."
+  echo "Paths under /usr are recommended but paths under /etc or /bin can also be specified as 'CHROOT:TARGET', e.g., '/etc/systemd/system/my.service:/usr/systemd/system/my.service' or '/bin/mybin:/usr/bin/mybin' supported."
+  echo "Since only the specified paths are available in the host, any files accessed by systemd for service units must also be specified."
+  echo "The binary itself will be able to access all files of the chroot as specifed in the CHROOT environment variable (current value is '${CHROOT}')."
+  echo "It will also be able to access all files of the host as specified in the HOST environment variable and to /etc, /var, /home if not disabled below (current value is '${HOST}')."
+  echo "The mapping of /etc, /var, /home from host or the chroot can be controlled with the ETCMAP/VARMAP/HOMEMAP environment variables by setting them to 'chroot' (current values are '${ETCMAP}', '${VARMAP}', '${HOMEMAP}')."
+  echo "The binaries will be spawned with bwrap if available for non-root users. When bwrap is missing, an almost equivalent combination of unshare commands is used."
+  echo "For testing, pass KEEP=1 as environment variable (current value is '${KEEP}') and run the binaries with [sudo] FLATWRAP_ROOT=SYSEXTNAME SYSEXTNAME/usr/bin/binary."
+  echo
+  echo "A temporary directory named SYSEXTNAME in the current folder will be created and deleted again."
+  echo "All files in the sysext image will be owned by root."
+  echo "To use a different architecture than amd64 pass 'ARCH=arm64' as environment variable (current value is '${ARCH}')."
+  "${SCRIPTFOLDER}"/bake.sh --help
+  exit 1
+fi
+
+FOLDER="$1"
+SYSEXTNAME="$2"
+shift
+shift
+PATHS=("$@")
+
+if [ "${ETCMAP}" = host ]; then
+  HOST+=" /etc"
+else
+  CHROOT+=" /etc"
+fi
+if [ "${VARMAP}" = host ]; then
+  HOST+=" /var"
+else
+  CHROOT+=" /var"
+fi
+if [ "${HOMEMAP}" = host ]; then
+  HOST+=" /home"
+else
+  CHROOT+=" /home"
+fi
+# Make sure to mount /var before /var/tmp
+HOST=$(echo "${HOST}" | tr ' ' '\n' | sort)
+CHROOT=$(echo "${CHROOT}" | tr ' ' '\n' | sort)
+
+rm -rf "${SYSEXTNAME}"
+mkdir -p "${SYSEXTNAME}/usr/local/${SYSEXTNAME}"
+mkdir -p "${SYSEXTNAME}/usr/local/${SYSEXTNAME}/mount-dir"
+
+cp -ar "${FOLDER}/." "${SYSEXTNAME}/usr/local/${SYSEXTNAME}"
+
+CARGS=() # CHROOT unshare bind mounts
+BWCARGS=() # CHROOT bwrap bind mounts
+for DIR in ${CHROOT}; do
+  CHROOTDIR="${SYSEXTNAME}/usr/local/${SYSEXTNAME}/${DIR}"
+  if [ ! -L "${CHROOTDIR}" ] && [ ! -e "${CHROOTDIR}" ]; then
+    continue
+  fi
+  CHROOTDIR=$(realpath -m --relative-base="${SYSEXTNAME}/usr/local/${SYSEXTNAME}" "${SYSEXTNAME}/usr/local/${SYSEXTNAME}/${DIR}")
+  CHROOTDIR="${SYSEXTNAME}/usr/local/${SYSEXTNAME}/${CHROOTDIR}"
+  CARGS+=(mkdir -p "\${FLATWRAP_ROOT-}/usr/local/${SYSEXTNAME}/mount-dir/${DIR}" "&&" mount --rbind "\${FLATWRAP_ROOT-}/usr/local/${SYSEXTNAME}/${DIR}" "\${FLATWRAP_ROOT-}/usr/local/${SYSEXTNAME}/mount-dir/${DIR}" "&&")
+  BWCARGS+=(--bind "\${FLATWRAP_ROOT-}/usr/local/${SYSEXTNAME}/${DIR}" "${DIR}")
+done
+HARGS=() # HOST unshare bind mounts
+BWHARGS=() # HOST bwrap bind mounts
+for DIR in ${HOST}; do
+  HARGS+=(mkdir -p "\${FLATWRAP_ROOT-}/usr/local/${SYSEXTNAME}/mount-dir/${DIR}" "&&" mount --rbind "${DIR}" "\${FLATWRAP_ROOT-}/usr/local/${SYSEXTNAME}/mount-dir/${DIR}" "&&")
+  BWHARGS+=(--bind "${DIR}" "${DIR}")
+done
+# The above could be moved into the helper below to allow controlling the mapping at runtime
+# and the helpers could be put into one file, controlled by a different first argument.
+
+# Helper for priv chroot setup
+tee "${SYSEXTNAME}/usr/local/${SYSEXTNAME}/unshare-helper" > /dev/null <<EOF
+#!/bin/sh
+# Instead of chroot we run unshare --root --wd to keep PWD (but no new namespaces are created)
+mount -t tmpfs tmpfs "\${FLATWRAP_ROOT-}/usr/local/${SYSEXTNAME}/mount-dir" && ${CARGS[@]} ${HARGS[@]} exec unshare --root "\${FLATWRAP_ROOT-}/usr/local/${SYSEXTNAME}/mount-dir" --wd "\${PWD}" "\$@"
+EOF
+chmod +x "${SYSEXTNAME}/usr/local/${SYSEXTNAME}/unshare-helper"
+
+# Helper for unpriv chroot setup (when bwrap is not available)
+tee "${SYSEXTNAME}/usr/local/${SYSEXTNAME}/unshare-helper-unpriv" > /dev/null <<EOF
+#!/bin/sh
+U="\$1"
+G="\$2"
+shift
+shift
+mount -t tmpfs tmpfs "\${FLATWRAP_ROOT-}/usr/local/${SYSEXTNAME}/mount-dir" && ${CARGS[@]} ${HARGS[@]} exec unshare --map-user="\$U" --map-user="\$G" -U --root "\${FLATWRAP_ROOT-}/usr/local/${SYSEXTNAME}/mount-dir" --wd "\${PWD}" "\$@"
+EOF
+chmod +x "${SYSEXTNAME}/usr/local/${SYSEXTNAME}/unshare-helper-unpriv"
+
+for ENTRY in "${PATHS[@]}"; do
+  NEWENTRY="${ENTRY}"
+  if [[ "${ENTRY}" != /usr/* ]]; then
+    NEWENTRY=$(echo "${ENTRY}" | cut -d : -f 2)
+    ENTRY=$(echo "${ENTRY}" | cut -d : -f 1)
+    if [ "${ENTRY}" = "${NEWENTRY}" ] || [ "${NEWENTRY}" = "" ] || [[ "${NEWENTRY}" != /usr/* ]]; then
+      echo "Error: '${ENTRY}' should be passed with ':/usr/TARGET'" >&2; exit 1
+    fi
+  fi
+  DIR=$(dirname "${NEWENTRY}")
+  mkdir -p "${SYSEXTNAME}/${DIR}"
+  if [ -L "${FOLDER}/${ENTRY}" ]; then
+    TARGET=$(realpath -m --relative-base="${FOLDER}" "${FOLDER}/${ENTRY}")
+    ln -fs "/usr/local/${SYSEXTNAME}/${TARGET}" "${SYSEXTNAME}/${NEWENTRY}"
+  elif [ -d "${FOLDER}/${ENTRY}" ] || [ ! -x "${FOLDER}/${ENTRY}" ]; then
+    NAME=$(basename "${NEWENTRY}")
+    ln -fs --no-target-directory "/usr/local/${SYSEXTNAME}/${ENTRY}" "${SYSEXTNAME}/${DIR}/${NAME}"
+  else
+      tee "${SYSEXTNAME}/${NEWENTRY}" > /dev/null <<EOF
+#!/bin/sh
+if [ "\$(id -u)" = 0 ]; then
+  exec unshare -m "\${FLATWRAP_ROOT-}/usr/local/${SYSEXTNAME}/unshare-helper" "${ENTRY}" "\$@"
+elif command -v bwrap >/dev/null && [ "\${NOBWRAP-}" = "" ]; then
+  exec bwrap ${BWCARGS[@]} ${BWHARGS[@]} "${ENTRY}" "\$@"
+else
+  exec unshare -m -U -r "\${FLATWRAP_ROOT-}/usr/local/${SYSEXTNAME}/unshare-helper-unpriv" "\$(id -u)" "\$(id -g)" "${ENTRY}" "\$@"
+fi
+EOF
+    chmod +x "${SYSEXTNAME}/${NEWENTRY}"
+  fi
+done
+
+RELOAD=1 "${SCRIPTFOLDER}"/bake.sh "${SYSEXTNAME}"
+if [ "${KEEP}" != 1 ]; then
+  rm -rf "${SYSEXTNAME}"
+fi

flix.sh

@@ -0,0 +1,138 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+export ARCH="${ARCH-amd64}"
+EXTRALIBS="${EXTRALIBS-}"
+KEEP="${KEEP-}"
+SCRIPTFOLDER="$(dirname "$(readlink -f "$0")")"
+
+if [ $# -lt 3 ] || [ "$1" = "-h" ] || [ "$1" = "--help" ]; then
+  echo "Usage: $0 FOLDER SYSEXTNAME PATHS..."
+  echo "The script will extract the specified binary paths (e.g., /usr/bin/nano) or resource paths (e.g., /usr/share/nano/) from FOLDER, resolve the dynamic libraries, and create a systemd-sysext squashfs image with the name SYSEXTNAME.raw in the current folder."
+  echo "Paths under /usr are recommended but paths under /etc or /bin can also be specified as 'CHROOT:TARGET', e.g., '/etc/systemd/system/my.service:/usr/systemd/system/my.service' or '/bin/mybin:/usr/bin/mybin' supported (but not if they are symlinks under /bin/)."
+  echo "Since dynamic libraries should not conflict, you must not pass libraries in PATHS."
+  echo "If a particular library is needed for dlopen, pass EXTRALIBS as space-separated environment variable (current value is '${EXTRALIBS}')."
+  echo "Note: The resolving of libraries copies them in one shared folder and might not cover all use cases."
+  echo "E.g., specifying a folder with binaries does not work, each one has to be specified separately."
+  echo "For testing, pass KEEP=1 as environment variable (current value is '${KEEP}') and run the binaries with bwrap --bind /proc /proc --bind SYSEXTNAME/usr /usr /usr/bin/BINARY."
+  echo
+  echo "A temporary directory named SYSEXTNAME in the current folder will be created and deleted again."
+  echo "All files in the sysext image will be owned by root."
+  echo "To use a different architecture than amd64 pass 'ARCH=arm64' as environment variable (current value is '${ARCH}')."
+  "${SCRIPTFOLDER}"/bake.sh --help
+  exit 1
+fi
+
+FOLDER="$1"
+SYSEXTNAME="$2"
+shift
+shift
+PATHS=("$@")
+
+if ! command -v patchelf >/dev/null; then
+  echo "Error: patchelf missing" >&2
+  exit 1
+fi
+
+# Map library name to found library location
+declare -A FOUND_DEPS=()
+find_deps() {
+  local FROM="$1"
+  local TO="$2"
+  local FILE="$3" # Should be the copied file
+  local DEP=
+  local FOUND=
+  local LIB_PATHS=
+  local NEW_RPATHS=
+  local RP=
+  LIB_PATHS=("${FROM}/lib64" "${FROM}/usr/lib64" "${FROM}/usr/local/lib64" "${FROM}/lib" "${FROM}/usr/lib" "${FROM}/usr/local/lib")
+  for RP in $({ cat "${FROM}"/etc/ld.so.conf.d/* "${FROM}"/etc/ld.so.conf 2> /dev/null || true ; } | { grep -Pv '^(#|include )' || true ; }); do
+    LIB_PATHS+=("${RP}")
+  done
+  for DEP in $(patchelf --print-needed "${FILE}"); do
+    if [ "${FOUND_DEPS["${DEP}"]-}" != "" ]; then
+      continue
+    fi
+    FOUND=$({ find "${LIB_PATHS[@]}" -name "${DEP}" 2>/dev/null || true ;} | head -n 1)
+    if [ "${FOUND}" = "" ]; then
+      echo "Error: Library ${DEP} not found in ${LIB_PATHS[*]}" >&2; exit 1
+    fi
+    FOUND=$(echo -n "${FOLDER}/"; realpath -m --relative-base="${FOLDER}" "${FOUND}")
+    cp -a "${FOUND}" "${TO}/usr/local/${SYSEXTNAME}/${DEP}"
+    FOUND_DEPS["${DEP}"]="${FOUND}"
+    find_deps "${FROM}" "${TO}" "${TO}/usr/local/${SYSEXTNAME}/${DEP}"
+  done
+  NEW_RPATHS="/usr/local/${SYSEXTNAME}:/usr/local/${SYSEXTNAME}/extralibs"
+  for RP in $(patchelf --print-rpath "${FILE}" | tr ':' ' '); do
+    if [[ "${RP}" == *"\$ORIGIN"* ]]; then
+      echo "Warning: Ignored rpath ${RP}"
+      continue
+    fi
+    if [ ! -e "${SYSEXTNAME}/usr/local/${SYSEXTNAME}/${RP}" ]; then
+      mkdir -p "${SYSEXTNAME}/usr/local/${SYSEXTNAME}/${RP}"
+      cp -ar "${FROM}/${RP}/." "${SYSEXTNAME}/usr/local/${SYSEXTNAME}/${RP}"
+    fi
+    NEW_RPATHS+=":/usr/local/${SYSEXTNAME}/${RP}"
+  done
+  patchelf --no-default-lib --set-rpath "${NEW_RPATHS}" "${FILE}"
+}
+
+rm -rf "${SYSEXTNAME}"
+mkdir -p "${SYSEXTNAME}/usr/local/${SYSEXTNAME}"
+
+for ENTRY in "${PATHS[@]}"; do
+  NEWENTRY="${ENTRY}"
+  if [[ "${ENTRY}" != /usr/* ]]; then
+    NEWENTRY=$(echo "${ENTRY}" | cut -d : -f 2)
+    ENTRY=$(echo "${ENTRY}" | cut -d : -f 1)
+    if [ "${ENTRY}" = "${NEWENTRY}" ] || [ "${NEWENTRY}" = "" ] || [[ "${NEWENTRY}" != /usr/* ]]; then
+      echo "Error: '${ENTRY}' should be passed with ':/usr/TARGET'" >&2; exit 1
+    fi
+  fi
+  DIR=$(dirname "${NEWENTRY}")
+  mkdir -p "${SYSEXTNAME}/${DIR}"
+  if [ ! -L "${FOLDER}/${ENTRY}" ] && [ -d "${FOLDER}/${ENTRY}" ]; then
+    cp -ar "${FOLDER}/${ENTRY}/." "${SYSEXTNAME}/${NEWENTRY}"
+  else
+    cp -a "${FOLDER}/${ENTRY}" "${SYSEXTNAME}/${NEWENTRY}"
+    if [ -L "${FOLDER}/${ENTRY}" ]; then
+      TARGET=$(realpath -m --relative-base="${FOLDER}" "${FOLDER}/${ENTRY}")
+      DIR=$(dirname "${TARGET}")
+      mkdir -p "${SYSEXTNAME}/${DIR}"
+      if [ -d "${FOLDER}/${TARGET}" ]; then
+        cp -ar "${FOLDER}/${TARGET}/." "${SYSEXTNAME}/${TARGET}"
+      else
+        cp -a "${FOLDER}/${TARGET}" "${SYSEXTNAME}/${TARGET}"
+        # Check if we need to patch the target file
+        ENTRY="${TARGET}"
+      fi
+    fi
+  fi
+  INTERP=
+  if [ ! -L "${SYSEXTNAME}/${NEWENTRY}" ] && [ -f "${SYSEXTNAME}/${NEWENTRY}" ]; then
+    INTERP=$(patchelf --print-interpreter "${SYSEXTNAME}/${NEWENTRY}" 2>/dev/null || true)
+  fi
+  if [ "${INTERP}" != "" ]; then
+    INTERP_NAME=$(basename "${INTERP}")
+    if [ ! -f "${SYSEXTNAME}/usr/local/${SYSEXTNAME}/${INTERP}" ]; then
+      INTERP=$(realpath -m --relative-base="${FOLDER}" "${FOLDER}/${INTERP}")
+      cp -a "${FOLDER}/${INTERP}" "${SYSEXTNAME}/usr/local/${SYSEXTNAME}/${INTERP_NAME}"
+    fi
+    patchelf --set-interpreter "/usr/local/${SYSEXTNAME}/${INTERP_NAME}" "${SYSEXTNAME}/${NEWENTRY}"
+    find_deps "${FOLDER}" "${SYSEXTNAME}" "${SYSEXTNAME}/${NEWENTRY}"
+  fi
+done
+for ENTRY in ${EXTRALIBS}; do
+  DIR=$(dirname "${ENTRY}")
+  NAME=$(basename "${ENTRY}")
+  mkdir -p "${SYSEXTNAME}/${DIR}"
+  cp -a "${FOLDER}/${ENTRY}" "${SYSEXTNAME}/usr/local/${SYSEXTNAME}/extralibs/${NAME}"
+  if [ ! -L "${FOLDER}/${ENTRY}" ] && [ -f "${FOLDER}/${ENTRY}" ]; then
+    find_deps "${FOLDER}" "${SYSEXTNAME}" "${FOLDER}/${ENTRY}"
+  fi
+done
+
+RELOAD=1 "${SCRIPTFOLDER}"/bake.sh "${SYSEXTNAME}"
+if [ "${KEEP}" != 1 ]; then
+  rm -rf "${SYSEXTNAME}"
+fi