From 22e06ae9f78c7ac2da3aa597f6bf5fb2d23536d1 Mon Sep 17 00:00:00 2001 From: Alexis Girault Date: Fri, 13 Dec 2024 18:32:05 -0500 Subject: [PATCH] Improvements to run list and run build (#639) * Improve ./run list results Signed-off-by: Alexis Girault * Allow to build operators with run script Also: - verify input validity using the improve listing from the last commit - allow passing a path to the app/operator dir, for improve UX by leveraging terminal autocompletion Signed-off-by: Alexis Girault * Allow to build packages with run script Signed-off-by: Alexis Girault * Style improvement to remaining _desc() Signed-off-by: Alexis Girault * Update README for build instructions Signed-off-by: Alexis Girault * Update README.md Co-authored-by: Tom Birdsong <40648863+tbirdso@users.noreply.github.com> Signed-off-by: Alexis Girault --------- Signed-off-by: Alexis Girault Signed-off-by: Alexis Girault Co-authored-by: Tom Birdsong <40648863+tbirdso@users.noreply.github.com> --- README.md | 83 +++++++++-------- run | 260 ++++++++++++++++++++++++++++++++---------------------- 2 files changed, 203 insertions(+), 140 deletions(-) diff --git a/README.md b/README.md index 0a07273a4..b94f7caa5 100644 --- a/README.md +++ b/README.md @@ -25,12 +25,13 @@ This is a central repository for the NVIDIA Holoscan AI sensor processing commun This repository is a collection of applications and extensions created by the Holoscan AI sensor processing community. The following directories make up the core of this repo: -- Example applications: Visit [`applications`](./applications) to explore an evolving collection of example +- **Example applications**: Visit [`applications`](./applications) to explore an evolving collection of example applications built on the NVIDIA Holoscan platform. Examples are available from NVIDIA, partners, and - community collaborators. -- Community components: Visit [`operators`](./operators/) and [`gxf_extensions`](./gxf_extensions) to explore + community collaborators, and provide a demonstration of the SDK capabilities. +- **Community components**: Visit [`operators`](./operators/) and [`gxf_extensions`](./gxf_extensions) to explore reusable Holoscan modules. -- Tutorials: Visit [`tutorials`](./tutorials/) for extended walkthroughs and tips for the Holoscan platform. +- **Package configurations**: Visit [`pkg`](./pkg/) for a list of debian package to generate, to distribute operators and applications for easier development. +- **Tutorials**: Visit [`tutorials`](./tutorials/) for extended walkthroughs and tips for the Holoscan platform. Visit the [Holoscan SDK User Guide](https://docs.nvidia.com/holoscan/sdk-user-guide/overview.html) to learn more about the NVIDIA Holoscan AI sensor processing platform. You can also chat with the [Holoscan-GPT](https://chatgpt.com/g/g-M6hMJimGa-holochatgpt) Large Language Model to learn about using Holoscan SDK, ask questions, and get code help. Holoscan-GPT requires an OpenAI account. @@ -52,7 +53,7 @@ We strongly recommend new users follow our [Container Build](#container-build-re If you prefer to build locally without `docker`, take a look at our [Native Build](./doc/developer.md#native-build) instructions. -Once your development environment is configured you may move on to [Building Applications](#building-applications). +Once your development environment is configured you may move on to [Building the Holohub components you are interested in](#building-operators-applications-and-packages). ## Container Build (Recommended) @@ -66,7 +67,7 @@ To build and run in a containerized environment you will need: You will also need to set up your NVIDIA NGC credentials at [ngc.nvidia.com](https://catalog.ngc.nvidia.com/). ### Fetch - + Clone the repository to your local system: ```sh $ git clone https://www.github.com/nvidia-holoscan/holohub.git @@ -104,7 +105,7 @@ Launch the Docker container environment: $ ./dev_container launch ``` -You are now ready to run applications! You may jump to the [Running Applications](#running-applications) section to get started. +You are now ready to [build Holohub operators, applications, or packages!](#building-operators-applications-and-packages) ***Note*** The `launch` option will use the default development container built using Holoscan SDK's container from NGC for the local GPU. The script will also inspect for available video devices (V4L2, AJA capture boards, Deltacast capture boards) and the presence of Deltacast's Videomaster SDK and map it into the development container. @@ -113,7 +114,7 @@ See also: [Advanced Launch Options](/doc/developer.md#advanced-launch-options-co ### Platform Notes (Container) The development container has been tested on the following platforms: -- x86_x64 workstation with multiple RTX GPUs +- x86_64 workstation with multiple RTX GPUs - Clara AGX Dev Kit (dGPU mode) - IGX Orin Dev Kit (dGPU and iGPU mode) - AGX Orin Dev Kit (iGPU) @@ -124,37 +125,46 @@ The development container has been tested on the following platforms: (2) When building Holoscan SDK on AGX Orin Dev Kit from source please add the option ```--cudaarchs all``` to the ```./run build``` command to include support for AGX Orin's iGPU. -# Building Applications +# Building Operators, Applications, and Packages -_Make sure you have installed the [prerequisites](#prerequisites) before attempting to build reference applications._ +> _Make sure you have either launched your [development container](#container-build-recommended) or [set up your local environment](./doc/developer.md#native-build) before attempting to build Holohub components._ -Reference applications based on the Holoscan platform may be found under the [applications](./applications/) directory. Reference applications aim to provide a demonstration of the SDK capabilities. +This repository provides a convenience `run` script to abstract some of the CMake build process below. -This repository provides a convenient `run` script to build and run applications in your development environment. -To build an application: +Run the following to list existing components available to build: -```bash - ./run build -``` + ```bash + ./run list + ``` + +Then run the following to build the component of your choice, using either its name or its path: + + ```bash + # Build using the component name + ./run build + # Ex: ./run build endoscopy_tool_tracking -When the build is successful you can [run applications](#running-applications). + # Build using the component path + ./run build .// + # Ex: ./run build ./applications/endoscopy_tool_tracking/ + ``` + +The build artifacts will be created under `./build/` by default to isolate them from other components which might have different build environment requirements. You can override this behavior and other defaults, see `./run build --help` for more details. # Running Applications To list all available applications you can run the following command: -```bash - ./run list -``` + + ```bash + ./run list_apps + ``` Then you can run the application using the command: -```bash - ./run launch -``` -For example, to run the tool tracking endoscopy application: -```bash - ./run launch endoscopy_tool_tracking -``` + ```bash + ./run launch + # Ex: ./run launch endoscopy_tool_tracking + ``` Several applications are implemented in both C++ and Python programming languages. You can request a specific implementation as a trailing argument to the `./run launch` command @@ -162,9 +172,9 @@ or omit the argument to use the default language. For instance, the following command will run the Python implementation of the tool tracking endoscopy application: -```bash - ./run launch endoscopy_tool_tracking python -``` + ```bash + ./run launch endoscopy_tool_tracking python + ``` The run script reads the "run" command from the metadata.json file for a given application and runs from the "workdir" directory. Make sure you build the application (if applicable) before running it. @@ -173,14 +183,15 @@ Make sure you build the application (if applicable) before running it. You can run the command below to reset your `build` directory: -```sh -./run clear_cache -``` + ```sh + ./run clear_cache + ``` In some cases you may also want to clear out datasets downloaded by applications to the `data` folder: -```sh -rm -rf ./data -``` + + ```sh + rm -rf ./data + ``` Note that many applications supply custom container environments with build and runtime dependencies. Failing to clean the build cache between different applications may result in unexpected behavior where build diff --git a/run b/run index f9b6c2be2..2f9c36ba7 100755 --- a/run +++ b/run @@ -25,7 +25,9 @@ HOLOHUB_PY_LIB="${HOLOHUB_PY_EXE/bin/lib}" YELLOW="\e[1;33m" RED="\e[1;31m" +CYAN="\e[1;36m" NOCOLOR="\e[0m" +BOLD="\e[1;37m" # Compare versions compare_version () { @@ -72,9 +74,9 @@ run_command() { local cmd="$*" if [ "${DO_DRY_RUN}" != "true" ]; then - echo -e "${YELLOW}[command]${NOCOLOR} ${cmd}" + echo -e "${CYAN}[command]${NOCOLOR} ${cmd}" else - echo -e "${YELLOW}[dryrun]${NOCOLOR} ${cmd}" + echo -e "${CYAN}[dryrun]${NOCOLOR} ${cmd}" fi [ "$(echo -n "$@")" = "" ] && return 1 # return 1 if there is no command available @@ -210,6 +212,10 @@ print_error() { echo -e "${RED}ERROR${NOCOLOR}:" $* } +print_warning() { + echo -e "${YELLOW}WARNING${NOCOLOR}:" $* +} + check_exit_code() { local exit_code=$1 @@ -259,12 +265,8 @@ install_cuda_dependencies_package() { } # Setup the environment -# Build HoloHub sample apps setup_desc() { - echo "" - echo "Install the required dependencies for HoloHub (sample applications)" - echo "Usage: ./run setup" - echo "" + echo -e "${BOLD}Install the standard set of dependencies for HoloHub applications${NOCOLOR}" } setup() { @@ -397,16 +399,19 @@ setup() { #=========================================================================================== # Lint -install_lint_deps_desc() { echo 'Install lint dependencies +install_lint_deps_desc() { + echo -e " + ${BOLD}Install lint dependencies${NOCOLOR} -This command will install the dependencies required to run the linting tools. + This command will install the dependencies required to run the linting tools. -- Python linting: ruff, isort, black, mypy -- CMake linting: cmakelint -- C++ linting: cpplint, clang-format -- Spelling: codespell -' + - Python linting: ruff, isort, black, mypy + - CMake linting: cmakelint + - C++ linting: cpplint, clang-format + - Spelling: codespell + " } + install_lint_deps() { # We use $(command) || exit_code=1 to run all linting tools, and exit # with failure after all commands were executed if any of them failed @@ -425,18 +430,20 @@ install_lint_deps() { exit $exit_code } -lint_desc() { echo 'Lint the repository +lint_desc() { +echo -e " +${BOLD}Lint the repository${NOCOLOR} Python linting: black, isort, ruff C++ linting: cpplint CMake linting: cmakelint Spelling: codespell -Arguments: - $@ - directories to lint (default: .) +${CYAN}USAGE:${NOCOLOR} ./run lint -Options: - --fix: to fix all linting errors' +${CYAN}Options:${NOCOLOR} + --fix: to fix all linting errors +" } lint() { @@ -534,35 +541,42 @@ lint() { #=========================================================================================== -# Build HoloHub sample apps +# Build HoloHub targets build_desc() { - echo "" - echo "Build the application(s) in HoloHub" - echo "Usage: ./run build [options]" - echo "Options:" - echo " --sdk : Provide path to the SDK" - echo " --with : Optional operators that should be built" - echo " If multiple operators are to be built, list must be" - echo " in quotes with operators separated by semicolons (;)" + echo + echo -e "${BOLD}Build a Holohub operator, application, or package, in its isolated build directory.${NOCOLOR}" + echo + echo -e "${CYAN}USAGE:${NOCOLOR}" + echo + echo " ./run build [options]" + echo " ./run build ./path/to/ [options]" + echo + echo -e "${CYAN}Options:${NOCOLOR}" + echo + echo " --sdk : Provide path to the SDK" + echo " --with : Optional operators that should be built" + echo " If multiple operators are to be built, list must be" + echo " in quotes with operators separated by semicolons (;)" echo " --type : Specify the type of build" echo " Default: release" echo " Associated environment variable: CMAKE_BUILD_TYPE" echo " --buildpath : Change the build path." echo " Default: build" echo " Associated environment variable: CMAKE_BUILD_PATH" - echo " --benchmark : Build for Holoscan Flow Benchmarking." - echo " --install : Install applications and its dependencies to" - echo " install/ if supported." + echo " --benchmark : Build for Holoscan Flow Benchmarking. Valid for" + echo " applications only" + echo " --install : Install the target and its dependencies to" + echo " install/ if supported." echo " --configure-args : Additional configuration arguments" echo " multiple arguments can be passed between quotes" echo " or using several --configure-args in the command line" echo " --parallel : Build in parallel using the given number of jobs." echo " Only needed to override the native build tool's default." echo " Example: --parallel $(($(nproc)-1))" - echo "" - echo "Prior to the first build use './run setup' to install the required dependencies" - echo "" + echo + echo "Prior to the first build use './run setup' to install the required dependencies." + echo } # See how many directories under the input path matches "install*" pattern @@ -666,75 +680,100 @@ build() { fi done - echo "Building Holohub" - - # Application to build + # Package, application or operator to build if [ "$1" == "" ]; then - print_error "Please specify which application to build." - echo "You can run ./run list for a full list of applications." + print_error "Please specify which package, application, or operator to build." + echo "You can run ./run list for a full list of options." exit 1 fi + project=$(basename "$1") + opt_prefix="" + is_app=false + is_pkg=false + if list_packages | grep -Fxq "$project"; then + opt_prefix="-D PKG_" + is_pkg=true + elif list_apps | grep -Fxq "$project"; then + opt_prefix="-D APP_" + is_app=true + elif list_operators | grep -Fxq "$project"; then + opt_prefix="-D OP_" + else + print_error "Unknown package, application, or operator: $1" + echo "You can run ./run list for a full list of options." + exit 1 + fi + echo "Building $project" + opt_flag="$opt_prefix$project:BOOL=1" # Assign default build path if [ "$build_path" == "" ]; then - build_path="build/$1" + build_path="build/$project" fi local app_source_root_path if [ $benchmark == true ]; then - app_source_root_path=$(get_app_source_root_dir $1) - run_command benchmarks/holoscan_flow_benchmarking/patch_application.sh ${app_source_root_path} + if [ $is_app == true ]; then + app_source_root_path=$(get_app_source_root_dir $project) + run_command benchmarks/holoscan_flow_benchmarking/patch_application.sh ${app_source_root_path} + else + print_warning "Requesting benchmarking with --benchmark is only available for applications, ignoring" + fi fi - echo "Building $1 application" - application="-DAPP_$1=1" - # We define the python path to make sure we grab the right one - cmake_extra_args="--no-warn-unused-cli -DPython3_EXECUTABLE=${HOLOHUB_PY_EXE} -DPython3_ROOT_DIR=${HOLOHUB_PY_LIB}" + cmake_extra_args="--no-warn-unused-cli -D Python3_EXECUTABLE=${HOLOHUB_PY_EXE} -D Python3_ROOT_DIR=${HOLOHUB_PY_LIB}" # We set the data directory to be outside the build directory - cmake_extra_args="$cmake_extra_args $configure_args -DHOLOHUB_DATA_DIR:PATH=${SCRIPT_DIR}/data" + cmake_extra_args="$cmake_extra_args $configure_args -D HOLOHUB_DATA_DIR:PATH=${SCRIPT_DIR}/data" # Sets the default path for cuda export PATH=$PATH:/usr/local/cuda/bin - run_command cmake -S . -B ${build_path} ${cmake_extra_args} -DCMAKE_BUILD_TYPE=${build_type} ${holoscan_sdk} ${application} + run_command cmake -S . -B ${build_path} ${cmake_extra_args} -D CMAKE_BUILD_TYPE=${build_type} ${holoscan_sdk} ${opt_flag} ret=$? - check_exit_code $ret "Error building application." + check_exit_code $ret "Error building $project." # Job concurrency determined by the underlying build tool unless a number is specified run_command cmake --build ${build_path} -j ${parallel_jobs} ret=$? - check_exit_code $ret "Error building application." + check_exit_code $ret "Error building $project." if [ $install == true ]; then run_command cmake --install ${build_path} ret=$? - check_exit_code $ret "Error installing application." + check_exit_code $ret "Error installing $project." fi - if [ $benchmark == true ]; then + if [ $benchmark == true ] && [ $is_app == true ]; then app_source_root_path=$(get_app_source_root_dir $1) run_command benchmarks/holoscan_flow_benchmarking/restore_application.sh ${app_source_root_path} fi - echo "Holohub build done." + if [ $is_pkg == true ]; then + for cpack_config in $(find ${build_path}/pkg -name "CPackConfig-*.cmake"); do + run_command cpack --config "$cpack_config" + done + fi + + echo "Build done." } # Launch a sample app launch_desc() { - echo "" - echo "Launch command allowing to run application directly." - echo "Usage: ./run launch [language: cpp|python] [--extra_args ]" - echo "Options:" - echo " --extra_args : additional arguments passed to the application command" - echo " --nsys_profile : profile using Nsight Systems" - echo " --verbose : output additional run command information to stdout" - echo "" - echo "" - echo "Use './run list' first to list the available applications" - echo "Note that applications might be listed but not built." - echo "Make sure you ./run build the application first" - echo "" + echo -e " + ${BOLD}Run an application.${NOCOLOR} + + ${CYAN}USAGE:${NOCOLOR} ./run launch [cpp|python] [options] + + ${CYAN}Options:${NOCOLOR} + --extra_args : additional arguments passed to the application command + --nsys_profile : profile using Nsight Systems + --verbose : output additional run command information to stdout + + + Use './run list' or './run list_apps' to list the available applications. + Make sure you call './run build ' to first build the application. + " } @@ -885,38 +924,50 @@ for k, v in obj[project_type]["run"].items(): #=========================================================================================== # Other options -# Build HoloHub sample apps list_desc() { - echo "" - echo "Display the list of applications to run." - echo "Usage: ./run list" - echo "" + echo -e " + ${BOLD}Display the list of Holohub targets.${NOCOLOR} + + Includes applications, operators, packages. + " } -list() { - echo "" - echo "***** HoloHub applications ******" +list_cmake_dir_options() { + cmake_function=$1 + cmakelists=$(find ${SCRIPT_DIR} -name "CMakeLists.txt") + cmake_add_dir=$(grep --no-filename $cmake_function $cmakelists) + echo "$cmake_add_dir" | sed -n "s|$cmake_function([[:space:]]*\([^[:space:])]*\).*|\1|p" | sort +} - apps=$(find ${SCRIPT_DIR}/applications -name 'metadata.json' | sort -d) - apps+=$(find ${SCRIPT_DIR}/benchmarks -name 'metadata.json' | sort -d) - for d in ${apps}; do - local appname=$(dirname $d | grep -Po '^'${SCRIPT_DIR}'/(applications|benchmarks)/\K[^ ]*') +list_apps() { + list_cmake_dir_options add_holohub_application +} - local language="${appname##*/}" - if [[ ${language} != "cpp" ]] && [[ ${language} != "python" ]]; then - language="" - filename="${appname}" - else - language="(${language})" - filename="${appname%/*}" - fi +list_operators() { + list_cmake_dir_options add_holohub_operator +} - application="${filename##*/}" +list_packages() { + list_cmake_dir_options add_holohub_package +} - echo "- ${application} ${language}" - done - echo "" - echo "Run ./run build to build a specific app." +list() { + echo + echo -e "${BOLD}== APPLICATIONS =================${NOCOLOR}" + echo + + list_apps + echo + echo -e "${BOLD}== OPERATORS ====================${NOCOLOR}" + echo + list_operators + echo + echo -e "${BOLD}== PACKAGES =====================${NOCOLOR}" + echo + list_packages + echo + echo -e "${BOLD}=================================${NOCOLOR}" + echo } # Returns the list of words for autocompletion @@ -939,7 +990,7 @@ clear_cache() { } clear_cache_desc() { - echo "Remove the build folders" + echo -e "${BOLD}Remove the build folders${NOCOLOR}" } parse_args() { @@ -995,18 +1046,19 @@ parse_args() { print_usage() { set +x echo - echo "USAGE: $0 [command] [arguments]..." - echo "" - echo "Global Arguments" - echo " --help, -h : Print help messages for [command]" - echo " --dryrun : Print commands to screen without running" - echo "" - echo "Command List" - echo " setup : Install HoloHub main required packages" - echo " clear_cache : Clear cache folders" - echo " list : List all the applications" - echo " build [application] [--sdk holoscan_sdk] : Build a specific application. Default is sample apps" - echo " launch : Run the application" + echo -e "${BOLD}USAGE: $0 [command] [arguments]...${NOCOLOR}" + echo + echo -e "${CYAN}Global Arguments${NOCOLOR}" + echo " --help, -h : Print help messages for [command]" + echo " --dryrun : Print commands to screen without running" + echo + echo -e "${CYAN}Commands${NOCOLOR}" + echo " setup : Install HoloHub main required packages" + echo " clear_cache : Clear cache folders" + echo " list : List all the available build targets" + echo " build : Build a specific package, application, or operator" + echo " launch : Run the application" + echo } print_cmd_help_messages() {