Creating an SDK recipe including cuda toolkit and supported compiler

[kalj]

We are transitioning from dunfell to a kirkstone based yocto build targetting a TX2. To get the correct host code compiler (gcc 8.5) for CUDA (10.2), I have understood that making recipies inherit cuda should be enough. However, we also have a SDK recipe which should include that compiler. Currently, I have not been able to get that to work; only the default compiler (gcc 11) gets included. What is the simplest way of getting everything needed for cuda development to show up in the SDK?

[madisongh]

The only thing I've had to add to my image recipes (I'm assuming you're using bitbake -c populate_sdk image-recipe to construct the SDK) is:

TOOLCHAIN_HOST_TASK:append:tegra = " nativesdk-packagegroup-cuda-sdk-host"

The other needed bits should be picked up automatically through the bbappends in recipes-core/packagegroups.

[kalj]

We have a custom recipe for making the SDK, and it works similar to this:

inherit populate_sdk

TOOLCHAIN_HOST_TASK += " \
    nativesdk-packagegroup-cuda-sdk-host \
	"

However, this is not working, the gcc-8.5 cross compiler does not get added. I am trying to make a minimal example which creates an SDK with CUDA and gcc-8.5.

We are defining our own custom tx2-based MACHINE, could that be causing any problems?

[madisongh]

It's more likely to be your recipe than the custom machine definition, but some debugging will tell you. You might need to start using bitbake -e on the recipe and it dependencies to see where things might be getting overridden/changed in a way that's removing the needed packages.

In particular, make sure your SDK recipe is dependent on packagegroup-cross-canadian-${MACHINE}, and that that packagegroup recipe is picking up the bbappend from the meta-tegra layer that appends gcc-8-cross-canadian-${TRANSLATED_TARGET_ARCH} to the RDEPENDS for that packagegroup.

[kalj]

Okay, this is new territory for me, but here we go. I tried bitbake -e, and get the following hits for packagegroup-cross-canadian-:

# $SDK_RDEPENDS
#   set /home/user/dev/svk/vic_yocto_svk/build/../poky/meta/classes/populate_sdk_base.bbclass:71
#     "${TOOLCHAIN_TARGET_TASK} ${TOOLCHAIN_HOST_TASK}"
SDK_RDEPENDS="packagegroup-core-standalone-sdk-target target-sdk-provides-dummy nativesdk-packagegroup-sdk-host packagegroup-cross-canadian-jetson-tx2-devkit-tx2i      nativesdk-packagegroup-cuda-sdk-host 	..."

TOOLCHAIN_HOST_TASK="nativesdk-packagegroup-sdk-host packagegroup-cross-canadian-jetson-tx2-devkit-tx2i      nativesdk-packagegroup-cuda-sdk-host ..."

And finally that do_populate_sdk has rdepends:

....
'packagegroup-core-standalone-sdk-target:do_package_write_rpm packagegroup-core-standalone-sdk-target:do_packagedata target-sdk-provides-dummy:do_package_write_rpm target-sdk-provides-dummy:do_packagedata nativesdk-packagegroup-sdk-host:do_package_write_rpm nativesdk-packagegroup-sdk-host:do_packagedata packagegroup-cross-canadian-jetson-tx2-devkit-tx2i:do_package_write_rpm packagegroup-cross-canadian-jetson-tx2-devkit-tx2i:do_packagedata nativesdk-packagegroup-cuda-sdk-host:do_package_write_rpm nativesdk-packagegroup-cuda-sdk-host:do_packagedata',

Found from:

_task_deps="{'tasks': ['do_patch', 'do_populate_sysroot', 'do_populate_sysroot_setscene', 'do_prepare_recipe_sysroot', 'do_listtasks', 'do_clean', 'do_checkuri', 'do_fetch', 'do_unpack', 'do_deploy_source_date_epoch_setscene', 'do_deploy_source_date_epoch', 'do_configure', 'do_compile', 'do_install', 'do_build', 'do_cleansstate', 'do_cleanall', 'do_rm_work_all', 'do_rm_work', 'do_build_without_rm_work', 'do_package_qa', 'do_package_qa_setscene', 'do_package', 'do_package_setscene', 'do_packagedata', 'do_packagedata_setscene', 'do_package_write_rpm_setscene', 'do_package_write_rpm', 'do_devshell', 'do_pydevshell', 'do_populate_lic', 'do_populate_lic_setscene', 'do_populate_sdk_setscene', 'do_populate_sdk'], 'parents': {'do_patch': ['do_unpack'], 'do_populate_sysroot': ['do_install'], 'do_populate_sysroot_setscene': [], 'do_prepare_recipe_sysroot': ['do_fetch'], 'do_listtasks': [], 'do_clean': [], 'do_checkuri': [], 'do_fetch': [], 'do_unpack': ['do_fetch'], 'do_deploy_source_date_epoch_setscene': [], 'do_deploy_source_date_epoch': ['do_patch'], 'do_configure': ['do_deploy_source_date_epoch', 'do_prepare_recipe_sysroot', 'do_patch'], 'do_compile': ['do_configure'], 'do_install': ['do_compile'], 'do_build': ['do_rm_work', 'do_populate_sdk', 'do_populate_lic', 'do_package_write_rpm', 'do_packagedata', 'do_package_qa', 'do_rm_work_all', 'do_populate_sysroot'], 'do_cleansstate': ['do_clean'], 'do_cleanall': ['do_cleansstate'], 'do_rm_work_all': ['do_rm_work'], 'do_rm_work': ['do_deploy', 'do_package_qa', 'do_package_write_rpm', 'do_packagedata', 'do_populate_lic', 'do_populate_sdk', 'do_populate_sysroot'], 'do_build_without_rm_work': ['do_deploy', 'do_package_qa', 'do_package_write_rpm', 'do_packagedata', 'do_populate_lic', 'do_populate_sdk', 'do_populate_sysroot'], 'do_package_qa': ['do_packagedata', 'do_package'], 'do_package_qa_setscene': [], 'do_package': ['do_install'], 'do_package_setscene': [], 'do_packagedata': ['do_package'], 'do_packagedata_setscene': [], 'do_package_write_rpm_setscene': [], 'do_package_write_rpm': ['do_packagedata', 'do_package', 'do_deploy_source_date_epoch'], 'do_devshell': ['do_patch', 'do_prepare_recipe_sysroot'], 'do_pydevshell': ['do_patch'], 'do_populate_lic': ['do_patch'], 'do_populate_lic_setscene': [], 'do_populate_sdk_setscene': [], 'do_populate_sdk': ['do_install']}, 'mcdepends': {}, 'depends': {'do_patch': 'quilt-native:do_populate_sysroot patch-replacement-native:do_populate_sysroot', 'do_populate_sysroot': ' virtual/aarch64-poky-linux-binutils:do_populate_sysroot', 'do_prepare_recipe_sysroot': ' virtual/fakeroot-native:do_populate_sysroot', 'do_install': ' virtual/fakeroot-native:do_populate_sysroot', 'do_package_qa': ' virtual/aarch64-poky-linux-binutils:do_populate_sysroot', 'do_package': ' virtual/fakeroot-native:do_populate_sysroot', 'do_package_setscene': ' virtual/fakeroot-native:do_populate_sysroot', 'do_package_write_rpm': ' ', 'do_devshell': ' virtual/fakeroot-native:do_populate_sysroot', 'do_populate_sdk': ' rpm-native:do_populate_sysroot dnf-native:do_populate_sysroot createrepo-c-native:do_populate_sysroot opkg-native:do_populate_sysroot virtual/fakeroot-native:do_populate_sysroot xz-native:do_populate_sysroot cross-localedef-native:do_populate_sysroot nativesdk-qemuwrapper-cross:do_populate_sysroot qemuwrapper-cross:do_populate_sysroot nativesdk-glibc-locale:do_populate_sysroot rpm-native:do_populate_sysroot dnf-native:do_populate_sysroot createrepo-c-native:do_populate_sysroot opkg-native:do_populate_sysroot'}, 'rdepends': {'do_populate_sdk': 'packagegroup-core-standalone-sdk-target:do_package_write_rpm packagegroup-core-standalone-sdk-target:do_packagedata target-sdk-provides-dummy:do_package_write_rpm target-sdk-provides-dummy:do_packagedata nativesdk-packagegroup-sdk-host:do_package_write_rpm nativesdk-packagegroup-sdk-host:do_packagedata packagegroup-cross-canadian-jetson-tx2-devkit-tx2i:do_package_write_rpm packagegroup-cross-canadian-jetson-tx2-devkit-tx2i:do_packagedata nativesdk-packagegroup-cuda-sdk-host:do_package_write_rpm nativesdk-packagegroup-cuda-sdk-host:do_packagedata nativesdk-packagegroup-cuda-sdk-host:do_package_write_rpm nativesdk-packagegroup-cuda-sdk-host:do_packagedata nativesdk-packagegroup-cuda-sdk-host:do_package_write_rpm nativesdk-packagegroup-cuda-sdk-host:do_packagedata nativesdk-packagegroup-cuda-sdk-host:do_package_write_rpm nativesdk-packagegroup-cuda-sdk-host:do_packagedata'}, 'deptask': {'do_prepare_recipe_sysroot': 'do_populate_sysroot', 'do_package_write_rpm': ''}, 'rdeptask': {'do_build': ' do_package_write_rpm', 'do_package_qa': '', 'do_package_write_rpm': ''}, 'recrdeptask': {'do_build': ' do_deploy', 'do_rm_work_all': 'do_rm_work', 'do_populate_sdk': ' do_packagedata do_package_write_rpm do_package_write_ipk do_package_write_deb'}, 'recideptask': {}, 'nostamp': {'do_listtasks': '1', 'do_clean': '1', 'do_checkuri': '1', 'do_cleansstate': '1', 'do_cleanall': '1', 'do_devshell': '1', 'do_pydevshell': '1'}, 'fakeroot': {'do_install': '1', 'do_package': '1', 'do_package_setscene': '1', 'do_populate_sdk': '1'}, 'noexec': {'do_build': '1', 'do_rm_work_all': '1', 'do_build_without_rm_work': '1'}, 'umask': {}}"

Continuing on for gcc-8-cross-canadian-, I don't get any hits for that. I do get the following hit for gcc-8-cross:

SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS=" *->patch-native *->quilt-native *->subversion-native *->git-native *->icecc-create-env-native gcc-cross-aarch64->linux-libc-headers ppp-dialin->ppp resolvconf->bash docbook-xsl-stylesheets->perl ca-certificates->openssl initramfs-framework->busybox initramfs-framework->eudev initramfs-framework->systemd initramfs-module-install-efi->dosfstools initramfs-module-install-efi->e2fsprogs initramfs-module-install-efi->parted initramfs-module-install-efi->util-linux initramfs-module-install->e2fsprogs initramfs-module-install->grub initramfs-module-install->parted initramfs-module-install->util-linux initramfs-module-setup-live->udev-extraconf grub-efi->grub-bootconf liberation-fonts->fontconfig cantarell-fonts->fontconfig ttf-bitstream-vera->fontconfig gnome-icon-theme->librsvg font-alias->font-util systemd-boot->systemd-bootconf systemd->systemd-conf weston->weston-init weston-init->weston weston-init->kbd connman->xl2tpd lttng-tools->lttng-modules adwaita-icon-theme->gdk-pixbuf adwaita-icon-theme->gtk+3 tegra-nvstartup->tegra-configs-nvstartup xserver-xorg-video-nvidia->tegra-configs-xorg alsa-state->tegra-configs-alsa tegra-boot-tools->tegra-bootpart-config weston->tegra-udrm-probeconf gstreamer1.0-omx-tegra->tegra-configs-omx-tegra libglvnd->tegra-libraries vulkan-loader->tegra-libraries cuda-cudart->tegra-libraries libcufft->tegra-libraries cuda-cuobjdump->tegra-libraries cuda-cupti->tegra-libraries libcurand->tegra-libraries libcusolver->tegra-libraries libcusparse->tegra-libraries cuda-driver->tegra-libraries cuda-gdb->tegra-libraries cuda-memcheck->tegra-libraries cuda-misc-headers->tegra-libraries libnpp->tegra-libraries cuda-nvcc->tegra-libraries cuda-nvcc-headers->tegra-libraries cuda-nvdisasm->tegra-libraries cuda-nvgraph->tegra-libraries cuda-nvml->tegra-libraries cuda-nvprof->tegra-libraries cuda-nvprune->tegra-libraries cuda-nvrtc->tegra-libraries cuda-nvtx->tegra-libraries libcublas->tegra-libraries tensorrt-core->tegra-libraries libdrm->libdrm-nvdc libdrm-nvdc->tegra-libraries tegra-nvs-service->tegra-nvs-base gcc-8-cross-aarch64->linux-libc-headers android-tools->android-tools-conf usb-modeswitch-data->usb-modeswitch lmsensors->lmsensors-config phoronix-test-suite->bash phoronix-test-suite->python3 phoronix-test-suite->php phoronix-test-suite->lsb-release phoronix-test-suite->util-linux phoronix-test-suite->busybox phoronix-test-suite->shared-mime-info phoronix-test-suite->desktop-file-utils terminus-font->fontconfig ttf-abyssinica->fontconfig ttf-arphic-uming->fontconfig ttf-dejavu->fontconfig ttf-droid->fontconfig ttf-gentium->fontconfig ttf-hunkyfonts->fontconfig ttf-inconsolata->fontconfig ttf-ipa->fontconfig ttf-liberation->fontconfig ttf-liberation-sans-narrow->fontconfig ttf-lklug->fontconfig ttf-lohit->fontconfig ttf-noto-emoji->fontconfig ttf-pt-sans->fontconfig ttf-roboto->fontconfig ttf-mplus->fontconfig ttf-sazanami->fontconfig ttf-takao->fontconfig ttf-tlwg->fontconfig ttf-ubuntu-font-family->fontconfig ttf-vlgothic->fontconfig ttf-wqy-zenhei->fontconfig source-han-sans-cn-fonts->fontconfig source-han-sans-jp-fonts->fontconfig source-han-sans-kr-fonts->fontconfig source-han-sans-tw-fonts->fontconfig source-code-pro-fonts->fontconfig wireguard-tools->wireguard-module wireless-regdb->crda dracut->virtual/kernel "

But that does not look all that relevant.

How does this look?

[kalj]

bitbake -e packagegroup-cross-canadian-jetson-tx2-devkit-tx2i did give some more hits:

# $RDEPENDS:packagegroup-cross-canadian-jetson-tx2-devkit-tx2i [4 operations]
#   set <YOCTO_ROOT>/build/../poky/meta/recipes-core/packagegroups/packagegroup-cross-canadian.bb:18
#     "    ${@all_multilib_tune_values(d, 'BINUTILS')}     ${@all_multilib_tune_values(d, 'GCC')}     ${@all_multilib_tune_values(d, 'GDB')}     meta-environment-${MACHINE}     "
#   :append[tegra] <YOCTO_ROOT>/build/../meta-tegra/recipes-core/packagegroups/packagegroup-cross-canadian.bbappend:1
#     " gcc-8-cross-canadian-${TRANSLATED_TARGET_ARCH}"
#   :remove[tegra] <YOCTO_ROOT>/build/../meta-boot2qt/meta-boot2qt-distro/dynamic-layers/tegra/recipes-core/packagegroups/packagegroup-cross-canadian.bbappend:1
#     " gcc-8-cross-canadian-${TRANSLATED_TARGET_ARCH}"
#   rename from RDEPENDS:${PN} data.py:104 [expandKeys]
#     "    ${@all_multilib_tune_values(d, 'BINUTILS')}     ${@all_multilib_tune_values(d, 'GCC')}     ${@all_multilib_tune_values(d, 'GDB')}     meta-environment-${MACHINE}     "
# pre-expansion value:
#   "    ${@all_multilib_tune_values(d, 'BINUTILS')}     ${@all_multilib_tune_values(d, 'GCC')}     ${@all_multilib_tune_values(d, 'GDB')}     meta-environment-${MACHINE}      gcc-8-cross-canadian-${TRANSLATED_TARGET_ARCH}"
RDEPENDS:packagegroup-cross-canadian-jetson-tx2-devkit-tx2i="    binutils-cross-canadian-aarch64     gcc-cross-canadian-aarch64     gdb-cross-canadian-aarch64     meta-environment-jetson-tx2-devkit-tx2i      "
#
# $RDEPENDS:packagegroup-cross-canadian-jetson-tx2-devkit-tx2i-dev [2 operations]
#   set <YOCTO_ROOT>/build/../poky/meta/conf/bitbake.conf:352
#     "${PN} (= ${EXTENDPKGV})"
#   rename from RDEPENDS:${PN}-dev data.py:104 [expandKeys]
#     "${PN} (= ${EXTENDPKGV})"
# pre-expansion value:
#   "${PN} (= ${EXTENDPKGV})"
RDEPENDS:packagegroup-cross-canadian-jetson-tx2-devkit-tx2i-dev="packagegroup-cross-canadian-jetson-tx2-devkit-tx2i (= 1.0-r0)"
#
# $RDEPENDS:packagegroup-cross-canadian-jetson-tx2-devkit-tx2i-staticdev [2 operations]
#   set <YOCTO_ROOT>/build/../poky/meta/conf/bitbake.conf:356
#     "${PN}-dev (= ${EXTENDPKGV})"
#   rename from RDEPENDS:${PN}-staticdev data.py:104 [expandKeys]
#     "${PN}-dev (= ${EXTENDPKGV})"
# pre-expansion value:
#   "${PN}-dev (= ${EXTENDPKGV})"
RDEPENDS:packagegroup-cross-canadian-jetson-tx2-devkit-tx2i-staticdev="packagegroup-cross-canadian-jetson-tx2-devkit-tx2i-dev (= 1.0-r0)"

Does this clarify anything?

I notice that <YOCTO_ROOT>/meta-boot2qt/meta-boot2qt-distro/dynamic-layers/tegra/recipes-core/packagegroups/packagegroup-cross-canadian.bbappend gets involved; could that cause problems?

[madisongh]

Yep, there's your problem. The bbappend in meta-boot2qt is removing what we add in our bbappend:

#   :remove[tegra] <YOCTO_ROOT>/build/../meta-boot2qt/meta-boot2qt-distro/dynamic-layers/tegra/recipes-core/packagegroups/packagegroup-cross-canadian.bbappend:1
#     " gcc-8-cross-canadian-${TRANSLATED_TARGET_ARCH}"

Looking at the commit message for their bbappend, it looks like Qt had reasons to do this that probably don't apply to your situation.

You might consider removing that layer from your builds altogether. If you can't do that, you'll need to work around the problem it's introducing, maybe by explicitly adding gcc-8-cross-canadian-${TRANSLATED_TARGET_ARCH} to your TOOLCHAIN_HOST_TASK setting.

[kalj]

Yes! That did it! Thanks very much for helping out! :D