Install autoware from source

Install autoware from source
Generated by DALL-E: "Luke Skywalker riding in an autonomous vehicle"

Previously I walked through installing autoware in a docker container, but I've run into an issue trying to run AWSim within a container, and so I decided to install autoware from source rather than running it inside a docker container.

This post is mainly about the stumbling blocks I hit along the way and how I got past them.  If you don't hit any of those same issues, you're probably off just following the official installation docs and coming back here if you happen to hit the same issues.  

A quick tl;dr:

  1. Fixing an issue when installing libvulkan1 from the "lunarg" repo rather than the Ubuntu focal repo.
  2. Add some symlinks to get past this error: fatal error: nvml.h: No such file or directory when building the system_monitor package.
  3. Doing a clean build to avoid compilation errors resulting from re-using the autoware/src directory between the host and the container.  
  4. Fixing compiler warnings that skipped compiling many autoware packages due to broken CUDA symlinks

System specs

  • System76 Oryx w/ GeForce 1070 Nvidia GPU
  • Ubuntu 20.04 w/ linux kernel 5.18.10
  • Nvidia driver 520 with the cuda-11-6 package installed

Install dependencies

Install Ros2

I followed these instructions, but hit a few errors along the way.

Error: "you have held broken packages" 👹

# apt install ros-${rosdistro}-${installation_type}
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 ros-galactic-desktop : Depends: ros-galactic-pcl-conversions but it is not going to be installed
                        Depends: ros-galactic-rqt-common-plugins but it is not going to be installed
                        Depends: ros-galactic-rviz-default-plugins but it is not going to be installed
                        Depends: ros-galactic-rviz2 but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

Fix attempt #1 - apt autoremove 🤡

I did notice this message from the previous apt update:

The following packages were automatically installed and are no longer required:
  libbsd0:i386 libfwupdplugin1 libnvidia-common-515 libx11-6:i386 libxau6:i386 libxcb1:i386 libxdmcp6:i386 libxext6:i386 libxmlb1
Use 'sudo apt autoremove' to remove them.

and according to this askubuntu post that fixed the same issues for users.  

Unfortunately that didn't fix the issue for me.

Tracing the root cause 🌴

I recursively tried to install packages until I found the broken package:

# apt install ros-galactic-rviz2
The following packages have unmet dependencies:
 ros-galactic-rviz2 : Depends: ros-galactic-rviz-common but it is not going to be installed
                      Depends: ros-galactic-rviz-default-plugins but it is not going to be installed
# apt install ros-galactic-rviz-common
The following packages have unmet dependencies:
 ros-galactic-rviz-common : Depends: qtbase5-dev but it is not going to be installed
                            Depends: ros-galactic-rviz-rendering but it is not going to be installed
# apt install qtbase5-dev
The following packages have unmet dependencies:
 qtbase5-dev : Depends: libvulkan-dev but it is not going to be installed
# apt install libvulkan-dev
The following packages have unmet dependencies:
 libvulkan-dev : Depends: libvulkan1 (= 1.2.131.2-1) but it is not going to be installed

To see what was going on with libvulkan1 I ran aptitude :

# aptitude versions libvulkan1
Package libvulkan1:
p A 1.2.131.2-1                                                                                                                                                          focal                                                                                                                                            500
i A 1.3.224.0~rc2-1lunarg18.04-1                                                                                                                                                                                                                                                                                          100

and it looks like what's happening is that:

  • I installed libvulkan1 version 1.3.x from the "lunarg" repo ("lunarg" is the org that makes the vulkan drivers, which is some kind of non-proprietary graphics hardware layer).
  • libvulkan-dev wants libulkan1 1.2.x and won't accept 1.3.x.  I wonder why the lunarg repo doesn't have it's own version of libvulkan-dev that can use 1.3.x?  

Fix attempt #2 - remove lunarg repo, reinstall ubuntu focal version of libvulkan1 😅

Trying to remove libvulkan1 wants to remove a ton of stuff:

# apt remove libvulkan1
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  cuda-cccl-11-6 cuda-command-line-tools-11-6 cuda-compiler-11-6 cuda-cudart-11-6 cuda-cudart-dev-11-6 cuda-cuobjdump-11-6 cuda-cupti-11-6 cuda-cupti-dev-11-6 cuda-cuxxfilt-11-6 cuda-documentation-11-6 cuda-driver-dev-11-6 cuda-gdb-11-6 cuda-libraries-11-6 cuda-libraries-dev-11-6 cuda-memcheck-11-6
  cuda-nsight-compute-11-6 cuda-nsight-systems-11-6 cuda-nvcc-11-6 cuda-nvdisasm-11-6 cuda-nvml-dev-11-6 cuda-nvprof-11-6 cuda-nvprune-11-6 cuda-nvrtc-11-6 cuda-nvrtc-dev-11-6 cuda-nvtx-11-6 cuda-samples-11-6 cuda-sanitizer-11-6 dkms fprintd gds-tools-11-6 gir1.2-graphene-1.0 libcddb2 libcufft-11-6
  libcufft-dev-11-6 libcufile-11-6 libcufile-dev-11-6 libcurand-11-6 libcurand-dev-11-6 libcusolver-11-6 libcusolver-dev-11-6 libcusparse-11-6 libcusparse-dev-11-6 libdc1394-22 libdca0 libdouble-conversion3 libdvbpsi10 libebml4v5 libebook-1.2-20 libebook-contacts-1.2-3 libecal-2.0-1 libedata-book-1.2-26
  libedata-cal-2.0-1 libfaad2 libfprint-2-2 libgsound0 libgssdp-1.2-0 libgstreamer-plugins-bad1.0-0 libgupnp-1.2-0 libgupnp-av-1.0-2 libgupnp-dlna-2.0-3 libice-dev libixml10 libkate1 liblua5.2-0 libmad0 libmatroska6v5 libmpcdec6 libnpp-11-6 libnpp-dev-11-6 libnvidia-cfg1-520 libnvidia-common-520
  libnvidia-decode-520 libnvidia-encode-520 libnvidia-extra-520 libnvjpeg-11-6 libnvjpeg-dev-11-6 libopengl0 libopenmpt-modplug1 libpam-fprintd libpcre2-16-0 libprotobuf-lite17 libproxy-tools libpthread-stubs0-dev libqt5core5a libqt5dbus5 libqt5gui5-gles libqt5network5 libqt5widgets5 libqt5x11extras5
  libresid-builder0c2a librygel-core-2.6-2 librygel-db-2.6-2 librygel-renderer-2.6-2 librygel-server-2.6-2 libsdl-image1.2 libsdl1.2debian libsidplay2 libsm-dev libsndio7.0 libspatialaudio0 libsrt1 libssh2-1 libupnp13 liburcu6 libva-wayland2 libvlc-bin libvlc5 libvlccore9 libwoff1 libx11-dev libx11-doc libxau-dev
  libxcb-xinerama0 libxcb-xinput0 libxcb1-dev libxdmcp-dev libxt-dev nsight-compute-2022.1.1 nsight-systems-2021.5.2 nvidia-compute-utils-520 nvidia-dkms-520 nvidia-kernel-source-520 nvidia-modprobe nvidia-settings nvidia-utils-520 qt5-gtk-platformtheme qttranslations5-l10n rygel screen-resolution-extra
  switcheroo-control vlc-bin vlc-data vlc-l10n vlc-plugin-base vlc-plugin-notify vlc-plugin-samba vlc-plugin-video-splitter x11proto-core-dev x11proto-dev xdg-dbus-proxy xorg-sgml-doctools xtrans-dev
Use 'apt autoremove' to remove them.
The following additional packages will be installed:
  libqt5gui5-gles
Suggested packages:
  qt5-image-formats-plugins qtwayland5
Recommended packages:
  libqt5svg5
The following packages will be REMOVED:
  apturl cheese cuda-11-6 cuda-demo-suite-11-6 cuda-drivers cuda-drivers-520 cuda-nsight-11-6 cuda-nvvp-11-6 cuda-runtime-11-6 cuda-toolkit-11-6 cuda-tools-11-6 cuda-visual-tools-11-6 default-jre evolution-data-server gdm3 gir1.2-clutter-1.0 gir1.2-clutter-gst-3.0 gir1.2-cogl-1.0 gir1.2-coglpango-1.0
  gir1.2-gst-plugins-base-1.0 gir1.2-gtkclutter-1.0 gir1.2-mutter-6 gir1.2-rb-3.0 gir1.2-totem-1.0 gir1.2-webkit2-4.0 gnome-calendar gnome-control-center gnome-getting-started-docs gnome-initial-setup gnome-online-accounts gnome-session-bin gnome-shell gnome-shell-extension-appindicator
  gnome-shell-extension-desktop-icons gnome-shell-extension-ubuntu-dock gnome-startup-applications gnome-todo gnome-user-docs gnome-user-guide gstreamer1.0-clutter-3.0 gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-vaapi libatk-wrapper-java libatk-wrapper-java-jni libcheese-gtk25 libcheese8 libclutter-1.0-0
  libclutter-gst-3.0-0 libclutter-gtk-1.0-0 libcogl-pango20 libcogl-path20 libcogl20 libedataserverui-1.2-2 libgl1 libgl1-mesa-dri libgl1-mesa-glx libglu1-mesa libglx-mesa0 libglx0 libgoa-backend-1.0-1 libgstreamer-gl1.0-0 libmutter-6-0 libnvidia-fbc1-520 libnvidia-gl-520 libplacebo7 libqt5gui5 libqt5svg5 libtotem0
  libvulkan1 libwebkit2gtk-4.0-37 libyelp0 mesa-utils mesa-vulkan-drivers mutter nautilus-share nvidia-driver-520 openjdk-11-jre openjdk-8-jdk openjdk-8-jre python3-gst-1.0 rhythmbox-plugins shotwell totem totem-plugins ubuntu-desktop ubuntu-desktop-minimal ubuntu-docs ubuntu-release-upgrader-gtk ubuntu-session
  update-manager update-notifier vlc vlc-plugin-qt vlc-plugin-skins2 vlc-plugin-video-output vlc-plugin-visualization vnc4server x11-utils xbase-clients xorg xserver-xephyr xserver-xorg xserver-xorg-core xserver-xorg-input-all xserver-xorg-input-libinput xserver-xorg-input-wacom xserver-xorg-video-all
  xserver-xorg-video-amdgpu xserver-xorg-video-ati xserver-xorg-video-fbdev xserver-xorg-video-intel xserver-xorg-video-nouveau xserver-xorg-video-nvidia-520 xserver-xorg-video-qxl xserver-xorg-video-radeon xserver-xorg-video-vesa xserver-xorg-video-vmware xwayland yelp zenity
The following NEW packages will be installed:
  libqt5gui5-gles
0 upgraded, 1 newly installed, 121 to remove and 11 not upgraded.
Need to get 2,931 kB of archives.
After this operation, 1,315 MB disk space will be freed.
Do you want to continue? [Y/n]

Instead I followed the instructions in this post to surgically modify the /var/lib/dpkg/status package database, removing this entry:

Package: libvulkan1
Status: install ok installed
Priority: optional
Section: graphics
Installed-Size: 503
Maintainer: John Zupin <johnz@lunarg.com>
Architecture: amd64
Multi-Arch: same
Source: vulkan-loader
Version: 1.3.224.0~rc2-1lunarg18.04-1
Replaces: libvulkan-dev (<< 1.2.141.0~rc3), vulkan-loader
Depends: libc6 (>= 2.17)
Breaks: libvulkan-dev (<< 1.2.141.0~rc3), vulkan-loader
Description: This package contains the Vulkan loader
Homepage: https://github.com/KhronosGroup/Vulkan-Loader

Then reinstalling via:

# apt update
# apt install libvulkan1 --reinstall
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  libvulkan1
0 upgraded, 1 newly installed, 0 to remove and 11 not upgraded.
Need to get 93.3 kB of archives.
After this operation, 370 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu focal/main amd64 libvulkan1 amd64 1.2.131.2-1 [93.3 kB]
Fetched 93.3 kB in 0s (231 kB/s)
Selecting previously unselected package libvulkan1:amd64.
(Reading database ... 328925 files and directories currently installed.)
Preparing to unpack .../libvulkan1_1.2.131.2-1_amd64.deb ...
Unpacking libvulkan1:amd64 (1.2.131.2-1) ...
Setting up libvulkan1:amd64 (1.2.131.2-1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.9) ...

Now aptitude shows a cleaner output:

# aptitude versions libvulkan1
Package libvulkan1:
i   1.2.131.2-1                                                                                                                                                          focal                                                                                                                                            500

Install ros packages works now

# apt install ros-${rosdistro}-${installation_type}
...
0 upgraded, 683 newly installed, 0 to remove and 11 not upgraded.
Need to get 300 MB of archives.
After this operation, 1,615 MB of additional disk space will be used.
Do you want to continue? [Y/n] y

Install ros 2 dev tools

Drama free

Install RMW implementation

Drama free

Install pacmod

Drama free

Install autoware core dependencies

Drama free

Install autoware universe dependencies

Drama free

Install pre-commit dependencies

Drama free

Install nvidia cuda

Already covered in installing autoware in a docker container

Install cudnn and tensorrt

Already covered in installing autoware in a docker container

Setup workspace

Follow these instructions.

Error building workspace

$ colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release
Starting >>> obstacle_avoidance_planner
[Processing: detected_object_validation, grid_map_filters, lane_departure_checker, map_based_prediction, multi_object_tracker, ndt_scan_matcher, object_merger, obstacle_avoidance_planner, pointcloud_preprocessor, route_handler, system_monitor, traffic_light_visualization]
[Processing: detected_object_validation, grid_map_filters, lane_departure_checker, map_based_prediction, multi_object_tracker, ndt_scan_matcher, object_merger, obstacle_avoidance_planner, pointcloud_preprocessor, route_handler, system_monitor, traffic_light_visualization]
[Processing: detected_object_validation, grid_map_filters, lane_departure_checker, map_based_prediction, multi_object_tracker, ndt_scan_matcher, object_merger, obstacle_avoidance_planner, pointcloud_preprocessor, route_handler, system_monitor, traffic_light_visualization]
--- stderr: system_monitor
make[2]: *** No rule to make target '/usr/lib/x86_64-linux-gnu/libnl-3.so', needed by 'libnet_monitor_lib.so'.  Stop.
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:448: CMakeFiles/net_monitor_lib.dir/all] Error 2
make: *** [Makefile:141: all] Error 2
---
Failed   <<< system_monitor [2min 51s, exited with code 2]
Aborted  <<< ndt_scan_matcher [4min 5s]
Aborted  <<< traffic_light_visualization [2min 50s]
Aborted  <<< route_handler [1min 41s]
Aborted  <<< map_based_prediction [1min 41s]
Aborted  <<< object_merger [1min 43s]
Aborted  <<< lane_departure_checker [1min 45s]
Aborted  <<< multi_object_tracker [3min 16s]
Aborted  <<< detected_object_validation [3min 22s]
Aborted  <<< obstacle_avoidance_planner [2min 14s]
Aborted  <<< pointcloud_preprocessor [8min 2s]
Aborted  <<< grid_map_filters [9min 59s]

Summary: 109 packages finished [13min 55s]
  1 package failed: system_monitor
  11 packages aborted: detected_object_validation grid_map_filters lane_departure_checker map_based_prediction multi_object_tracker ndt_scan_matcher object_merger obstacle_avoidance_planner pointcloud_preprocessor route_handler traffic_light_visualization
  21 packages had stderr output: autoware_point_types costmap_generator detected_object_validation dummy_perception_publisher grid_map_pcl lidar_apollo_instance_segmentation lidar_centerpoint map_loader map_tf_generator ndt ndt_omp ndt_pcl_modified ndt_scan_matcher pointcloud_preprocessor pointcloud_to_laserscan shape_estimation system_monitor tensorrt_yolo tier4_pcl_extensions tvm_utility velodyne_pointcloud
  117 packages not processed

Workaround attempt

This workaround got past some of the build errors:

$ ln -s /lib/x86_64-linux-gnu/libnl-3.so /usr/lib/x86_64-linux-gnu/libnl-3.so
$ ln -s /lib/x86_64-linux-gnu/libnl-genl-3.so /usr/lib/x86_64-linux-gnu/libnl-genl-3.so

but on the rebuild I hit this issue:

$ colcon build --packages-select system_monitor --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release
Starting >>> system_monitor
--- stderr: system_monitor
In file included from /home/tleyden/Development/autoware/src/universe/autoware.universe/system/system_monitor/src/gpu_monitor/nvml_gpu_monitor.cpp:20:
/home/tleyden/Development/autoware/src/universe/autoware.universe/system/system_monitor/include/system_monitor/gpu_monitor/nvml_gpu_monitor.hpp:25:10: fatal error: nvml.h: No such file or directory
   25 | #include <nvml.h>
      |          ^~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/gpu_monitor_lib.dir/build.make:96: CMakeFiles/gpu_monitor_lib.dir/src/gpu_monitor/nvml_gpu_monitor.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:683: CMakeFiles/gpu_monitor_lib.dir/all] Error 2
make: *** [Makefile:161: all] Error 2
---
Failed   <<< system_monitor [29.7s, exited with code 2]

Summary: 0 packages finished [30.3s]
  1 package failed: system_monitor
  1 package had stderr output: system_monitor

I do have an nvml.h file, actually a few of them:

$ locate nvml.h
/usr/include/hwloc/nvml.h
/usr/local/cuda-11.6/targets/x86_64-linux/include/nvml.h
/usr/local/cuda-9.0/include/nvml.h

Compare against container

I switched into my autoware container and tried to build that package, but am getting a different error:

$ colcon build --packages-select system_monitor --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release
Starting >>> system_monitor
[Processing: system_monitor]
[Processing: system_monitor]
[Processing: system_monitor]
--- stderr: system_monitor
make[2]: *** No rule to make target '/usr/lib/x86_64-linux-gnu/libnvidia-ml.so', needed by 'libgpu_monitor_lib.so'.  Stop.
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:663: CMakeFiles/gpu_monitor_lib.dir/all] Error 2
make: *** [Makefile:161: all] Error 2
---
Failed   <<< system_monitor [1min 37s, exited with code 2]

Summary: 0 packages finished [1min 38s]
  1 package failed: system_monitor
  1 package had stderr output: system_monitor

The puzzling thing is that prior to this the full colcon build succeeded.

The file /usr/lib/x86_64-linux-gnu/libnvidia-ml.so exists on the host outside of the container, but not in the container.

It seems like the container is just missing a symlink for some reason.  It has:

$ ls -alh /usr/lib/x86_64-linux-gnu/ | grep -i nvidia-ml
lrwxrwxrwx  1 root root    25 Nov  2 17:00 libnvidia-ml.so.1 -> libnvidia-ml.so.520.61.05
-rw-r--r--  1 root root  1.7M Sep 28 22:30 libnvidia-ml.so.520.61.05

while on the host:

$ ls -alh /usr/lib/x86_64-linux-gnu/ | grep -i nvidia-ml
lrwxrwxrwx   1 root root    17 Sep 29 02:22 libnvidia-ml.so -> libnvidia-ml.so.1
lrwxrwxrwx   1 root root    25 Sep 29 02:22 libnvidia-ml.so.1 -> libnvidia-ml.so.520.61.05
-rw-r--r--   1 root root  1.7M Sep 28 22:30 libnvidia-ml.so.520.61.05

Container build failure workaround

This workaround worked on the container:

$ sudo ln -s /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1 /usr/lib/x86_64-linux-gnu/libnvidia-ml.so
$ colcon build --packages-select system_monitor --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release
Starting >>> system_monitor
Finished <<< system_monitor [22.8s]

Summary: 1 package finished [23.4s]

I'm still puzzled why the workaround was needed though.  

I did just realize something though: I'm working on a shared directory between the host and the container 😬.  This could cause all sorts of weird behavior.

From now on, I will make sure to keep these isolated.

Clean and rebuild workspace on the host

Based on these troubleshooting instructions I'm going to do a clean build on the host machine:

$ rm -rf build/ install/ log/
$ colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release

The system_monitor packages is now building, and made it past the fatal error: nvml.h: No such file or directory error mentioned above.  As it turns out, the symlink workaround above wasn't even needed, I was able to remove those symlinks and do a clean rebuild without any errors.

The build finished but there was some stderr output to the build (full output):

Summary: 238 packages finished [33min 47s]
  44 packages had stderr output: autoware_point_types bag_time_manager_rviz_plugin behavior_velocity_planner compare_map_segmentation costmap_generator detected_object_validation detection_by_tracker dummy_perception_publisher elevation_map_loader euclidean_cluster front_vehicle_velocity_estimator grid_map_pcl grid_map_sdf ground_segmentation image_projection_based_fusion lanelet2_map_preprocessor lidar_apollo_instance_segmentation lidar_apollo_segmentation_tvm lidar_apollo_segmentation_tvm_nodes lidar_centerpoint livox_tag_filter map_loader map_tf_generator ndt_omp ndt_scan_matcher obstacle_collision_checker obstacle_stop_planner obstacle_velocity_limiter occupancy_grid_map_outlier_filter pointcloud_preprocessor pointcloud_to_laserscan pose_initializer probabilistic_occupancy_grid_map shape_estimation simulator_compatibility_test surround_obstacle_checker tensorrt_yolo tier4_pcl_extensions tier4_traffic_light_rviz_plugin traffic_light_classifier traffic_light_ssd_fine_detector trtexec_vendor tvm_utility velodyne_pointcloud

Address CUDA not found errors

In the colcon build output, I'm seeing errors like this:

CUDA_TOOLKIT_ROOT_DIR not found or specified
CUDA NOT FOUND
TensorrtYolo won't be built, CUDA and/or TensorRT were not found.

On my $PATH I have /usr/local/cuda/bin which matches up with the autoware cuda install instructions, however that directory is symlinked to another dir:

$ ls -alh /usr/local/cuda
lrwxrwxrwx 1 root root 22 Oct 21 13:44 /usr/local/cuda -> /etc/alternatives/cuda

which is symlinked to a version of cuda I don't even have installed! 🤷‍♂️  I'm using cuda-11.6 not 11.8

$ ls -alh /etc/alternatives/cuda
lrwxrwxrwx 1 root root 20 Oct 31 08:10 /etc/alternatives/cuda -> /usr/local/cuda-11.8

I compared against the autoware container, and indeed it's pointing to the right version of cuda:

$ ls -alh /etc/alternatives/cuda
lrwxrwxrwx 1 root root 20 Oct 27 15:46 /etc/alternatives/cuda -> /usr/local/cuda-11.6

Fixing the host by re-pointing the symlink:

$ sudo rm /etc/alternatives/cuda
$ sudo ln -s /usr/local/cuda-11.6 /etc/alternatives/cuda
$ ls -alh /etc/alternatives/cuda
lrwxrwxrwx 1 root root 20 Nov  2 18:47 /etc/alternatives/cuda -> /usr/local/cuda-11.6

Clean and rebuild workspace again on the host

$ rm -rf build/ install/ log/
$ colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release
...
Summary: 238 packages finished [33min 46s]
  41 packages had stderr output: autoware_point_types bag_time_manager_rviz_plugin behavior_velocity_planner compare_map_segmentation costmap_generator detected_object_validation detection_by_tracker dummy_perception_publisher elevation_map_loader euclidean_cluster front_vehicle_velocity_estimator grid_map_pcl grid_map_sdf ground_segmentation image_projection_based_fusion lanelet2_map_preprocessor lidar_apollo_instance_segmentation lidar_apollo_segmentation_tvm lidar_apollo_segmentation_tvm_nodes lidar_centerpoint livox_tag_filter map_loader map_tf_generator ndt_omp ndt_scan_matcher obstacle_collision_checker obstacle_stop_planner obstacle_velocity_limiter occupancy_grid_map_outlier_filter pointcloud_preprocessor pointcloud_to_laserscan pose_initializer probabilistic_occupancy_grid_map shape_estimation simulator_compatibility_test surround_obstacle_checker tier4_pcl_extensions tier4_traffic_light_rviz_plugin trtexec_vendor tvm_utility velodyne_pointcloud

Now it worked!  The CUDA errors are gone, and aside from a few warnings, everything compiled.  (full output / full error)