Additional patches for Ubuntu 20 (nasa#627)

* Update Ubuntu 20 armhf patches

* Move debian creation to a separate directory
* Add patch for boringssl
* Add patch for fastboot

* Update cmake build scripts to the latest version for focal

CMake maintainers released an updated version of the debian we base
from. This new version overrides ours because of naming conventions.
Therefore we must rebase our debian generation script.

* Add script to compile armhf custom debians

We do not need to get dependencies when in chroot since everything
is provided.

This is a strip-down version of the build_install_debians script.

* Set LANG so perl does not complain all the time on chroot

* Add OpenCV patch to handle log escape char issue

Somehow this was not added to the initial commit for Ubuntu 20
support. Fixing that mistake now.

* Add more patches for cmake compilation

* Fix a compilation issue caused by git backporting a new security
  feature
* Fix a few tests that were failing on qemu chroot
* Skip one test that can only be fixed by recompiling yet another package

* Add generic build_debians script

It supports building only, building and installing, or building
and installing with dependencies on both amd64 and armhf.

* Update readme to explain how to build debians on chroot

* Replace references to make_xenial with make_chroot

* Use the new repo to get our custom debians

* Update path for astrobee key

* Minor doc updates

* Bind ssh tunnel to localhost

* Minor updates to changelog to make lintian happy

* Remove unnecessary .dir files

* Add dh lintian overrides

* Add more lintian compliance updates

* Check for errors
* Reference license

* Add localization_analysis headers to install

Needed for our amd64 debian build

* Update Astrobee Debian packaging

* Add support for Ubuntu 16 and 20
* Clean up rules file and adapt to catkin (as possible)
* Support armhf, config only, and amd64 builds
* Adaptive control file
* Adaptive install file

* Update build_debian.sh to support Ubuntu 16 and 20, plus amd64

* Update for compatibility with Ubuntu 16

* Change python2 dependency to python for Ubuntu xenial

* Add traj_opt_ros to install

* Make sure we are extending only what we need

This also avoids conflicts between native and debian_native
profiles

* Change buid_debian argument from --amd64 to --native

* Move Ubuntu 20 patches out of this repo.

* Update platform submodule pointer

---------

Co-authored-by: Ruben Garcia <[email protected]>

debian/astrobee-comms.lintian-overrides

@@ -0,0 +1,15 @@
+# This will actually have no effect on lintian because of the current profile
+# settings for this vendor (ubuntu). If you want to ignore these errors
+# you must modify the following file: /usr/share/lintian/profiles/ubuntu/main.profile
+#
+# Add to the end:
+#
+# Overridable: yes
+# Tags: dir-or-file-in-opt,
+#
+# See https://bugs.launchpad.net/ubuntu/+source/lintian/+bug/824122 for more.
+
+dir-or-file-in-opt
+
+# Ignore until we figure out how to use shlibs on chroot
+missing-dependency-on-libc

debian/astrobee-config.lintian-overrides

@@ -0,0 +1,2 @@
+# See astrobee-comms.lintian-overrides for details
+dir-or-file-in-opt

debian/astrobee-config.postinst

@@ -1,5 +1,7 @@
 #!/bin/sh
 
+set -e
+
 if [ "$1" = "configure" ]; then
   chmod -R g+rwX /opt/astrobee/config
   if [ $(getent group users) ]; then

debian/astrobee-dev.lintian-overrides

@@ -0,0 +1,2 @@
+# See astrobee-comms.lintian-overrides for details
+dir-or-file-in-opt

debian/astrobee0.install

@@ -8,6 +8,6 @@ opt/astrobee/version.txt
 opt/astrobee/.catkin
 opt/astrobee/bin/*
 opt/astrobee/lib/*.so
-opt/astrobee/lib/python2.7/*
+opt/astrobee/lib/python*/*
 opt/astrobee/lib/astrobee/*
 opt/astrobee/share/*

debian/astrobee0.lintian-overrides

@@ -0,0 +1,4 @@
+# See astrobee-comms.lintian-overrides for details
+dir-or-file-in-opt
+# Ignore until we figure out how to use shlibs on chroot
+missing-dependency-on-libc

debian/changelog

@@ -107,7 +107,7 @@ astrobee (0.14.1) testing; urgency=medium
   * Bug fixes
   * Note, Perching does not work in this release, use another version for this.
 
-  -- Astrobee Flight Software <[email protected]>  Wed, 03 Mar 2021 14:55:02 -0800
+ -- Astrobee Flight Software <[email protected]>  Wed, 03 Mar 2021 14:55:02 -0800
 
 astrobee (0.14.0) testing; urgency=medium
 
@@ -130,7 +130,7 @@ astrobee (0.13.0) testing; urgency=medium
   * Cross compiling with docker
   * Various other minor fixes
 
- -- Astrobee Flight Software <[email protected]>   Fri, 15 Jan 2021 10:37:25 -0800
+ -- Astrobee Flight Software <[email protected]>  Fri, 15 Jan 2021 10:37:25 -0800
 
 astrobee (0.12.0) testing; urgency=medium
 
@@ -215,7 +215,7 @@ astrobee (0.9.3) testing; urgency=medium
     associated logging and fault generation)
   * Re-positioned the dock location on ISS from the mapping activity.
 
- -- Astrobee Developer <astrobee@avm>  Mon, 08 Jul 2019 18:33:16 -0700
+ -- Astrobee Flight Software <astrobee[email protected]>  Mon, 08 Jul 2019 18:33:16 -0700
 
 astrobee (0.9.2) testing; urgency=medium
 
@@ -232,7 +232,7 @@ astrobee (0.9.1) testing; urgency=medium
   * Camera driver double buffer fix
   * As usual, other bug fixes too
 
- -- Astrobee Developer <astrobee@avm>  Wed, 29 May 2019 10:18:34 -0700
+ -- Astrobee Flight Software <astrobee[email protected]>  Wed, 29 May 2019 10:18:34 -0700
 
 astrobee (0.8.0) testing; urgency=medium
 

debian/control

@@ -13,7 +13,7 @@ Section: libdevel
 Architecture: any
 Depends: astrobee0 (= ${binary:Version}), ${misc:Depends}
   libgoogle-glog-dev, libgtest-dev, libgflags-dev,
-  libace-dev, libqt4-dev,
+  libace-dev, libqt4-dev | qtbase5-dev,
   protobuf-compiler, libv4l-dev, libeigen3-dev, libluajit-5.1-dev,
   libjsoncpp-dev, libtinyxml-dev, libyaml-cpp-dev, libusb-1.0-0-dev,
   libalvar-dev, libdbow2-dev, libgtsam-dev, libopenmvg-dev, libroyale-dev, libceres-dev, rti-dev,
@@ -31,42 +31,43 @@ Description: Astrobee flight software configuration.
 Package: astrobee-comms
 Section: comm
 Architecture: any
-Depends: ${shlibs:Depends}, ${misc:Depends}
+Depends: ${misc:Depends}
   astrobee-config (>= ${binary:Version}),
   rti (>=1.0), libmiro0 (>=0.1), libsoracore1 (>=1.0),
-  ros-kinetic-ros-base, ros-kinetic-cpp-common,
-  ros-kinetic-common-msgs, ros-kinetic-nodelet, ros-kinetic-image-transport,
-  ros-kinetic-compressed-image-transport, ros-kinetic-tf2,
-  ros-kinetic-tf2-geometry-msgs, ros-kinetic-tf2-msgs, ros-kinetic-tf2-ros,
-  ros-kinetic-tf2-sensor-msgs,
+  ros-${ros-distro}-ros-base, ros-${ros-distro}-cpp-common,
+  ros-${ros-distro}-common-msgs, ros-${ros-distro}-nodelet, ros-${ros-distro}-image-transport,
+  ros-${ros-distro}-compressed-image-transport, ros-${ros-distro}-tf2,
+  ros-${ros-distro}-tf2-geometry-msgs, ros-${ros-distro}-tf2-msgs, ros-${ros-distro}-tf2-ros,
+  ros-${ros-distro}-tf2-sensor-msgs,
   libgoogle-glog0v5
-Description: The communications for Astrobee flight software.
+Description: Astrobee flight software communications.
  The communications portion of the Astrobee flight software.
 
 Package: astrobee0
 Architecture: any
-Depends: ${shlibs:Depends}, ${misc:Depends}
+Depends: ${misc:Depends}
+  ${ros-python},
   astrobee-config (>= ${binary:Version}),
   astrobee-comms (>= ${binary:Version}),
   libalvar2 (>=2.0), libdbow21 (>=0.1), libgtsam (>=4.0), libopenmvg1 (>=1.0), libroyale1 (>=1.0),
   libceres1 (>=1.0), rti (>=1.0), libmiro0 (>=0.1), libsoracore1 (>=1.0),
   libdecomputil0 (>=0.1), libjps3d0 (>=0.1),
   libluajit-5.1-2,
-  ros-kinetic-ros-base, ros-kinetic-cpp-common, ros-kinetic-rosbag,
-  ros-kinetic-common-msgs, ros-kinetic-nodelet, ros-kinetic-pluginlib,
-  ros-kinetic-actionlib, ros-kinetic-image-transport,
-  ros-kinetic-compressed-image-transport, ros-kinetic-tf2,
-  ros-kinetic-tf2-eigen, ros-kinetic-tf2-geometry-msgs,
-  ros-kinetic-tf2-kdl, ros-kinetic-tf2-msgs, ros-kinetic-tf2-ros,
-  ros-kinetic-tf2-sensor-msgs, ros-kinetic-message-filters,
-  ros-kinetic-robot-state-publisher, ros-kinetic-rosconsole, ros-kinetic-roscpp,
-  ros-kinetic-rosgraph, ros-kinetic-rosgraph-msgs,
-  ros-kinetic-roslaunch, ros-kinetic-rosmaster, ros-kinetic-rosmsg,
-  ros-kinetic-rosnode, ros-kinetic-rosout, ros-kinetic-rosparam,
-  ros-kinetic-rosservice, ros-kinetic-rostopic, ros-kinetic-roswtf,
-  ros-kinetic-std-srvs, ros-kinetic-topic-tools, ros-kinetic-xmlrpcpp,
-  ros-kinetic-opencv3,
-  libjsoncpp1, libprotoc9v5, libgoogle-glog0v5,
-  libtinyxml2.6.2v5, libyaml-cpp0.5v5
+  ros-${ros-distro}-ros-base, ros-${ros-distro}-cpp-common, ros-${ros-distro}-rosbag,
+  ros-${ros-distro}-common-msgs, ros-${ros-distro}-nodelet, ros-${ros-distro}-pluginlib,
+  ros-${ros-distro}-actionlib, ros-${ros-distro}-image-transport,
+  ros-${ros-distro}-compressed-image-transport, ros-${ros-distro}-tf2,
+  ros-${ros-distro}-tf2-eigen, ros-${ros-distro}-tf2-geometry-msgs,
+  ros-${ros-distro}-tf2-kdl, ros-${ros-distro}-tf2-msgs, ros-${ros-distro}-tf2-ros,
+  ros-${ros-distro}-tf2-sensor-msgs, ros-${ros-distro}-message-filters,
+  ros-${ros-distro}-robot-state-publisher, ros-${ros-distro}-rosconsole, ros-${ros-distro}-roscpp,
+  ros-${ros-distro}-rosgraph, ros-${ros-distro}-rosgraph-msgs,
+  ros-${ros-distro}-roslaunch, ros-${ros-distro}-rosmaster, ros-${ros-distro}-rosmsg,
+  ros-${ros-distro}-rosnode, ros-${ros-distro}-rosout, ros-${ros-distro}-rosparam,
+  ros-${ros-distro}-rosservice, ros-${ros-distro}-rostopic, ros-${ros-distro}-roswtf,
+  ros-${ros-distro}-std-srvs, ros-${ros-distro}-topic-tools, ros-${ros-distro}-xmlrpcpp,
+  ros-kinetic-opencv3 | libopencv3.3.1,
+  libjsoncpp1, libprotoc9v5 | libprotoc17, libgoogle-glog0v5,
+  libtinyxml2.6.2v5, libyaml-cpp0.5v5 | libyaml-cpp0.6
 Description: Astrobee flight software.
  The Astrobee flight software.

debian/copyright

@@ -1,6 +1,6 @@
 Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
 Upstream-Name: astrobee
-Source: <url://example.com>
+Source: https://github.com/nasa/astrobee
 
 Files: *
 Copyright: 2017 Astrobee Team <[email protected]>
@@ -26,4 +26,7 @@ Files: debian/*
 Copyright: 2017 Astrobee Team <[email protected]>
 License: Apache-2
 
+On Debian systems, the complete text of the Apache License, Version 2
+can be found in "/usr/share/common-licenses/Apache-2.0".
+
 License: Apache-2

debian/rules

@@ -3,61 +3,59 @@
 # output every command that modifies files on the build system.
 #export DH_VERBOSE = 1
 
-
-# see FEATURE AREAS in dpkg-buildflags(1)
-#export DEB_BUILD_OPTIONS = parallel=`nproc`
-#export DEB_BUILD_MAINT_OPTIONS = hardening=+all
-
-# see ENVIRONMENT in dpkg-buildflags(1)
-# package maintainers to append CFLAGS
-#export DEB_CFLAGS_MAINT_APPEND  = -Wall -pedantic
-# package maintainers to append LDFLAGS
-#export DEB_LDFLAGS_MAINT_APPEND = -Wl,--as-needed
-export EXTRA_CMAKE_OPTS
-export LIB_SEARCH_OPTIONS
-
 ifeq ($(DEB_HOST_ARCH),armhf)
-#   export EXTRA_CMAKE_OPTS=-DCMAKE_TOOLCHAIN_FILE=../scripts/build/ubuntu_cross.cmake -DUSE_CTC=true
-	# stripping doesn't work with the cross compile
-  #export DEB_BUILD_OPTIONS += nostrip
-	export LIB_SEARCH_OPTIONS = -l$(ARMHF_CHROOT_DIR)/opt/rti/ndds/lib/armv6vfphLinux3.xgcc4.7.2:$(ARMHF_CHROOT_DIR)/opt/ros/kinetic/lib:$(ARMHF_CHROOT_DIR)/opt/ros/kinetic/lib/arm-linux-gnueabihf:$(ARMHF_CHROOT_DIR)/lib:$(ARMHF_CHROOT_DIR)/usr/lib:$(ARMHF_CHROOT_DIR)/usr/lib/arm-linux-gnueabihf:$(ARMHF_CHROOT_DIR)/lib/arm-linux-gnueabihf -- --ignore-missing-info --admindir=$(ARMHF_CHROOT_DIR)/var/lib/dpkg 
-	export DEB_MAINT_CC_SET = $(ARMHF_TOOLCHAIN)/bin/arm-linux-gnueabihf-gcc
-	export DEB_MAINT_CXX_SET = $(ARMHF_TOOLCHAIN)/bin/arm-linux-gnueabihf-g++
+	export FSW_CONFIG_FLAGS = -a -w debian -n debian -p src/debian/tmp
 else
-	export LIB_SEARCH_OPTIONS = -l/opt/ros/kinetic/lib:/opt/ros/kinetic/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu/gazebo-7/plugins
+	export FSW_CONFIG_FLAGS = -l -w debian_native -n debian_native -p src/debian/tmp/opt/astrobee
+	# dpkg-buildpackage and catkin were both adding flags apparently, ignore dpkg's
+	export DEB_CXXFLAGS_MAINT_SET
+endif
+
+export FSW_PYTHON_DEP = python
+ifeq ($(ROS_PYTHON_VERSION),3)
+	export FSW_PYTHON_DEP = python3
 endif
 
-# skip build for config files
-build-indep: ;
-	mkdir -p build
+# This is a Makefile variable that will only be used by native since armhf uses --extend
+CMAKE_PREFIX_PATH = $(CURDIR)/cmake:/opt/ros/$(ROS_DISTRO)
 
 %:
-	dh $@ --buildsystem=cmake --builddirectory=build/ --parallel
+	# We are using catkin now, but debhelper does not exactly support it
+	# Since debhelper xenial does not support 'none', set cmake instead
+	dh $@ --buildsystem=cmake
 
-# we always need release, otherwise it is too slow
-override_dh_auto_configure:
+# only configure for architecture dependent
+override_dh_auto_configure-arch:
 	# configure the build
-	cd .. && \
-	./src/scripts/configure.sh -a -w debian -p src/debian/tmp -n debian
-override_dh_auto_build: ;
-
-# the install command is already invoked in the build
-override_dh_auto_install:
-	cd .. && \
-	catkin build
-
-override_dh_install:
+	cd .. && src/scripts/configure.sh  $(FSW_CONFIG_FLAGS)
+	# Make sure we install
+	catkin config --install
+
+# build and install for architecture dependent only
+override_dh_auto_install-arch:
+	# Make sure we only extend what we need
+	cd .. && CMAKE_PREFIX_PATH=$(CMAKE_PREFIX_PATH) catkin build
+
+# override only for arch dependent, for astrobee-config let it use default
+override_dh_install-arch:
+	# Clean pycache before copying directories
+	pyclean debian || py3clean debian
+	# Move DDS stuff out of debian/tmp (astrobee-comms.files)
+	# so we don't intall it again to astrobee0
 	dh_movefiles
+	# Install the rest of the files as defined in <package>.install
 	dh_install
 
-override_dh_auto_test: ;
-
-#override_dh_auto_install-idep:
-#	@
-#
-#override_dh_auto_install:
-#	dh_auto_install
-#
+# dpkg-shlibdebs and dpkq-query have issues finding dependencies in our chroot
+# Skip this step since it was not populating {shlibs:Depends} anyway.
 override_dh_shlibdeps:
-	dh_shlibdeps $(LIB_SEARCH_OPTIONS)
 
+override_dh_gencontrol:
+	# Generate control file with the correct ros distribution and python for dependencies
+	dh_gencontrol -- -Vros-distro=$(ROS_DISTRO) -Vros-python=$(FSW_PYTHON_DEP)
+
+# skip for 'arch independent' (astrobee-config)
+override_dh_auto_configure-indep override_dh_auto_install-indep:
+
+# skip for all
+override_dh_auto_build override_dh_auto_test:

doc/general_documentation/NASA_INSTALL.md

@@ -64,7 +64,7 @@ entering your password (`m` is used to tunnel to `astrobee.ndc.nasa.gov`):
 The command should succeed without entering your password. Once this is verified,
 exit this session on `m` with `<ctrl>+D`.
 
-- These notes apply to `install_desktop_16.04_packages.sh` and `make_xenial.sh`
+- These notes apply to `install_desktop_16.04_packages.sh` and `make_chroot.sh`
 
 
 ### Checkout the project source code
@@ -132,7 +132,7 @@ Next, download the cross toolchain and install the chroot:
     mkdir -p $ARMHF_TOOLCHAIN
     cd $HOME/arm_cross
     $ASTROBEE_WS/src/submodules/platform/fetch_toolchain.sh
-    $ASTROBEE_WS/src/submodules/platform/rootfs/make_xenial.sh dev $ARMHF_CHROOT_DIR
+    $ASTROBEE_WS/src/submodules/platform/rootfs/make_chroot.sh xenial dev $ARMHF_CHROOT_DIR
 
 *Note: The last script shown above needs the packages `qemu-user-static` (not
 `qemu-arm-static`) and `multistrap` to be installed (can be installed through apt).*

mobility/planner_qp/traj_opt_ros/CMakeLists.txt

@@ -92,6 +92,16 @@ install(TARGETS traj_ros_bridge
   RUNTIME DESTINATION ${CATKIN_GLOBAL_BIN_DESTINATION}
 )
 
+if (NOT USE_CTC)
+  if(rviz_QT_VERSION)
+    install(TARGETS ${PROJECT_NAME}
+      ARCHIVE DESTINATION ${CATKIN_PACKAGE_LIB_DESTINATION}
+      LIBRARY DESTINATION ${CATKIN_PACKAGE_LIB_DESTINATION}
+      RUNTIME DESTINATION ${CATKIN_GLOBAL_BIN_DESTINATION}
+    )
+  endif()
+endif (NOT USE_CTC)
+
 # Mark nodelet_plugin for installation
 install(FILES nodelet_plugins.xml
         DESTINATION ${CATKIN_PACKAGE_SHARE_DESTINATION}

scripts/build/build_debian.sh

@@ -1,19 +1,27 @@
 #!/bin/bash
 
 DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+DIST=$(. /etc/os-release && echo $UBUNTU_CODENAME)
 
 if [ -n "$(git status --porcelain)" ]; then 
   echo "You should not build Debians for a dirty source tree!"
   echo "Make sure all your changes are committed AND pushed to the server..."
   exit -1
 fi
 
-EXTRA_FLAGS="-b -a armhf"
+EXTRA_FLAGS="-b -aarmhf"
+
+# In some cases we may want to build for amd64 (e.g. astrobee-comms for users)
+if [[ $* == *--native* ]]; then
+  EXTRA_FLAGS="-b"
+fi
+
 if [[ $* == *--config* ]]; then
   EXTRA_FLAGS="-A"
 fi
 
 pushd $DIR/../..
-export CMAKE_TOOLCHAIN_FILE=${DIR}/ubuntu_cross.cmake
-DEB_BUILD_OPTIONS="parallel=20" debuild -e ARMHF_CHROOT_DIR -e ARMHF_TOOLCHAIN -e CMAKE_TOOLCHAIN_FILE -e CMAKE_PREFIX_PATH -us -uc $EXTRA_FLAGS 
+DEBEMAIL="[email protected]" DEBFULLNAME="Astrobee Flight Software" dch -l"+$DIST" -D"$DIST" "Set distribution '$DIST' for local build"
+debuild -e ARMHF_CHROOT_DIR -e ARMHF_TOOLCHAIN -e CMAKE_TOOLCHAIN_FILE -e CMAKE_PREFIX_PATH -e ROS_DISTRO -e ROS_PYTHON_VERSION -us -uc $EXTRA_FLAGS
+git checkout debian/changelog
 popd > /dev/null

scripts/docker/readme.md

@@ -283,7 +283,7 @@ Next, download the cross toolchain and install the chroot:
 mkdir -p $ARMHF_TOOLCHAIN
 cd $HOME/arm_cross
 $SOURCE_PATH/submodules/platform/fetch_toolchain.sh
-$SOURCE_PATH/submodules/platform/rootfs/make_xenial.sh dev $ARMHF_CHROOT_DIR
+$SOURCE_PATH/submodules/platform/rootfs/make_chroot.sh xenial dev $ARMHF_CHROOT_DIR
 ```
 
 From the root of the repository, run:

scripts/setup/add_local_repository.sh

@@ -33,4 +33,4 @@ sudo touch $arssrc
 #sudo /bin/bash -c "echo \"deb [arch=amd64] http://127.0.0.1:8765/software xenial main\" > $arssrc" || exit 1
 #sudo /bin/bash -c "echo \"deb-src http://127.0.0.1:8765/software xenial main\" >> $arssrc" || exit 1
 
-sudo apt-key add $scriptdir/../../submodules/platform/rootfs/keys/astrobee.key || exit 1
+sudo apt-key add $scriptdir/../../submodules/platform/rootfs/multistrap/keys/astrobee.key || exit 1