沉默杀手
交叉编译 PCL
|cmake/qmake|
Word count: 2.2k|Reading time: 10 min

在x86架构的Ubuntu22.04交叉编译PCL源码，以能在ARM aarch64系统上运行，也就是全志MR527的TinaLinux 5.15.123 aarch64

先看完成的状态，也就是ARM平台的PCL库文件链接关系:

libpcl_kdtree.so

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
ldd  /usr/lib/aarch64-linux-gnu/libpcl_kdtree.so
    linux-vdso.so.1 (0x0000ffff8d240000)

    libpcl_common.so.1.10 => /lib/aarch64-linux-gnu/libpcl_common.so.1.10 (0x0000ffff8cfb7000)
    liblz4.so.1 => /lib/aarch64-linux-gnu/liblz4.so.1 (0x0000ffff8cf89000)


    libstdc++.so.6 => /lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000ffff8cda4000)
    libgomp.so.1 => /lib/aarch64-linux-gnu/libgomp.so.1 (0x0000ffff8cd56000)
    libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000ffff8cd32000)
    libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffff8cbbf000)
    /lib/ld-linux-aarch64.so.1 (0x0000ffff8d210000)
    libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000ffff8cb14000)
    libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000ffff8cae3000)
    libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000ffff8cacf000)

libpcl_common.so

1
2
3
4
5
6
7
8
9
10
ldd  /usr/lib/aarch64-linux-gnu/libpcl_common.so
    linux-vdso.so.1 (0x0000ffff975e8000)
    libstdc++.so.6 => /lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000ffff972d0000)
    libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000ffff97225000)
    libgomp.so.1 => /lib/aarch64-linux-gnu/libgomp.so.1 (0x0000ffff971d7000)
    libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000ffff971b3000)
    libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000ffff97182000)
    libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffff9700f000)
    /lib/ld-linux-aarch64.so.1 (0x0000ffff975b8000)
    libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2

交叉编译 PCL

在终端找到/opt/cmake/bin，启动cmake-gui，这里是新安装的cmake 3.27. 选择PCL的源码目录和build目录，选择交叉编译平台。没有显示需求，所以cmake-gui不需要选择 OpenGL和VTK

The C compiler identification is unknown

既然是交叉编译，那就得用跨平台的编译器，注意在启动cmake-gui的终端要提前设置环境变量:

1
2
3
4
5
export ARCH=arm64
export TOOLCHAIN_PATH=/home/user/toolschain/gcc-ubuntu-9.3.0-2020.03-x86_64-aarch64-linux-gnu
export LD_LIBRARY_PATH=/home/user/toolschain/gcc-ubuntu-9.3.0-2020.03-x86_64-aarch64-linux-gnu/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH 
export CROSS_COMPILE=/home/user/toolschain/gcc-ubuntu-9.3.0-2020.03-x86_64-aarch64-linux-gnu/bin/aarch64-linux-gnu-

不重要的报警

configure PCL时出现的部分信息，可以无视

1
2
3
4
Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE)                 # OMIT
Could NOT find LIBUSB_1 (missing: LIBUSB_1_LIBRARY LIBUSB_1_INCLUDE_DIR)  # OMIT
Could NOT find ZLIB (missing: ZLIB_LIBRARY ZLIB_INCLUDE_DIR) 
Could NOT find PNG (missing: PNG_LIBRARY PNG_PNG_INCLUDE_DIR)

编译的过程中，发现需要处理Boost, Flann, lz4, hdf5等库，它们也需要交叉编译。

交叉编译Boost

PCL需要Boost，cmake时会提示无法找到Boost的错误，不是本机的x86 Boost，是aarch64的Boost，也就是说需要交叉编译Boost。

下载Boost1.74源码，找到bootstrap.sh，然后执行

1
./bootstrap.sh  --with-libraries=system,filesystem,thread,date_time,iostreams --with-toolset=gcc

修改project-config.jam文件，把using gcc部分修改为
using gcc : arm : /home/user/toolschain/gcc-ubuntu-9.3.0-2020.03-x86_64-aarch64-linux-gnu/bin/aarch64-linux-gnu-gcc ;

找到gcc.jam文件，在430行左右，增加一行

1
2
3
4
5
6
# On Windows, fPIC is the default, and specifying -fPIC explicitly leads
# to a warning.
local non-windows = [ set.difference $(all-os) : cygwin windows ] ;
compile-link-flags <link>shared/<target-os>$(non-windows) : -fPIC ;
# 增加下面这行
compile-link-flags <link>static/<target-os>$(non-windows) : -fPIC ;

此步骤的主要目的是打开-fPIC，避免PCL在编译时找不到boost库的.a文件

进行编译并安装boost:

1
sudo ./b2 cxxflags=-fPIC cflags=-fPIC -a install

编译时可能会报错: error while loading shared libraries: libisl.so.22: cannot open shared object file: No such file

将交叉编译的toolschain地址的这几个库文件都复制到/usr/lib/x86_64-linux-gnu: libctf-arm64.so.0, libopcodes-2.34-arm64.so, libbfd-2.34-arm64.so, libisl.so.22。 如果没报错就不用了。

编译结束，我把生成的Boost库的文件都放到了/usr/local/lib。也就是/usr/lib/x86_64-linux-gnu中的libboost开头的库文件仍是x86的，/usr/local/lib中的libboost开头的库文件是AArch64架构。对于ARM的so库文件，在x86上用ldd命令查看会出现 not a dynamic executable

修改PCL中的pcl_find_boost.cmake文件

1
2
3
4
set(Boost_DIR  "/usr/lib/x86_64-linux-gnu/cmake")
 # Required boost modules
set(BOOST_REQUIRED_MODULES filesystem date_time iostreams system)
find_package(Boost 1.74.0 REQUIRED COMPONENTS ${BOOST_REQUIRED_MODULES})

lz4 1.9.1 arm64版本的交叉编译

下载源码后执行

1
2
3
make CC=/home/user/toolschain/gcc-ubuntu-9.3.0-2020.03-x86_64-aarch64-linux-gnu/bin/aarch64-linux-gnu-gcc
mkdir lz4_arm
make install PREFIX=$(pwd)/lz4_arm

运行ldconfig， 出现 /sbin/ldconfig.real: /lib/x86_64-linux-gnu/liblz4.so.1 is not a symbolic link

运行file /lib/liblz4.so，出现liblz4.so: broken symbolic link to liblz4.s

目前解决: 把地平线x3M的 /lib/liblz4.so 和 /lib/liblz4.so.1 复制到我的本机的 /lib目录， /usr/lib/x86_64-linux-gnu的几个 liblz4.so 仍然是 x86 平台。

hdf5的问题

编译flann时，链接hdf5出错

1
2
3
/home/user/toolschain/gcc-ubuntu-9.3.0-2020.03-x86_64-aarch64-linux-gnu/bin/../lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld:   /usr/lib/x86_64-linux-gnu/hdf5/serial/libhdf5.so:   error adding symbols: file in wrong format

collect2: error: ld returned 1 exit status

显然需要的是ARM的hdf5，这个库的交叉编译特别麻烦，最后解决方法是从一台ARM平台的主机上找到 hdf5文件夹，放到了 /usr/lib/x86_64-linux-gnu

如果交叉编译hdf5，参考:

  1. 使用cmake-gui进行configure和generate,第一次configure会失败,重新configure即可
  2. 进行make,两次,编译失败
  3. 拷贝bin/H5detect bin/H5make_libsettings libhdf5.settings到arm平台
  4. 在arm平台修改文件执行权限,执行H5detect和H5make_libsettings,把程序输出分别保存到H5Tinit.c和H5lib_settings.c
  5. 拷贝H5Tinit.c和H5lib_settings.c到主机的编译目录下,继续编译

交叉编译flann库

在一台ARM平台的设备上，flann库文件是这样的

1
2
3
4
5
6
7
8
9
/usr/lib/aarch64-linux-gnu/libflann.so
/usr/lib/aarch64-linux-gnu/libflann.so.1.9
/usr/lib/aarch64-linux-gnu/libflann.so.1.9.1

/usr/lib/aarch64-linux-gnu/libflann_cpp.so
/usr/lib/aarch64-linux-gnu/libflann_cpp.so.1.9
/usr/lib/aarch64-linux-gnu/libflann_cpp.so.1.9.1
/usr/lib/aarch64-linux-gnu/libflann_cpp_s.a
/usr/lib/aarch64-linux-gnu/libflann_s.a

链接关系:
libflann.so
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
ldd /usr/lib/aarch64-linux-gnu/libflann.so
    linux-vdso.so.1 (0x0000ffff85887000)
    libstdc++.so.6 => /lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000ffff8511b000)
    libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000ffff85070000)
    # 缺
    libgomp.so.1 => /lib/aarch64-linux-gnu/libgomp.so.1 (0x0000ffff85022000)

    libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000ffff84ffe000)
    libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffff84e8b000)
    /lib/ld-linux-aarch64.so.1 (0x0000ffff85857000)
    # 目前缺这两个
    libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000ffff84e77000)
    libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000ffff84e46000)


ldd /usr/lib/aarch64-linux-gnu/libflann_cpp.so
    linux-vdso.so.1 (0x0000ffff9fb1f000)
    # 目前需要交叉编译
    liblz4.so.1 => /lib/aarch64-linux-gnu/liblz4.so.1 (0x0000ffff9fa68000)
    libstdc++.so.6 => /lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000ffff9f883000)
    # 目前缺
    libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000ffff9f852000)
    libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffff9f6df000)
    /lib/ld-linux-aarch64.so.1 (0x0000ffff9faef000)
    libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000ffff9f634000)
    libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1

如果之前缺hdf5，编译会报警 hdf5 library not found, not compiling flann_example.cpp

解决方法: 把ARM平台的的flann库文件复制到了 /usr/lib/x86_64-linux-gnu，原来的x86的flann库文件复制到了新文件夹 /usr/lib/x86_64-linux-gnu/x86_flann

如果要交叉编译，需要修改CMakeLists:

1
2
3
4
5
pkg_check_modules(LZ4 REQUIRED liblz4)
#include_directories(${LZ4_INCLUDE_DIRS})
include_directories(/home/user/Downloads/lz4-1.9.1/lz4_arm/include)

link_directories(/home/user/Downloads/lz4-1.9.1/lz4_arm/lib)

现在继续使用cmake-gui交叉编译PCL，在界面中没有选择所有模块，执行Configure和Generate后，会有日志:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
编译的模块:
  common    kdtree     octree    search
  sample_consensus     filters    2d
  geometry     io      features   ml
  segmentation    surface     keypoints
  tracking     stereo    tools

不会编译的模块:
  visualization: VTK was not found.
  registration: Disabled manually.
  recognition: Requires registration.
  apps: Disabled: registration missing.
  outofcore: Requires visualization.
  examples: Code examples are disabled by default.
  people: Requires visualization.
  simulation: Disabled: visualization missing.
  global_tests: No reason
  tools: Disabled: registration missing.

然后去build文件夹找每个模块的Makefile，比如pcl-pcl-1.10.0/build/kdtree，然后执行make即可。

PCL交叉编译得到的so文件特别大，有的到了800M。使用strip瘦身，结果报错 strip: Unable to recognise the format of the input file ，因为so文件是aarch64的，应当使用gcc-ubuntu-9.3.0-2020.03-x86_64-aarch64-linux-gnu/aarch64-linux-gnu/bin/strip，结果让libpcl_features.so.1.10.0从797M 下降到 39M。

所有编译生成的库文件在pcl-pcl-1.10.0/build/lib，把这些库文件放到开发板上，比如路径/pcl_libs，所有Boost的库文件也放过去，比如/boost_libs。


测试程序

在本机上写一个PCL的测试程序，CMake部分:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
cmake_minimum_required(VERSION 3.5)

project(test_cross_pcl LANGUAGES CXX)

set(CMAKE_CXX_STANDARD 14)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

find_package(Boost COMPONENTS system filesystem thread REQUIRED)

include_directories(
    include
    /usr/include/eigen3

    /home/user/Downloads/pcl-pcl-1.10.0/common/include
    /home/user/Downloads/pcl-pcl-1.10.0/build/include
    /home/user/Downloads/pcl-pcl-1.10.0/io/include
    /home/user/Downloads/pcl-pcl-1.10.0/filters/include
)
LINK_DIRECTORIES(/home/user/Downloads/pcl-pcl-1.10.0/build/lib)
LINK_DIRECTORIES(/home/user/toolschain/gcc-ubuntu-9.3.0-2020.03-x86_64-aarch64-linux-gnu/aarch64-linux-gnu/lib)

add_executable(test_cross_pcl main.cpp)
target_link_libraries(test_cross_pcl
    -lpcl_common
    -lpcl_io
    -lpcl_filters
    -lpthread
)

不能使用本机上的编译器，还用cmake-gui进行交叉编译: configure, generate, 然后到对应的build文件夹里执行make，把生成的可执行文件放到开发板。

执行export LD_LIBRARY_PATH=/pcl_libs:/boost_libs:$LD_LIBRARY_PATH，也就是指定链接库的地址，再执行就成功了。

参考:Windows 10上源码编译PCL 1.8.1支持VTK和QT，可视化三维点云
交叉编译Boost

解决vcpkg无法交叉编译arm64版本 HDF5 库的问题
编译HDF5
Linux安装HDF5及遇到的问题总结

FLANN 1.9.2 源码编译
PCL与VTK