Side-load on embedded Linux with Buildroot

Embedded Linux boards and devices often come with a shortage of tools installed. Often enough they don’t use a “normal” Linux distribution that can be easily extended by the end user by means of installing additional packages via dnf, apt, pacman, etc. Old/ancient versions of the Linux kernel is also very common.

In my previous post I wrote about using evemu to record data coming from the input subsytem and replay it on my computer in order to develop a Remote Controller software for ArduPilot. Without surprise, evemu isn’t included in the tools that come with Disco or SkyController 2. They also don’t have a package manager neither. I could use their alchemy build system to regenerate the entire distro, but that is much more work than I wanted: just add a few binaries there to extend rather than to replace what is currently there.

I turned back to Buildroot for that. It has a great variety of software already “packaged”, it’s pretty straight forward to use and has good integration for toolchains. I have 2 use cases for it:

The first allows me to build software already integrated in Buildroot to extend the ones available in Disco while with the second it’s possible to build my own external projects: dema-rc and ArduPilot.

Using pre-packaged software

You can head to https://buildroot.org/download.html and download a stable release, but I prefer to just clone their git repository, which allows me to switch to different branches/tags as the need arises.

I’m building additional software for Disco and SkyController 2 from Parrot. Both of them use a P7 processor that is an ARM Cortex A9. To configure Buildroot we use kconfig, which will be familiar for those who work(ed) in the Linux kernel. We just call make menuconfig and select the options we want.

$ git clone git://git.buildroot.net/buildroot
$ cd buildroot
$ make menuconfig

The options for me are:

- Target options -->
  - Target Architecture: ARM little endian
  - Target Architecture variant: cortex-A9
  - Target ABI: EABIhf
  - Floating point strategy: VFPv3-D16

On the SkyController 2 I have very little space for the additional software (8 MB) so I like to enable these options, too:

- Build options -->
  - Strip target binaries
  - gcc optimization level: optimize for size
  - libraries: shared only

Now the toolchain part. You can either bring your own toolchain (e.g. provided by the manufacturer of your board) or you can use Buildroot’s own integrated toolchains. I prefer the second since it’s much easier than trying to find where the different vendors make them available for download and making sure that toolchain can build correctly the software already packaged. Also, Buildroot often has a more recent and better maintained version.

You may face some problems, though: if you use a much more recent version, your libc may contain different symbol versions than the libc available on the board. You will either have to make the new libc available on the board together with your programs (more on that later) or you may simply choose an older toolchain version to avoid the problem. Another problem that may happen is if the libc you are using or other software make use of syscalls that are not available in the (ancient) Linux kernel the vendor made available for that board. In my case it’s either version 3.10 or 4.0, depending on the version of the firmware I’m using on Disco/SK2. My advice: stay with a version close to what is used on the board, or be In my case I select the toolchain like below:

- Toolchain
  - Toolchain type: External toolchain
  - Toolchain: Linaro ARM 2018.05

After that, just select the software you want under the Target packages section and build with make. All needed software will be downloaded and built.

Since we are not really creating a complete rootfs for a board, I don’t care about building a kernel or creating a filesystem image. All I want is to build a few packages and have the toolchain available for my own projects.

After building those packages you can inspect output/target/ for the binaries and libraries compiled. In the end you want to bring (some of) those to the target board. You can copy the ones you need to the root filesystem in the board, making them available right away, or you can dedicate a separate directory for those. I prefer the second approach (hence the name “side-load” in this post) since then I’m sure I’m not overwriting a system library/binary and changing its behavior. When I want to execute them, setting LD_LIBRARY_PATH is sufficient in most of the cases, with the exception of when a configuration file is needed and the program tries to search for it in e.g. /etc. If you don’t want to copy everything, readelf helps to find the dependent libraries:

$ arm-linux-gnueabihf-readelf -d output/target/usr/bin/evemu-record  | grep NEEDED
0x00000001 (NEEDED)                     Shared library: [libevemu.so.3]
0x00000001 (NEEDED)                     Shared library: [libevdev.so.2]
0x00000001 (NEEDED)                     Shared library: [libm.so.6]
0x00000001 (NEEDED)                     Shared library: [libc.so.6]

For my needs I can use the system’s libm and libc, and just copy libevemu, libevdev, and evemu-record to /data/rootfs/usr/{lib,bin}/. Alchemy build system used by Disco comes from Android and inherits the filesystem layout: / is read-only and /data/ read-write. As said before, setting LD_LIBRARY_PATH=/data/rootfs/usr/lib is all I need most of the time.

Piggyback on Buildroot toolchain integration

Once we have Buildroot configured we could just integrate our software in there and make it part of the normal “distro”. However I prefer keeping my development software outside of any package management or rootfs builder: even though an incremental build in Buildroot is fast, it’s slower than an incremental build of just my single program. Also, having the toolchain setup to build simple test programs while I’m hacking on the board helps a lot.

The toolchain we set up in Buildroot is a cross toolchain. In order to use it all we need is to follow the instructions for our build system to cross-compile our software. Each build system has a slightly different way to do that. We also need a small tweak in Buildroot’s toolchain, because for some reason we don’t get a symlink for pkg-config in the usual triplet scheme like for other binaries:

$ cd ~/p/buildroot/output/host/bin
$ ln -s pkg-config arm-linux-gnueabihf-pkg-config

dema-rc uses meson as build system. You can find its instructions for cross compilation in its manual. I also suggest taking a look on instructions for other projects using meson. Examples: mesa and also its specific instructions for RPI. My configuration file for meson (`~/.local/share/meson/cross/disco.txt) looks like this:

[binaries]
c =         '/home/lucas/p/buildroot/output/host/bin/arm-linux-gnueabihf-gcc'
cpp =       '/home/lucas/p/buildroot/output/host/bin/arm-linux-gnueabihf-g++'
ar =        '/home/lucas/p/buildroot/output/host/bin/arm-linux-gnueabihf-ar'
strip =     '/home/lucas/p/buildroot/output/host/bin/arm-linux-gnueabihf-strip'
pkgconfig = '/home/lucas/p/buildroot/output/host/bin/arm-linux-gnueabihf-pkg-config'

[properties]
sys_root = '/home/lucas/p/buildroot/output/target/'

[host_machine]
system = 'linux'
cpu_family = 'arm'
cpu = 'armv7a'
endian = 'little'

For ArduPilot, since each build targets a certain board, we already set the extra configuration. The only thing you need is to have the compiler with the triplet scheme in your PATH. Then you compile for the disco board as usual:

$ ./waf configure --board disco
$ ./waf plane

More information about using a cross toolchain and generate a rootfs can be found in the crosstool-ng instructions.

blogroll

social