Embedded Linux boards and devices often come with a shortage of tools
installed. Often enough they don’t use a “normal” Linux distribution that can
be easily extended by the end user by means of installing additional packages
via dnf, apt, pacman, etc. Old/ancient versions of the Linux kernel is also
very common.
In my previous post I wrote
about using evemu to record data coming from the input subsytem and replay
it on my computer in order to develop a Remote Controller software
for ArduPilot. Without surprise, evemu isn’t included in the tools that come
with Disco or SkyController 2. They also don’t have a package manager neither.
I could use their alchemy build system to regenerate the entire
distro, but that is much more work than I wanted: just add a few binaries there
to extend rather than to replace what is currently there.
I turned back to Buildroot for that. It has a great
variety of software already “packaged”, it’s pretty straight forward to use
and has good integration for toolchains. I have 2 use cases for it:
The first allows me to build software already integrated in Buildroot to extend the ones
available in Disco while with the second it’s possible to build my own external projects:
dema-rc and ArduPilot.
Using pre-packaged software
You can head to
https://buildroot.org/download.html
and download a stable release, but I prefer to just clone their git
repository, which allows me to switch to different branches/tags as
the need arises.
I’m building additional software for Disco and SkyController 2 from
Parrot. Both of them use a P7 processor that is an ARM Cortex A9. To
configure Buildroot we use kconfig, which will be familiar for those
who work(ed) in the Linux kernel. We just call make menuconfig
and select the options we want.
$ git clone git://git.buildroot.net/buildroot
$ cd buildroot
$ make menuconfig
The options for me are:
- Target options -->
- Target Architecture: ARM little endian
- Target Architecture variant: cortex-A9
- Target ABI: EABIhf
- Floating point strategy: VFPv3-D16
On the SkyController 2 I have very little space for the additional
software (8 MB) so I like to enable these options, too:
- Build options -->
- Strip target binaries
- gcc optimization level: optimize for size
- libraries: shared only
Now the toolchain part. You can either bring your own toolchain (e.g. provided
by the manufacturer of your board) or you can use Buildroot’s own integrated
toolchains. I prefer the second since it’s much easier than trying to find
where the different vendors make them available for download and making sure
that toolchain can build correctly the software already packaged. Also, Buildroot
often has a more recent and better maintained version.
You may face some problems, though: if you use a much more recent version, your
libc may contain different symbol versions than the libc available on the board.
You will either have to make the new libc available on the board together with
your programs (more on that later) or you may simply choose an older toolchain
version to avoid the problem. Another problem that may happen is if the libc
you are using or other software make use of syscalls that are not available
in the (ancient) Linux kernel the vendor made available for that board.
In my case it’s either version 3.10 or 4.0, depending on the version of the
firmware I’m using on Disco/SK2. My advice: stay with a version close to
what is used on the board, or be In my case I select the toolchain
like below:
- Toolchain
- Toolchain type: External toolchain
- Toolchain: Linaro ARM 2018.05
After that, just select the software you want under the Target packages
section and build with make. All needed software will be downloaded and built.
Since we are not really creating a complete rootfs for a board, I don’t
care about building a kernel or creating a filesystem image. All I want is to
build a few packages and have the toolchain available for my own projects.
After building those packages you can inspect output/target/ for the binaries
and libraries compiled. In the end you want to bring (some of) those to the
target board. You can copy the ones you need to the root filesystem in the board,
making them available right away, or you can dedicate a separate directory for
those. I prefer the second approach (hence the name “side-load” in this post)
since then I’m sure I’m not overwriting a system library/binary and changing its
behavior. When I want to execute them, setting LD_LIBRARY_PATH is sufficient
in most of the cases, with the exception of when a configuration file is needed
and the program tries to search for it in e.g. /etc. If you don’t want to
copy everything, readelf helps to find the dependent libraries:
$ arm-linux-gnueabihf-readelf -d output/target/usr/bin/evemu-record | grep NEEDED
0x00000001 (NEEDED) Shared library: [libevemu.so.3]
0x00000001 (NEEDED) Shared library: [libevdev.so.2]
0x00000001 (NEEDED) Shared library: [libm.so.6]
0x00000001 (NEEDED) Shared library: [libc.so.6]
For my needs I can use the system’s libm and libc, and just copy libevemu,
libevdev, and evemu-record to /data/rootfs/usr/{lib,bin}/. Alchemy build
system used by Disco comes from Android and inherits the filesystem layout:
/ is read-only and /data/ read-write. As said before, setting
LD_LIBRARY_PATH=/data/rootfs/usr/lib is all I need most of the time.
Once we have Buildroot configured we could just integrate our software in
there and make it part of the normal “distro”. However I prefer keeping
my development software outside of any package management or rootfs
builder: even though an incremental build in Buildroot is fast, it’s
slower than an incremental build of just my single program. Also, having
the toolchain setup to build simple test programs while I’m hacking on
the board helps a lot.
The toolchain we set up in Buildroot is a cross toolchain. In order to
use it all we need is to follow the instructions for our build system
to cross-compile our software. Each build system has a slightly different
way to do that. We also need a small tweak in Buildroot’s toolchain,
because for some reason we don’t get a symlink for pkg-config in the
usual triplet scheme like for other binaries:
$ cd ~/p/buildroot/output/host/bin
$ ln -s pkg-config arm-linux-gnueabihf-pkg-config
dema-rc uses meson as build system. You can find its instructions for cross
compilation in its manual.
I also suggest taking a look on instructions for other projects using meson.
Examples: mesa and also its specific instructions
for RPI.
My configuration file for meson (`~/.local/share/meson/cross/disco.txt)
looks like this:
[binaries]
c = '/home/lucas/p/buildroot/output/host/bin/arm-linux-gnueabihf-gcc'
cpp = '/home/lucas/p/buildroot/output/host/bin/arm-linux-gnueabihf-g++'
ar = '/home/lucas/p/buildroot/output/host/bin/arm-linux-gnueabihf-ar'
strip = '/home/lucas/p/buildroot/output/host/bin/arm-linux-gnueabihf-strip'
pkgconfig = '/home/lucas/p/buildroot/output/host/bin/arm-linux-gnueabihf-pkg-config'
[properties]
sys_root = '/home/lucas/p/buildroot/output/target/'
[host_machine]
system = 'linux'
cpu_family = 'arm'
cpu = 'armv7a'
endian = 'little'
For ArduPilot, since each build targets a certain board, we already set the extra
configuration. The only thing you need is to have the compiler with the triplet scheme
in your PATH. Then you compile for the disco board as usual:
$ ./waf configure --board disco
$ ./waf plane
More information about using a cross toolchain and generate a rootfs can be found in
the crosstool-ng instructions.