Qt Creator cross-platform development in Stretch: exploration

pdo debian eng qt-cross

I have been asked to create a system that makes it easy to do cross-architecture Qt development with Debian Stretch.

The goal is this:

Do development with Qt Creator, including remote debugging on the target architecture
Compile using native code, to avoid running the compilers inside an emulator making the build slower
Leverage all of Debian as development environment, using existing Debian packages for build-dependencies
Set everything up just by installing a single .deb package, which ideally gives Qt Creator a new kit that just works

I did some exploration on this some time ago, and the Qt/KDE and cross-build people did some work on this, too.

Now I'm trying to build on all that. I need to target Stretch so I cannot build on the recent improvements in Qt packaging, but I can at least use the experience that went into those changes.

I have two sample Qt application to try and cross-build, one that depends on a non-Qt library (libmosquittopp-dev), and one that depends on a nasty Qt library (qtwebengine5-dev).

This post has the notes from the first day of trying out different strategies.

It begins

I imported the two projects in Qt creator, installed their amd64 dependencies and make sure they build for the current system, with the default kit, no cross-building yet.

That works, good.

Now let's see about the rest:

dpkg --add-architecture armhf
apt install crossbuild-essential-armhf

A cross-build kit for Qt Creator

I created a new armhf kit for Qt Creator:

Device type: Generic Linux Device
Device: my hi-fi
Sysroot: blank
C compiler: /usr/bin/arm-linux-gnueabihf-gcc
C++ compiler: /usr/bin/arm-linux-gnueabihf-g++
Qt version: /usr/lib/arm-linux-gnueabihf/qt5/bin/qmake

I ran qmake from Qt Creator and go this:

10:59:49: Starting: "/usr/lib/arm-linux-gnueabihf/qt5/bin/qmake" …/project.pro -spec linux-g++ CONFIG+=debug CONFIG+=qml_debug
sh: 1: /usr/lib/arm-linux-gnueabihf/qt5/bin/rcc: not found
10:59:50: The process "/usr/lib/arm-linux-gnueabihf/qt5/bin/qmake" exited normally.

rcc is provided by qtbase5-dev-tools, which cannot be coinstalled alongside other architectures' version of itself:

# apt install qtbase5-dev-tools:armhf
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  qtbase5-dev-tools
The following NEW packages will be installed:
  qtbase5-dev-tools:armhf
0 upgraded, 1 newly installed, 1 to remove and 0 not upgraded.
Need to get 651 kB of archives.
After this operation, 985 kB disk space will be freed.
Do you want to continue? [Y/n] q

I'll try with a very dirty hack:

# cd /usr/lib/arm-linux-gnueabihf/qt5/bin
# ln -s `which rcc` .
# ln -s `which uic` .
# ln -s `which moc` .
# ls -la
total 1944
drwxr-xr-x 2 root root    4096 Nov 29 11:04 .
drwxr-xr-x 7 root root    4096 Nov 28 17:05 ..
lrwxrwxrwx 1 root root      12 Nov 29 11:04 moc -> /usr/bin/moc
-rwxr-xr-x 1 root root 1982208 Jan 11  2017 qmake
lrwxrwxrwx 1 root root      12 Nov 29 11:04 rcc -> /usr/bin/rcc
lrwxrwxrwx 1 root root      12 Nov 29 11:04 uic -> /usr/bin/uic

This is ugly:

It places files in /usr outside /usr/local that are not maintained by the package manager
It places amd64 executables in /usr/lib/arm-linux-gnueabihf which should contain armhf code

Let's see what happens in Qt Creator:

11:14:35: Starting: "/usr/lib/arm-linux-gnueabihf/qt5/bin/qmake" …/project.pro -spec linux-g++ CONFIG+=debug CONFIG+=qml_debug
11:14:35: The process "/usr/lib/arm-linux-gnueabihf/qt5/bin/qmake" exited normally.
11:14:35: Starting: "/usr/bin/make" qmake_all
make: Nothing to be done for 'qmake_all'.
11:14:36: The process "/usr/bin/make" exited normally.

and build:

11:15:29: Starting: "/usr/bin/make"
g++ -c -pipe -std=c++11 -g -Wall -W -D_REENTRANT -fPIC -DQT_DEPRECATED_WARNINGS -DQT_QML_DEBUG -DQT_SVG_LIB -DQT_WIDGETS_LIB -DQT_GUI_LIB -DQT_CORE_LIB -I../project -I. -isystem /usr/include/arm-linux-gnueabihf/qt5 -isystem /usr/include/arm-linux-gnueabihf/qt5/QtSvg -isystem /usr/include/arm-linux-gnueabihf/qt5/QtWidgets -isystem /usr/include/arm-linux-gnueabihf/qt5/QtGui -isystem /usr/include/arm-linux-gnueabihf/qt5/QtCore -I. -I/usr/lib/arm-linux-gnueabihf/qt5/mkspecs/linux-g++ -o main.o ../project/main.cpp

Unfortunately, it is using g++ even though I configured the kit to use /usr/bin/arm-linux-gnueabihf-* instead.

Trying an explicit override in the .pro:

QMAKE_CXX = /usr/bin/arm-linux-gnueabihf-g++
QMAKE_LINK = /usr/bin/arm-linux-gnueabihf-g++

And the project builds and runs fine on the Raspberry Pi, using a kit and two simple overrides in the .pro, all build dependencies as packaged by Debian, and a toolchain that entirely runs on native code.

Entirely?

$ file /usr/lib/arm-linux-gnueabihf/qt5/bin/qmake
/usr/lib/arm-linux-gnueabihf/qt5/bin/qmake: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=1f5b063926570702f5568b1e5cec6c70d214fc73, stripped
$ dpkg-architecture -q DEB_HOST_ARCH
amd64
$ /usr/lib/arm-linux-gnueabihf/qt5/bin/qmake -v
QMake version 3.0
Using Qt version 5.7.1 in /usr/lib/arm-linux-gnueabihf

Indeed qmake is armhf code that runs happily on my amd64 laptop because I accidentally have qemu-user-static installed.

This strategy produces results, although it depends on a dirty hack. To summarise it:

Depend on crossbuild-essential-armhf, qemu-user-static, qemu-system-arm
Install :armhf versions of Qt and all build-dependencies
Symlink native moc, rcc, uic into the armhf bin directory (argh!)
Configure a kit as before
Override QMAKE_CXX and QMAKE_LINK in the .pro

Other things to investigate, besides removing the need for that dirty hack:

Find out if there is a way to package and install a kit for Qt Creator
Find out why qmake is ignoring the compiler settings from the kit and needs overrides in the .pro. Ideally the .pro should be unmodified, so that it can be used for all builds

A native qmake

Would it be possible to build a native version of qmake tweaked to point to all the right bits out of the box?

$ apt source qtbase-opensource-src-5.7.1+dfsg
$ qtbase-opensource-src-5.7.1+dfsg$
$ DEB_HOST_MULTIARCH=arm-linux-gnueabihf DEB_HOST_ARCH_BITS=32 debian/rules override_dh_auto_configure
MAKEFLAGS="-j1" ./configure \
            -confirm-license \
            -prefix "/usr" \
            -bindir "/usr/lib/arm-linux-gnueabihf/qt5/bin" \
            -libdir "/usr/lib/arm-linux-gnueabihf" \
            -docdir "/usr/share/qt5/doc" \
            -headerdir "/usr/include/arm-linux-gnueabihf/qt5" \
            -datadir "/usr/share/qt5" \
            -archdatadir "/usr/lib/arm-linux-gnueabihf/qt5" \
            -plugindir "/usr/lib/arm-linux-gnueabihf/qt5/plugins" \
            -importdir "/usr/lib/arm-linux-gnueabihf/qt5/imports" \
            -translationdir "/usr/share/qt5/translations" \
            -hostdatadir "/usr/lib/arm-linux-gnueabihf/qt5" \
            -sysconfdir "/etc/xdg" \
            -examplesdir "/usr/lib/arm-linux-gnueabihf/qt5/examples" \
            -opensource \
            -plugin-sql-mysql \
            -plugin-sql-odbc \
            -plugin-sql-psql \
            -plugin-sql-sqlite \
            -no-sql-sqlite2 \
            -plugin-sql-tds \
            -system-sqlite \
            -platform linux-g++ \
            -system-harfbuzz \
            -system-zlib \
            -system-libpng \
            -system-libjpeg \
            -system-doubleconversion \
            -openssl \
            -no-rpath \
            -verbose \
            -optimized-qmake \
            -dbus-linked \
            -no-strip \
            -no-separate-debug-info \
            -qpa xcb \
            -xcb \
            -glib \
            -icu \
            -accessibility \
            -compile-examples \
            -no-directfb \
            -gstreamer 1.0 \
            -plugin-sql-ibase -opengl desktop \
…
$ file bin/qmake
bin/qmake: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=ba868730cf34c54d7ddf6df0ab4b6ce5c7d6f2a0, not stripped
$ bin/qmake -v
QMake version 3.0
Using Qt version 5.7.1 in /usr/lib/arm-linux-gnueabihf

./configure in Qt builds qmake, and there is no need to run make afterwards. Good.

DEB_HOST_ARCH_BITS=32 is a hack I added to avoid debian/rules using -platform linux-g++-64 instead of -platform linux-g++.

Let's try to use that in Qt Creator:

sudo cp bin/qmake /usr/local/bin/qmake-arm-linux-gnueabihf

Qt Creator autodetects the new qmake and offers it as one of the available versions of Qt. Nice.

The need for the symlink hack is still there:

11:46:51: Starting: "/usr/local/bin/qmake-arm-linux-gnueabihf" ../project.pro -spec linux-g++-64 CONFIG+=debug CONFIG+=qml_debug
sh: 1: /usr/lib/arm-linux-gnueabihf/qt5/bin/rcc: not found

So is the need for the QMAKE_CXX and QMAKE_LINK overrides in the .pro.

Still, this way I could remove qemu-user-static from my system and the project still builds on my laptop and runs on my Raspberry Pi.

The qemu dependency is not needed anymore, the rest of the problems still stand. I wonder, since I'm rebuilding qmake, if there's a way to tell it to use the compilers I want, and the tools I want, removing the need for the dirty hack and the overrides in the .pro files.

qtwebengine5-dev

How about the other project that depends on qtwebengine5-dev?

# apt install qtwebengine5-dev qtwebengine5-dev:armhf
Reading package lists... Done
Building dependency tree
Reading state information... Done
qtwebengine5-dev is already the newest version (5.7.1+dfsg-6.1).
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 qtwebengine5-dev:armhf : Depends: libqt5webengine5:armhf (= 5.7.1+dfsg-6.1) but it is not going to be installed
                          Depends: libqt5webenginecore5:armhf (= 5.7.1+dfsg-6.1) but it is not going to be installed
                          Depends: libqt5webenginewidgets5:armhf (= 5.7.1+dfsg-6.1) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

It turns out that something in the big chain of dependencies of qtwebengine5-dev makes the amd64 and armhf versions not coinstallable.

It seems that I have to abandon the idea of installing armhf build-dependencies in the main development system, and I need to start considering to leave the host system untouched to do the native builds, and use a chroot for the cross-compilers and the cross-build-dependencies.

In the next post, I'll see how that goes.

Credits

This has been done as part of my work with Truelite.