| 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607 |
- <!DOCTYPE html><head><meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no" /><meta name="keywords" content="GNU, Emacs, Libre Software, Hurd, Guile, Guix" /><meta name="description" content="GNUcode.me is a website focusing on libre software projects, especially the GNU project." /><link type="application/atom+xml" rel="alternate" title="GNUcode.me -- Feed" href="/feed.xml" /><a rel="me" href="https://fosstodon.org/@thegnuguy"></a><link type="text/css" href="css/footer.min.css" rel="stylesheet"></link><link type="text/css" href="css/header.min.css" rel="stylesheet"></link><link type="text/css" href="css/main.min.css" rel="stylesheet"></link><title>Creating New Build Systems — GNUcode.me</title></head><body><header><nav><ul><li><a href="index.html">GNUcode.me</a></li><li><a href="services.html">Services</a></li><li><a href="about.html">About</a></li><li><a href="business-ideas.html">Business-ideas</a></li></ul></nav></header><h1>Creating New Build Systems</h1><main><section class="basic-section-padding"><article><h3>by Mitchell Schmeisser <mitchellschmeisser@librem.one> — February 24, 2023</h3><div><p>In Guix each package must specify a so-called <code>build-system</code>, which
- knows how to bring a package from its inputs to deployment.
- Build systems are responsible for setting up the environment and performing
- build actions within that environment. The most ubiquitous of these is the
- <a href="https://www.gnu.org/software/automake/manual/html_node/GNU-Build-System.html"><code>gnu-build-system</code></a>.
- Guix builds packages using this build system via the usual
- <code>./configure && make && make install</code> process.</p><p>Any package can alter its build system by removing some steps or
- adding extra ones. This is extremely common and almost every package
- makes some adjustment to the build process. A <code>build-system</code> in Guix
- hides away some of the common configuration choices. For example, there is
- no need to specify <code>make</code> or <code>gcc</code> as native inputs when using the <code>gnu-build-system</code>,
- because they are added implicitly when a package is lowered into a <em>bag</em>.</p><h2>Anatomy of a Guix Build System</h2><p>The job of a build system is to compile our <em>packages</em> into <em>bags</em>.
- Bags are a lower level representation of a package without all the bells and whistles
- (Makes sense since we are implementing them here),
- the bags are further refined to derivations which are used by the
- build daemon to create an isolated environment suitable for our
- <em>build phases</em>.</p><p>Below is how guix defines a build system.
- It's surprisingly simple with just three items, two of which are for human consumption.</p><pre><code class="language-scheme">(define-record-type* <build-system> build-system make-build-system
- build-system?
- (name build-system-name) ; symbol
- (description build-system-description) ; short description
- (lower build-system-lower)) ; args ... -> bags</code></pre><p>The last field <code>lower</code> is a function which takes the list of arguments
- given in the <code>(package ... (arguments ...))</code> field.
- The keyword arguments that we are allowed to supply when writing the
- package are defined by the build-system.</p><h1>Code Strata</h1><p>Guix builds are implemented in two parts.</p><ol><li><p>Code that compiles <code>packages->derivations</code>.</p><p>Derivations are the language the Guix Daemon speaks.
- They describe everything about how to <em>derive</em> our package
- from the inputs to the environment and all the code on how to drive
- the build tools.
- This code is run in a poorly defined "user" environment.
- Guix produces derivations that actually can be influenced by
- undeclared aspects of the environment, like manually installed Guile
- packages or code added with the `-L` flag.</p></li><li><p>The guix daemon runs the builder code in as isolated and reproducible build environment to produce the package from its inputs.</p><p>This code is executed in an explicitly defined build environment
- with nothing being introduced from the host.</p></li></ol><p>Code that runs in the host environment <em>stages</em> code, which will run in isolation.
- This is where G-Expressions really shine.
- They provide the syntax to describe this relationship.</p><h1>Build Phases</h1><p>All programs are built more or less the same way.</p><ol><li><p>Unpack the source code.</p><p>Whether it's tarball or a version controlled repository, the guix daemon must
- copy the software's source tree into our build environment.</p></li><li><p>Patch the shebangs.</p><p>Many projects contain scripts written to aid the build process.
- In Linux, executable scripts can contain a so-called <em>shebang</em>,
- which contains an absolute path to the program, which is meant to
- interpret it: e.g. <code>#!/bin/sh</code>. Most distributions provide a
- POSIX compliant shell interpreter at this location. Guix System does not do this.
- Instead, Guix System's <code>sh</code> is yet another component in the store, so all of these files must
- be patched to point to the new location, which is only known at
- build time. We cannot rely on our <code>PATH</code> to store this information,
- because the Linux kernel does not respect such things.</p></li><li><p>Configure the build.</p><p>Whether it is autotools or CMake or ninja, etc., if you are relying
- on tools from the host system, then you have a step, which enables the
- host system to tell you where to find those tools.</p></li><li><p>Patch the generated shebangs.</p><p>Sometimes the configure phases creates scripts, which run during the build phase,
- these often contain references to <code>/bin/sh</code>, and so guix must patch these scripts as well.</p></li><li><p>Build!</p><p>That's right folks we are off to the races, and the program is building.
- Usually this takes the form of a call to <code>make</code>, with a handful of
- flags and last minute definitions, but there are other more <em>exotic</em> methods.</p></li><li><p>Test.</p><p>Now that guix built our software, we should test it before we sent software
- updates to users. This helps the guix project catch and eleminate software
- bugs before they impact guix users. Not all packages have tests, and guix
- developers occasionally disables some packages' testsuites, but the
- guix policy is to run the software's testsuite, if it is exists.</p></li><li><p>Install.</p><p>Now that we've verified everything works as expected, it is time to run
- <code>make install</code> or the equivalent. This phase copies our build artifacts into
- their final resting place in the store.</p></li><li><p>Validate the Installation.</p><p>Here guix checks that our outputs do not contain any component paths, which
- are not specified by the package inputs. That would lead to incomplete
- deployment, harm <a href="https://reproducible-builds.org/">reproducible builds</a>,
- and would be "bad".</p></li></ol><p>There are more steps, but they are not universally applicable.
- Of course no generic model, such as this, captures all the chaos of the
- world, and every package has exceptions to it.</p><p>Guix implements <em>build phases</em> as a list of lambdas.
- As such our package can modify this list and add/delete/replace
- lambdas as they require. It is so common Guix provides a syntax
- for manipulating build phases: <code>modify-phases</code>.</p><p>A build system contains a default set of phases called <code>%standard-phases</code>.
- Every build system starts with the <code>gnu-build-system</code>, <code>%standard-phases</code>,
- and modifies it to their needs.
- It is then provided to the packages using that build system.</p><p>The <code>cmake-build-system</code> is nearly identical to the <code>gnu-build-system</code>
- except two phases.</p><pre><code class="language-scheme">;;
- ;; Excerpt from `guix/build/cmake-build-system.scm`
- ;;
- (define %standard-phases
- ;; Everything is the same as the GNU Build System, except for the `configure'
- ;; and 'check' phases.
- (modify-phases gnu:%standard-phases
- (delete 'bootstrap)
- (replace 'check check)
- (replace 'configure configure)))</code></pre><h2>The Zephyr Build System</h2><p>Zephyr uses <code>cmake</code> to perform <em>physical component composition</em>.
- It searches the filesystem and generates scripts, which the toolchain will
- use to successfully combine those components into a firmware image.</p><p>The fact that Zephyr provides this mechanism is one reason I chose to
- target zephyr in the first place.</p><p>This separation of projects in an embedded context is a really great thing.
- It brings many of the advantages to the Linux world such as
- code re-use, smaller binaries, more efficient cache/RAM usage, etc.
- It also allows us to work as independent groups and compose
- contributions from many teams.</p><p>It also brings all of the complexity. Suddenly all of the problems
- that plague traditional deployment now apply to our embedded
- system. The fact that the libraries are statically linked at compile
- time instead of dynamically at runtime is simply an implementation detail.</p><p>We now have dependencies which we must track and compose in the correct environments.
- The Zephyr Project offers a tool to help manage this new complexity: <a href="https://docs.zephyrproject.org/latest/develop/west/index.html">The West Meta-tool</a>!
- To call it a "meta-tool" is an abuse of language. It is not even meta
- enough to bootstrap itself and relies on <a href="https://docs.zephyrproject.org/latest/develop/getting_started/index.html">other package managers</a> to get
- started.
- In fact, it is not even suitable for managing multiple embedded projects as a composition!
- The project recommends <a href="https://docs.zephyrproject.org/latest/build/sysbuild/index.html">Yet Another Tool</a> for this very common use case.</p><p>In fact, West does not really bring anything new to the table, and we can
- replace it in the same way we can replace every other language specific package
- manager, like node (js), Cabol (haskell), Dub (D), etc. <strong>PUTTING SOFTWARE ON
- THE SYSTEM IS THE JOB OF THE PACKAGE MANAGER!</strong>.</p><p>West is the way it is for a reason. It is very practical to design the package
- manager in this way, because it enables Windows users to access the build
- environment. A more in-depth discussion on the material conditions, which lead
- to this or that design decision of the West tool is beyond the scope of this
- post. Let's instead explain how to provide a meta-tool and bootstrap complex
- embedded systems, from the ground up, in a flexible, composable, and
- <em>reproducible</em> way.</p><h1>Why not use <code>cmake-build-system</code>?</h1><p>A fair question! Zephyr's CMake scripts require additional information
- about the build to be provided at configure time. Most tedius for
- zephyr-packages is the <code>ZEPHYR_MODULES</code> variable, which must be formatted and
- passed to CMake on the command line.</p><h1>Host Side</h1><p>Our job at this level is to take packages described using the <code>package</code> syntax and
- lower it into a derivation that the guix-daemon can understand.</p><p>Here is the derivation for building hello world for the <code>frdm_k64f</code> (hashes removed for brevity).
- The <code>package</code> syntax provides a human friendly veneer over this garbage.</p><pre><code class="language-sh">Derive
- ([("debug","/gnu/store/...-zephyr-hello-world-newlib-frdm-k64f-3.1.0-0.zephyr--debug","","")
- ,("out","/gnu/store/...-zephyr-hello-world-newlib-frdm-k64f-3.1.0-0.zephyr-","","")]
- ,[("/gnu/store/...-newlib-nano-3.3.drv",["out"])
- ,("/gnu/store/...-hal-nxp-3.1.0-0.708c958.drv",["out"])
- ,("/gnu/store/...-make-4.3.drv",["out"])
- ,("/gnu/store/...-findutils-4.8.0.drv",["out"])
- ,("/gnu/store/...-grep-3.6.drv",["out"])
- ,("/gnu/store/...-sed-4.8.drv",["out"])
- ,("/gnu/store/...-ld-wrapper-0.drv",["out"])
- ,("/gnu/store/...-bash-minimal-5.1.8.drv",["out"])
- ,("/gnu/store/...-hal-cmsis-5.8.0-0.093de61.drv",["out"])
- ,("/gnu/store/...-gawk-5.1.0.drv",["out"])
- ,("/gnu/store/...-gzip-1.10.drv",["out"])
- ,("/gnu/store/...-python-six-1.16.0.drv",["out"])
- ,("/gnu/store/...-zephyr-3.1.0-0.zephyr--checkout.drv",["out"])
- ,("/gnu/store/...-linux-libre-headers-5.10.35.drv",["out"])
- ,("/gnu/store/...-python-3.9.9.drv",["out"])
- ,("/gnu/store/...-diffutils-3.8.drv",["out"])
- ,("/gnu/store/...-arm-zephyr-eabi-nano-toolchain-12.1.0.drv",["out"])
- ,("/gnu/store/...-python-pyelftools-0.28.drv",["out"])
- ,("/gnu/store/...-guile-3.0.7.drv",["out"])
- ,("/gnu/store/...-python-dateutil-2.8.2.drv",["out"])
- ,("/gnu/store/...-patch-2.7.6.drv",["out"])
- ,("/gnu/store/...-gcc-10.3.0.drv",["out"])
- ,("/gnu/store/...-bzip2-1.0.8.drv",["out"])
- ,("/gnu/store/...-dtc-1.6.1.drv",["out"])
- ,("/gnu/store/...-gcc-cross-sans-libc-arm-zephyr-eabi-12.1.0.drv",["out"])
- ,("/gnu/store/...-cmake-minimal-3.21.4.drv",["out"])
- ,("/gnu/store/...-python-pyyaml-6.0.drv",["out"])
- ,("/gnu/store/...-python-packaging-21.3.drv",["out"])
- ,("/gnu/store/...-arm-zephyr-eabi-binutils-2.38.drv",["out"])
- ,("/gnu/store/...-glibc-2.33.drv",["out","static"])
- ,("/gnu/store/...-qemu-7.2.0.drv",["out"])
- ,("/gnu/store/...-ninja-1.10.2.drv",["out"])
- ,("/gnu/store/...-tar-1.34.drv",["out"])
- ,("/gnu/store/...-xz-5.2.5.drv",["out"])
- ,("/gnu/store/...-binutils-2.37.drv",["out"])
- ,("/gnu/store/...-python-pykwalify-1.7.0.drv",["out"])
- ,("/gnu/store/...-zephyr-3.1.0-0.zephyr-.drv",["out"])
- ,("/gnu/store/...-glibc-utf8-locales-2.33.drv",["out"])
- ,("/gnu/store/...-gdb-arm-zephyr-eabi-12.1.drv",["out"])
- ,("/gnu/store/...-module-import-compiled.drv",["out"])
- ,("/gnu/store/...-file-5.39.drv",["out"])
- ,("/gnu/store/...-python-pyparsing-3.0.6.drv",["out"])
- ,("/gnu/store/...-python-docopt-0.6.2.drv",["out"])
- ,("/gnu/store/...-arm-zephyr-eabi-sdk-0.15.0.drv",["out"])
- ,("/gnu/store/...-coreutils-8.32.drv",["out"])]
- ,["/gnu/store/...-zephyr-hello-world-newlib-frdm-k64f-3.1.0-0.zephyr--builder","/gnu/store/...-module-import"]
- ,"x86_64-linux","/gnu/store/...-guile-3.0.7/bin/guile",["--no-auto-compile"
- ,"-L","/gnu/store/...-module-import"
- ,"-C","/gnu/store/...-module-import-compiled"
- ,"/gnu/store/...-zephyr-hello-world-newlib-frdm-k64f-3.1.0-0.zephyr--builder"]
- ,[("debug","/gnu/store/...-zephyr-hello-world-newlib-frdm-k64f-3.1.0-0.zephyr--debug")
- ,("out","/gnu/store/...-zephyr-hello-world-newlib-frdm-k64f-3.1.0-0.zephyr-")])</code></pre><h3>Lowering packages to bags</h3><p>We must provide the build system with a function which the following lambda form.</p><pre><code class="language-scheme">(lambda* (name #:key #:allow-other-keys) ...)</code></pre><p>This means it takes one required argument <code>name</code> and any amount of keys.
- Individual procedures can specify keys they are interested in such as
- <code>inputs</code> or <code>outputs</code>.</p><p>Which keys are ultimately supported is defined by our <code>lower</code> function
- and our <em>build phases</em>.</p><pre><code class="language-scheme">;; Use module-ref instead of referencing the variables directly
- ;; to avoid circular dependencies.
- (define %zephyr-build-system-modules
- `((mfs build zephyr-build-system)
- ,@%cmake-build-system-modules))
- (define default-zephyr-base
- (module-ref (resolve-interface '(mfs packages zephyr))
- 'zephyr))
- (define default-zephyr-sdk
- (module-ref (resolve-interface '(mfs packages zephyr))
- 'arm-zephyr-eabi-sdk))
- (define default-ninja
- (module-ref (resolve-interface '(gnu packages ninja))
- 'ninja))
- (define default-cmake
- (module-ref (resolve-interface '(gnu packages cmake))
- 'cmake-minimal))
- (define* (lower name
- #:key source inputs native-inputs outputs system target
- (zephyr default-zephyr-base)
- (sdk default-zephyr-sdk)
- (ninja default-ninja)
- (cmake default-cmake)
- #:allow-other-keys
- #:rest arguments)
- "Return a bag for NAME."
- (define private-keywords `(#:zephyr #:inputs #:native-inputs #:target))
- (bag
- (name name)
- (system system)
- (target target)
- (build-inputs `(,@(if source `(("source" ,source)) '())
- ,@`(("cmake" ,cmake))
- ,@`(("zephyr-sdk" ,sdk))
- ,@`(("zephyr" ,zephyr))
- ,@`(("ninja" ,ninja))
- ,@native-inputs
- ,@(standard-packages)))
- ;; Inputs need to be available at build time
- ;; since everything is statically linked.
- (host-inputs inputs)
- (outputs outputs)
- (build zephyr-build)
- (arguments (strip-keyword-arguments private-keywords arguments))))</code></pre><p>Here our <code>lower</code> function provides default values for the packages
- every zephyr package needs, the SDK, CMake, and <code>ZEPHYR_BASE</code> and adds
- them to the build-inputs.</p><p>Notice we also strip out some keywords, which do not get passed to the
- build function, because they get included as part of the broader
- abstractions the build system provides.</p><p>At this step it would be great to have a parser which could work out
- the required sdk from a <code>.config,</code> but this requires compiling the kconfig,
- which requires at least the sdk cmake files.
- There might be a way to make it happen, but until then if a board needs a different
- sdk, then they can specify it in an argument keyword.</p><h3>Lowering Bags to Derivations</h3><p>Here is the definition for the actual build procedure. There is a lot
- of abstract trickery going on here, so do not worry if you don't understand it,
- I barely understand it!
- It's mostly copy and pasted from the CMake build system.</p><pre><code class="language-scheme">(define* (zephyr-build name inputs
- #:key guile source
- board
- (outputs '("out")) (configure-flags ''())
- (search-paths '())
- (make-flags ''())
- (out-of-source? #t)
- (tests? #f)
- (test-target "test")
- (parallel-build? #t) (parallel-tests? #t)
- (validate-runpath? #f)
- (patch-shebangs? #t)
- (phases '%standard-phases)
- (system (%current-system))
- (substitutable? #t)
- (imported-modules %zephyr-build-system-modules)
- ;; The modules referenced here contain code
- ;; which will be staged in the build environment with us.
- ;; Our build gexp down below will only be able to access this code
- ;; and we must be careful not to reference anything else.
- (modules '((zephyr build zephyr-build-system)
- (guix build utils))))
- "Build SOURCE using CMAKE, and with INPUTS. This assumes that SOURCE
- provides a 'CMakeLists.txt' file as its build system."
- ;; This is the build gexp. It handles staging values from our host
- ;; system into code that our build system can run.
- (define build
- (with-imported-modules imported-modules
- #~(begin
- (use-modules #$@(sexp->gexp modules))
- #$(with-build-variables inputs outputs
- #~(zephyr-build #:source #+source
- #:system #$system
- #:outputs %outputs
- #:inputs %build-inputs
- #:board #$board
- #:search-paths '#$(sexp->gexp
- (map search-path-specification->sexp
- search-paths))
- #:phases #$(if (pair? phases)
- (sexp->gexp phases)
- phases)
- #:configure-flags #$(if (pair? configure-flags)
- (sexp->gexp configure-flags)
- configure-flags)
- #:make-flags #$make-flags
- #:out-of-source? #$out-of-source?
- #:tests? #$tests?
- #:test-target #$test-target
- #:parallel-build? #$parallel-build?
- #:parallel-tests? #$parallel-tests?
- #:validate-runpath? #$validate-runpath?
- #:patch-shebangs? #$patch-shebangs?
- #:strip-binaries? #f)))))
- (mlet %store-monad ((guile (package->derivation (or guile (default-guile))
- system #:graft? #f)))
- (gexp->derivation name build
- #:system system
- #:target #f
- #:graft? #f
- #:substitutable? substitutable?
- #:guile-for-build guile)))</code></pre><p>Finally we define our build system which the package definitions can reference.</p><pre><code class="language-scheme">(define zephyr-build-system
- (build-system
- (name 'zephyr)
- (description "The standard Zephyr build system")
- (lower lower)))</code></pre><p>Easy right?</p><h1>Build Side</h1><p>The build side is not as complex as you might initially expect.
- Our build system is almost exactly the same as the CMake build system
- except our configure phase passes different values to CMake.
- Our job is much easier.</p><h3>Locating Modules</h3><p>Zephyr CMake requires the zephyr <em>modules</em> which are needed for the
- build to be supplied on the command line.
- Unfortunately for us, the
- <a href="https://docs.zephyrproject.org/3.1.0/develop/application/index.html#important-build-vars">documentation</a>
- is wrong, and the <code>ZEPHYR_MODULES</code> environment variable is not honored.
- Thus we must implement some other solution for locating modules, until
- this that is fixed.</p><p><strong>Input Scanning</strong> - Lucky for us we are keeping detailed information
- about our dependencies. It is a simple matter to write a file tree
- walker which collects all the zephyr modules in our inputs.</p><pre><code class="language-scheme">(define* (find-zephyr-modules directories)
- "Return the list of directories containing zephyr/module.yml found
- under DIRECTORY, recursively. Return the empty list if DIRECTORY is
- not accessible."
- (define (module-directory file)
- (dirname (dirname file)))
- (define (enter? name stat result)
- ;; Skip version control directories.
- ;; Shouldn't be in the store but you never know.
- (not (member (basename name) '(".git" ".svn" "CVS"))))
- (define (leaf name stat result)
- ;; Add module root directory to results
- (if (and (string= "module.yml" (basename name))
- (string= "zephyr" (basename (dirname name))))
- (cons (module-directory name) result)
- result))
- (define (down name stat result) result)
- (define (up name stat result) result)
- (define (skip name stat result) result)
- (define (find-modules directory)
- (file-system-fold enter? leaf down up skip error
- '() (canonicalize-path directory)))
- (append-map find-modules directories))
- (define (zephyr-modules-cmake-argument modules)
- "Return a proper CMake list from MODULES, a list of filepaths"
- (format #f "-DZEPHYR_MODULES='~{~a~^;~}'" modules))
- </code></pre><p>Here are two functions. The first one <code>find-zephyr-modules</code> walks a
- list of directories (package inputs) and returns a list of every module it finds.
- The second one is just for syntactic convenience when writing the CMake invokation.
- This is also slightly more robust than West's module discovery, because it allows for
- a single repository to provide multiple modules which are not <em>technically</em> required
- to be at the top level.</p><p>From here we just need to provide alternate implementations of <code>configure</code> and <code>install</code>.</p><pre><code class="language-scheme">(define* (configure #:key outputs (configure-flags '())
- inputs (out-of-source? #t)
- build-type
- #:allow-other-keys)
- "Configure the given package."
- (let* ((out (assoc-ref outputs "out"))
- (abs-srcdir (getcwd))
- (srcdir (if out-of-source?
- (string-append "../" (basename abs-srcdir))
- ".")))
- (format #t "source directory: ~s (relative from build: ~s)~%"
- abs-srcdir srcdir)
- (when out-of-source?
- (mkdir "../build")
- (chdir "../build"))
- (format #t "build directory: ~s~%" (getcwd))
- ;; this is required because zephyr tries to optimize
- ;; future calls to the build scripts by keep a cache.
- (setenv "XDG_CACHE_HOME" (getcwd))
- (let ((args `(,srcdir
- ,@(if build-type
- (list (string-append "-DCMAKE_BUILD_TYPE="
- build-type))
- '())
- ;; enable verbose output from builds
- "-DCMAKE_VERBOSE_MAKEFILE=ON"
- ,(zephyr-modules-cmake-argument
- (find-zephyr-modules (map cdr inputs)))
- ,@configure-flags)))
- (format #t "running 'cmake' with arguments ~s~%" args)
- (apply invoke "cmake" args))))
- (define* (install #:key outputs #:allow-other-keys)
- (let* ((out (string-append (assoc-ref outputs "out") "/lib/firmware"))
- (dbg (string-append (assoc-ref outputs "debug") "/share/zephyr")))
- (mkdir-p out)
- (mkdir-p dbg)
- (copy-file "zephyr/.config" (string-append dbg "/config"))
- (copy-file "zephyr/zephyr.map" (string-append dbg "/zephyr.map"))
- (copy-file "zephyr/zephyr.elf" (string-append out "/zephyr.elf"))
- (copy-file "zephyr/zephyr.bin" (string-append out "/zephyr.bin"))))
- ;; Define new standard-phases
- (define %standard-phases
- (modify-phases cmake:%standard-phases
- (replace 'configure configure)
- (replace 'install install)))
- ;; Call cmake build with our new phases
- (define* (zephyr-build #:key inputs (phases %standard-phases)
- #:allow-other-keys #:rest args)
- (apply cmake:cmake-build #:inputs inputs #:phases phases args))</code></pre><p>One thing to note is the "debug" output. This exists so we don't
- retain references to our build environment and make the file system
- closure huge. If you put all of the build outputs in the same store
- path, then the deployment closure will grow from 2MB to 833MB.</p><h1>Defining Zephyr Packages</h1><p>Now that we have a proper build system, it's time to define some packages!</p><h3>Zephyr Base</h3><p>Zephyr base contains the Zephyr source code. It is equivalent (in my mind
- anyway) to the linux kernel, in that packages' definitions' specifications,
- which target the linux kernel, can be minimal.</p><p>The selection of operating system actually comes from the toolchain. For
- example, we build linux packages with the <a href="https://www.gnu.org/software/autoconf/manual/autoconf-2.65/html_node/Specifying-Target-Triplets.html">gnu
- triplet</a>.
- When we select the <code>arm-linux-gnueabihf,</code> we are specifying our operating system.</p><p>It is the same for Zephyr. When we build for zephyr we use the <code>arm-zephyr-eabi</code>
- toolchain. However, unlike linux applications, zephyr applications are
- embedded firmware images and are generally statically linked.
- Thus this package just consists of its source code and is not compiled directly.
- We cannot compile it now because applications/modules provide the required Kconfig
- options.</p><pre><code class="language-scheme">(define-public zephyr
- (let ((version "3.1.0")
- (commit "zephyr-v3.1.0"))
- (package
- (name "zephyr")
- (version version)
- (home-page "https://zephyrproject.org")
- (source (origin (method git-fetch)
- (uri (git-reference
- (url "https://github.com/zephyrproject-rtos/zephyr")
- (commit commit)))
- (file-name (git-file-name name version))
- (sha256
- (base32 "1yl5y9757xc3l037k3g1dynispv6j5zqfnzrhsqh9cx4qzd485lx"))
- (patches
- ;; this patch makes this package work in a symlinked profile
- (search-patches "zephyr-3.1-linker-gen-abs-path.patch"))))
- (build-system copy-build-system)
- (arguments
- `(#:install-plan
- '(("." "zephyr-workspace/zephyr"))
- #:phases
- (modify-phases %standard-phases
- (add-after 'unpack 'patch-cmake-scripts
- (lambda* _
- (format #t "~a~&" (getcwd))
- ;; Some cmake scripts assume the presence of a
- ;; git repository in the source directory.
- ;; We will just hard-code that information now
- (substitute* "CMakeLists.txt"
- (("if\\(DEFINED BUILD_VERSION\\)" all)
- (format #f "set(BUILD_VERSION \"~a-~a\")~&~a"
- ,version ,commit all))))))))
- (propagated-inputs
- (list python-3
- python-pyelftools
- python-pykwalify
- python-pyyaml
- python-packaging))
- (native-search-paths
- (list (search-path-specification
- (variable "ZEPHYR_BASE")
- (files '("zephyr-workspace/zephyr")))))
- (synopsis "Source code for zephyr rtos")
- (description "Zephyr rtos source code.")
- (license license:apsl2))))
- (define-public zephyr-3.2.0-rc3
- (package (inherit zephyr)
- (version "3.2.0-rc3")
- (source (origin (method git-fetch)
- (uri (git-reference
- (url "https://github.com/zephyrproject-rtos/zephyr")
- (commit "v3.2.0-rc3")))
- (file-name (git-file-name "zephyr" version))
- (sha256
- (base32 "06ksd9zj4j19jq0zg3lms13jx0gxzjc41433zgb91cnd2cqmn5cb"))
- (patches
- (search-patches "zephyr-3.1-linker-gen-abs-path.patch"))))))</code></pre><p>Here we use the <code>copy-build-system</code> which takes a list of source destination
- pairs. In our case, we just copy everything to the output directory, but not
- before patching some files to accomodate our special environment.</p><p>While developing this I wanted to test some toolchain/board features
- on the latest release of Zephyr. I included an example of that package
- definition to show how we can easily accomodate side by side package
- variants and experiment without breaking anything.</p><h3>Modules</h3><p>It's finally time to define some firmware!
- Zephyr packages some examples in <code>$ZEPHYR_BASE/samples</code> including a
- basic hello world. The k64 development board is already supported
- by Zephyr so building the example is trivial.</p><p>In order to actually target the k64 we need to two modules, the NXP
- hardware abstraction layer, and CMSIS.
- Looking at <code>$ZEPHYR_BASE/west.yml</code> we can see the repositories
- and commits which contain these modules. This is how West does
- dependency management.</p><p>Defining these packages is not so bad (see footnote 1).</p><pre><code class="language-scheme">(define-public hal-cmsis
- (package
- (name "hal-cmsis")
- (version "5.8.0")
- (home-page "https://developer.arm.com/tools-and-software/embedded/cmsis")
- (source (origin
- (method git-fetch)
- (uri (git-reference
- (url "https://github.com/zephyrproject-rtos/cmsis")
- (commit "093de61c2a7d12dc9253daf8692f61f793a9254a")))
- (file-name (git-file-name name version))
- (sha256
- (base32 "0f7cipnwllna7iknsnz273jkvrly16yr6wm4y2018i6njpqh67wi"))))
- (build-system zephyr-module-build-system)
- (arguments `(#:workspace-path "/modules/hal/cmsis"))
- (synopsis "Zephyr module providing the Common Microcontroller
- Software Interface Standard")
- (description "Zephyr module providing the Common Microcontroller
- Software Interface Standard")
- (license license:apsl2)))
- (define-public hal-nxp
- (package
- (name "hal-nxp")
- (version "3.1.0")
- (home-page "https://nxp.com")
- (source (origin
- (method git-fetch)
- (uri (git-reference
- (url "https://github.com/zephyrproject-rtos/hal_nxp")
- (commit "708c95825b0d5279620935a1356299fff5dfbc6e")))
- (file-name (git-file-name name version))
- (sha256
- (base32 "1c0i26bpk6cyhr1q4af183jclfmxshv4d15i7k3cz7brzb12m8q1"))))
- (build-system zephyr-module-build-system)
- (arguments `(#:workspace-path "/modules/hal/nxp"))
- (native-search-paths
- (list (search-path-specification
- (variable "ZEPHYR_MODULES")
- (files `(,(string-append %zephyr-workspace-name module-path)))
- (separator ";"))))
- (synopsis "Zephyr module for NXP Hardware Abstraction Layer")
- (description "Provides sources for NXP HAL zephyr module")
- (license license:bsd-3)))</code></pre><p>With these two modules defined we can write <code>zephyr-hello-world-frdm-k64f</code>.</p><h3>Hello world</h3><pre><code class="language-scheme">(define-public zephyr-hello-world-frdm-k64f
- (package
- (name "zephyr-hello-world-frdm-k64f")
- (version (package-version zephyr))
- (home-page "https://zephyrproject.org")
- (source (file-append (package-source zephyr)
- "/samples/hello_world"))
- (build-system zephyr-build-system)
- (arguments
- '(#:configure-flags '("-DBOARD=frdm_k64f")))
- (outputs '("out" "debug"))
- (inputs
- (list hal-cmsis
- hal-nxp))
- (synopsis "Hello world example from Zephyr Project")
- (description "Sample package for zephyr project")
- (license license:apsl2)))</code></pre><h3>Building</h3><p>Our above definition can be built using the following:
- When testing package definitions, I use the <code>-L</code> flag to point to the local
- repository to load the new packages. I will be omitting that flag as
- if I had ~guix pull~ed successfully from <a href="https://github.com/paperclip4465/guix-zephyr">guix-zephyr</a>.</p><pre><code class="language-sh">guix build zephyr-hello-world-frdm-k64f
- /gnu/store/...-zephyr-hello-world-frdm-k64f-3.1.0-debug
- /gnu/store/...-zephyr-hello-world-frdm-k64f-3.1.0</code></pre><p>This actually doesn't fully test our toolchain. The hello world
- example, by default, will use a zephyr provided
- minimal implementation of the C library and will not link against
- newlib.</p><pre><code class="language-scheme">(define-public zephyr-hello-world-newlib-frdm-k64f
- (package
- (inherit zephyr-hello-world-frdm-k64f)
- (name "zephyr-hello-world-newlib-frdm-k64f")
- (arguments
- (substitute-keyword-arguments (package-arguments zephyr-hello-world-frdm-k64f)
- ((#:configure-flags flags)
- `(append
- '("-DCONFIG_MINIMAL_LIBC=n"
- "-DCONFIG_NEWLIB_LIBC=y")
- ,flags))))))
- guix build zephyr-hello-world-newlib-frdm-k64f
- /gnu/store/...-zephyr-hello-world-newlib-frdm-k64f-3.1.0-debug
- /gnu/store/...-zephyr-hello-world-newlib-frdm-k64f-3.1.0</code></pre><p>Woohoo!</p><h1>Further Musings</h1><p>One thing I learned while going through the pains of getting this
- working is that even though the components are "modular" there is
- still a lot rigid interdependencies, especially on the zephyr base.
- Just having two versions of Zephyr in the code base made component
- composition fragile.
- Modules rely on specific features from the kernel.
- This is hidden from developers normally by west automagically walking
- the `west.yml` of all of the declared dependencies recursively to
- discover the graph.</p><p>While there are many benefits to a modularized build system, a monolithic build
- system, like Zephyr, does have many benefits too.</p><p>Part of the problem comes from the domain itself. If you really want to be able
- to target the most resource constrained systems and deal with the "industrial
- mess" that comes from every board being unique, you have to be as generic and
- flexible as possible, which is hard in a guix-like modular build system.</p><p>Superficially the problem is solved in the same way Linux solved it, using
- device trees and having a very stable userland interface. However, unlike Linux
- where the device tree is compiled to a binary blob and interpreted by drivers at
- runtime, Zephyr device trees are compiled to a <code>.h</code> file and are mostly
- interpreted by the C pre-processor using an elaborate set of macros.</p><p>It goes beyond simply abstracting the hardware using clever hacks.
- Zephyr applications (and any zephyr module) can also introduce new
- "kernel" code, configuration options, and even linker script fragments
- at build time.
- Essentially the Zephyr CMake build system acts like a reverse ld.
- Instead of linking libraries after compilation, it discovers these
- things before gcc is ever invoked and provides additional code
- generation steps.</p><p>Zephyr does not have a stable "userland" interface for the same
- reason Linux does not have a stable "kernel module" interface.
- Because Zephyr applications are so tightly coupled to the hardware
- they run on it is not uncommon to bypass Zephyr utilities and directly
- touch hardware and memory.</p><p>In this way they are more related to kernel modules
- than userspace applications such as GNU Hello.</p><p>Perhaps there is a lispy way to track several zephyr releases without reducing
- the ability to freely modify components in the usual ways...I invite you dear
- reader to developer code to explore that possibility.</p><p>It is use-able for Guix anyway.</p><h1>Why a Special Build System? Why not <code>--target=frdm_k64f</code>?</h1><p>That is a fair question!
- At first glance you might imagine the following incantation:</p><pre><code class="language-sh">guix build --target=arm-zephyr-eabi hello</code></pre><p>The problem with calling the toolchain directly is that the architecture
- is specified by the board selection. It is not generally useful to
- compile a board to a different architecture.</p><p>Perhaps maybe something like this then.</p><pre><code class="language-sh">guix build --target=frdm_k64f hello</code></pre><p>The above command tells GNU hello to link against arm-zephyr-newlib and run on
- a specific board. The problem is that while this <em>may</em> work for GNU Hello, it
- will not work for anything, which requires inputs that are discovered by normal
- methods. Only packages which target zephyr explicitly could benefit from such an
- interface, and at that point, you may as well record which board is being targeted
- in the package definition.</p><p>In general not all zephyr applications can run on every board zephyr
- can run on, so the usefulness of the above command is dubious.</p><p>I think if you have firmware which targets multiple boards it is
- better to define a package for every board. It is likely every board
- will require special configuration flags anyway.</p><h1>Conclusion</h1><p>Zephyr has a very complex build process which can be difficult to
- understand and frustrating to set up.</p><p>Using Guix to define firmware packages makes these problems disappear.
- It is trivial to create a channel which contains all of the quirks of
- your platform and share it with a team or student.</p><p>Packages defined this way serve as a reproducible starting point for
- future hardware revisions and variations.</p><h1>Footnotes</h1><ol><li>I also made a <code>zephyr-module-build-system</code> as well which is just the
- <code>copy-build-system</code> that mimics the default zephyr workspace layout as provided
- by west. This way we do not need to provide the same install-plan for every
- module we package. However as I use the <code>copy-build-system</code> more often, it
- doesn't really provide much over just using the copy-build system.</li></ol></div></article></section></main><footer><p>© 2020 Joshua Branson. The text on this site is free culture under the Creative Commons Attribution Share-Alike 4.0 International license.</p><p>This website is build with Haunt, a static site generator written in Guile Scheme. Source code is <a href="https://notabug.org/jbranso/gnucode.me">available.</a></p><p>The color theme of this website is based off of the famous <a href="#3f3f3f" target="_blank">zenburn</a> theme.</p></footer></body>
|
