bake

A simple, bash-based build system

This is based on the observation that many Makefiles actually call out to bash to do the real work. That includes many tasks such as creating output directories, checking for the existence of files, and many others that the Make language simply doesn't allow for. This kinda defeats the purpose of having a Make language since the moment a bash function is called directly, we lose portability to run the Makefile on any other shell. Make is a strange kind of language with subtleties that are lost on many first-time and veteran users alike. Like the difference between '=' and ':=', and when to use each one is not easy to explain to a newbie. In fact, many makefiles are not even written by hand anymore - they are generated by other scripts (like autoconf), so in fact you have to learn at least one other language (sometimes you have to learn M4 macros as well) in addition to Make in order to write a portable makefile. Moreover, the syntax and functionality of make are not that different from bash itself, which begs the question 'why use two (or more) languages instead of one?' Thus, 'bake' is the result of bash+make => a bash-based build system

Make also tries to be useful by referring to a lot of built-in rules that try to guess what I'm doing based on the most common programming practices. Unfortunately, many of these common practices are not really that common (when was the last time you actually write a yacc parser script?) and it takes time for make to run through all of these rules which are (the majority of the time) useless. That slows down the build, leading people to write other tools to avoid all this overhead.

How To Install

'bake' will run just the same as any bash script. Personally, I like to clone the git repository somewhere in my home directory, and then create a soft link to another directory that contains all my scripts that is in my path. For example:

~/projects$ git clone https://github.com/jnider/bake.git
~/projects$ mkdir ~/bin
~/projects$ export PATH=$PATH:~/bin
~/projects$ ln -s $(realpath bake/bake) ~/bin/bake

How to Use

The input to 'bake' is called a recipe. Recipes give the precise instructions on how to build your particular program. This is the equivalent of a Makefile or CMakeLists.txt file in other build systems. Recipe files are regular bash shell scripts, but you can rely on existing infrastructure to get your project up and running very quickly. Bake will automatically look for a file called 'recipe' in the current directory. If you want to use a different name, you can pass the -f flag like this:

bake -f other-file

Environment

You will often want to change the behaviour of the build without resorting to invasive changes in the code or recipe. Just like other build tools, 'bake' can read environment variables that can be passed on the command line. For example, many projects have a 'debug' and 'release' mode. The mode can be controlled by defining the environment variable DEBUG to change the compilation flags, such as reducing the compiler optimization level or including conditionally compiled print statements that give more insight into what your program is doing at runtime. Since 'bake' is just a 'bash' script, you can pass environment variables in 2 familiar ways: export them for all subshells to see, or prepend them to the command line for just this particular run.

export ARCH=arm32
bake

or

ARCH=arm32 bake

Targets

Targets are how you tell 'bake' what to build. It could be the name of an output file, or a random name that you make up - it doesn't really matter. You can specify the target you want to build on the command line like this:

bake my-target

That's the fun part of 'bake' - if you already know bash, then you can use the commands and syntax you are already familiar with. If you don't provide a target, 'bake' will try to find one called 'main'. If you want to be able to run 'bake' from the command line without any other parameters, add a rule to your recipe that sets your output file as a dependency of 'main'. You can see the section below on Dependencies for more details on how to do this.

'bake' looks for a target called 'main' automatically if no parameters are provided on the command line. You can add a target called 'main' with a single dependency so 'bake' will do what you expect like this:

main=video.elf

How to Write A Recipe

If you are starting your own project, you will need to write your own recipe. Although not necessary, I like to get the benefits of syntax highlighting out of my editor (vim) for free, so I start off my recipes with:

#!/bin/bash

Recipes generally consist of one or more targets, their dependencies and the rules needed to generate each target. Common targets (such as generating .o files from .c files) have some language-specific support, described in the Languages section. You can also make your own custom rules if you need to do something different. That is described in the Rules section.

Languages

There is some support for performing common tasks on common languages. The support consists of pre-formed rules for known file types and compilation patterns. These rules are flexible and accept parameters for things like compilation flags, source directory, target directory, compiler name, etc.

To see all of the supported languages in your version:

bake languages

You can use the language support in your recipe by including those rules in your recipe like this example for C++:

_language cpp

Input Files

Often there will be a list of input files that you wish to perform actions on such as compiling, assembling, etc. These lists can be generated in many ways ranging from a completely manual process (where each filename is written explicitly) to a complicated set of automatic functions that find files based on search criteria. No matter how the list is generated, it should be stored in a Bash array. The following example specifies a list of .c files:

c_src=(main.c usb.c loader.c video.c)

Another more automated method may find all of the .c files in the current directory:

c_src=$(ls *.c)

A 3rd example finds all of the .c files in all of the subdirectories:

c_src=$(find . -name '*.c')

This list can be processed further to generate a list of targets.

Input Files for Linking

The linker expects a list of object files to be linked together to create the final executable. In a simple project, this list is equivalent to the dependency list of the executable, and the built-in linker rule will use it automatically.

# dependencies and linkable objects are the same
out_elf=(main.o another.o)

In a more complicated project, the executable can be dependent on libraries, linker scripts, and other files. In this case, the dependencies include all of the files, but the list of linkable objects is only a subset of that list. We can specify two different lists, using the built-in 'objects' array. In this way, the command line to the linker can be specified more precisely, where options and libraries preceed the link objects.

# only the object files
elf_objs=(main.o another.o)
objects[out.elf]=elf_objs

# dependencies include object files as well as other files
out_elf=(linker.ld somelib.a ${elf_objs[@]})

Dependencies

If the target is dependent on other files (like in the C language, where .o files depend on .c and .h files) you can use an array variable to specify the dependencies in your recipe file. The name of the array is important - Bake will look for a variable that matches the target name in order to know its dependencies. But there is a small snag; it is not legal syntax to use certain characters in Bash variable names that are legal to use in filenames. The most trivial example is the '.' character. This is a very common character to use in filenames, but it cannot be used in a Bash variable name. Instead, Bash (and therefore recipe writers as well) must replace these characters with and underscore '_'. Thus for a target main.o, its dependencies will be found in the array variable main_o. There are 3 such characters:

• . (period)
• / (forward slash)
• - (hyphen)

In addition, Bash variable names cannot have a leading digit, but directory names can. In that case, the directory name will be prepended with an underscore '_'.

See function filter_name() for more details

Just like writing lists of input files, dependencies are Bash arrays:

main_o=(main.c header.h)

As the project grows, building it can become more complicated. You might want to support multiple source directories, or a variable build directory name. That makes static dependency lists difficult to maintain or even produce. To make things easier, you can use the dependency redirection array. A global associative array called 'dep_names' holds variable names containing the actual dependencies. The target name can be practically any string and can use variables. For example, if you want to support a user-defined build directory, you can use a variable. Let's call it $BUILDDIR for this example. You then specify all targets relative to that build directory, e.g. $BUILDDIR/main.o. You can then tell 'bake' where to find the dependencies of this target like this:

dep_names[$BUILDDIR/main.o]=deps_main_o
deps_main_o=(main.c main.h)

Notice that the array subscript does not need to be filtered or altered in any way. No matter the value of $BUILDDIR, 'bake' will know how to find the dependency array 'deps_main_o'. All of the dependencies are relative to the source directory, which is the directory where 'bash' started (by default).

Rules

Rules run when 'bake' determines something is out of date. Generally, something is out of date if it doesn't exist, or it has a dependency that has a more recent timestamp in the file system. A rule is just a regular bash function, but the function name must start with the prefix "rule_". The 'bake' script will search for all functions that start with that prefix when it is trying to run a rule. That way, you have the freedom to write any other functions you want, and they will not be picked up as rules. If 'bake' determines that a rule should be run on a particular target, it will try all rules that it can find, one by one, until it finds one that accepts the name of that target. When you write a rule, you are responsible for the code that tells 'bake' if the target matches or not. Generally, it looks something like this:

function rule_clean_roottask()
{
	local target=$1
	[[ "$target" == "roottask_clean" ]] || return -1

	echo "Cleaning output directory $dest_path"
	cmd="rm -rf $dest_path"
	[[ $__verbose ]] && echo $cmd
	$cmd
}

Independent Rules

Sometimes you want a rule to run every time, without any dependencies. One example is a 'tag' file, which should be updated every time you run the rule. You don't want to explicitly add all other files as dependencies since you might not want to update the tag file on every build, but you do want it updated even if the tag file already exists. To do so, you add a dependency to your target that will never exist, and build a rule to run for that new dependency. This is similar behaviour to declaring a target .PHONY in a Makefile. In the example below, 'rebuild_tags' is a dependency of 'tags' that will never exist. When 'bake' tries to build it, it will run the rule 'rule_rebuild_tags'.

tags=(rebuild_tags)

function rule_rebuild_tags()
{
   local target=$1
   [[ $target == "rebuild_tags" ]] || return

   echo "Building tag file"
   ctags -R -f tags .
}

How to Debug

This is actually two different questions - (1) how to debug your recipe, and (2) how to debug Bake itself. Fortunately, both of these things are just Bash scripts, which means you can inspect the source code with any text editor if it comes to that. But before jumping the gun, there are other things that you can try.

Print Out Your Variables

When you and bash disagree on what the contents of a variable are, it invariably leads to a misunderstanding that manifests as a misbehaviour (AKA bug). Make sure you and bash agree by using 'echo' to dump out the value of the variable at various points in your script. Some common mistakes are listed below.

1. Variable name masking (scope) Variables are generally global in bash. Even when you are inside a function you still have access to all of the global variables. If you want to reuse a variable name, you have to use the keyword 'local'.
2. Array variables When you reference an array variable, make sure you use the correct syntax. For example, if you have a list called 'c_src', referencing it as '$c_src' is legal, but will only reference the first item in the list. To get all of the items, you must write it as: ${c_src[@]}

Use the -v option

If 'bake' is invoked with the -v flag, it will print out all of the decisions it is taking so that you can trace through the execution. It will also set the variable $__verbose that you can use in your own rules to get a higher level of detail.

Examples

Probably the best thing about Make is the documentation. Probably the best thing about the documentation is the examples. I found the examples extremely useful, and so will endeavour to reproduce that success by providing useful examples and code snippets to make learning 'bake' easy.

Rule for compiling 'c' files with 'gcc'

In general, you should use the built-in language support to compile 'c' files (see language support above). This is meant as a simple example to explain a language most people are likely to be familiar with. Even if you don't know C, you will probably understand the compilation process.

The name of the function doesn't really matter, as long as it starts with the word 'rule'. The first line declares a local variable named 'target', which gets the target name that bake is trying to build. The next step is to determine the file type by inspecting the name. I use the standard bash ## operator to remove everything in the name up to the last dot. Whatever is left after the dot is compared to "o" to see if we are trying to build an .o file. If so, we assume that the source file is a C file with the same stem, but a .c extension. Then it is trivial to use these two variable names in a command that executes 'gcc'. In this example, the result of the compilation is not checked directly but 'bake' will see the return code when the rule exits and will not continue to the next step if it has failed. If the target file is not built for some reason (error in the source code, compiler crashes, etc), 'bake' will quit with an error message about a failed dependency for the parent target.

function rule_c
{
	local target=$1
	local flags=#{2:-$CFLAGS}
	[ ${target##*.} == "o" ] || return -1
	local src=${target%%.o}.c

	echo "building $target from $src"
	gcc $flags -c $src -o $target
}