Driving Embedded Firmware C Project on CruiseControl

Timo Punkka ( tpunkka at users.sourceforge.net )
ngWare

Continuous integration and automated unit tests are long known good programming practices. Agile methods put a strong emphasis on disciplined use of these practices. CruiseControl is an open source continuous integration server software. Embedded Unit (embUnit) is an unit test framework targeted for embedded firmware C projects. This article describes how to set up a buid system with automated build, continuous integration, and automated unit tests for embedded firmware C project.

Driving Embedded Firmware C Project on CruiseControl

Getting Started

Prequisities:

JAVA SDK (download)
CVS (WinCVS)
Embedded Unit
(optional, but recommended) CygWin
Understanding of C, CVS

Download and install CruiseControl. This example uses the folder structure familiar from Lasse Koskela's "Driving on CruiseControl" articles. If you also choose to do so, change the 'Destination Folder' setting as shown below.

Remember to have your JAVA_HOME variable set, for example in CygWin:

$ export JAVA_HOME="c:\j2sdk1.4.2_13"

The current distribution (2.5) has a server and Ant coming with it, so it is a breeze to launch even for hardcore firmware programmer. Go to CruiseControl root and run the .bat file.

.
$ /cruisecontrol.bat

Browse to http://localhost:8080 to see how CruiseControl presents the build results from Java project example, connectfour. The demo project is not managed with version control so your changes will not trigger a new build.

Automated Build, Version Control, and Continuous Integration

We are ready move to the C zone and actually do automated continuous integration. First we need something to integrate. We will somewhat reuse the sample files from Embedded Unit distribution. However we also want a main file and an automated build script for it. This example uses GNU Make for automated build.

You can find the whole example project and files needed to configure CruiseControl here (zip'ed). This example expects that you have extracted the zip to C:\

You are able to build the example project with GNU Make:

$ cd /c/embunit_cruisecontrol/project

$ make -f project.mak -B

Note that in order to build for embedded target, you need your target compiler installed in your build server as well.

(Example uses gcc to build for PC architecture)

Next we want our project to be version controlled so that CruiseControl can actually do the continuous integration. To do that, we need to create a version repository, import our code into the repository, and then checkout a new working copy of the source into our CruiseControl work area and another copy for development under version control.

$ make -f project.mak clean

$ export CVSROOT=/c/CIA/RepositoryServer

$ cvs -d /c/CIA/RepositoryServer init

$ cvs import -m "First C project" project sts init

$ mkdir /c/CIA/WorkArea

$ cvs checkout -d /c/CIA/WorkArea/project project

$ mkdir /c/CIA/Developer1

$cvs checkout -d /c/CIA/Developer1/project project

You should now have a structure like below:

CruiseControl's config.xml is where the magic happens, so replace the demo version with the one given below. You have it in your extracted folder embunit_cruisecontrol/cruisecontrol/config, with name config1.xml (if you use this, you have to rename it to exclude the '1'). Replace the demo version in CruiseControl root with this one. It is propably a good idea to move the old versio away by renaming it, and not to overwrite it. For details about the config.xml see "Driving on CruiseControl" and CruiseControl Configuration Reference.

The main difference when moving to use GNU Make instead of Java and Ant in CruiseControl demo, is the use of generic exec builder.You can see an example of its attributes in the example.

<cruisecontrol>
<project name="project">

    <listeners>
      <currentbuildstatuslistener file="logs/project/status.txt" />
    </listeners>

    <bootstrappers>
      <cvsbootstrapper localWorkingCopy="../../WorkArea/project" />
    </bootstrappers>

    <modificationset quietperiod="60">
      <cvs LocalWorkingCopy="../../WorkArea/project" />
    </modificationset>

    <schedule interval="900">
      <exec command="make.exe" workingdir="../../WorkArea/project" args="-f project.mak -B" errorstr="failed" />
    </schedule>

</project>
</cruisecontrol>

Run cruisecontrol.bat again, find your way to http://localhost:8080, and see your project being under cruisecontrol.

Click the project link to see the results from the first build.

Now let's see for the first time what all this is needed for. Browse to development copy of our poject at Developer1/project. Make a change to one file, for example main.c and commit the changes to version control. You should need to wait maximum of 15 minutes (schedule interval defined in config.xml) to see a new build triggered for your changes. And it worked! You can see the results in the browser. The report contains the commit message from the version control system.

Unit Tests

In the previous example we had a simple build under automated continuous integration. Yet we had the the text "No Tests Run, This project doesn't have any tests" in the results. Let's do something about this. Embedded Unit can generate results in XML output format. CruiseControl can be configured to merge these results into its build report. So we want the testframe to report the results in XML so that we can use them in CruiseControl. This is explained in Embedded Unit Documentation, but below is a modified AllTests.c file, which runs the tests via XML outputter if we give '-X' in command line when executing the tests. Otherwise it uses the simple outputter. This comes handy when you want to quickly run your tests when developing (hopefully in TDD), and CruiseControl can run the same tests with '-X' option.

#include <embUnit/embUnit.h>

TestRef CounterTest_tests(void);
TestRef PersonTest_tests(void);

int main (int argc, const char* argv[])
{
if ( argc > 1 &&
       argv[1][0] == '-' &&
       argv[1][1] == 'X' )
{
    TextUIRunner_setOutputter(XMLOutputter_outputter());

    TextUIRunner_start();
    TextUIRunner_runTest(CounterTest_tests());
    TextUIRunner_runTest(PersonTest_tests());
    TextUIRunner_end();
}
else
{
    TestRunner_start();
    TestRunner_runTest(CounterTest_tests());
    TestRunner_runTest(PersonTest_tests());
    TestRunner_end();
}
return (int)0;
}

I have modified the makefile which is used to build the tests a little bit from the current EmbUnit examples to include the textUI library. It also uses the EMB_UNIT_DIR environment variable in defining the include and library sources. Other than that I wanted to keep the example makefile as it is.

CC = gcc
CFLAGS = -O
INCLUDES = -I.. -I$(EMB_UNIT_DIR)
LIBS = $(EMB_UNIT_DIR)/lib
RM = rm
TARGET = alltests
OBJS = AllTests.o counter.o counterTest.o person.o personTest.o

all: $(TARGET)

$(TARGET): $(OBJS)
    $(CC) -o $@ $(OBJS) -L$(LIBS) -lembUnit -ltextUI

.c.o:
    $(CC) $(CFLAGS) $(INCLUDES) -c $<

clean:
    -$(RM) $(TARGET) $(OBJS)

.PHONY: clean all

The main make file, project.mak, defines a special target for CruiseControl. It will compile the target, the unit tests (by executing the above makefile), then execute the unit tests and pass the unit test XML results into file testResults/TEST-alltests.xml.

CC = gcc
CFLAGS = -O
INCLUDES =.
LIBS =
RM = rm
TARGET = project.exe
OBJS = main.o counter.o person.o

$(TARGET): $(OBJS)
    $(CC) -o $@ $(OBJS)

all: $(TARGET) test

test:
    $(MAKE) -f makefile -B
    ./alltests.exe

CruiseControl: $(TARGET) initResultFolder
    $(MAKE) -f makefile -B
    ./alltests.exe -X > testResults/TEST-alltests.xml

initResultFolder:
    mkdir -p testResults
    rm -f testResults/*.*

.c.o:
    $(CC) $(CFLAGS) $(INCLUDES) -c $<

clean:
    -$(RM) $(TARGET) $(OBJS)

.PHONY: clean all

Let's modify CruiseControl's config.xml a little bit to actually compile for CruiseControl target and merge the test results into our build report.

<cruisecontrol>
<project name="project">

    <listeners>
      <currentbuildstatuslistener file="logs/project/status.txt" />
    </listeners>

    <bootstrappers>
      <cvsbootstrapper localWorkingCopy="../../WorkArea/project" />
    </bootstrappers>

    <modificationset quietperiod="60">
      <cvs LocalWorkingCopy="../../WorkArea/project" />
    </modificationset>

    <schedule interval="900">
      <exec command="make.exe" workingdir="../../WorkArea/project" args="-f project.mak CruiseControl -B" errorstr="failed" />
    </schedule>

    <log dir="logs/project">
      <merge dir="../../WorkArea/project/testResults" />
    </log>

</project>
</cruisecontrol>

We are almost done, but not quite. CruiseControl has XSL transformers to format the appearance of these reports. The transformers included in distribution are created for Ant and JUnit. So it does not know anything about how embUnit reports tests. You can find the transformers for Embedded Unit in the exctracted folder, testdetails.xsl and unittests.xsl. Replace the files circled in the picture below with these new ones.

OK, Let's launch CruiseControl again. Browse to http://localhost:8080 and force a new build for your project. Let's see the results. You can now see the results of the unit tests as well. If you want more details, you can click the Tests Results -button on top of the page.

Let's make one test to fail to show how CruiseControl indicates this. We just change the expected value of counter after the initialization to be 1 instead of 0. Not the best example, but it is the first assertion in counterTest.c. Commit the modified code into repository (normally you would never commit broken code into repository. Never.) and wait for CI server to integrate. Now the main build results page has a red bar and details of failed test. Test Results screen shows the name of the failing test in red, and details can be found by clicking Failed>> -link.

counterTest.c

LINE 18: TEST_ASSERT_EQUAL_INT(0, Counter_value(counterRef));

LINE 18: TEST_ASSERT_EQUAL_INT(1, Counter_value(counterRef));

Beyond

Publishers: Telling Others About It

The importance of continuous integration server is highlighted when multiple developers are working on shared code. It is convenient to somehow tell others about each change. CruiseControl has a mechanism called Publisher to configure additional reporting mechanisms. In this example we will look at html email publisher Lets add this section to our config.xml. After each build, success or failure, developer1 will get a html email report.

<cruisecontrol>
<project name="project">

    <listeners>
      <currentbuildstatuslistener file="logs/project/status.txt" />
    </listeners>

    <bootstrappers>
      <cvsbootstrapper localWorkingCopy="../../WorkArea/project" />
    </bootstrappers>

    <modificationset quietperiod="60">
      <cvs LocalWorkingCopy="../../WorkArea/project" />
    </modificationset>

    <schedule interval="900">
      <exec command="make.exe" workingdir="../../WorkArea/project" args="-f project.mak CruiseControl -B" errorstr="failed" />
    </schedule>

    <log dir="logs/project">
      <merge dir="../../WorkArea/project/testResults" />
    </log>

    <publishers>
      <htmlemail mailhost="smtp.yourdomain.com" returnaddress="cruisecontrol@yourdomain.com"
                 subjectprefix="[CruiseControl]"
                 xsldir="C:\CIA\BuildServer\CruiseControl\webapps\cruisecontrol\xsl"
                 css="C:\CIA\BuildServer\CruiseControl\webapps\cruisecontrol\css\cruisecontrol.css">
        <failure address="developer1@yourdomain.com" />
        <success address="developer1@yourdomain.com" />
      </htmlemail>
</publishers>

</project>
</cruisecontrol>

Archiving the artefacts

Sometimes it is a good idea to archive results of each build automatically. For this we can use another Publisher, artifactspublisher. We add one more section to our config file. This copies the defined file project.exe to our logs/project folder. It actually creates a subfolder with timestamp as a name, and uses this to archive current version of target, project.exe. Very convenient.

<cruisecontrol>
<project name="project">

    <listeners>
      <currentbuildstatuslistener file="logs/project/status.txt" />
    </listeners>

    <bootstrappers>
      <cvsbootstrapper localWorkingCopy="../../WorkArea/project" />
    </bootstrappers>

    <modificationset quietperiod="60">
      <cvs LocalWorkingCopy="../../WorkArea/project" />
    </modificationset>

    <schedule interval="900">
      <exec command="make.exe" workingdir="../../WorkArea/project" args="-f project.mak CruiseControl -B" errorstr="failed" />
    </schedule>

    <log dir="logs/project">
      <merge dir="../../WorkArea/project/testResults" />
    </log>

    <publishers>
      <htmlemail mailhost="smtp.kolumbus.fi" returnaddress="cruisecontrol@yourdomain.com"
                 subjectprefix="[CruiseControl]"
                 xsldir="C:\CIA\BuildServer\CruiseControl\webapps\cruisecontrol\xsl"
                 css="C:\CIA\BuildServer\CruiseControl\webapps\cruisecontrol\css\cruisecontrol.css">
        <failure address="timo.punkka@kolumbus.fi" />
        <success address="timo.punkka@kolumbus.fi" />
      </htmlemail>

      <artifactspublisher file="../../WorkArea/project/project.exe" dest="logs/project/" />

    </publishers>

</project>
</cruisecontrol>

Build Indicators: Making It Visible

If the team is working in open workspace it is a good idea to add something simpler and more visible to indicate build results than just emails. Lava lamps are the classic. Below are some links to what people have been doing.

The classic X10 version, "Bubble, Bubble, Build's In Trouble"
Simpler firmware solution by Mark Michaeli and their team (Building Status Using Lava Lamps).
Even more straightforward firmware solution by our team (Bubble, Bubble...).

Links to Related Stuff

Martin Fowler: Continuous Integration
Effective Test Driven Development for Embedded Software (pdf)
Test Driven Development of Embedded Systems Using Existing Software Test Infrastucture (pdf)
Test Driven Development in C (pdf)
CATS and Catsrunner
Dr. Dobb's : "Embedded Agile: A Case Study In Numbers"
Atomic Object: Embedded Corner