Writing integration tests for a Go CLI application
I’ve recently open sourced a small Go CLI application for random data generation. I’m working on an application for which I continuously need testing data, so I did some research on the subject. As I couldn’t find exactly what I was looking for, I decided to write a small Go CLI application to solve this problem. CLI applications are one of my favorite use cases for Go, and I knew I’d enjoy writing it. A few weeks after I released fakedata, Kevin Gimbel found a bug. The issue was a direct consequence of a feature we introduced on the day Kevin found the bug. It was an obvious regression, and fakedata clearly had no tests covering the feature end to end. It made me think about testing how to test a CLI application end to end. I wanted to do something like:
- create a binary of the CLI app
- run the binary with some specific argument
- assert correct behavior
I managed to get it done here and decided to share the key ingredients that made this a simple and effective way of writing integration tests for a Go CLI application. I created an example application for the sake of the discussion, it’s available here.
I use make
to build my Go applications, it suits the task well and it gives me
a very short command to build an entire project (as build
is the de-facto
default target, building a project is as easy as typing make
and hitting
enter). In this case, the
Makefile
builds a tiny program called echo-args
. The program works like this:
$ echo-args # no arguments, no output
$
$ echo-args ciao # it prints each argument on its own line
$ ciao
$ echo-args ciao hello
ciao
hello
$ echo -shout ciao # it shouts at you if you ask it to
CIAO
This behavior gives me four different test cases. The first step of the approach I just discussed is to build the binary. Here is the code for that:
func TestMain(m *testing.M) {
err := os.Chdir("..")
if err != nil {
fmt.Printf("could not change dir: %v", err)
os.Exit(1)
}
make := exec.Command("make")
err = make.Run()
if err != nil {
fmt.Printf("could not make binary for %s: %v", binaryName, err)
os.Exit(1)
}
os.Exit(m.Run())
}
Two things are worth mentioning:
- TestMain is the recommended way to do setup and teardown of tests
os.Chdir("..")
is ugly but practical. It works for my use case, but it’s neither a general or a robust solution.
Since this code runs before any of my tests, I can assume there’s going to be a binary of my Go Cli application I can use to run the program and assert the correctness of its output. Here is the test I wrote to cover the four test cases:
func TestCliArgs(t *testing.T) {
tests := []struct {
name string
args []string
fixture string
}{
{"no arguments", []string{}, "no-args.golden"},
{"one argument", []string{"ciao"}, "one-argument.golden"},
{"multiple arguments", []string{"ciao", "hello"}, "multiple-arguments.golden"},
{"shout arg", []string{"--shout", "ciao"}, "shout-arg.golden"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
dir, err := os.Getwd()
if err != nil {
t.Fatal(err)
}
cmd := exec.Command(path.Join(dir, binaryName), tt.args...)
output, err := cmd.CombinedOutput()
if err != nil {
t.Fatal(err)
}
if *update {
writeFixture(t, tt.fixture, output)
}
actual := string(output)
expected := loadFixture(t, tt.fixture)
if !reflect.DeepEqual(actual, expected) {
t.Fatalf("actual = %s, expected = %s", actual, expected)
}
})
}
}
The tests use two of my favorite practices I use while testing Go programs:
- Golden files
- TableDrivenTests
While Table Drive tests is officially documented, I couldn’t find anything about golden files. The basic idea is to:
- store on disk (in so-called golden files) the expected (and possibly complex) output of the code under test
- then use a simple comparison of the actual output and the content of corresponding golden file.
It’s a good practice to use a command line flag to automatically update the golden file when the specified behavior changes, so that you can:
go test integration/cli_test.go -update
and then check the golden file before running the tests again. Here is the output on my machine:
go-cli-integration-tests master ➜ make test
go test ./...
? github.com/lucapette/go-cli-integration-tests/cmd [no test files]
ok github.com/lucapette/go-cli-integration-tests/integration 0.267s
While it’s obvious this is a trivial example, I think it’s still pretty amazing
that building the binary, running it four times, loading four files, and
asserting the correctness of the problem takes as little time as 0.267s
. The
integration tests run nicely on
travis too.
Thanks to TableDrivenTests, covering more use cases is simple. Imagining that
echo-args
now accepts a reverse
flag, I can ensure the flag works with one
more line:
tests := []struct {
name string
args []string
fixture string
}{
// existing tests not shown
{"reversed", []string{"--reverse", "ciao", "hello"}, "reversed.golden"},
}
I really like it takes so little effort to test a relatively complex behavior.
The tests make only two assumptions: how to build the binary (running make
in
a specific directory) and the name of the binary. If I changed the internals of
the program completely, the tests would still be untouched. The tests need to be
updated only if the behavior changes, which is the only kind of testing I’m
comfortable with. Just to highlight the point of this benefit even more: the
binary doesn’t even have to be a Go program.
I enjoyed using this technique and I hope it helps some of you too. Happy testing with Go!