Blog

Simplified Results Reporting with Sonobuoy 0.15.1

John Schnake

Aug 7, 2019

Sonobuoy 0.15.1 introduces an entirely new feature that promises to make it easier than ever to see the results of your plug-ins.

The new CLI command, sonobuoy results, targets a results tarball and can report on which tests pass, fail, or get skipped. Unlike the sonobuoy e2e command, this new command has been generalized to work with different types of plug-ins and can be extended to work with different formats.

The Canonical Data Format

Because we want to be able to report on the actual results of any plug-in, we need to be able to present those results in a standardized way. To accomplish this, we generated a new, simple data format for capturing results.

type Item struct {
    Name     string
    Status   string
    Metadata map[string]string
    Items    []Item
}

This format is self-referential so that we can represent the results as a tree, which is a natural structure for multiple relationships like nested subtests and daemonsets running the same set of tests on each node. The tree structure also allows us to bubble up failures so that you can see the most important information (whether the plug-in passed or failed) by looking at a single item.

Getting Results From Any Plug-in

Originally, when Sonobuoy introduced the sonobuoy e2e command, it was written specifically for the e2e plug-in. This customization meant that the CLI tool would make several assumptions:

• The plug-in was a job-type plug-in, not a daemonset
• The results would be in plugins/e2e/results/junit_01.xml
• The data would be in the JUnit XML format

As a result of these assumptions, a user or tool would need to know all these details to check the results of a single plug-in.

In the new standardized approach, we decided to leverage the aggregator server since it has all the plug-in definitions and is already controlling the results data. By adding a post-processing step after results are gathered, we can create, organize, and summarize the results so that downstream clients can more easily process arbitrary results.

To define where results are located and what the format is, we added two fields to the plug-in definition:

    ResultFormat string
    ResultFiles []string (optional)

The ResultFormat field allows the user running the plug-in to specify a post-processing format. Currently, we support two formats: junit and raw. These options correspond to the output of the two most common plug-ins: the e2e plug-in and the systemd-logs plug-in.

Each plug-in format applies its own heuristic to determine which files to target and how to process them. The junit format will attempt to parse every *.xml file as a junit result file. This way you don’t even have to know the name of the file to process the results, and we can support plug-ins which create multiple junit outputs.

The raw format is appropriate when you don’t really want specific processing on the resulting files but want access to them, such as when you’re creating logs or reports. The post-processing will walk the files generated by the plug-in and record their locations within the tarball so that tools can look them up more easily.

The ResultFiles field enables you to manually specify which files to process as results so that you don’t have to rely on our heuristics to find your result files (for example, if you produce junit results but don’t give them the .xml extension).

Accessing the Results

We’ve added the sonobuoy results command to analyze these new, processed results. Let’s take a look at a few examples.

If you want a human-readable report of the results that lists summary data as well as failures, run the following command:

$ sonobuoy results $tarball --plugin e2e

Here’s an example of what the results might look like:

Plugin: e2e
Status: passed
Total: 3586
Passed: 1
Failed: 1
Skipped: 3584

Failures:
<list of test names>

Note: The --plugin flag defaults to e2e, so you don’t need to add it if you are inspecting e2e results.

To see detailed data about all the results, you can change the default --mode=report to --mode=detailed. The detailed mode will write all the leaf nodes of the tree as JSON objects so that you can pipe them easily into a tool like jq:

$ sonobuoy results $outfile --mode=detailed
{"name":"[sig-storage] In-tree Volumes...","status":"skipped","meta":{"path":"e2e|junit_01.xml"}}
…

Note: In this format, we added a path to the object in the tree within the meta field to clarify exactly where the results are from.

If you target a daemonset plug-in, you might want to print only a certain section of the tree instead of all the leaf nodes. This is why we added the --node flag:

$ sonobuoy results $outfile --mode=detailed --node=kind-control-plane --plugin systemd-logs
kind-control-plane|systemd_logs {"_HOSTNAME":"kind-control-plane",...}
...

In the example above, you may notice we didn’t explicitly dump the results as a JSON object like we did with the junit results. This approach lets you access the raw file contents. For clarity, the Sonobuoy CLI is outputting two things: the tree path to the file in question and the file’s contents.

But doesn’t the prefix prevent handling the raw file contents easily? Yes it does, so if you want to access just the file data itself, use the --skip-prefix flag. For instance, if you want the first log item in the systemd logs for a particular node, run the following command:

$ sonobuoy results $outfile --mode=detailed --node=kind-control-plane --plugin systemd-logs --skip-prefix | head -n1 | jq

This approach makes the systemd-logs plug-in much more useful since the post-processing is simplified. You can now quickly scan for errors or keywords by dumping and filtering the results as desired.

Lastly, if you really want to take a look at the whole, post-processed results file, you can use the --mode=dump flag:

$ sonobuoy results $outfile --mode=dump
name: e2e
status: passed
items:
- name: junit_01.xml
  status: passed
  meta:
    file: results/global/junit_01.xml
  items:
  - name: '[sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: blockfs]
…
...

TL;DR

Every plug-in goes through a post-processing step in which we aggregate results into a file at a known location: plugins/<name>/sonobuoy_results.yaml The new command sonobuoy results can print out that data for any plug-in. It can report how many tests passed or failed, print detailed information about every test, or be used to access files generated by a plug-in without having to handle the entire tarball.

We hope these new features will help you understand and script around your test runs.

A big thank you goes out to the Sonobuoy community for your continuous feedback; we had a lot of community contributions for this release and want to give a special thanks to MengmengZHANG and johscheuer for their contributions.

Join the Sonobuoy community:

• Get updates on Twitter ( @projectsonobuoy)
• Chat with us on Slack ( #sonobuoy on Kubernetes)
• Join the Kubernetes Software Conformance Working Group: github.com/cncf/k8s-conformance