Conformance Testing - 1.11+


With such a wide array of Kubernetes distributions available, conformance tests help ensure that a Kubernetes cluster meets the minimal set of features. They are a subset of end-to-end (e2e) tests that should pass on any Kubernetes cluster.

A conformance-passing cluster provides the following guarantees:

  • Best practices: Your Kubernetes is properly configured. This is useful to know whether you are running a distribution out of the box or handling your own custom setup.

  • Predictability: All your cluster behavior is well-documented. Available features in the official Kubernetes documentation can be taken as a given. Unexpected bugs should be rare, because distribution-specific issues are weeded out during the conformance tests.

  • Interoperability: Workloads from other conforming clusters can be ported into your cluster, or vice versa. This standardization of Kubernetes is a key advantage of open source software, and allows you to avoid vendor lock-in.

Individual Kubernetes distributions may offer additional features beyond conformance testing, but if you change distributions, these features can’t be expected to be provided.

NOTE: Kubernetes documentation also describes the concept of “node conformance tests”. Although they are useful, these tests are more component-focused than system-wide. They validate only the behavior of a specific node, not cluster behavior as a whole.

See the official documentation for Kubernetes’s existing conformance tests.

Integration with Sonobuoy

Sonobuoy’s plugin architecture enables you to integrate conformance test results into your reporting. The e2e tests can be configured with the plugin. The default configuration runs the basic set of conformance tests against a local provider.

To customize the set of tests that are run as part of the report, the following environmental variables can be set in the plugin-specific YAML config. Typically you set this via a flow that involves the gen command:

VariableDefault ValueDescription
E2E_FOCUS“Conformance”The test suite to run.NOTE: Because the real conformance suite can take up to an hour to run, the quickstart example’s e2e config specifies just a single test, “Pods should be submitted and removed”.
E2E_SKIP“Alpha|Disruptive|Feature|Flaky”Which subset of tests to skip
E2E_PROVIDER“local”The platform that the cluster is running on
E2E_DRYRUN""Setting any non-empty value results in skipping the actual test execution but reporting the tests as successful. This is helpful when fine tuning your focus/skip regular expressions to target the desired tests.

NOTE: The length of time it takes to run conformance can vary based on the size of your cluster—the timeout can be adjusted in the Server.timeoutseconds field of the Sonobuoy config.json.

Getting Started

To help you get started, see the documentation.