Fuzzing for CI Workflows

Stephen Dolan recently presented crowbar at the 2017 OCaml Workshop. Crowbar bridges a gap between property-based testing frameworks and instrumentation-based automated testing techniques. Tests written in Crowbar can be executed by the wildly popular and successful American Fuzzy Lop fuzzer. (For more on testing OCaml code with AFL, see the afl-persistent README, Crowbar’s examples, or a user’s DHCP library tests.)

American Fuzzy Lop provides a command-line tool for running tests, afl-fuzz. afl-fuzz will invoke one test instance to run a given test continuously on different inputs, keeping track of which inputs generated crashes (or, for Crowbar, were counterexamples for a given proposition). The default settings for afl-fuzz don’t terminate when crashes are discovered, or when new execution paths haven’t been found. The user is expected to terminate all the instances themselves when they’re satisfied that enough work has been done (as indicated, colorfully, in the excellent GUI), and then inspect the files left by afl-fuzz during the run.

The workflow is a great fit for a security researcher trying to find vulnerabilities in existing, released software (especially one who wants to assign work to all of the CPUs they have access to explicitly), but it doesn’t fit well into a continuous testing workflow at all. ocaml-bun is a tool for running Crowbar tests in your CI environment, so fuzzing is as easy as unit testing – both to perform, and to inform decisions with.

Goals for ocaml-bun

  • Deliver “counterexample found, here it is” or “no counterexamples found in the allotted time” in automated run contexts.
  • Require no persistent storage for the user to be able to analyze and reproduce crash inputs and execution.
  • Make full use of available resources both in shared cloud environments and self-hosted test environments.
  • Play nicely with container testing environments.

What’s ocaml-bun do so far?

Easy Fixes

Some of these problems are fixable with the default afl-fuzz by just setting some environment variables: AFL_BENCH_UNTIL_CRASH causes afl-fuzz to exit after finding one crash, and AFL_EXIT_WHEN_DONE causes afl-fuzz to exit “when all existing paths have been fuzzed and there were no new finds for a while” (in other words: as far as we can tell, we’re unlikely to find a counterexample to the properties being tested).

For non-parallel afl-fuzz runs, these environment variables and a bit of shell scripting are sufficient to integrate afl-fuzz into a traditional CI workflow, but this doesn’t make good use of the available resources in most contexts - even free-tier TravisCI jobs have two CPU cores available, and we’d really like to use them.


A single afl-fuzz instance wishes to bind to a particular free CPU core and execute there; in order to use multiple cores, one needs to launch multiple afl-fuzz instances (but not too many). afl-fuzz instances can be easily configured to share information about their exploration of the available paths, but not to coordinate on exit conditions: if one afl-fuzz finds a crash and exits, the others will continue fuzzing until they find another crash or are satisfied that no others exist. ocaml-bun will start as many afl-fuzz instances in parallel as AFL’s own tools report can usefully run on a system, then force all of the instances to finish once one reports that it’s found a crash.

Reporting Results

When a particular piece of input is discovered to cause a crash, afl-fuzz saves it in a directory; the analyst running the test is expected to re-run the test with this input and observe the crash behavior. This workflow doesn’t work at all in ephemeral environments like cloud-based CI platforms, where the person working on the software has no access to the filesystem used by the test system. ocaml-bun exfiltrates any discovered inputs that caused crashes by outputting them in a copy-paste friendly format, so that even if you can only get data out from a web console, you’ll still be able to get the input that caused the crash.

Fun modules to test: the OCaml standard library

A test framework isn’t very useful without something to test. OCaml 4.06.0 recently entered feature freeze and includes new functions in the standard library, so it seemed reasonable to target some interesting modules in that codebase.

The ocaml-test-stdlib repository contains some Crowbar tests for the Set and Map modules, including tests adapted from the unit tests for Map.update added for 4.06.0 . These tests use several modules from the standard library as inputs for the Make functors of Map and Set. Int, String, Char, Nativeint, and Uchar are all potential candidates for Map keys or values, as well as Set elements. Tests for all of these modules (although not all combinations for Map keys and values) are generated, and have an equal chance of being chosen for execution.

stdlib tests as a proxy for CI

Testing the OCaml standard library in a feature-frozen release isn’t a perfect proxy for the workflow we’d like to support in ocaml-bun. Fortunately, continually iterating on the tests themselves is a fairly good proxy for iterating on the code behind them - we have many deploy cycles, and sometimes tests have bugs in them which necessitate retaining and using the crashy input to make improvements.

So far, we’ve run these tests on a machine with 8 CPU cores, and found no property violations in either beta of 4.06.0. We ran the same set of tests against the known-buggy 4.04.0, and were pleased to see many property violations in the Set module, so we’re confident that at least some classes of property violation are detected.

Where This Tooling Fits

We can’t prove that an implementation is correct with ocaml-bun, afl-fuzz, or tools built on similar technologies. Instead, we hope to provide a tool which:

  • fits easily into CI workflows
  • requires little developer effort to write useful tests
  • rewards any test effort with outsized likelihood of discovering useful errors
  • delivers results with a minimum of fuss for maximum convenience

We hope that the ease of using ocaml-bun and afl-fuzz is complementary to approaches which require more developer involvement and attention like verification.

ocaml-bun is currently in active development and much about it may change before settling, but if you’d like to give it a try, you are welcome to visit the repository!

Related Posts