Cache and Continuity
Since 0.39.0, Mull brings a new capability: caching of mutation results.
Caching has two different modes: explicit caching and implicit caching (continuity).
The rest of the guide is based on CMake Integration. Make sure you have a local build ready.
Implicit caching (continuity)
Section titled “Implicit caching (continuity)”Implicit caching is enabled by default and doesn’t require any changes to the config or Mull setup.
The first, cold run records the mutation results in a temporary cache file:
mull-runner --reporters SQLite build.mull/bin/chrono-test[info] Using config /private/tmp/sc-U7p0Z3/mull.yml[info] Using cache: /tmp/mull/6b67f650f7541d4c9fc593ee8e182163-a63aa7fddd7528de6caf9411b399c0cc.sqlite[info] Warm up run (threads: 1) [################################] 1/1. Finished in 142ms[info] Applying file path filter (threads: 8) [################################] 147/147. Finished in 1ms[info] Baseline run (threads: 1) [################################] 1/1. Finished in 126ms[info] Running mutants (threads: 8) [################################] 147/147. Finished in 4.50s[info] Results can be found at './1795040071.sqlite'[info] Surviving mutants: 101[info] Total execution time: 4.87sThe second run reuses the same results and simply skips execution:
mull-runner --reporters SQLite build.mull/bin/chrono-test[info] Using config /private/tmp/sc-U7p0Z3/mull.yml[info] Using cache: /tmp/mull/6b67f650f7541d4c9fc593ee8e182163-a63aa7fddd7528de6caf9411b399c0cc.sqlite[info] Warm up run (threads: 1) [################################] 1/1. Finished in 144ms[info] Applying file path filter (threads: 8) [################################] 147/147. Finished in 1ms[info] Reusing 147 cached result(s)[info] Results can be found at './1795040151.sqlite'[info] Surviving mutants: 101[info] Total execution time: 216msThe same mechanism applies when you want to restart an interrupted run:
mull-runner --reporters SQLite build.mull/bin/chrono-test[info] Using config /private/tmp/sc-U7p0Z3/mull.yml[info] Using cache: /tmp/mull/6b67f650f7541d4c9fc593ee8e182163-a63aa7fddd7528de6caf9411b399c0cc.sqlite[info] Warm up run (threads: 1) [################################] 1/1. Finished in 141ms[info] Applying file path filter (threads: 8) [################################] 147/147. Finished in 1ms[info] Baseline run (threads: 1) [################################] 1/1. Finished in 127ms[info] Running mutants (threads: 8) [############--------------------] 59/147. 1.59s^Cmull-runner --reporters SQLite build.mull/bin/chrono-test[info] Using config /private/tmp/sc-U7p0Z3/mull.yml[info] Using cache: /tmp/mull/6b67f650f7541d4c9fc593ee8e182163-a63aa7fddd7528de6caf9411b399c0cc.sqlite[info] Warm up run (threads: 1) [################################] 1/1. Finished in 127ms[info] Applying file path filter (threads: 8) [################################] 147/147. Finished in 1ms[info] Reusing 59 cached result(s)[info] Baseline run (threads: 1) [################################] 1/1. Finished in 126ms[info] Running mutants (threads: 8) [################################] 88/88. Finished in 2.77s[info] Results can be found at './1795040222.sqlite'[info] Surviving mutants: 101[info] Total execution time: 3.07sThis can be disabled via the --no-cache flag.
Explicit caching
Section titled “Explicit caching”Explicit caching has different behaviour and serves different use cases.
To enable explicit caching you have to specify the path to the cache via the --cache-path foobar.sqlite CLI flag.
Sharing of the results across different binaries
Section titled “Sharing of the results across different binaries”The first use case covers a scenario where several test suites/binaries share some mutants. If test A killed a mutant, then there is no need to check whether test B also kills it.
The first run (format-test) kills ~100 mutants.
The second run (chrono-test) can simply reuse that information and skip many already detected ones.
The original, isolated run of chrono-test took ~4.8s, while the cached one completed in ~2.6s.
mull-runner --reporters SQLite --cache-path shared.sqlite build.mull/bin/format-test[info] Using config /private/tmp/sc-U7p0Z3/mull.yml[info] Using cache: shared.sqlite[info] Warm up run (threads: 1) [################################] 1/1. Finished in 47ms[info] Applying file path filter (threads: 8) [################################] 141/141. Finished in 1ms[info] Baseline run (threads: 1) [################################] 1/1. Finished in 32ms[info] Running mutants (threads: 8) [################################] 141/141. Finished in 1.23s[info] Results can be found at './1795040852.sqlite'[info] Surviving mutants: 47[info] Total execution time: 1.32smull-runner --reporters SQLite --cache-path shared.sqlite build.mull/bin/chrono-test[info] Using config /private/tmp/sc-U7p0Z3/mull.yml[info] Using cache: shared.sqlite[info] Warm up run (threads: 1) [################################] 1/1. Finished in 137ms[info] Applying file path filter (threads: 8) [################################] 147/147. Finished in 1ms[info] Reusing 81 cached result(s)[info] Baseline run (threads: 1) [################################] 1/1. Finished in 125ms[info] Running mutants (threads: 8) [################################] 66/66. Finished in 2.28s[info] Results can be found at './1795040903.sqlite'[info] Surviving mutants: 45[info] Total execution time: 2.60sIncremental test suite improvement
Section titled “Incremental test suite improvement”The second use case targets incremental improvement of a test suite.
Consider the following scenario: you run the test suite and it detects 90 out of 100 mutants, leaving 10 survivors.
You add a new test case and only want to check whether any of those 10 mutants are killed now.
The explicit cache enables this: mull detects the rebuilt binary and reruns only the mutants that were not killed (survivors and timeouts).
mull-runner --reporters SQLite --cache-path inc.sqlite build.mull/bin/format-test[info] Using config /private/tmp/sc-U7p0Z3/mull.yml[info] Using cache: inc.sqlite[info] Warm up run (threads: 1) [################################] 1/1. Finished in 46ms[info] Applying file path filter (threads: 8) [################################] 141/141. Finished in 1ms[info] Baseline run (threads: 1) [################################] 1/1. Finished in 32ms[info] Running mutants (threads: 8) [################################] 141/141. Finished in 1.24s[info] Results can be found at './1795041413.sqlite'[info] Surviving mutants: 47[info] Total execution time: 1.34s# Rebuild the format-test binarymull-runner --reporters SQLite --cache-path inc.sqlite build.mull/bin/format-test[info] Using config /private/tmp/sc-U7p0Z3/mull.yml[info] Using cache: inc.sqlite[info] Warm up run (threads: 1) [################################] 1/1. Finished in 455ms[info] Applying file path filter (threads: 8) [################################] 141/141. Finished in 1ms[info] Reusing 86 cached result(s), 52 outdated by a rebuild[info] Baseline run (threads: 1) [################################] 1/1. Finished in 33ms[info] Running mutants (threads: 8) [################################] 55/55. Finished in 879ms[info] Results can be found at './1795041473.sqlite'[info] Surviving mutants: 46[info] Total execution time: 1.40sCaveats
Section titled “Caveats”Caching is hard. mull fingerprints a run by hashing the test binary itself, the dynamic libraries that contain mutants, and the test program (when it is a file).
Runs with different arguments keep separate results.
If the test results depend on anything else (e.g., env, date/time, network availability, etc.), then it is advised to disable caching altogether via the --no-cache CLI flag.
Wrapper test programs
Section titled “Wrapper test programs”Wrappers such as ctest or scripts need extra care: mull doesn’t fingerprint the test binaries they run.
If you only change tests outside of the binary with mutants, mull will reuse old survivors, so delete the cache or use --no-cache.
Within a single fresh build (e.g., a CI job), a shared explicit cache is safe: a mutant killed via one binary is killed for all of them.