Advanced Testing and Verification

Why it matters

Testing and Maintainability covered example-based unit tests, test doubles, and determinism. Beyond examples, a family of techniques checks broader properties, actively hunts for weaknesses in a test suite itself, and — for a tool that transforms code rather than just running it — a fixture-based testing approach of its own.

Property-based testing

An example-based test checks one specific input/output pair. A property-based test instead states an invariant that should hold for any valid input, and a library generates many varied inputs (including deliberately tricky edge cases) to check it:

JavaScriptA property-based test, checking an invariant across generated inputs

import fc from 'fast-check';

// Property: reversing an array twice always returns the original array
fc.assert(
  fc.property(fc.array(fc.integer()), (arr) => {
    const reversed = [...arr].reverse();
    const reversedTwice = [...reversed].reverse();
    return JSON.stringify(reversedTwice) === JSON.stringify(arr);
  }),
);

The library runs this property against dozens or hundreds of generated arrays — varying length, values, and edge cases (empty array, single element, duplicates) — and, on failure, automatically shrinks the failing input down to the smallest example that still reproduces it, far more useful for debugging than whatever large random input first triggered the failure.

Mutation testing: testing your tests

Code coverage measures which lines ran during a test suite — it says nothing about whether the test actually verified anything meaningful about the code that ran. Mutation testing checks this directly: it automatically introduces small bugs (“mutants”) into the code — flipping a < to <=, changing + to -, deleting a line — and reruns the test suite against each mutant.

What a mutation testing run reports
Outcome	Meaning
Mutant killed	A test failed against the mutated code — the suite genuinely caught this specific bug
Mutant survived	Every test still passed against the mutated (broken) code — the suite has a real gap here

// Original
function isAdult(age) {
  return age >= 18;
}

// A mutant: >= changed to >
function isAdult(age) {
  return age > 18;
}
// If no test checks age === 18 specifically, this mutant survives —
// revealing that 100% line coverage of isAdult still missed a real boundary case.

Fuzz and differential testing

Fuzz testing feeds a function large volumes of randomized, often malformed input, specifically looking for crashes or unexpected exceptions — most valuable for code parsing untrusted input (a file format parser, a request body parser) where a genuinely malformed input is a realistic, expected scenario, not a hypothetical.

Differential testing runs two independent implementations of the same behavior against the same inputs and compares their outputs — useful when migrating between implementations (an old parser and a new one) or validating a fast, optimized version against a simple, obviously-correct reference implementation.

// Differential testing: comparing a fast implementation against a simple reference
function referenceSum(arr) {
  let total = 0;
  for (const n of arr) total += n;
  return total;
}

function optimizedSum(arr) {
  return arr.reduce((total, n) => total + n, 0); // "optimized" here just for illustration
}

fc.assert(
  fc.property(fc.array(fc.integer()), (arr) => {
    return referenceSum(arr) === optimizedSum(arr);
  }),
);

Hermetic builds and reproducibility

A hermetic build produces an identical output given the same inputs, regardless of what else is on the machine running it — no dependency on ambient system state, unpinned tool versions, network access during the build, or the current date/time.

Codemods and static analysis

A codemod is a program that transforms source code programmatically — a scripted, repeatable refactor applied across an entire codebase, built on parsing source into an AST (recall this is covered more fully in the next lesson), transforming it, and printing it back out:

// A simplified codemod sketch: rename every call to oldFunctionName to newFunctionName
function codemod(sourceCode) {
  const ast = parse(sourceCode);
  traverse(ast, {
    CallExpression(path) {
      if (path.node.callee.name === 'oldFunctionName') {
        path.node.callee.name = 'newFunctionName';
      }
    },
  });
  return generate(ast);
}

A trustworthy codemod is tested against fixtures — real (or realistic) before/after code samples — checked for idempotency (running it twice produces the same result as running it once, so accidentally running it again causes no harm), and shipped with rollback guidance for anyone who needs to undo it.

Check your understanding

Or reveal the answer without checking

Answer:Every line ran during testing, but no test actually asserts on the specific boundary value where >= and > disagree — a real gap coverage alone didn't reveal
A surviving mutant means the test suite didn't distinguish between the original and mutated behavior — here, no test checks the exact boundary value (age === 18) where >= and > produce different results. Full line coverage doesn't guarantee this; mutation testing specifically surfaces it.

Or reveal the answer without checking

Answer:Malformed, adversarial input is a realistic, expected scenario for such a function — fuzzing systematically searches for inputs that crash or misbehave it, rather than relying on the test author to imagine every malformed case
A parser for untrusted input will genuinely encounter malformed data in production — fuzz testing automates the search for inputs that break it, covering a much larger space than a human author manually writing malformed-input examples could realistically anticipate.

Practice: warm-up

Write one property (in your own words, no library required) that should hold for any correct implementation of a sortArray(arr) function, beyond “the output looks sorted.”

Stuck? Reveal one hint at a time.

Hint 1
A sort that just returns an empty array, or a fixed array, would "look sorted" too — what property specifically rules that out?
Hint 2
Think about what must be true relating the multiset of the OUTPUT to the multiset of the INPUT.

Reveal an example property

Try the problem yourself before reading this. There is often more than one reasonable approach — treat this as one worked example, not the only correct answer.

Property: sortArray(arr) contains exactly the same elements as arr (same multiset —
same values, same counts, just possibly reordered), for any array.

This rules out a "sort" that returns [] or a hard-coded constant array, which would
otherwise trivially satisfy "the output looks sorted" alone.

Practice: apply it

Sketch (as pseudocode or real code) a codemod that finds every var declaration in a file and converts it to let. What fixture-based test structure would you use to verify it?