Property-based testing — JS: Advanced Testing

• Frameworks
• Conclusion

Property-based testing is an approach to functional testing that helps to verify whether the function under test corresponds to a given property. You don't need to set all the tests cases for this approach. Its task is to match the characteristics at output with a given property.

A property is a statement that can be represented in pseudocode as follows:

for all (x, y, ...)
such as precondition(x, y, ...) holds
property(x, y, ...) is true

We describe an invariant in the style of “for any data such that ... condition is satisfied” and, unlike usual tests, we don't explicitly specify all test cases, we only describe the conditions that they must satisfy.

Suppose we have a divide() function that finds the quotient of two numbers.

const divide = (a, b) => a / b;

Let's write an ordinary test for this function:

const {equal} = require('assert');

equal(divide(4, 2), 2);
equal(divide(18, 3), 6);

Here it's simple, we pass the numbers 18 and 3 to the input, and we expect to get 6. The tests pass. But here we only tested the function on two pairs of input data. The test shows that the function works correctly only in these two cases. But it may turn out that with a different pair of numbers, the function works in a different way unexpectedly. To solve this problem, we need to write a test that focuses not on the input and output parameters, but on the properties as a whole. These properties must be true for any correct implementation.

The division operation has a property: the distributive property on the right. It means that dividing the sum of a and b by c is equal to a / c + b / c.

We use this in tests. Let's not get hung up on specific values and use a random number generator to obtain test data.

const a = Math.random() * 1000;
const b = Math.random() * 1000;
const c = Math.random() * 1000;

const left = divide(a + b, c);
const right = divide(a, c) + divide(b, c);

We'll run this test several times in a loop, and at a certain point, we'll get a set of test data where all three numbers equal zero. The test will fall because dividing zero by zero gives NaN, and NaN is not equal to NaN. So we understand that we need to add a check for zeros to the function.

AssertionError [ERR_ASSERTION]: NaN == NaN

We wrote an ordinary test, but we used arbitrary values instead of those taken from our head and were able to run the test many times on different input data. In this way, we've checked the specification itself, i.e., what the function should do, not its behavior in individual cases. This is property-based testing.

In real life, no one runs tests in a loop by manually putting in values. There are ready-made frameworks for this. The data is generated automatically by the property-based testing framework based on the described properties. If after a certain number of runs with random data that meet the description, the condition is met, the test is considered passed. Otherwise, the framework completes the test with an error.

Let's look at what the advantages of property-based testing are:

• It covers all possible data. The frameworks automatically generate data based on the described properties. In theory, this feature allows you to cover all possible types of input data: for example, the entire range of strings or integers.

• Whenever a failure occurs, the framework tries to shorten the test case. For example, if the failure condition is the presence of a given character in the string, the framework should return a single-character string that contains only that character. This is a serious advantage of property-testing - if failure occurs, the test stops working on a minimal example, not on a set of input data.

• Reproducibility: Before each test run, initial values are created so that if the test fails, you can reproduce the test on the same data set.

It's important to note that property testing does not replace unit testing. It should be treated as an additional level of tests, which will help reduce the time to check your code is written properly compared with other approaches.

Frameworks

The idea of property testing was first implemented in the QuickCheck framework in Haskell. There are several libraries for JavaScript too, one of them is fast-check.

To install it, run this command:

npm install fast-check --save-dev

Let's use it to test the contains(), which checks if a substring is contained in a string. Strings have two properties that we can use:

• A string always contains itself as a substring
• The string a + b + c always contains its substring b, regardless of the contents of b, regardless of content a, b and c

import fc from 'fast-check';

// Code being tested
const contains = (text, pattern) => text.indexOf(pattern) >= 0;

// Defining the properties

test('string should always contain itself', () => {
  fc.assert(
    fc.property(
      fc.string(),
      text => contains(text, text)
    )
  );
});

test('string should always contain its substring', () => {
  fc.assert(
    fc.property(
      fc.string(), fc.string(), fc.string(),
      (a, b, c) => contains(a + b + c, b)
    )
  );
});

Let's examine the structure of the test in more detail

fc.assert(<property>(, parameters)) — performs a test and checks that the property remains true for all a, b and c strings created by the library. When a failure occurs, this line is responsible for reducing the test case to a minimum size to make it easier for the user. By default, it performs a property check on 100 pieces of generated input data.

fc.property(<...arbitraries>, <predicate>) — defines a property. arbitraries — are values that are responsible for building input data, and predicate — is a function that tests input data. The predicate predicate should either return a logical value or return nothing and terminate the test in case of failure.

fc.string() is a string generator that's responsible for creating and shortening test values.

If desired, you can extract the generated values to check the properties by replacing fc.assert with fc.sample:

fc.sample(
  fc.property(
    fc.string(), fc.string(), fc.string(),
    (a, b, c) => contains(a + b + c, b)
  )
);

The generated data will look something like this:

{a: ") | 2", b: "", c: "$ & RJh %%"}
{a: "\\\" ", b:" Y \\\ "\\\" ", c:" $ S # K3 "}
{a:" $ ", b:" \\\\ cx% wf ", c:" 't4qRA "}
{a:" ", b:" ", c:" n? H. 0% "}
{a:" 6_ # 7 ", b:" b ", c:" 4% E "}
...

Now let's try testing a knowingly wrong implementation of contains(). We use it as an example to show what the framework generates in case of failure and how it shortens the input:

const contains = (pattern, text) => {
  return text.substr(1).indexOf(pattern) !== -1;
};

The framework generates a certain set of data. As soon as the test sees a failure, it triggers the shortening process. When testing the example above, a failure occurs:

Error: Property failed after 20 tests
{ seed: 1783957873, path: "19:1:0:1:1", endOnFailure: true }
Counterexample: [""," ",""]
Shrunk 4 time(s)
Got error: Property failed by returning false

Conclusion

Property-based testing is a useful and powerful tool. We shouldn't abandon classic testing methods, but we can combine them with property-based testing. For example, you can cover the basic functionality with classic example-based tests and also cover the critical functions with property-based tests.

Recommended materials

1. Property-based testing framework