Mutation testing with Foundry

Mutation testing with Foundry can be a gem for your protocol stress tests, if you know how to use it.

In short mutation testing is a method to check the quality of the test suite by intentionally introducing bugs into the code and ensuring the tests catch the bug.

For example, you got a func that can be called only by the owner. Later you wrote a simple test to check it:

If this is only tested in a situation where the owner was the one calling the funciton then deleting onlyOwner will not cause the test to fail. If the onlyOwner modifier doesn’t block non-owners, then the unit tests won’t catch it

I hope you understand the idea.

So there are can be some other ways to hold a mutation testing for your protocol:

1. Inverting inequality comparisons (== to !=);
2. Changing constant values or swapping string constants for empty strings
3. Replace true with false
4. Deleting lines

But...

Developers are really good today and they write great tests. We see tests that will revert if a user with no role calls a function with a modifier, or separate tests to check a revert or if/else option. etc. So in most cases such mutation tests are useless.

However, not always

There are three other types of mutation testing that can add great value to your protocol, especially when combined with an invariant tests!

1. Changes in bitwise operations;
2. Swapping arithmetic operators;
3. Swapping lines of code;

Let's talk about each one.

Imagine the situation where your protocol relies on the tons of bitwise / bitshifting operations. You wrote a test that meets your requirements. But was it good enough to pass if any bit operator would be changed?

Swapping arithmetic operators can be useful when dealing with a complex math operations to get the actual amount of tokens to send to your user. A lot of tests should be written to check the correct calculations for the code in the snippet.

Swapping lines of code, my favorite one, can help you to find a Crit! From the code snippet above try to think what type of vulnerability you can avoid by swapping the line with the mint() and burn() functions?

Mutation testing is a powerful method if you understand how to use it wisely.

Follow me to get more ideas of how to make your protocol more secure.