Test Suites

What is a Test Suite?

A test suite is a grouping of one or more test cases. Test suites organize your test cases into meaningful groups, answering important questions like:
  • How does my model's performance differ across demographic groups?
  • How do different lighting conditions affect my model's performance?
  • How robust is my model to various input perturbations?

Test Suite Best Practices & FAQ

How many test cases should I have in my test suite?
In general we recommend including between 5 and 50 test cases in a test suite. However, these numbers are just guidelines — one single test case is a good place to start, and over time many test suites (such as regression test suites) will accrue hundreds of test cases.
How many test suites should I test a model on?
As many as are necessary! Regardless of the number of test suites a given image is in, you will only ever have to process the image and upload inferences to Kolena a single time. Testing on multiple test suites at a time is encouraged:
test(model, TestSuite("A"), TestSuite("B"), ...) # test on any number of suites at once
Can I include the same test case in multiple test suites?
Yes. A test case may be present in as many test suites as desired.
Should I update my test suites or keep them frozen?
Update your test suites often! And don't worry about losing the ability to compare apples to apples — in the web platform two models tested on different versions of the same test suite can be compared on only the test cases they've both been tested on.

See Also