Test Suites
A test suite is a grouping of one or more test cases. Test suites organize your test cases into meaningful groups, answering important questions like:
- How does my model's performance differ across demographic groups?
- How do different lighting conditions affect my model's performance?
- How robust is my model to various input perturbations?
In general we recommend including between 5 and 50 test cases in a test suite. However, these numbers are just guidelines — one single test case is a good place to start, and over time many test suites (such as regression test suites) will accrue hundreds of test cases.
As many as are necessary! Regardless of the number of test suites a given image is in, you will only ever have to process the image and upload inferences to Kolena a single time. Testing on multiple test suites at a time is encouraged:
test(model, TestSuite("A"), TestSuite("B"), ...) # test on any number of suites at once
Update your test suites often! And don't worry about losing the ability to compare apples to apples — in the web platform two models tested on different versions of the same test suite can be compared on only the test cases they've both been tested on.
Last modified 8mo ago