Skip to content

Quickstart#

Install Kolena to set up rigorous and repeatable model testing in minutes.

In this quickstart guide, we'll use the object_detection_2d example integration to demonstrate the how to curate test data and test models in Kolena.

Install kolena#

Install the kolena Python package to programmatically interact with Kolena:

pip install kolena

poetry add kolena

Clone the Examples#

The kolenaIO/kolena repository contains a number of example integrations to clone and run directly:

To get started, clone the kolena repository:

git clone https://github.com/kolenaIO/kolena.git

With the repository cloned, let's set up the object_detection_2d example:

cd kolena/examples/object_detection_2d
poetry update && poetry install

Now we're up and running and can start creating test suites and testing models.

Create Test Suites#

Each of the example integrations comes with scripts for two flows:

  1. seed_test_suite.py: Create test cases and test suite(s) from a source dataset
  2. seed_test_run.py: Test model(s) on the created test suites

Before running seed_test_suite.py, let's first configure our environment by populating the KOLENA_TOKEN environment variable. Visit the Developer page to generate an API token and copy and paste the code snippet into your environment:

export KOLENA_TOKEN="********"

We can now create test suites using the provided seeding script:

poetry run python3 object_detection_2d/seed_test_suite.py

After this script has completed, we can visit the Test Suites page to view our newly created test suites.

In this object_detection_2d example, we've created test suites stratifying the COCO 2014 validation set (which is stored as a CSV in S3) into test cases by brightness and bounding box size. In this example will be looking at the following labels:

["bicycle", "car", "motorcycle", "bus", "train", "truck", "traffic light", "fire hydrant", "stop sign"]

Test a Model#

After we've created test suites, the final step is to test models on these test suites. The object_detection_2d example provides the following models to choose from {yolo_r, yolo_x, mask_rcnn, faster_rcnn, yolo_v4s, yolo_v3} for this step:

poetry run python3 object_detection_2d/seed_test_run.py "yolo_v4s"

Note: Testing additional models

In this example, model results have already been extracted and are stored in CSV files in S3. To run a new model, plug it into the infer method in seed_test_run.py.

Once this script has completed, click the results link in your console or visit Results to view the test results for this newly tested model.

Conclusion#

In this quickstart, we used an example integration from kolenaIO/kolena to create test suites from the COCO dataset and test the open-source yolo_v4s model on these test suites.

This example shows us how to define an ML problem as a workflow for testing in Kolena, and can be arbitrarily extended with additional metrics, plots, visualizations, and data.