Skip to content

Base Definitions#

Legacy Warning

This module is considered legacy and should not be used for new projects.

Please see kolena.workflow for customizable and extensible definitions to use for all new projects.

Base definitions shared between kolena.classification and kolena.detection.

BaseTestCase(name, workflow, version=None, description=None, images=None, reset=False) #

Bases: ABC, Frozen, WithTelemetry

A test case holds a set of images to compute performance metrics against.

Parameters:

Name Type Description Default
name str

Name of the test case to create or load.

 required
version Optional[int]

Optionally specify the version of the test case to load. When absent, the latest version is loaded. Ignored when creating new test cases.

 None
description Optional[str]

Optionally specify a description for a newly created test case. For existing test cases, this description can be edited via TestCase.edit.

 None
images Optional[List[_TestImageClass]]

Optionally provide a list of images and associated ground truths to populate a new test case. For existing test cases, images can be edited via TestCase.edit.

 None

name: str instance-attribute #

The unique name of this test case.

version: int instance-attribute #

The version of this test case. A test case's version is automatically incremented whenever it is edited via TestCase.edit.

description: str instance-attribute #

Free-form, human-readable description of this test case. Can be edited at any time via TestCase.edit.

Editor(description, reset) #

Interface to edit a test case. Create with TestCase.edit.

description(description) #

Update the description of this test case.

Parameters:

Name Type Description Default
description str

The new test case description.

 required
add(image) #

Add a test image to the test case, targeting the ground_truths held by the image. When the test image already exists in the test case, its ground truth is overwritten.

To filter the ground truths associated with a test image, see TestImage.filter.

Parameters:

Name Type Description Default
image _TestImageClass

The test image to add to the test case, holding corresponding ground truths.

 required
remove(image) #

Remove the image from the test case.

Parameters:

Name Type Description Default
image _TestImageClass

The image to remove.

 required

Raises:

Type Description
KeyError

The image is not in the test case.

load_images() #

Load all test images with their associated ground truths in this test case.

iter_images() #

Iterate through all images with their associated ground truths in this test case.

create(name, description=None, images=None) classmethod #

Create a new test case with the provided name.

Parameters:

Name Type Description Default
name str

The name of the new test case to create.

 required
description Optional[str]

Optional free-form description of the test case to create.

 None
images Optional[List[_TestImageClass]]

Optionally specify a set of images to populate the test case.

 None

Returns:

Type Description
BaseTestCase

The newly created test case.

load(name, version=None) classmethod #

Load an existing test case with the provided name.

Parameters:

Name Type Description Default
name str

The name of the test case to load.

 required
version Optional[int]

Optionally specify a particular version of the test case to load. Defaults to the latest version when unset.

 None

Returns:

Type Description
BaseTestCase

The loaded test case.

edit(reset=False) #

Edit this test case in a context:

with test_case.edit() as editor:
    # perform as many editing actions as desired
    editor.add(...)
    editor.remove(...)

Changes are committed to Kolena when the context is exited.

Parameters:

Name Type Description Default
reset bool

Clear all existing test samples in the test case.

 False

BaseTestSuite(name, workflow, version=None, description=None, test_cases=None, reset=False) #

Bases: ABC, Frozen, WithTelemetry

A test suite groups together one or more test cases.

Parameters:

Name Type Description Default
name str

The name of the test suite to create or load.

 required
version Optional[int]

Optionally specify the version of the test suite to load. When absent, the latest version is loaded. Ignored when creating new test suites.

 None
description Optional[str]

Optionally specify a description for a newly created test suite. For existing test suites, this description can be edited via TestSuite.edit.

 None
test_cases Optional[List[BaseTestCase]]

optionally specify a list of test cases to populate a new test suite. For existing test suites, test cases can be edited via TestSuite.edit.

 None

Editor(test_cases, description, reset) #

Interface to edit a test suite. Create with TestSuite.edit.

description(description) #

Update the description of the test suite.

Parameters:

Name Type Description Default
description str

The new description of the test suite.

 required
add(test_case) #

Add the provided TestCase to the test suite. If a different version of the test case already exists in this test suite, it is replaced.

Parameters:

Name Type Description Default
test_case BaseTestCase

The test case to add to the test suite.

 required
remove(test_case) #

Remove the provided TestCase from the test suite. Any version of this test case in this test suite will be removed; the version does not need to match exactly.

Parameters:

Name Type Description Default
test_case BaseTestCase

The test case to be removed from the test suite.

 required
merge(test_case) #

Deprecated: since 0.56.0

Replaced by idempotent behavior in TestSuite.Editor.add.

Add the provided TestCase to the test suite, replacing any previous version of the test case that may be present in the suite.

Parameters:

Name Type Description Default
test_case BaseTestCase

The test case to be merged into the test suite.

 required

create(name, description=None, test_cases=None) classmethod #

Create a new test suite with the provided name.

Parameters:

Name Type Description Default
name str

The name of the new test suite to create.

 required
description Optional[str]

Optional free-form description of the test suite to create.

 None
test_cases Optional[List[BaseTestCase]]

Optionally specify a set of test cases to populate the test suite.

 None

Returns:

Type Description
BaseTestSuite

The newly created test suite.

load(name, version=None) classmethod #

Load an existing test suite with the provided name.

Parameters:

Name Type Description Default
name str

The name of the test suite to load.

 required
version Optional[int]

Optionally specify a particular version of the test suite to load. Defaults to the latest version when unset.

 None

Returns:

Type Description
BaseTestSuite

The loaded test suite.

edit(reset=False) #

Edit this test suite in a context:

with test_suite.edit() as editor:
    # perform as many editing actions as desired
    editor.add(...)
    editor.remove(...)

Changes are committed to Kolena when the context is exited.

Parameters:

Name Type Description Default
reset bool

Clear all existing test cases in the test suite.

 False

Returns:

Type Description
Iterator[Editor]

Context-managed TestSuite.Editor instance exposing editing functionality for this test suite.

BaseModel(name, workflow, metadata=None) #

Bases: ABC, Frozen, WithTelemetry

The base class for kolena.classification.Model and kolena.detection.Model.

name: str = name instance-attribute #

Unique name of the model, potentially containing information about the architecture, training dataset, configuration, framework, commit hash, etc.

metadata: Dict[str, Any] = loaded.metadata instance-attribute #

Unstructured metadata associated with the model.

load_inferences(test_object) #

Retrieve the uploaded inferences with identical ground truth labels for each image in a test case or test suite.

Returns:

Type Description
List[Tuple[_TestImageClass, Optional[List[_InferenceClass]]]]

List of all images and inferences from this model on the provided test_object.

iter_inferences(test_object) #

Iterate the uploaded inferences with identical ground truth labels for each image in a test case or test suite.

Returns:

Type Description
Iterator[Tuple[_TestImageClass, Optional[List[_InferenceClass]]]]

Iterator over all images and inferences from this model on the provided test_object.

load_inferences_by_test_case(test_suite) #

Retrieve the uploaded inferences of a test suite for each image, grouped by test case.

CustomMetricsCallback = Callable[[List[SampleInferences]], CustomMetrics] module-attribute #

Signature for a custom metrics computation function.

BaseTestRun(model, test_suite, config=None, custom_metrics_callback=None, reset=False) #

Bases: ABC, Frozen, WithTelemetry

The base class for kolena.classification.TestRun and kolena.detection.TestRun.

add_inferences(image, inferences) #

Add inferences for a test image to the test run results.

Parameters:

Name Type Description Default
image _TestImageClass

The image that inferences are evaluated on.

 required
inferences Optional[List[_InferenceClass]]

List of inferences corresponding to the image.

 required

iter_images() #

Returns an iterator of all remaining images that need inferences evaluated.

load_images(batch_size=BatchSize.LOAD_SAMPLES.value) #

Returns a list of images that still need inferences evaluated, bounded in count by batch_size. Note that image ground truths will be excluded from the returned batch of images.

Parameters:

Name Type Description Default
batch_size int

The maximum number of images to retrieve.

 BatchSize.LOAD_SAMPLES.value