kolena.classification

Legacy Warning

The kolena.classification module is considered legacy and should not be used for new projects.

Please see kolena.workflow for customizable and extensible definitions to use for all new projects.

kolena.classification supports the following types of classification models:

Classification Type	Description
Binary	Classification model predicts a single class, using a threshold on prediction confidence to bisect the test set
Multi-label	Classification model predicts multiple classes, with each prediction over a threshold considered positive (i.e. ensemble of binary classifiers)

Quick Links

• kolena.classification.TestImage: create images for testing
• kolena.classification.TestCase: create and manage test cases
• kolena.classification.TestSuite: create and manage test suites
• kolena.classification.TestRun: test models on test suites
• kolena.classification.Model: create models for testing

CustomMetricsCallback = Callable[[List[SampleInferences]], CustomMetrics] module-attribute

Signature for a custom metrics computation function.

AccuracyOptimal()

Bases: TestConfig

Test configuration that sets the default display threshold in Kolena to the threshold that corresponds to the highest accuracy score across all images within the test suite being tested.

This threshold is evaluated and set per label for test suites with multiple labels.

FixedGlobalThreshold(fixed_threshold)

Bases: TestConfig

Test configuration that sets the default display threshold in Kolena to a fixed global threshold for all labels within the test suite.

fixed_threshold: float = fixed_threshold instance-attribute

The threshold used as the default when visualizing results in Kolena. Must be between 0 and 1.

InferenceModel(name, infer, metadata=None)

Bases: Model

A Model with a special infer member performing inference on a provided TestImage.

infer: Callable[[TestImage], Optional[List[Tuple[str, float]]]] instance-attribute

A function transforming an input TestImage to zero or more (label, confidence) tuples representing model predictions.

Model(name, metadata=None)

Bases: BaseModel

The descriptor for a classification model in Kolena.

For additional functionality, see the associated base class documentation.

metadata: Dict[str, Any] instance-attribute

Unstructured metadata associated with the model.

name: str instance-attribute

Unique name of the model, potentially containing information about the architecture, training dataset, configuration, framework, commit hash, etc.

TestCase(name, version=None, description=None, images=None, reset=False)

Bases: BaseTestCase

A test case is the base grouping of test data in the Kolena platform.

Fundamentally, a test case can be thought of as a benchmark dataset. Metrics are computed for each test case.

A test case may be as large or as small as necessary. A test case may have millions of images for high-level results across a large population. Alternatively, a test case may have only one or a handful of images for laser-focus on a specific scenario.

For additional functionality, see the associated base class documentation.

Parameters:

Name	Type	Description	Default
name	str	The name of the test case. If a test case by this name already exists, that test case is loaded.	required
version	Optional[int]	Optionally specify the version of the test case to load. Ignored when a test case by the provided name does not already exist.	None
description	Optional[str]	Optionally specify a description for the new test case. Ignored when a test case with the provided name already exists.	None
images	Optional[List[_TestImageClass]]	Optionally provide a list of TestImage images used to seed a new test case. Ignored when a test case with the provided name already exists.	None

TestConfig

Bases: _TestConfig

Base class for testing configurations.

See concrete implementations FixedGlobalThreshold and AccuracyOptimal for details.

TestImage(locator, dataset=None, labels=None, metadata=None)

Bases: BaseTestImage

An image with associated ground truth labels for testing.

dataset: str instance-attribute

The source dataset this image belongs to.

labels: List[str] = labels or [] instance-attribute

Zero or more ground truth labels for this image. For binary classifiers, an arbitrary string such as positive may be used. Not surfaced during testing.

locator: str instance-attribute

Pointer to the bucket location of this image, e.g. gs://my-bucket/my-dataset/example.png.

metadata: Dict[str, MetadataElement] instance-attribute

Arbitrary metadata associated with this image. This metadata is surfaced during testing and may be used as model inputs as necessary.

Certain metadata values can be visualized in the web platform when viewing results:

• Annotation objects are overlaid on the main image
• Asset objects containing locators pointing to images, e.g. gs://my-bucket/my-dataset/example-1channel.png, are displayed

See the metadata documentation for more details.

filter(predicate)

Return a copy of this test image with ground truth labels filtered to only those that match the provided predicate.

Parameters:

Name	Type	Description	Default
predicate	Callable[[str], bool]	Function accepting a string label and returning a boolean indicating whether or not to include the ground truth label.	required

Returns:

Type	Description
TestImage	A new test image with labels filtered by the predicate.

TestRun(model, test_suite, test_config=None, custom_metrics_callback=None, reset=False)

Bases: BaseTestRun

Interface to run tests for a Model on a TestSuite. Any in-progress tests for this model on these suites are resumed.

For a streamlined interface, see test.

Parameters:

Name	Type	Description	Default
model	Model	The model being tested.	required
test_suite	TestSuite	The test suite on which to test the model.	required
test_config	Optional[TestConfig]	Optionally specify a configuration, e.g. FixedGlobalThreshold, to customize the metrics evaluation logic for this test run. Defaults to AccuracyOptimal if unspecified.	None
custom_metrics_callback	Optional[CustomMetricsCallback[_TestImageClass, _InferenceClass]]	Optionally specify a callback function to compute custom metrics for each test-case. The callback would be passed inferences of images in each testcase and should return a dictionary with metric name as key and metric value as value.	None
reset	bool	Overwrites existing inferences if set.	False

TestSuite(name, version=None, description=None, test_cases=None, reset=False)

Bases: BaseTestSuite

A test suite is a grouping of TestCase objects.

Testing on test suites is performed via test. Metrics are computed across all samples in a test suite and also for each individual test case within the suite.

For additional functionality, see the associated base class documentation.

Parameters:

Name	Type	Description	Default
name	str	The name of the test suite. If a test suite by this name already exists, that test suite is loaded.	required
version	Optional[int]	Optionally specify the version of the test suite to load. Ignored when the a suite by the provided name does not already exist.	None
description	Optional[str]	Optionally specify a description for the new test suite. Ignored when a test suite with the provided name already exists.	None
test_cases	Optional[List[TestCase]]	Optionally provide a list of TestCase tests used to seed a new test suite. Ignored when a test suite with the provided name already exists.	None

description: str instance-attribute

Free-form description of this test suite. May be edited at any time via TestSuite.edit.

name: str instance-attribute

Unique name of the test suite.

test_cases: List[TestCase] instance-attribute

The TestCase objects in this test suite. May be edited at any time via TestSuite.edit.

version: int instance-attribute

The version of the test suite. Version is automatically incremented whenever the test suite is modified via TestSuite.edit.

test(model, test_suite, test_config=None, custom_metrics_callback=None, reset=False)

Test the provided InferenceModel on a TestSuite. Any tests already in progress for this model on these suites are resumed.

Parameters:

Name	Type	Description	Default
model	InferenceModel	The model being tested, complete with an InferenceModel.infer function to perform inference.	required
test_suite	TestSuite	The test suite on which to test the model.	required
test_config	Optional[TestConfig]	Optionally specify a configuration, e.g. FixedGlobalThreshold, to customize the metrics evaluation logic for this test run. Defaults to AccuracyOptimal if unspecified.	None
custom_metrics_callback	Optional[CustomMetricsCallback[TestImage, Tuple[str, float]]]	Optionally specify a callback function to compute custom metrics for each test case. The callback would be passed inferences of images in each testcase and should return a dictionary with metric name as key and metric value as value.	None
reset	bool	Overwrites existing inferences if set.	False

Metadata

Metadata associated with a TestImage.

from kolena.classification import TestImage
from kolena.classification.metadata import Landmarks, BoundingBox, Asset

test_image = TestImage("s3://bucket/path/to/image.png", metadata=dict(
    input_landmarks=Landmarks([(0,0), (10, 10), (20, 20), (30, 30), (40, 40)]),
    input_bounding_box=BoundingBox((0, 0), (100, 100)),
    image_grayscale=Asset("s3://bucket/path/to/image_grayscale.png"),
))

Annotation()

Bases: Frozen, Serializable

An annotation associated with an image.

Annotations are surfaced during testing along with the image locator and any other metadata associated with an image. In the web platform, annotations are overlaid on top of images when visualizing results.

Asset(locator)

Bases: Frozen, Serializable

An asset living in your shared bucket. Assets are surfaced during testing along with any other metadata associated with a given test image.

In the web platform, certain assets such as PNG and JPG images are viewable when visualizing results in the gallery.

locator: str = locator instance-attribute

Location of this asset in shared bucket, e.g. s3://my-bucket/path/to/image.png.

BoundingBox(top_left, bottom_right)

Bases: Annotation

An annotation comprising a bounding box around an object in an image.

bottom_right: Tuple[float, float] = bottom_right instance-attribute

Point in (x, y) pixel coordinates representing the bottom right corner of the bounding box.

top_left: Tuple[float, float] = top_left instance-attribute

Point in (x, y) pixel coordinates representing the top left corner of the bounding box.

Landmarks(points)

Bases: Annotation

An annotation comprising an arbitrary-length set of landmarks corresponding to some object in an image, e.g. face landmarks used for pose estimation.

points: List[Tuple[float, float]] = points instance-attribute

Any number of (x, y) points in pixel coordinates representing a set of landmarks.