Skip to content

kolena.detection#

Legacy Warning

The kolena.detection module is considered legacy and should not be used for new projects.

Please see kolena.workflow for customizable and extensible definitions to use for all new projects.

Object detection example from the Common Objects in Context (COCO) dataset.

Object detection models attempt to localize and classify objects in an image. Kolena supports single-class and multi-class object detection models identifying objects with rectangular (object detection) and arbitrary (instance segmentation) geometry.

Note

Instance Segmentation and Object Detection are functionally equivalent, differing only in the geometry of the detected object. For brevity, this documentation discusses object detection only.

CustomMetricsCallback = Callable[[List[SampleInferences]], CustomMetrics] module-attribute #

Signature for a custom metrics computation function.

F1Optimal(iou_threshold=0.5) #

Bases: TestConfig

Test configuration that sets the default display threshold in Kolena to the threshold that corresponds to the highest F1 score across all images within the test suite.

This threshold is evaluated and set per class for test suites with multiple classes.

iou_threshold: float = iou_threshold instance-attribute #

The minimum intersection over union (IoU) score between an inference and a ground truth for it to qualify as a potential match. Must be between 0 and 1.

FixedGlobalThreshold(fixed_threshold, iou_threshold=0.5) #

Bases: TestConfig

Test configuration that sets the default display threshold in Kolena to a fixed global threshold for all classes within the test suit.

fixed_threshold: float instance-attribute #

The threshold used as the default for all classes when visualizing results in Kolena. Must be between 0 and 1.

iou_threshold: float = iou_threshold instance-attribute #

The minimum intersection over union (IoU) score between an inference and a ground truth for it to qualify as a potential match. Must be between 0 and 1.

GroundTruth() #

Bases: Serializable, Frozen

Base class for ground truths associated with an image.

See concrete implementations BoundingBox and SegmentationMask for details.

Inference #

Bases: _Inference

Base class for model inferences associated with an image.

See concrete implementations BoundingBox, and SegmentationMask for details.

InferenceModel(name, infer, metadata=None) #

Bases: Model

Extension of Model with an infer method to perform inference.

See documentation parent Model for details.

infer: Callable[[TestImage], Optional[List[Inference]]] instance-attribute #

Function transforming a TestImage into a list of zero or more Inference objects.

Model(name, metadata=None) #

Bases: BaseModel

The descriptor for an object detection model within Kolena.

For additional functionality, see the associated base class documentation.

metadata: Dict[str, Any] instance-attribute #

Unstructured metadata associated with the model.

name: str instance-attribute #

Unique name of the model within the platform. If the provided model name has already been registered, that model and its metadata are loaded upon instantiation.

TestCase(name, version=None, description=None, images=None, reset=False) #

Bases: BaseTestCase

A test case holds a set of images to compute performance metrics against.

For additional functionality, see the associated base class documentation.

Parameters:

Name Type Description Default
name str

Name of the test case to create or load.

 required
version Optional[int]

Optionally specify the version of the test case to load. When absent, the latest version is loaded. Ignored when creating new test cases.

 None
description Optional[str]

Optionally specify a description for a newly created test case. For existing test cases, this description can be edited via TestCase.edit.

 None
images Optional[List[_TestImageClass]]

Optionally provide a list of images and associated ground truths to populate a new test case. For existing test cases, images can be edited via TestCase.edit.

 None

TestConfig #

Bases: _TestConfig

Base class for testing configurations.

See concrete implementations FixedGlobalThreshold and F1Optimal for details.

TestImage(locator, dataset=None, ground_truths=None, metadata=None) #

Bases: BaseTestImage

Test image comprising a single image locator.

dataset: str instance-attribute #

Dataset this test image belongs to. Empty when unspecified.

ground_truths: List[GroundTruth] = ground_truths or [] instance-attribute #

List of GroundTruth annotations associated with this image.

locator: str instance-attribute #

Bucket locator for the provided test sample, e.g. gs://my-bucket/path/to/image.png.

metadata: Dict[str, MetadataElement] instance-attribute #

Metadata associated with this test image. Surfaced during test runs.

filter(predicate) #

Return a copy of this test image with ground truths filtered to only those that match the provided predicate.

Parameters:

Name Type Description Default
predicate Callable[[GroundTruth], bool]

Function accepting a GroundTruth and returning a boolean indicating whether or not to include the ground truth.

 required

Returns:

Type Description
TestImage

A new test image with ground truths filtered by the predicate.

TestRun(model, test_suite, test_config=None, custom_metrics_callback=None, reset=False) #

Bases: BaseTestRun

Interface to run tests for a Model on a TestSuite.

For a streamlined interface, see test.

Changes are committed to Kolena during execution and when the context is exited.

Parameters:

Name Type Description Default
model Model

The model being tested.

 required
test_suite TestSuite

The test suite on which to test the model.

 required
test_config Optional[TestConfig]

Optionally specify a configuration, e.g. FixedGlobalThreshold to customize the metrics evaluation logic for this test run. Defaults to F1Optimal with an iou_threshold of 0.5 if unspecified.

 None
custom_metrics_callback Optional[CustomMetricsCallback[_TestImageClass, _InferenceClass]]

Optionally specify a callback function to compute custom metrics for each test case. The callback would be passed inferences of images in each testcase and should return a dictionary with metric name as key and metric value as value.

 None
reset bool

Overwrites existing inferences if set.

 False

TestSuite(name, version=None, description=None, test_cases=None, reset=False) #

Bases: BaseTestSuite

A test suite groups together one or more test cases.

For additional functionality, see the associated base class documentation.

Parameters:

Name Type Description Default
name str

Name of the test suite to create or load.

 required
version Optional[int]

Optionally specify the version of the test suite to load. When absent, the latest version is loaded. Ignored when creating new test suites.

 None
description Optional[str]

Optionally specify a description for a newly created test suite. For existing test suites, this description can be edited via TestSuite.edit.

 None
test_cases Optional[List[TestCase]]

Optionally specify a list of test cases to populate a new test suite. For existing test suites, test cases can be edited via TestSuite.edit.

 None

description: str instance-attribute #

Free-form, human-readable description of the test suite. Can be edited at any time via TestSuite.edit.

name: str instance-attribute #

Unique name for this test suite.

test_cases: List[TestCase] instance-attribute #

The test cases within this test suite.

version: int instance-attribute #

The version of the test suite. This version is automatically incremented each time the suite is edited with TestSuite.edit.

iter_images(dataset=None) #

Deprecated: since 0.26.0

Please use TestCase.iter_images instead.

Return iterator over TestImage samples registered in Kolena. Images are lazily loaded in chunks to facilitate working with large datasets that are cumbersome to hold in memory.

Parameters:

Name Type Description Default
dataset Optional[str]

Optionally specify the single dataset to be retrieved. By default, images from all datasets are returned.

 None

load_images(dataset=None) #

Deprecated: since 0.26.0

Please use TestCase.load_images instead.

Load a list of TestImage samples registered in Kolena.

Parameters:

Name Type Description Default
dataset Optional[str]

Optionally specify the single dataset to be retrieved. By default, images from all datasets are returned.

 None

test(model, test_suite, test_config=None, custom_metrics_callback=None, reset=False) #

Test the provided InferenceModel on the provided TestSuite. Any tests already in progress for this model on these suites are resumed.

Parameters:

Name Type Description Default
model InferenceModel

The model to test.

 required
test_suite TestSuite

The test suite on which to test the model.

 required
test_config Optional[TestConfig]

Optionally specify a configuration, e.g. FixedGlobalThreshold, to customize metrics evaluation logic for this test run. Defaults to F1Optimal with an iou_threshold of 0.5 if unspecified.

 None
custom_metrics_callback Optional[CustomMetricsCallback[TestImage, Inference]]

Optionally specify a callback function to compute custom metrics for each test case. The callback would be passed inferences of images in each testcase and should return a dictionary with metric name as key and metric value as value.

 None
reset bool

Overwrites existing inferences if set.

 False

Ground Truth#

BoundingBox(label, top_left, bottom_right, difficult=False) #

Bases: GroundTruth

Ground truth data object representing a bounding box.

Point coordinates should be in (x, y) format, as absolute pixel values.

bottom_right: Tuple[float, float] = bottom_right instance-attribute #

Point in (x, y) pixel coordinates representing the bottom right corner of the bounding box.

difficult: bool = difficult instance-attribute #

A bounding box marked as difficult indicates that the object is considered difficult to recognize and should be ignored when evaluating metrics.

label: str = label instance-attribute #

Label to associate with this bounding box.

top_left: Tuple[float, float] = top_left instance-attribute #

Point in (x, y) pixel coordinates representing the top left corner of the bounding box.

GroundTruth() #

Bases: Serializable, Frozen

Base class for ground truths associated with an image.

See concrete implementations BoundingBox and SegmentationMask for details.

SegmentationMask(label, points, difficult=False) #

Bases: GroundTruth

Ground truth data object representing a detection mask.

Point coordinates should be in (x, y) format, as absolute pixel values.

Raises:

Type Description
ValueError

When fewer than three points are provided.

difficult: bool = difficult instance-attribute #

A segmentation mask marked as difficult indicates that the object is considered difficult to recognize and should be ignored when evaluating metrics.

label: str = label instance-attribute #

Label to associate with this segmentation mask.

points: List[Tuple[float, float]] = points instance-attribute #

Polygon corresponding to the vertices of the segmentation mask. Must have at least three distinct elements, and may not cross itself or touch itself at any point.

Inference#

BoundingBox(label, confidence, top_left, bottom_right) #

Bases: Inference

Inference representing a bounding box.

Point coordinates should be in (x, y) format, as absolute pixel values.

bottom_right: Tuple[float, float] = bottom_right instance-attribute #

Point in (x, y) pixel coordinates representing the bottom right corner of the bounding box.

confidence: float = confidence instance-attribute #

Confidence score associated with this inference.

label: str = label instance-attribute #

Label to associate with this bounding box.

top_left: Tuple[float, float] = top_left instance-attribute #

Point in (x, y) pixel coordinates representing the top left corner of the bounding box.

Inference #

Bases: _Inference

Base class for model inferences associated with an image.

See concrete implementations BoundingBox, and SegmentationMask for details.

SegmentationMask(label, confidence, points) #

Bases: Inference

Inference data object representing a segmentation mask.

Point coordinates should be in (x, y) format, as absolute pixel values.

Raises:

Type Description
ValueError

When fewer than three points are provided.

confidence: float = confidence instance-attribute #

Confidence score associated with this inference.

label: str = label instance-attribute #

Label to associate with this segmentation mask.

points: List[Tuple[float, float]] = points instance-attribute #

Polygon corresponding to the vertices of the segmentation mask. Must have at least three distinct elements, and may not cross itself or touch itself at any point.

Metadata#

Metadata associated with a TestImage.

from kolena.detection import TestImage
from kolena.detection.metadata import Landmarks, BoundingBox, Asset

test_image = TestImage("s3://bucket/path/to/image.png", metadata=dict(
    input_landmarks=Landmarks([(0,0), (10, 10), (20, 20), (30, 30), (40, 40)]),
    input_bounding_box=BoundingBox((0, 0), (100, 100)),
    image_grayscale=Asset("s3://bucket/path/to/image_grayscale.png"),
))

Annotation() #

Bases: Frozen, Serializable

An annotation associated with an image.

Annotations are surfaced during testing along with the image locator and any other metadata associated with an image. In the web platform, annotations are overlaid on top of images when visualizing results.

Asset(locator) #

Bases: Frozen, Serializable

An asset living in your shared bucket. Assets are surfaced during testing along with any other metadata associated with a given test image.

In the web platform, certain assets such as PNG and JPG images are viewable when visualizing results in the gallery.

locator: str = locator instance-attribute #

Location of this asset in shared bucket, e.g. s3://my-bucket/path/to/image.png.

BoundingBox(top_left, bottom_right) #

Bases: Annotation

An annotation comprising a bounding box around an object in an image.

bottom_right: Tuple[float, float] = bottom_right instance-attribute #

Point in (x, y) pixel coordinates representing the bottom right corner of the bounding box.

top_left: Tuple[float, float] = top_left instance-attribute #

Point in (x, y) pixel coordinates representing the top left corner of the bounding box.

Landmarks(points) #

Bases: Annotation

An annotation comprising an arbitrary-length set of landmarks corresponding to some object in an image, e.g. face landmarks used for pose estimation.

points: List[Tuple[float, float]] = points instance-attribute #

Any number of (x, y) points in pixel coordinates representing a set of landmarks.