kolena.workflow.TestCase

TestCase(name, version=None, description=None, test_samples=None, reset=False)

Bases: Frozen, WithTelemetry

A test case holds a list of test samples paired with ground truths representing a testing dataset or a slice of a testing dataset.

Rather than importing this class directly, use the TestCase type definition returned from define_workflow.

workflow: Workflow instance-attribute

The workflow of this test case. Automatically populated when constructing via test case type returned from define_workflow.

name: str instance-attribute

The unique name of this test case. Cannot be changed after creation.

version: int instance-attribute

The version of this test case. A test case's version is automatically incremented whenever it is edited via TestCase.edit.

description: str instance-attribute

Free-form, human-readable description of this test case. Can be edited at any time via TestCase.edit.

Editor(description, reset)

description(description)

Update the description of the test case.

add(test_sample, ground_truth)

Add a test sample to the test case. When the test sample already exists in the test case, its ground truth is overwritten with the ground truth provided here.

Parameters:

Name	Type	Description	Default
test_sample	TestSample	The test sample to add.	required
ground_truth	GroundTruth	The ground truth for the test sample.	required

remove(test_sample)

Remove a test sample from the test case. Does nothing if the test sample is not in the test case.

Parameters:

Name	Type	Description	Default
test_sample	TestSample	The test sample to remove.	required

create(name, description=None, test_samples=None) classmethod

Create a new test case with the provided name.

Parameters:

Name	Type	Description	Default
name	str	The name of the new test case to create.	required
description	Optional[str]	Optional free-form description of the test case to create.	None
test_samples	Optional[List[Tuple[TestSample, GroundTruth]]]	Optionally specify a set of test samples and ground truths to populate the test case.	None

Returns:

Type	Description
TestCase	The newly created test case.

load(name, version=None) classmethod

Load an existing test case with the provided name.

Parameters:

Name	Type	Description	Default
name	str	The name of the test case to load.	required
version	Optional[int]	Optionally specify a particular version of the test case to load. Defaults to the latest version when unset.	None

Returns:

Type	Description
TestCase	The loaded test case.

load_test_samples()

Load all TestSamples and GroundTruths contained in this test case.

Returns:

Type	Description
List[Tuple[TestSample, GroundTruth]]	A list of each test sample, paired with its ground truth, in this test case.

iter_test_samples()

Iterate through all TestSamples and GroundTruths contained in this test case.

Returns:

Type	Description
Iterator[Tuple[TestSample, GroundTruth]]	An iterator yielding each test sample, paired with its ground truth, in this test case.

edit(reset=False)

Edit this test case in a context:

with test_case.edit() as editor:
    # perform as many editing actions as desired
    editor.add(...)
    editor.remove(...)

Changes are committed to the Kolena platform when the context is exited.

Parameters:

Name	Type	Description	Default
reset	bool	Clear all existing test samples in the test case.	False

init_many(data, reset=False) classmethod

Experimental

This function is considered experimental, so beware that it is subject to changes even in patch releases.

Create, load or edit multiple test cases.

test_cases = TestCase.init_many([
    ("test-case 1", [(test_sample_1, ground_truth_1), ...]),
    ("test-case 2", [(test_sample_2, ground_truth_2), ...])
])

test_suite = TestSuite("my test suite", test_cases=test_cases)

Changes are committed to the Kolena platform together. If there is an error, none of the edits would take effect.

Parameters:

Name	Type	Description	Default
data	List[Tuple[str, List[Tuple[TestSample, GroundTruth]]]]	A list of tuples where each tuple is a test case name and a set of test samples and ground truths tuples for the test case.	required
reset	bool	If a test case of the same name already exists, overwrite with the provided test_samples.	False

Returns:

Type	Description
List[TestCase]	The test cases.