kolena.workflow.test

TestRun(model, test_suite, evaluator=None, configurations=None, reset=False)

Bases: Frozen, WithTelemetry

A Model tested on a TestSuite using a specific Evaluator implementation.

Parameters:

Name	Type	Description	Default
model	Model	The model being tested.	required
test_suite	TestSuite	The test suite on which to test the model.	required
evaluator	Optional[Union[Evaluator, BasicEvaluatorFunction]]	An optional evaluator implementation. Requires a previously configured server-side evaluator to default to if omitted. (Please see BasicEvaluatorFunction for type definition.)	None
configurations	Optional[List[EvaluatorConfiguration]]	a list of configurations to use when running the evaluator.	None
reset	bool	overwrites existing inferences if set.	False

run()

Run the testing process, first extracting inferences for all test samples in the test suite then performing evaluation.

load_test_samples()

Load the test samples in the test suite that do not yet have inferences uploaded.

Returns:

Type	Description
List[TestSample]	a list of all test samples in the test suite still requiring inferences.

iter_test_samples()

Iterate through the test samples in the test suite that do not yet have inferences uploaded.

Returns:

Type	Description
Iterator[TestSample]	an iterator over each test sample still requiring inferences.

upload_inferences(inferences)

Upload inferences from a model.

Parameters:

Name	Type	Description	Default
inferences	List[Tuple[TestSample, Inference]]	the inferences, paired with their corresponding test samples, to upload.	required

evaluate()

Perform evaluation by computing metrics for individual test samples, in aggregate across test cases, and across the complete test suite at each EvaluatorConfiguration.

test(model, test_suite, evaluator=None, configurations=None, reset=False)

Test a Model on a TestSuite using a specific Evaluator implementation.

from kolena.workflow import test

test(model, test_suite, evaluator, reset=True)

Parameters:

Name	Type	Description	Default
model	Model	The model being tested, implementing the infer method.	required
test_suite	TestSuite	The test suite on which to test the model.	required
evaluator	Optional[Union[Evaluator, BasicEvaluatorFunction]]	An optional evaluator implementation. Requires a previously configured server-side evaluator to default to if omitted. (Please see BasicEvaluatorFunction for type definition.)	None
configurations	Optional[List[EvaluatorConfiguration]]	A list of configurations to use when running the evaluator.	None
reset	bool	Overwrites existing inferences if set.	False