kolena._experimental.workflow.ThresholdedMetrics

Experimental Feature

This pre-built workflow is an experimental feature. Experimental features are under active development and may occasionally undergo API-breaking changes.

Thresholded Metrics play a crucial role in specific contexts of model evaluation, particularly when the performance of models is assessed based on varying threshold values. In such scenarios, these metrics are vital for accurately interpreting the effectiveness of the models, as different thresholds can lead to markedly different performance outcomes.

ThresholdedMetrics(threshold) dataclass

Bases: TypedDataObject[_MetricsType]

Represents metrics tied to a specific threshold.

List[ThresholdedMetrics] should be used as a field type within MetricsTestSample from the kolena.workflow module. This list is meant to hold metric values associated with distinct thresholds. These metrics are expected to be uniform across TestSample instances within a single test execution.

ThresholdedMetrics prohibits the use of dictionary objects as field values and guarantees that the threshold values remain immutable once set. For application within a particular workflow, subclassing is required to define relevant metrics fields.

Usage example:

from kolena.workflow import MetricsTestSample
from kolena._experimental.workflow import ThresholdedMetrics

@dataclass(frozen=True)
class ClassThresholdedMetrics(ThresholdedMetrics):
    precision: float
    recall: float
    f1: float

@dataclass(frozen=True)
class TestSampleMetrics(MetricsTestSample):
    car: List[ClassThresholdedMetrics]
    pedestrian: List[ClassThresholdedMetrics]

# Creating an instance of metrics
metric = TestSampleMetrics(
    car=[
        ClassThresholdedMetrics(threshold=0.3, precision=0.5, recall=0.8, f1=0.615),
        ClassThresholdedMetrics(threshold=0.4, precision=0.6, recall=0.6, f1=0.6),
        ClassThresholdedMetrics(threshold=0.5, precision=0.8, recall=0.4, f1=0.533),
        # ...
    ],
    pedestrian=[
        ClassThresholdedMetrics(threshold=0.3, precision=0.6, recall=0.9, f1=0.72),
        ClassThresholdedMetrics(threshold=0.4, precision=0.7, recall=0.7, f1=0.7),
        ClassThresholdedMetrics(threshold=0.5, precision=0.8, recall=0.6, f1=0.686),
        # ...
    ],
)

Raises: TypeError: If any of the field values is a dictionary.