This pre-built workflow is an experimental feature. Experimental features are under active development and may occasionally undergo API-breaking changes.
Thresholded Metrics play a crucial role in specific contexts of model evaluation, particularly when the performance of models is assessed based on varying threshold values. In such scenarios, these metrics are vital for accurately interpreting the effectiveness of the models, as different thresholds can lead to markedly different performance outcomes.
Represents metrics tied to a specific threshold.
List[ThresholdedMetrics] should be used as a field type within
kolena.workflow module. This list is meant to hold metric values
associated with distinct thresholds. These metrics are expected to be uniform across
instances within a single test execution.
ThresholdedMetrics prohibits the use of dictionary objects as field values and guarantees that
the threshold values remain immutable once set. For application within a particular workflow,
subclassing is required to define relevant metrics fields.
from kolena.workflow import MetricsTestSample from kolena._experimental.workflow import ThresholdedMetrics @dataclass(frozen=True) class ClassThresholdedMetrics(ThresholdedMetrics): precision: float recall: float f1: float @dataclass(frozen=True) class TestSampleMetrics(MetricsTestSample): car: List[ClassThresholdedMetrics] pedestrian: List[ClassThresholdedMetrics] # Creating an instance of metrics metric = TestSampleMetrics( car=[ ClassThresholdedMetrics(threshold=0.3, precision=0.5, recall=0.8, f1=0.615), ClassThresholdedMetrics(threshold=0.4, precision=0.6, recall=0.6, f1=0.6), ClassThresholdedMetrics(threshold=0.5, precision=0.8, recall=0.4, f1=0.533), # ... ], pedestrian=[ ClassThresholdedMetrics(threshold=0.3, precision=0.6, recall=0.9, f1=0.72), ClassThresholdedMetrics(threshold=0.4, precision=0.7, recall=0.7, f1=0.7), ClassThresholdedMetrics(threshold=0.5, precision=0.8, recall=0.6, f1=0.686), # ... ], )
Raises: TypeError: If any of the field values is a dictionary.