quaterion.eval.base_metric module¶

class BaseMetric(distance_metric_name: Distance = Distance.COSINE)[source]¶

Bases: object

Base class for evaluation metrics

Provides a default implementation for distance matrix calculation.

Parameters:

distance_metric_name – name of a distance metric to calculate distance or similarity matrices. Available names could be found in Distance.

compute(*args, **kwargs) → Tensor[source]¶

Compute metric value

Parameters:

• args –

• metric. (kwargs - contain embeddings and targets required to compute) –

Returns:

torch.Tensor - computed metric

evaluate() → Tensor[source]¶

Perform metric computation with accumulated state

precompute(embeddings: Tensor, **targets) → Tuple[Tensor, Tensor][source]¶

Prepares data for computation

Compute distance matrix and final labels based on groups.

Parameters:

• embeddings – embeddings to compute metric value

• targets – objects to compute final labels

Returns:

torch.Tensor, torch.Tensor - labels and distance matrix

static prepare_labels(**targets) → Tensor[source]¶

Compute metric labels

Parameters:

**targets – objects to compute final labels. **targets in PairMetric consists of labels, pairs and subgroups, in GroupMetric - of groups.

Returns:

targets – torch.Tensor - labels to be used during metric computation

raw_compute(distance_matrix: Tensor, labels: Tensor) → Tensor[source]¶

Perform metric computation on ready distance_matrix and labels

This method does not make any data and labels preparation. It is assumed that distance_matrix has already been calculated, required changes such masking distance from an element to itself have already been applied and corresponding labels have been prepared.

Parameters:

• distance_matrix – distance matrix ready to metric computation

• labels – labels ready to metric computation with the same shape as distance_matrix. For PairMetric values are taken from SimilarityPairSample.score, for GroupMetric the possible values are in {0, 1}.

Returns:

torch.Tensor - calculated metric value