Shortcuts

quaterion.eval.base_metric module

class BaseMetric(distance_metric_name: Distance = Distance.COSINE)[source]

Bases: object

Base class for evaluation metrics

Provides a default implementation for distance matrix calculation.

Parameters:

distance_metric_name – name of a distance metric to calculate distance or similarity matrices. Available names could be found in Distance.

compute(*args, **kwargs) Tensor[source]

Compute metric value

Parameters:
  • args

  • metric. (kwargs - contain embeddings and targets required to compute) –

Returns:

torch.Tensor - computed metric

evaluate() Tensor[source]

Perform metric computation with accumulated state

precompute(embeddings: Tensor, **targets) Tuple[Tensor, Tensor][source]

Prepares data for computation

Compute distance matrix and final labels based on groups.

Parameters:
  • embeddings – embeddings to compute metric value

  • targets – objects to compute final labels

Returns:

torch.Tensor, torch.Tensor - labels and distance matrix

static prepare_labels(**targets) Tensor[source]

Compute metric labels

Parameters:

**targets – objects to compute final labels. **targets in PairMetric consists of labels, pairs and subgroups, in GroupMetric - of groups.

Returns:

targets – torch.Tensor - labels to be used during metric computation

raw_compute(distance_matrix: Tensor, labels: Tensor) Tensor[source]

Perform metric computation on ready distance_matrix and labels

This method does not make any data and labels preparation. It is assumed that distance_matrix has already been calculated, required changes such masking distance from an element to itself have already been applied and corresponding labels have been prepared.

Parameters:
  • distance_matrix – distance matrix ready to metric computation

  • labels – labels ready to metric computation with the same shape as distance_matrix. For PairMetric values are taken from SimilarityPairSample.score, for GroupMetric the possible values are in {0, 1}.

Returns:

torch.Tensor - calculated metric value