quaterion.eval.base_metric module

class BaseMetric(distance_metric_name: Distance = Distance.COSINE)[source]

Bases: object

Base class for evaluation metrics

Provides a default implementation for distance matrix calculation.


distance_metric_name – name of a distance metric to calculate distance or similarity matrices. Available names could be found in Distance.

compute(*args, **kwargs) Tensor[source]

Compute metric value

  • args

  • metric. (kwargs - contain embeddings and targets required to compute) –


torch.Tensor - computed metric

evaluate() Tensor[source]

Perform metric computation with accumulated state

precompute(embeddings: Tensor, **targets) Tuple[Tensor, Tensor][source]

Prepares data for computation

Compute distance matrix and final labels based on groups.

  • embeddings – embeddings to compute metric value

  • targets – objects to compute final labels


torch.Tensor, torch.Tensor - labels and distance matrix

static prepare_labels(**targets) Tensor[source]

Compute metric labels


**targets – objects to compute final labels. **targets in PairMetric consists of labels, pairs and subgroups, in GroupMetric - of groups.


targets – torch.Tensor - labels to be used during metric computation

raw_compute(distance_matrix: Tensor, labels: Tensor) Tensor[source]

Perform metric computation on ready distance_matrix and labels

This method does not make any data and labels preparation. It is assumed that distance_matrix has already been calculated, required changes such masking distance from an element to itself have already been applied and corresponding labels have been prepared.

  • distance_matrix – distance matrix ready to metric computation

  • labels – labels ready to metric computation with the same shape as distance_matrix. For PairMetric values are taken from SimilarityPairSample.score, for GroupMetric the possible values are in {0, 1}.


torch.Tensor - calculated metric value


Learn more about Qdrant vector search project and ecosystem

Discover Qdrant

Similarity Learning

Explore practical problem solving with Similarity Learning

Learn Similarity Learning


Find people dealing with similar problems and get answers to your questions

Join Community