quaterion.eval.group.retrieval_r_precision module

class RetrievalRPrecision(distance_metric_name: Distance = Distance.COSINE)

Bases: GroupMetric

Compute the retrieval R-precision score for group based data

Retrieval R-Precision is the ratio of r/R, where R is the number of the relevant documents for a given query in the collection, and r is the number of the truly relevant documents found in the R highest scored results for that query.

Parameters:

distance_metric_name – name of a distance metric to calculate distance or similarity matrices. Available names could be found in Distance.

Example

Suppose that a collection contains 20 relevant documents for our query, and the model can retrieve 15 of them in the 20 highest scored results, then Retrieval R-Precision is calculated as r/R = 15/20 = 0.75.

raw_compute(distance_matrix: Tensor, labels: Tensor) → Tensor

Perform metric computation on ready distance_matrix and labels

This method does not make any data and labels preparation. It is assumed that distance_matrix has already been calculated, required changes such masking distance from an element to itself have already been applied and corresponding labels have been prepared.

Parameters:

• distance_matrix – distance matrix ready to metric computation

• labels – labels ready to metric computation with the same shape as distance_matrix. Possible values are in {0, 1}.

Returns:

torch.Tensor - calculated metric value

retrieval_r_precision(distance_matrix: Tensor, labels: Tensor)

Calculates retrieval r precision given distance matrix and labels

Parameters:

• distance_matrix – distance matrix having max possible distance value on a diagonal

• labels – labels matrix having False or 0. on a diagonal

Returns:

torch.Tensor – mean retrieval r precision