aac_metrics.functional.fense module

class FENSEScores

Bases: dict

fense: Tensor
fer: Tensor
sbert_sim: Tensor
fense(
candidates: list[str],
mult_references: list[list[str]],
return_all_scores: bool = True,
*,
sbert_model: str | SentenceTransformer = 'paraphrase-TinyBERT-L6-v2',
echecker: str | BERTFlatClassifier = 'echecker_clotho_audiocaps_base',
echecker_tokenizer: AutoTokenizer | None = None,
error_threshold: float = 0.9,
device: str | device | None = 'cuda_if_available',
batch_size: int | None = 32,
reset_state: bool = True,
return_probs: bool = False,
penalty: float = 0.9,
verbose: int = 0,
) → tuple[FENSEScores, FENSEScores] | Tensor[source]

Fluency ENhanced Sentence-bert Evaluation (FENSE)

Parameters:

  • candidates – The list of sentences to evaluate.

  • mult_references – The list of list of sentences used as target.

  • return_all_scores – If True, returns a tuple containing the globals and locals scores. Otherwise returns a scalar tensor containing the main global score. defaults to True.

  • sbert_model – The sentence BERT model used to extract sentence embeddings for cosine-similarity. defaults to “paraphrase-TinyBERT-L6-v2”.

  • echecker – The echecker model used to detect fluency errors. Can be “echecker_clotho_audiocaps_base”, “echecker_clotho_audiocaps_tiny”, “none” or None. defaults to “echecker_clotho_audiocaps_base”.

  • echecker_tokenizer – The tokenizer of the echecker model. If None and echecker is not None, this value will be inferred with echecker.model_type. defaults to None.

  • error_threshold – The threshold used to detect fluency errors for echecker model. defaults to 0.9.

  • penalty – The penalty coefficient applied. Higher value means to lower the cos-sim scores when an error is detected. defaults to 0.9.

  • device – The PyTorch device used to run pre-trained models. If “cuda_if_available”, it will use cuda if available. defaults to “cuda_if_available”.

  • batch_size – The batch size of the sBERT and echecker models. defaults to 32.

  • reset_state – If True, reset the state of the PyTorch global generator after the initialization of the pre-trained models. defaults to True.

  • return_probs – If True, return each individual error probability given by the fluency detector model. defaults to False.

  • verbose – The verbose level. defaults to 0.

Returns:

A tuple of globals and locals scores or a scalar tensor with the main global score.