aac_metrics.functional.fense module

class FENSEScores

Bases: dict

fense : Tensor
fer : Tensor
sbert_sim : Tensor
fense(
candidates: list[str],
mult_references: list[list[str]],
return_all_scores: bool = True,
*,
sbert_model: str | SentenceTransformer = 'paraphrase-TinyBERT-L6-v2',
echecker: str | BERTFlatClassifier = 'echecker_clotho_audiocaps_base',
echecker_tokenizer: AutoTokenizer | None = None,
error_threshold: float = 0.9,
device: str | device | None = 'cuda_if_available',
batch_size: int | None = 32,
reset_state: bool = True,
return_probs: bool = False,
penalty: float = 0.9,
verbose: int = 0,
) → tuple[FENSEScores, FENSEScores] | Tensor[source]

Fluency ENhanced Sentence-bert Evaluation (FENSE)

Parameters:
candidates: list[str]

The list of sentences to evaluate.

mult_references: list[list[str]]

The list of list of sentences used as target.

return_all_scores: bool = True

If True, returns a tuple containing the globals and locals scores. Otherwise returns a scalar tensor containing the main global score. defaults to True.

sbert_model: str | SentenceTransformer = 'paraphrase-TinyBERT-L6-v2'

The sentence BERT model used to extract sentence embeddings for cosine-similarity. defaults to “paraphrase-TinyBERT-L6-v2”.

echecker: str | BERTFlatClassifier = 'echecker_clotho_audiocaps_base'

The echecker model used to detect fluency errors. Can be “echecker_clotho_audiocaps_base”, “echecker_clotho_audiocaps_tiny”, “none” or None. defaults to “echecker_clotho_audiocaps_base”.

echecker_tokenizer: AutoTokenizer | None = None

The tokenizer of the echecker model. If None and echecker is not None, this value will be inferred with echecker.model_type. defaults to None.

error_threshold: float = 0.9

The threshold used to detect fluency errors for echecker model. defaults to 0.9.

penalty: float = 0.9

The penalty coefficient applied. Higher value means to lower the cos-sim scores when an error is detected. defaults to 0.9.

device: str | device | None = 'cuda_if_available'

The PyTorch device used to run pre-trained models. If “cuda_if_available”, it will use cuda if available. defaults to “cuda_if_available”.

batch_size: int | None = 32

The batch size of the sBERT and echecker models. defaults to 32.

reset_state: bool = True

If True, reset the state of the PyTorch global generator after the initialization of the pre-trained models. defaults to True.

return_probs: bool = False

If True, return each individual error probability given by the fluency detector model. defaults to False.

verbose: int = 0

The verbose level. defaults to 0.

Returns:

A tuple of globals and locals scores or a scalar tensor with the main global score.