aac_metrics.functional.fer module

class BERTFlatClassifier(
model_type: str,
num_classes: int = 5,
)[source]

Bases: Module

forward(
input_ids: Tensor | None = None,
attention_mask: Tensor | None = None,
token_type_ids: Tensor | None = None,
**kwargs,
) → Tensor[source]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

classmethod from_pretrained(
model_name: str = 'echecker_clotho_audiocaps_base',
device: str | device | None = 'cuda_if_available',
use_proxy: bool = False,
proxies: dict[str, str] | None = None,
verbose: int = 0,
) → BERTFlatClassifier[source]
class FERScores

Bases: dict

fer: Tensor
fer(
candidates: list[str],
return_all_scores: bool = True,
*,
echecker: str | BERTFlatClassifier = 'echecker_clotho_audiocaps_base',
echecker_tokenizer: AutoTokenizer | None = None,
error_threshold: float = 0.9,
device: str | device | None = 'cuda_if_available',
batch_size: int | None = 32,
reset_state: bool = True,
return_probs: bool = False,
verbose: int = 0,
) → tuple[FERScores, FERScores] | Tensor[source]

Return Fluency Error Rate (FER) detected by a pre-trained BERT model.

Parameters:

  • candidates – The list of sentences to evaluate.

  • mult_references – The list of list of sentences used as target.

  • return_all_scores – If True, returns a tuple containing the globals and locals scores. Otherwise returns a scalar tensor containing the main global score. defaults to True.

  • echecker – The echecker model used to detect fluency errors. Can be “echecker_clotho_audiocaps_base”, “echecker_clotho_audiocaps_tiny”, “none” or None. defaults to “echecker_clotho_audiocaps_base”.

  • echecker_tokenizer – The tokenizer of the echecker model. If None and echecker is not None, this value will be inferred with echecker.model_type. defaults to None.

  • error_threshold – The threshold used to detect fluency errors for echecker model. defaults to 0.9.

  • device – The PyTorch device used to run pre-trained models. If “cuda_if_available”, it will use cuda if available. defaults to “cuda_if_available”.

  • batch_size – The batch size of the echecker models. defaults to 32.

  • reset_state – If True, reset the state of the PyTorch global generator after the initialization of the pre-trained models. defaults to True.

  • return_probs – If True, return each individual error probability given by the fluency detector model. defaults to False.

  • verbose – The verbose level. defaults to 0.

Returns:

A tuple of globals and locals scores or a scalar tensor with the main global score.