📊 Metrics#

class text_machina.src.metrics.base.Metric(task_type, name)[source]#

Bases: ABC

Base class for metrics.

run(dataset, path, **kwargs)[source]#

Calls _run, _save and _log to compute a metric, store and log its outputs.

Parameters:

  • dataset (Dataset) – A labeled dataset on which to compute the metric.

  • path (Path) – A path where to save results.

  • **kwargs – Additional arguments to pass to _run.

Return type:

None

class text_machina.src.metrics.mauve.MAUVEMetric(task_type, name)[source]#

Bases: Metric

Implements the MAUVE metric: https://arxiv.org/abs/2102.01454

Supported tasks: detection.

check_kwargs_ok(kwargs)[source]#
Return type:

None

class text_machina.src.metrics.perplexity.PerplexityMetric(task_type, name)[source]#

Bases: Metric

Implements the perplexity metric.

Supported tasks: detection, attribution, and boundary.

class text_machina.src.metrics.repetition_diversity.RepetitionDiversityMetric(task_type, name)[source]#

Bases: Metric

Implements the repetition and diversity metrics. See Sec. 4.1.2: https://arxiv.org/pdf/2202.06417.pdf

Supported tasks: detection, attribution, and boundary.

repetition_and_diversity(text, ns)[source]#
Return type:

Dict

class text_machina.src.metrics.simple_model.SimpleModelMetric(task_type, name)[source]#

Bases: Metric

Implements simple baseline models evaluated with stratified k-fold validation.

Supported tasks: detection, attribution, and boundary.

text_machina.src.metrics.simple_model.regression_report(y_true, y_pred)[source]#

Computes a regression report similarly to the classification report offered by scikit-learn.

Parameters:

  • y_true (np.ndarray) – the true labels.

  • y_pred (np.ndarray) – the predicted labels.

Returns:

the report.

Return type:

pd.DataFrame

class text_machina.src.metrics.token_classification.TokenClassificationMetric(task_type, name)[source]#

Bases: Metric

Implements a HF token classification model for evaluating a mixcase dataset.

Supported tasks: boundary and mixcase.

text_machina.src.metrics.token_classification.eval(predictions, references, label_mapping)[source]#

Evaluates using seqeval from HF metrics.

Parameters:

  • predictions (List[List[int]]) – list of predictions for each example.

  • references (List[List[int]]) – list of gold labels for each example.

  • label_mapping (Dict[int, str]) – label mapping int to str labels.

Return type:

Dict[str, float]

Returns:

Dict[str, float] -> dictionary with metric values.

text_machina.src.metrics.token_classification.fit(model, dataset, tokenizer, training_args)[source]#

Fits a model on a dataset.

Parameters:

  • model (AutoModelForTokenClassification) – a model.

  • dataset (Dataset) – a training dataset.

  • tokenizer (AutoTokenizer) – a tokenizer.

  • training_args (Dict) – args to be passed to the HF’s Trainer.

Return type:

None

text_machina.src.metrics.token_classification.predict(model, dataset, tokenizer)[source]#

Predicts a dataset using a model.

Parameters:

  • model (AutoModelForTokenClassification) – a model.

  • dataset (Dataset) – a test dataset.

  • tokenizer (AutoTokenizer) – a tokenizer.

Returns:

list of predicted labels for each example.

Return type:

List[List[int]]

text_machina.src.metrics.token_classification.prepare_dataset(dataset, task_type, tokenizer, test_size, label_mapping)[source]#

Prepares the dataset (tokenization, prepare labels, splitting, etc.).

Parameters:

  • dataset (Dataset) – a dataset.

  • tokenizer (AutoTokenizer) – a tokenizer.

  • test_size (float) – proportion reserved for the test set.

  • label_mapping (Dict[str, int]) – label mapping str to int labels.

Returns:

a dataset with train and test splits.

Return type:

DatasetDict

text_machina.src.metrics.token_classification.prepare_tags_for_boundary(offset_mappings, labels, label_mapping)[source]#

Prepares the labels for boundary tasks to be addressed as token classification tasks.

This fn is designed to work with the map HF’s function using batched=True.

Parameters:

  • offset_mappings (List[List[List[int]]]) – offset mappings of each text.

  • labels (List[int) – labels of each text.

  • label_mapping (Dict[str, int]) – label mapping str to int labels.

Returns:

the tags for each text in the batch.

Return type:

Dict[str, List[List[int]]]

text_machina.src.metrics.token_classification.prepare_tags_for_mixcase(offset_mappings, labels, label_mapping)[source]#

Prepares the labels for mixcase tasks to be addressed as token classification tasks.

This fn is designed to work with the map HF’s function using batched=True.

Parameters:

  • offset_mappings (List[List[List[int]]]) – offset mappings of each text.

  • labels (List[List[Dict]]) – labels of each text.

  • label_mapping (Dict[str, int]) – label mapping str to int labels.

Returns:

the tags for each text in the batch.

Return type:

Dict[str, List[List[int]]]