hf_evaluate_packages
- class aitoolbox.experiment.result_package.hf_evaluate_packages.HFEvaluateResultPackage(hf_evaluate_metric, use_models_additional_results=True, **kwargs)[source]
Bases:
AbstractResultPackage
HuggingFace Evaluate Metrics Result Package
Result package wrapping around the evaluation metrics provided in the HuggingFace Evaluate package.
All the metric result names will have the ‘_HFEvaluate’ appended at the end to help distinguish them.
Github: https://github.com/huggingface/evaluate
More info on how to use the metrics: https://huggingface.co/docs/evaluate/index
- Parameters:
hf_evaluate_metric (evaluate.EvaluationModule) – HF Evaluate metric to be used by the result package
use_models_additional_results (bool) – Should the additional results from the model (in addition to predictions and references) normally returned from the get_predictions() function be added as the additional input to the HF Evaluate metric to perform the evaluation calculation.
**kwargs – additional parameters or inputs to the HF Evaluate metric being calculated. These can be generally inputs available already at the start before making model predictions and thus don’t need to be gathered from the train/prediction loop.
- prepare_results_dict()[source]
Perform result package building
Mostly this consists of executing calculation of selected performance metrics and returning their result dicts. If you want to use multiple performance metrics you have to combine them in the single self.results_dict at the end by doing this:
return {**metric_dict_1, **metric_dict_2}
- Returns:
calculated result dict
- Return type: