performance_eval

class aitoolbox.torchtrain.callbacks.performance_eval.ModelPerformanceEvaluation(result_package, args, on_each_epoch=True, on_train_data=False, on_val_data=True, eval_frequency=None, if_available_output_to_project_dir=True)

Bases: AbstractCallback

Track performance metrics from result_package and store them into TrainLoop’s history

This callback is different from those for model and experiment saving where performance evaluations are also calculated. Here we only want to calculate performance and store it in memory into TrainLoop’s history dict.

It is a more lightweight, on the go performance tracking without the need for the full project folder structure construction.

Parameters:

• result_package (AbstractResultPackage) – result package to be evaluated

• args (dict) – used hyper-parameters

• on_each_epoch (bool) – calculate performance results just at the end of training or at the end of each epoch

• on_train_data (bool) – should the evaluation be done on the training dataset

• on_val_data (bool) – should the evaluation be done on the validation dataset

• eval_frequency (int or None) – evaluation is done every specified number of epochs. Useful when predictions are quite expensive and are slowing down the overall training

• if_available_output_to_project_dir (bool) – if using train loop version which builds project local folder structure for saving checkpoints or creation of end of training reports, by setting if_available_output_to_project_dir to True the potential additional metadata result outputs from the result_package will be saved in the folder inside the main project folder. In this case the result_package’s output folder shouldn’t be full path but just the folder name and the full folder path pointing inside the corresponding project folder will be automatically created. If such a functionality should to be prevented and manual full additional metadata results dump folder is needed potentially outside the project folder, then set this argument to False and specify a full folder path.

on_train_end()

Logic executed at the end of the overall training

Returns:

None

on_epoch_end()

Logic executed at the end of the epoch

Returns:

None

evaluate_model_performance(prefix='')

Calculate performance based on the provided result packages

Parameters:

prefix (str) – additional prefix for metric names that will get saved into the training history

Returns:

None

store_evaluated_metrics_to_history(prefix='')

Save the calculated performance results into the training history

The performance results are saved into the training history after they are calculated by the before called evaluate_model_performance() function.

Parameters:

prefix (str) – additional prefix for metric names that will get saved into the training history

Returns:

None

on_train_loop_registration()

Execute callback initialization / preparation after the train_loop_object becomes available

Returns:

None

class aitoolbox.torchtrain.callbacks.performance_eval.ModelPerformancePrintReport(metrics, on_each_epoch=True, report_frequency=None, strict_metric_reporting=True, list_tracked_metrics=False)

Bases: AbstractCallback

Print the model performance to the console

Best used in combination with the callback which actually calculates some performance evaluation metrics, such as ModelPerformanceEvaluation. Otherwise, we are limited only to automatic loss calculation reporting.

When listing callbacks for the TrainLoop it is important to list the ModelPerformanceEvaluation before this ModelPerformancePrintReport. This ensures that the calculated results are present in the TrainLoop.train_history before there is an attempt to print them.

Parameters:

• metrics (list) – list of string metric names which should be presented in the printed report

• on_each_epoch (bool) – present results just at the end of training or at the end of each epoch

• report_frequency (int or None) – evaluation is done every specified number of epochs. Useful when predictions are quite expensive and are slowing down the overall training

• strict_metric_reporting (bool) – if False ignore missing metric in the TrainLoop.train_history, if True, in case of missing metric throw and exception and thus interrupt the training loop

• list_tracked_metrics (bool) – should all tracked metrics names be listed

on_train_end()

Logic executed at the end of the overall training

Returns:

None

on_epoch_end()

Logic executed at the end of the epoch

Returns:

None

print_performance_report(prefix='')

Print the model performance

Parameters:

prefix (str) – additional prefix for metric names that will get saved into the training history

Returns:

None

class aitoolbox.torchtrain.callbacks.performance_eval.TrainHistoryFormatter(input_metric_getter, output_metric_setter, epoch_end=True, train_end=False, strict_metric_extract=True)

Bases: AbstractCallback

Format stored training history results

Parameters:

• input_metric_getter (lambda) – extract full history for the desired metric, not just the last history input. Return should be represented as a list.

• output_metric_setter (lambda) – take the extracted full history of a metric and convert it as desired. Return new / transformed metric name and transformed metric result.

• epoch_end (bool) – should the formatting be executed at the end of the epoch

• train_end (bool) – should the formatting be executed at the end of the training process

• strict_metric_extract (bool) – in case of (quality) problems should exception be raised on just the notification printed to console

on_epoch_end()

Logic executed at the end of the epoch

Returns:

None

on_train_end()

Logic executed at the end of the overall training

Returns:

None

format_history()

check_if_history_updated(train_end_phase)

class aitoolbox.torchtrain.callbacks.performance_eval.MetricHistoryRename(input_metric_path, new_metric_name, epoch_end=True, train_end=False, strict_metric_extract=True)

Bases: TrainHistoryFormatter

Specific interface for TrainHistoryFormatter which renames the metric in the training history

Parameters:

• input_metric_path (str or lambda) – if using lambda, extract full history for the desired metric, not just the last history input. Return should be represented as a list.

• new_metric_name (str) – the new metric name

• epoch_end (bool) – should the formatting be executed at the end of the epoch

• train_end (bool) – should the formatting be executed at the end of the training process

• strict_metric_extract (bool) – in case of (quality) problems should exception be raised on just the notification printed to console

class aitoolbox.torchtrain.callbacks.performance_eval.ModelTrainHistoryBaseCB(callback_name, execution_order=0, epoch_end=True, train_end=False, file_format='', project_name=None, experiment_name=None, local_model_result_folder_path=None, cloud_save_mode=None, bucket_name=None, cloud_dir_prefix=None)

Bases: AbstractExperimentCallback

Base callback class to be inherited from when reporting train performance history

Parameters:

• callback_name (str) – name of the callback

• execution_order (int) – order of the callback execution. If all the used callbacks have the orders set to 0, then the callbacks are executed in the order they were registered.

• epoch_end (bool) – should plot after every epoch

• train_end (bool) – should plot at the end of the training

• file_format (str) – output file format

• project_name (str or None) – root name of the project

• experiment_name (str or None) – name of the particular experiment

• local_model_result_folder_path (str or None) – root local path where project folder will be created

• cloud_save_mode (str or None) – Storage destination selector. For AWS S3: ‘s3’ / ‘aws_s3’ / ‘aws’ For Google Cloud Storage: ‘gcs’ / ‘google_storage’ / ‘google storage’ Everything else results just in local storage to disk

• bucket_name (str) – name of the bucket in the cloud storage

• cloud_dir_prefix (str) – path to the folder inside the bucket where the experiments are going to be saved

prepare_results_saver()

Initialize the required results saver

Returns:

None

class aitoolbox.torchtrain.callbacks.performance_eval.ModelTrainHistoryPlot(epoch_end=True, train_end=False, file_format='png', project_name=None, experiment_name=None, local_model_result_folder_path=None, cloud_save_mode=None, bucket_name=None, cloud_dir_prefix=None)

Bases: ModelTrainHistoryBaseCB

Plot the evaluated performance metric history

Parameters:

• epoch_end (bool) – should plot after every epoch

• train_end (bool) – should plot at the end of the training

• file_format (str) – output file format. Can be either ‘png’ for saving separate images or ‘pdf’ for combining all the plots into a single pdf file.

• project_name (str or None) – root name of the project

• experiment_name (str or None) – name of the particular experiment

• local_model_result_folder_path (str or None) – root local path where project folder will be created

• cloud_save_mode (str or None) – Storage destination selector. For AWS S3: ‘s3’ / ‘aws_s3’ / ‘aws’ For Google Cloud Storage: ‘gcs’ / ‘google_storage’ / ‘google storage’ Everything else results just in local storage to disk

• bucket_name (str) – name of the bucket in the cloud storage

• cloud_dir_prefix (str) – path to the folder inside the bucket where the experiments are going to be saved

on_train_loop_registration()

Execute callback initialization / preparation after the train_loop_object becomes available

Returns:

None

on_epoch_end()

Logic executed at the end of the epoch

Returns:

None

on_train_end()

Logic executed at the end of the overall training

Returns:

None

plot_current_train_history(prefix='')

Plot current training history snapshot in the encapsulating TrainLoop

Parameters:

prefix (str) – plots folder name prefix

Returns:

None

class aitoolbox.torchtrain.callbacks.performance_eval.ModelTrainHistoryFileWriter(epoch_end=True, train_end=False, file_format='txt', project_name=None, experiment_name=None, local_model_result_folder_path=None, cloud_save_mode=None, bucket_name=None, cloud_dir_prefix=None)

Bases: ModelTrainHistoryBaseCB

Write evaluated performance metric history to the text file

Parameters:

• epoch_end (bool) – should plot after every epoch

• train_end (bool) – should plot at the end of the training

• file_format (str) – output file format. Can be either ‘txt’ human-readable output or ‘tsv’ for a tabular format or ‘csv’ for comma separated format.

• project_name (str or None) – root name of the project

• experiment_name (str or None) – name of the particular experiment

• local_model_result_folder_path (str or None) – root local path where project folder will be created

• cloud_save_mode (str or None) – Storage destination selector. For AWS S3: ‘s3’ / ‘aws_s3’ / ‘aws’ For Google Cloud Storage: ‘gcs’ / ‘google_storage’ / ‘google storage’ Everything else results just in local storage to disk

• bucket_name (str) – name of the bucket in the cloud storage

• cloud_dir_prefix (str) – path to the folder inside the bucket where the experiments are going to be saved

on_train_loop_registration()

Execute callback initialization / preparation after the train_loop_object becomes available

Returns:

None

on_epoch_end()

Logic executed at the end of the epoch

Returns:

None

on_train_end()

Logic executed at the end of the overall training

Returns:

None

write_current_train_history(prefix='')

Write to text file the current training history snapshot in the encapsulating TrainLoop

Parameters:

prefix (str) – history text file name prefix

Returns:

None