local_model_save

class aitoolbox.experiment.local_save.local_model_save.AbstractLocalModelSaver

Bases: ABC

abstract save_model(model, project_name, experiment_name, experiment_timestamp=None, epoch=None, iteration_idx=None, protect_existing_folder=True)

Model saving method which all the model savers have to implement to give an expected API to other components

Parameters:

• model (keras.Model or dict) – model representation. If used with PyTorch it is a simple dict under the hood. In the case of Keras training this would be the keras Model.

• project_name (str) – root name of the project

• experiment_name (str) – name of the particular experiment

• experiment_timestamp (str or None) – time stamp at the start of training

• epoch (int or None) – in which epoch the model is being saved

• iteration_idx (int or None) – at which training iteration the model is being saved

• protect_existing_folder (bool) – can override potentially already existing folder or not

Returns:

model_name, model_local_path

Return type:

(str, str)

class aitoolbox.experiment.local_save.local_model_save.BaseLocalModelSaver(local_model_result_folder_path='~/project/model_result', checkpoint_model=False)

Bases: object

Base functionality for all the local model savers

Parameters:

• local_model_result_folder_path (str) – root local path where project folder will be created

• checkpoint_model (bool) – if the model is coming from the mid-training checkpoint

create_experiment_local_models_folder(project_name, experiment_name, experiment_timestamp)

Creates experiment local folder hierarchy and place the ‘models’ folder in it

Parameters:

• project_name (str) – root name of the project

• experiment_name (str) – name of the particular experiment

• experiment_timestamp (str) – time stamp at the start of training

Returns:

path to the created models folder in the experiment base folder

Return type:

str

class aitoolbox.experiment.local_save.local_model_save.PyTorchLocalModelSaver(local_model_result_folder_path='~/project/model_result', checkpoint_model=False)

Bases: AbstractLocalModelSaver, BaseLocalModelSaver

PyTorch experiment local model saver

Parameters:

• local_model_result_folder_path (str) – root local path where project folder will be created

• checkpoint_model (bool) – if the model is coming from the mid-training checkpoint

save_model(model, project_name, experiment_name, experiment_timestamp=None, epoch=None, iteration_idx=None, protect_existing_folder=True)

Save the PyTorch model representation dict to the local drive

Parameters:

• model (dict) – PyTorch model represented as a dict of weights, optimizer state and other necessary info.

• project_name (str) – root name of the project

• experiment_name (str) – name of the particular experiment

• experiment_timestamp (str or None) – time stamp at the start of training

• epoch (int or None) – in which epoch the model is being saved

• iteration_idx (int or None) – at which training iteration the model is being saved

• protect_existing_folder (bool) – can override potentially already existing folder or not

Returns:

model_name, model_local_path

Return type:

(str, str)

static check_model_dict_contents(model)

Check if PyTorch model save dict contains all the necessary elements for the training state reconstruction

Parameters:

model (dict) – PyTorch model represented as a dict of weights, optimizer state and other necessary info.

Raises:

ValueError –

Returns:

None

class aitoolbox.experiment.local_save.local_model_save.KerasLocalModelSaver(local_model_result_folder_path='~/project/model_result', checkpoint_model=False)

Bases: AbstractLocalModelSaver, BaseLocalModelSaver

Keras experiment local model saver

Parameters:

• local_model_result_folder_path (str) – root local path where project folder will be created

• checkpoint_model (bool) – if the model is coming from the mid-training checkpoint

save_model(model, project_name, experiment_name, experiment_timestamp=None, epoch=None, iteration_idx=None, protect_existing_folder=True)

Save the Keras model to the local drive

Parameters:

• model (keras.Model) – Keras model

• project_name (str) – root name of the project

• experiment_name (str) – name of the particular experiment

• experiment_timestamp (str or None) – time stamp at the start of training

• epoch (int or None) – in which epoch the model is being saved

• iteration_idx (int or None) – at which training iteration the model is being saved

• protect_existing_folder (bool) – can override potentially already existing folder or not

Returns:

model_name, model_local_path

Return type:

(str, str)

class aitoolbox.experiment.local_save.local_model_save.LocalSubOptimalModelRemover(metric_name, num_best_kept=2)

Bases: object

Removes the tracked saved models which become suboptimal when new models are trained in subsequent epochs

Useful when interested in saving the limited local disk space, especially when dealing with large model which take a lot of disk space.

Parameters:

• metric_name (str) – one of the metric names that will be calculated and will appear in the train_history dict in the TrainLoop

• num_best_kept (int) – number of best performing models which are kept when removing suboptimal model checkpoints

decide_if_remove_suboptimal_model(history, new_model_dump_paths)

Make decision if suboptimal model should be removed due to the introduction of the new and better model

Parameters:

• history (aitoolbox.experiment.training_history.TrainingHistory) – training performance history

• new_model_dump_paths (list) – new saved models paths which will begin to be tracked

Returns:

None

static rm_suboptimal_model(rm_model_paths)

Utility to remove the file

Parameters:

rm_model_paths (list) – list of string paths

Returns:

None