local_model_save

class aitoolbox.experiment.local_save.local_model_save.AbstractLocalModelSaver[source]

Bases: abc.ABC

abstract save_model(model, project_name, experiment_name, experiment_timestamp=None, epoch=None, iteration_idx=None, protect_existing_folder=True)[source]

Model saving method which all the model savers have to implement to give an expected API to other components

Parameters
  • model (keras.Model or dict) – model representation. If used with PyTorch it is a simple dict under the hood. In the case of Keras training this would be the keras Model.

  • project_name (str) – root name of the project

  • experiment_name (str) – name of the particular experiment

  • experiment_timestamp (str or None) – time stamp at the start of training

  • epoch (int or None) – in which epoch the model is being saved

  • iteration_idx (int or None) – at which training iteration the model is being saved

  • protect_existing_folder (bool) – can override potentially already existing folder or not

Returns

model_name, model_local_path

Return type

(str, str)

class aitoolbox.experiment.local_save.local_model_save.BaseLocalModelSaver(local_model_result_folder_path='~/project/model_result', checkpoint_model=False)[source]

Bases: object

Base functionality for all the local model savers

Parameters
  • local_model_result_folder_path (str) – root local path where project folder will be created

  • checkpoint_model (bool) – if the model is coming from the mid-training checkpoint

create_experiment_local_models_folder(project_name, experiment_name, experiment_timestamp)[source]

Creates experiment local folder hierarchy and place the ‘models’ folder in it

Parameters
  • project_name (str) – root name of the project

  • experiment_name (str) – name of the particular experiment

  • experiment_timestamp (str) – time stamp at the start of training

Returns

path to the created models folder in the experiment base folder

Return type

str

class aitoolbox.experiment.local_save.local_model_save.PyTorchLocalModelSaver(local_model_result_folder_path='~/project/model_result', checkpoint_model=False)[source]

Bases: aitoolbox.experiment.local_save.local_model_save.AbstractLocalModelSaver, aitoolbox.experiment.local_save.local_model_save.BaseLocalModelSaver

PyTorch experiment local model saver

Parameters
  • local_model_result_folder_path (str) – root local path where project folder will be created

  • checkpoint_model (bool) – if the model is coming from the mid-training checkpoint

save_model(model, project_name, experiment_name, experiment_timestamp=None, epoch=None, iteration_idx=None, protect_existing_folder=True)[source]

Save the PyTorch model representation dict to the local drive

Parameters
  • model (dict) – PyTorch model represented as a dict of weights, optimizer state and other necessary info.

  • project_name (str) – root name of the project

  • experiment_name (str) – name of the particular experiment

  • experiment_timestamp (str or None) – time stamp at the start of training

  • epoch (int or None) – in which epoch the model is being saved

  • iteration_idx (int or None) – at which training iteration the model is being saved

  • protect_existing_folder (bool) – can override potentially already existing folder or not

Returns

model_name, model_local_path

Return type

(str, str)

static check_model_dict_contents(model)[source]

Check if PyTorch model save dict contains all the necessary elements for the training state reconstruction

Parameters

model (dict) – PyTorch model represented as a dict of weights, optimizer state and other necessary info.

Raises

ValueError

Returns

None

class aitoolbox.experiment.local_save.local_model_save.KerasLocalModelSaver(local_model_result_folder_path='~/project/model_result', checkpoint_model=False)[source]

Bases: aitoolbox.experiment.local_save.local_model_save.AbstractLocalModelSaver, aitoolbox.experiment.local_save.local_model_save.BaseLocalModelSaver

Keras experiment local model saver

Parameters
  • local_model_result_folder_path (str) – root local path where project folder will be created

  • checkpoint_model (bool) – if the model is coming from the mid-training checkpoint

save_model(model, project_name, experiment_name, experiment_timestamp=None, epoch=None, iteration_idx=None, protect_existing_folder=True)[source]

Save the Keras model to the local drive

Parameters
  • model (keras.Model) – Keras model

  • project_name (str) – root name of the project

  • experiment_name (str) – name of the particular experiment

  • experiment_timestamp (str or None) – time stamp at the start of training

  • epoch (int or None) – in which epoch the model is being saved

  • iteration_idx (int or None) – at which training iteration the model is being saved

  • protect_existing_folder (bool) – can override potentially already existing folder or not

Returns

model_name, model_local_path

Return type

(str, str)

class aitoolbox.experiment.local_save.local_model_save.LocalSubOptimalModelRemover(metric_name, num_best_kept=2)[source]

Bases: object

Removes the tracked saved models which become suboptimal when new models are trained in subsequent epochs

Useful when interested in saving the limited local disk space, especially when dealing with large model which take a lot of disk space.

Parameters
  • metric_name (str) – one of the metric names that will be calculated and will appear in the train_history dict in the TrainLoop

  • num_best_kept (int) – number of best performing models which are kept when removing suboptimal model checkpoints

decide_if_remove_suboptimal_model(history, new_model_dump_paths)[source]

Make decision if suboptimal model should be removed due to the introduction of the new and better model

Parameters
Returns

None

static rm_suboptimal_model(rm_model_paths)[source]

Utility to remove the file

Parameters

rm_model_paths (list) – list of string paths

Returns

None