model_save

class aitoolbox.cloud.AWS.model_save.AbstractModelSaver[source]

Bases: ABC

abstract save_model(model, project_name, experiment_name, experiment_timestamp=None, epoch=None, iteration_idx=None, protect_existing_folder=True)[source]
Parameters:

  • model –

  • project_name (str) –

  • experiment_name (str) –

  • experiment_timestamp (str or None) –

  • epoch (int or None) –

  • iteration_idx (int or None) –

  • protect_existing_folder (bool) –

Returns:

model_s3_path, experiment_timestamp, model_local_path

Return type:

(str, str, str)

class aitoolbox.cloud.AWS.model_save.BaseModelSaver(bucket_name='model-result', cloud_dir_prefix='', checkpoint_model=False)[source]

Bases: BaseDataSaver

Base model saving to AWS S3 functionality

Parameters:

  • bucket_name (str) – S3 bucket into which the files will be saved

  • cloud_dir_prefix (str) – destination folder path inside selected bucket

  • checkpoint_model (bool) – if the model that is going to be saved is final model or mid-training checkpoint

create_experiment_cloud_storage_folder_structure(project_name, experiment_name, experiment_timestamp)[source]
Parameters:

  • project_name (str) – root name of the project

  • experiment_name (str) – name of the particular experiment

  • experiment_timestamp (str) – time stamp at the start of training

Returns:

experiment cloud path

Return type:

str

class aitoolbox.cloud.AWS.model_save.PyTorchS3ModelSaver(bucket_name='model-result', cloud_dir_prefix='', local_model_result_folder_path='~/project/model_result', checkpoint_model=False)[source]

Bases: AbstractModelSaver, BaseModelSaver

PyTorch AWS S3 model saving

Parameters:

  • bucket_name (str) – name of the bucket in the S3 to which the models will be saved

  • cloud_dir_prefix (str) – destination folder path inside selected bucket

  • local_model_result_folder_path (str) – root local path where project folder will be created

  • checkpoint_model (bool) – if the model being saved is checkpoint model or final end of training model

save_model(model, project_name, experiment_name, experiment_timestamp=None, epoch=None, iteration_idx=None, protect_existing_folder=True)[source]

Save PyTorch model representation to AWS S3

Parameters:

  • model (dict) – PyTorch model representation dict

  • project_name (str) – root name of the project

  • experiment_name (str) – name of the particular experiment

  • experiment_timestamp (str or None) – time stamp at the start of training

  • epoch (int or None) – epoch number

  • iteration_idx (int or None) – at which training iteration the model is being saved

  • protect_existing_folder (bool) – can override potentially already existing folder or not

Returns:

model_s3_path, experiment_timestamp, model_local_path

Return type:

(str, str, str)

Examples

local_model_result_folder_path = '~/project/model_results'
m_saver = PyTorchLocalModelSaver(local_model_result_folder_path=local_model_result_folder_path)
m_saver.save_model(model=model,
                   project_name='QA_QAngaroo',
                   experiment_name='FastQA_RNN_concat_model_GLOVE',
                   protect_existing_folder=False)
class aitoolbox.cloud.AWS.model_save.KerasS3ModelSaver(bucket_name='model-result', cloud_dir_prefix='', local_model_result_folder_path='~/project/model_result', checkpoint_model=False)[source]

Bases: AbstractModelSaver, BaseModelSaver

Keras AWS S3 model saving

Parameters:

  • bucket_name (str) – name of the bucket in the S3 to which the models will be saved

  • cloud_dir_prefix (str) – destination folder path inside selected bucket

  • local_model_result_folder_path (str) – root local path where project folder will be created

  • checkpoint_model (bool) – if the model being saved is checkpoint model or final end of training model

save_model(model, project_name, experiment_name, experiment_timestamp=None, epoch=None, iteration_idx=None, protect_existing_folder=True)[source]

Save Keras model to AWS S3

Parameters:

  • model (keras.Model) –

  • project_name (str) – root name of the project

  • experiment_name (str) – name of the particular experiment

  • experiment_timestamp (str or None) – time stamp at the start of training

  • epoch (int or None) – epoch number

  • iteration_idx (int or None) – at which training iteration the model is being saved

  • protect_existing_folder (bool) – can override potentially already existing folder or not

Returns:

model_s3_path, experiment_timestamp, model_local_path

Return type:

(str, str, str)

Examples

local_model_result_folder_path = '~/project/model_results'
m_saver = KerasS3ModelSaver(local_model_result_folder_path=local_model_result_folder_path)
m_saver.save_model(model=model,
                   project_name='QA_QAngaroo',
                   experiment_name='FastQA_RNN_concat_model_GLOVE',
                   protect_existing_folder=False)