abstract_result_packages
- class aitoolbox.experiment.result_package.abstract_result_packages.AbstractResultPackage(pkg_name=None, strict_content_check=False, np_array=True, **kwargs)[source]
Bases:
ABC
Base Result package used to derive specific result packages from
Functions which the user should potentially override in a specific result package:
- Parameters:
pkg_name (str or None) – result package name used just for clarity
strict_content_check (bool) – should just print warning or raise the error and crash
how the inputs should be handled. Should the package try to automatically guess, or you want to manually decide whether to leave the inputs as they are or convert them to np.array. Possible options: True, False, ‘auto’
Be slightly careful with ‘auto’ as it sometimes doesn’t work, so it is preferable to explicitly use True/False
**kwargs (dict) – additional package_metadata for the result package
- abstract prepare_results_dict()[source]
Perform result package building
Mostly this consists of executing calculation of selected performance metrics and returning their result dicts. If you want to use multiple performance metrics you have to combine them in the single self.results_dict at the end by doing this:
return {**metric_dict_1, **metric_dict_2}
- Returns:
calculated result dict
- Return type:
- prepare_result_package(y_true, y_predicted, hyperparameters=None, **kwargs)[source]
Prepares the result package by taking labels and running them through the specified metrics
This function is automatically called from the torchtrain callbacks to evaluate the provided callback. The main feature of this function is the call to the user-derived prepare_results_dict() function of the implemented result package where the metrics evaluation logic is implemented.
- Parameters:
y_true (numpy.ndarray or list) – ground truth targets
y_predicted (numpy.ndarray or list) – predicted targets
hyperparameters (dict or None) – dictionary filled with the set hyperparameters
**kwargs (dict) – additional results for the result package
- Returns:
None
- static auto_y_input_array_convert(y_array)[source]
Try to automatically decide if array should be left as it is or convert to np.array
Not working in all the situations so relying on it at all times is not recommended. Especially for costly experiments rely rather on your own judgement and explicitly define if np.array conversion is needed.
TODO: make it smarter so ‘auto’ option can be used more often
- get_hyperparameters()[source]
Get hyperparameters in a dict form
- Returns:
hyperparameters dict
- Return type:
- get_additional_results_dump_paths()[source]
Return paths to the additional results which are stored to local drive when the package is evaluated
For example if package plots attention heatmaps and saves pictures to disk, this function will return paths to these picture files. This is achieved via the call to the user-implemented function list_additional_results_dump_paths().
- Returns:
list of lists of string paths if it is not None. Each element of the list should be list of: [[results_file_name, results_file_local_path], … [,]]
- Return type:
list or None
- list_additional_results_dump_paths()[source]
Specify the list of metadata files you also want to save & upload to s3 during the experiment saving procedure
By default, there are no additional files that are saved as the return is None. If you want to save your specific additional files produced during the training procedure, then override this method specifying the file paths.
If you want to save a whole folder of files, use zip_additional_results_dump() function to zip it into a single file and save this zip instead.
The specified files are any additional data you would want to include into the experiment folder in addition to the model save files and performance evaluation report files. For example a zip of attention heatmap pictures in the machine translation projects.
- Returns:
list of lists of string paths if it is not None. Each element of the list should be list of: [[results_file_name, results_file_local_path], … [,]]
- Return type:
list or None
- set_experiment_dir_path_for_additional_results(project_name, experiment_name, experiment_timestamp, local_model_result_folder_path)[source]
Set experiment folder path after potential timestamps have already been generated.
Experiment folder setting for additional metadata results output is needed only in certain result packages, for example in QuestionAnswerResultPackage where the self.output_text_dir initially has only the name of the folder where the results text predictions for each example should be stored. This function when implemented reforms the folder name so that it becomes a full path placing the folder inside the experiment folder (for which the timestamp at the start of train loop is needed).
Another use of this function is in MachineTranslationResultPackage where the attention heatmap pictures are stored as additional metadata results.
As can be seen from the fact that the train loop mechanism is mentioned, this method’s functionality is primarily used for PyTorch experiments.
- qa_check_additional_results_dump_paths()[source]
Quality check the additional results path
- Returns:
None
- warn_about_result_data_problem(msg)[source]
Generic function for writing out warnings
Either just printing out the warning or throw the error exception.
- Parameters:
msg (str) – warning message either printed or written in the raised error
- Raises:
- Returns:
None
- static zip_additional_results_dump(source_dir_path, zip_path)[source]
Utility function for zipping a folder into .zip archive
- __add__(other)[source]
Concatenate result packages
Combine results from both result packages into a single one.
- Parameters:
other (AbstractResultPackage or dict) – another result package to be concatenated
- Returns:
merged result package
- Return type:
- __radd__(other)[source]
Concatenate result package
- Parameters:
other (AbstractResultPackage or dict) – another result package to be concatenated
- Returns:
merged result package
- Return type:
- add_merge_multi_pkg_wrap(other_object)[source]
Result package merge
- Parameters:
other_object (AbstractResultPackage or dict) – another result package to be merged with the current package
- Returns:
merged result package
- Return type:
- static _create_other_object_pkg(other_object)[source]
Util to deep copy and wrap results into the simple result package
- Parameters:
other_object (AbstractResultPackage or dict) – results package or results dict
- Returns:
deep copy of results wrapped in the simple result package
- Return type:
- __iadd__(other)[source]
Append result package
- Parameters:
other (AbstractResultPackage or dict) – another result package to be appended to the current package
- Returns:
merged result package
- Return type:
- add_merge_dicts(other)[source]
Append result package to the current one
- Parameters:
other (AbstractResultPackage or dict) – another result package to be appended to the current package
- Returns:
merged result package
- Return type:
- class aitoolbox.experiment.result_package.abstract_result_packages.PreCalculatedResultPackage(results_dict, strict_content_check=False, **kwargs)[source]
Bases:
AbstractResultPackage
Result package which doesn’t have any evaluation logic but just accepts pre-calculated results dict
- Parameters:
- prepare_results_dict()[source]
Perform result package building
Mostly this consists of executing calculation of selected performance metrics and returning their result dicts. If you want to use multiple performance metrics you have to combine them in the single self.results_dict at the end by doing this:
return {**metric_dict_1, **metric_dict_2}
- Returns:
calculated result dict
- Return type:
- class aitoolbox.experiment.result_package.abstract_result_packages.MultipleResultPackageWrapper(strict_content_check=False, **kwargs)[source]
Bases:
AbstractResultPackage
Wrapper result package which combines multiple evaluated result packages into a single result package
- Parameters:
- prepare_result_package(result_packages, hyperparameters=None, **kwargs)[source]
Prepares the multiple result package by merging the results from both result packages
- Parameters:
result_packages (list) – list of result packages where each of them is object inherited from aitoolbox.experiment.result_package.abstract_result_packages.AbstractResultPackage. If you want to add raw results in dict form, this dict first needs to be wrapped into
PreCalculatedResultPackage
to satisfy the result package object requirement.hyperparameters (dict or None) – hyperparameters dict
**kwargs – result package additional meta-data
- Returns:
None
- prepare_results_dict()[source]
Perform result package building
Mostly this consists of executing calculation of selected performance metrics and returning their result dicts. If you want to use multiple performance metrics you have to combine them in the single self.results_dict at the end by doing this:
return {**metric_dict_1, **metric_dict_2}
- Returns:
calculated result dict
- Return type:
- get_additional_results_dump_paths()[source]
Return paths to the additional results which are stored to local drive when the package is evaluated
For example if package plots attention heatmaps and saves pictures to disk, this function will return paths to these picture files. This is achieved via the call to the user-implemented function list_additional_results_dump_paths().
- Returns:
list of lists of string paths if it is not None. Each element of the list should be list of: [[results_file_name, results_file_local_path], … [,]]
- Return type:
list or None
- __len__()[source]
Get number of result packages inside the multi result package wrapper
- Returns:
number of result packages inside this multi package wrapper
- Return type:
- add_merge_multi_pkg_wrap(other_object)[source]
Result package merge
- Parameters:
other_object (AbstractResultPackage or dict) – another result package to be merged with the current package
- Returns:
merged result package
- Return type: