paddlespeech.t2s.training.experiment module

class paddlespeech.t2s.training.experiment.ExperimentBase(config, args)

Bases: object

An experiment template in order to structure the training code and take care of saving, loading, logging, visualization stuffs. It's intended to be flexible and simple.

So it only handles output directory (create directory for the output, create a checkpoint directory, dump the config in use and create visualizer and logger) in a standard way without enforcing any input-output protocols to the model and dataloader. It leaves the main part for the user to implement their own (setup the model, criterion, optimizer, define a training step, define a validation function and customize all the text and visual logs).

It does not save too much boilerplate code. The users still have to write the forward/backward/update mannually, but they are free to add non-standard behaviors if needed.

We have some conventions to follow. 1. Experiment should have model, optimizer, train_loader and valid_loader, config and args attributes. 2. The config should have a training field, which has valid_interval, save_interval and max_iteration keys. It is used as the trigger to invoke validation, checkpointing and stop of the experiment. 3. There are four methods, namely train_batch, valid, setup_model and setup_dataloader that should be implemented.

Feel free to add/overwrite other methods and standalone functions if you need.

Args:

config (yacs.config.CfgNode):

The configuration used for the experiment.

args (argparse.Namespace):

The parsed command line arguments.

Examples:

>>> def main_sp(config, args):
>>>     exp = Experiment(config, args)
>>>     exp.setup()
>>>     exe.resume_or_load()
>>>     exp.run()
>>>
>>> config = get_cfg_defaults()
>>> parser = default_argument_parser()
>>> args = parser.parse_args()
>>> if args.config:
>>>     config.merge_from_file(args.config)
>>> if args.opts:
>>>     config.merge_from_list(args.opts)
>>> config.freeze()
>>>
>>> if args.ngpu > 1:
>>>     dist.spawn(main_sp, args=(config, args), nprocs=args.ngpu)
>>> else:
>>>     main_sp(config, args)

Attributes:

parallel

A flag indicating whether the experiment should run with multiprocessing.

Methods

close()	Close visualizer to avoid hanging after training
dump_config()	Save the configuration used for this experiment.
init_parallel()	Init environment for multiprocess training.
new_epoch()	Reset the train loader and increment epoch.
read_batch()	Read a batch from the train_loader.
resume_or_load()	Resume from latest checkpoint at checkpoints in the output directory or load a specified checkpoint.
run()	The routine of the experiment after setup.
save()	Save checkpoint (model parameters and optimizer states).
setup()	Setup the experiment.
setup_checkpointer()	Create a directory used to save checkpoints into.
setup_dataloader()	Setup training dataloader and validation dataloader.
setup_logger()	Initialize a text logger to log the experiment.
setup_model()	Setup model, criterion and optimizer, etc.
setup_output_dir()	Create a directory used for output.
setup_visualizer()	Initialize a visualizer to log the experiment.
train()	The training process.
train_batch()	The training loop.
valid()	The validation.

close()

Close visualizer to avoid hanging after training

dump_config()

Save the configuration used for this experiment.

It is saved in to config.yaml in the output directory at the beginning of the experiment.

init_parallel()

Init environment for multiprocess training.

new_epoch()

Reset the train loader and increment epoch.

property parallel

A flag indicating whether the experiment should run with multiprocessing.

read_batch()

Read a batch from the train_loader.

Returns:

List[Tensor]

A batch.

resume_or_load()

Resume from latest checkpoint at checkpoints in the output directory or load a specified checkpoint.

If args.checkpoint_path is not None, load the checkpoint, else resume training.

run()

The routine of the experiment after setup. This method is intended to be used by the user.

save()

Save checkpoint (model parameters and optimizer states).

setup()

Setup the experiment.

setup_checkpointer()

Create a directory used to save checkpoints into.

It is "checkpoints" inside the output directory.

setup_dataloader()

Setup training dataloader and validation dataloader. A subclass should implement this method.

setup_logger()

Initialize a text logger to log the experiment.

Each process has its own text logger. The logging message is write to the standard output and a text file named worker_n.log in the output directory, where n means the rank of the process.

setup_model()

Setup model, criterion and optimizer, etc. A subclass should implement this method.

setup_output_dir()

Create a directory used for output.

setup_visualizer()

Initialize a visualizer to log the experiment.

The visual log is saved in the output directory.

Notes

Only the main process has a visualizer with it. Use multiple visualizers in multiprocess to write to a same log file may cause unexpected behaviors.

train()

The training process.

It includes forward/backward/update and periodical validation and saving.

train_batch()

The training loop. A subclass should implement this method.

valid()

The validation. A subclass should implement this method.