paddlespeech.s2t.training.trainer module

class paddlespeech.s2t.training.trainer.Trainer(config, args)

Bases: object

An experiment template in order to structure the training code and take care of saving, loading, logging, visualization stuffs. It's intended to be flexible and simple.

So it only handles output directory (create directory for the output, create a checkpoint directory, dump the config in use and create visualizer and logger) in a standard way without enforcing any input-output protocols to the model and dataloader. It leaves the main part for the user to implement their own (setup the model, criterion, optimizer, define a training step, define a validation function and customize all the text and visual logs). It does not save too much boilerplate code. The users still have to write the forward/backward/update mannually, but they are free to add non-standard behaviors if needed. We have some conventions to follow. 1. Experiment should have model, optimizer, train_loader and valid_loader, config and args attributes. 2. The config should have a training field, which has valid_interval, save_interval and max_iteration keys. It is used as the trigger to invoke validation, checkpointing and stop of the experiment. 3. There are four methods, namely train_batch, valid, setup_model and setup_dataloader that should be implemented. Feel free to add/overwrite other methods and standalone functions if you need.

Parameters:

config: yacs.config.CfgNode

The configuration used for the experiment.

args: argparse.Namespace

The parsed command line arguments.

Examples

--------

>>> def main_sp(config, args):

>>> exp = Trainer(config, args)

>>> exp.setup()

>>> exp.run()

>>>

>>> config = get_cfg_defaults()

>>> parser = default_argument_parser()

>>> args = parser.parse_args()

>>> if args.config:

>>> config.merge_from_file(args.config)

>>> if args.opts:

>>> config.merge_from_list(args.opts)

>>> config.freeze()

>>>

>>> if args.ngpu > 1:

>>> dist.spawn(main_sp, args=(config, args), nprocs=args.ngpu)

>>> else:

>>> main_sp(config, args)

Attributes:

parallel

A flag indicating whether the experiment should run with multiprocessing.

train

Methods

align()	The align.
destory()	Close visualizer to avoid hanging after training
do_train()	The training process control by epoch.
dump_config()	Save the configuration used for this experiment.
export()	The test.
maybe_batch_sampler_step()	batch_sampler seed by epoch
new_epoch()	Reset the train loader seed and increment epoch.
restore()	Resume from latest checkpoint at checkpoints in the output directory or load a specified checkpoint.
resume_or_scratch()	Resume from latest checkpoint at checkpoints in the output directory or load a specified checkpoint.
run()	The routine of the experiment after setup.
run_align()	Do CTC alignment
run_export()	Do Model Export
run_test()	Do Test/Decode
save([tag, infos])	Save checkpoint (model parameters and optimizer states).
setup()	Setup the experiment.
setup_dataloader()	Setup training dataloader and validation dataloader.
setup_model()	Setup model, criterion and optimizer, etc.
setup_output_dir()	Create a directory used for output.
setup_visualizer()	Initialize a visualizer to log the experiment.
test()	The test.
train_batch()	The training loop.
valid()	The validation.

after_train_batch
before_train
eval

after_train_batch()

align()

The align. A subclass should implement this method in Tester.

before_train()

destory()

Close visualizer to avoid hanging after training

do_train()

The training process control by epoch.

dump_config()

Save the configuration used for this experiment.

It is saved in to config.yaml in the output directory at the beginning of the experiment.

eval()

export()

The test. A subclass should implement this method in Tester.

maybe_batch_sampler_step()

batch_sampler seed by epoch

new_epoch()

Reset the train loader seed and increment epoch.

property parallel

A flag indicating whether the experiment should run with multiprocessing.

restore()

Resume from latest checkpoint at checkpoints in the output directory or load a specified checkpoint.

If args.checkpoint_path is not None, load the checkpoint, else resume training.

resume_or_scratch()

Resume from latest checkpoint at checkpoints in the output directory or load a specified checkpoint.

If args.checkpoint_path is not None, load the checkpoint, else resume training.

run()

The routine of the experiment after setup. This method is intended to be used by the user.

run_align()

Do CTC alignment

run_export()

Do Model Export

run_test()

Do Test/Decode

save(tag=None, infos: dict = None)

Save checkpoint (model parameters and optimizer states).

Args:

tag (int or str, optional): None for step, else using tag, e.g epoch. Defaults to None. infos (dict, optional): meta data to save. Defaults to None.

setup()

Setup the experiment.

setup_dataloader()

Setup training dataloader and validation dataloader. A subclass should implement this method.

setup_model()

Setup model, criterion and optimizer, etc. A subclass should implement this method.

setup_output_dir()

Create a directory used for output.

setup_visualizer()

Initialize a visualizer to log the experiment.

The visual log is saved in the output directory.

Notes

Only the main process has a visualizer with it. Use multiple visualizers in multiprocess to write to a same log file may cause unexpected behaviors.

test()

The test. A subclass should implement this method in Tester.

property train

train_batch()

The training loop. A subclass should implement this method.

valid()

The validation. A subclass should implement this method.