paddlespeech.s2t.io.batchfy module

paddlespeech.s2t.io.batchfy.make_batchset(data, batch_size=0, max_length_in=inf, max_length_out=inf, num_batches=0, min_batch_size=1, shortest_first=False, batch_sort_key='input', count='auto', batch_bins=0, batch_frames_in=0, batch_frames_out=0, batch_frames_inout=0, iaxis=0, oaxis=0)[source]

Make batch set from json dictionary

if utts have "category" value,

>>> data = [{'category': 'A', 'input': ..., 'utt':'utt1'},
...         {'category': 'B', 'input': ..., 'utt':'utt2'},
...         {'category': 'B', 'input': ..., 'utt':'utt3'},
...         {'category': 'A', 'input': ..., 'utt':'utt4'}]
>>> make_batchset(data, batchsize=2, ...)
[[('utt1', ...), ('utt4', ...)], [('utt2', ...), ('utt3': ...)]]

Note that if any utts doesn't have "category", perform as same as batchfy_by_{count}

Parameters:

  • data (List[Dict[str, Any]]) -- dictionary loaded from data.json

  • batch_size (int) -- maximum number of sequences in a minibatch.

  • batch_bins (int) -- maximum number of bins (frames x dim) in a minibatch.

  • batch_frames_in (int) -- maximum number of input frames in a minibatch.

  • batch_frames_out (int) -- maximum number of output frames in a minibatch.

  • batch_frames_out -- maximum number of input+output frames in a minibatch.

  • count (str) -- strategy to count maximum size of batch. For choices, see io.batchfy.BATCH_COUNT_CHOICES

  • max_length_in (int) -- maximum length of input to decide adaptive batch size

  • max_length_out (int) -- maximum length of output to decide adaptive batch size

  • num_batches (int) -- # number of batches to use (for debug)

  • min_batch_size (int) -- minimum batch size (for multi-gpu)

  • shortest_first (bool) -- Sort from batch with shortest samples to longest if true, otherwise reverse

  • batch_sort_key (str) -- how to sort data before creating minibatches ["input", "output", "shuffle"]

  • swap_io (bool) -- if True, use "input" as output and "output" as input in data dict

  • mt (bool) -- if True, use 0-axis of "output" as output and 1-axis of "output" as input in data dict

  • iaxis (int) -- dimension to access input (for ASR, TTS iaxis=0, for MT iaxis="1".)

  • oaxis (int) -- dimension to access output (for ASR, TTS, MT oaxis=0, reserved for future research, -1 means all axis.)

Returns:

List[List[Tuple[str, dict]]] list of batches