Why it matters

Data pipeline feeds GPUs. Stalled data = wasted GPU.

Advertisement

The architecture

Tokenize + shuffle + shard.

Stream from S3 or FS.

Prefetch to GPU.

LLM training data pipelineTokenize + shuffleoffline or streamShard + cachefor parallel readsPrefetch to GPUno stallsParquet, WebDataset, Mosaic StreamingDataset; TB/hr throughput needed
LLM training data pipeline.
Advertisement

How it works end to end

Parquet or WebDataset.

Mosaic StreamingDataset.

Prefetch.