PARALLELISM MANUAL TO PARALLEL DATA PARALLELISM

Distributed Data Parallelism (DDP, often abbreviated as DDp) represents a significant technique for scaling ML model training across several devices, like GPUs or machines. This approach involves replicating the entire model onto each worker and then splitting the data into smaller subsets which