On this tutorial, we’ll learn to use nn.parallel.DistributedDataParallel for coaching our fashions in a number of GPUs. We’ll take a minimal instance of coaching a picture classifier and see how we are able to pace up the coaching.
Let’s begin with some imports.
import torch
import torchvision
import torchvision.transforms as transforms
import torch.nn as nn
import torch.nn.purposeful as F
import torch.optim as optim
import time
We’ll use the CIFAR10 in all our experiments with a batch measurement of 256.
def create_data_loader_cifar10():
rework = transforms.Compose(
[
transforms.RandomCrop(32),
transforms.RandomHorizontalFlip(),
transforms.ToTensor(),
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
batch_size = 256
trainset = torchvision.datasets.CIFAR10(root='./information', practice=True,
obtain=True, rework=rework)
trainloader = torch.utils.information.DataLoader(trainset, batch_size=batch_size,
shuffle=True, num_workers=10, pin_memory=True)
testset = torchvision.datasets.CIFAR10(root='./information', practice=False,
obtain=True, rework=rework)
testloader = torch.utils.information.DataLoader(testset, batch_size=batch_size,
shuffle=False, num_workers=10)
return trainloader, testloader
We’ll first practice the mannequin on a single Nvidia A100 GPU for 1 epoch. Customary pytorch stuff right here, nothing new. The tutorial relies on the official tutorial from Pytorch’s docs.
def practice(internet, trainloader):
print("Begin coaching...")
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(internet.parameters(), lr=0.001, momentum=0.9)
epochs = 1
num_of_batches = len(trainloader)
for epoch in vary(epochs):
running_loss = 0.0
for i, information in enumerate(trainloader, 0):
inputs, labels = information
photos, labels = inputs.cuda(), labels.cuda()
optimizer.zero_grad()
outputs = internet(photos)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
running_loss += loss.merchandise()
print(f'[Epoch {epoch + 1}/{epochs}] loss: {running_loss / num_of_batches:.3f}')
print('Completed Coaching')
The take a look at operate is equally outlined. The primary script will simply put every little thing collectively:
if __name__ == '__main__':
begin = time.time()
PATH = './cifar_net.pth'
trainloader, testloader = create_data_loader_cifar10()
internet = torchvision.fashions.resnet50(False).cuda()
start_train = time.time()
practice(internet, trainloader)
end_train = time.time()
torch.save(internet.state_dict(), PATH)
take a look at(internet, PATH, testloader)
finish = time.time()
seconds = (finish - begin)
seconds_train = (end_train - start_train)
print(f"Complete elapsed time: {seconds:.2f} seconds,
Prepare 1 epoch {seconds_train:.2f} seconds")
We use a resnet50 to measure the efficiency of a decent-sized community.
Now let’s practice the mannequin:
$ python -m train_1gpu
Accuracy of the community on the 10000 take a look at photos: 27 %
Complete elapsed time: 69.03 seconds, Prepare 1 epoch 13.08 seconds
Okay, time to get to optimization work.
Code is obtainable on GitHub. In case you are planning to solidify your Pytorch information, there are two superb books that we extremely suggest: Deep studying with PyTorch from Manning Publications and Machine Studying with PyTorch and Scikit-Study by Sebastian Raschka. You may at all times use the 35% low cost code blaisummer21 for all Manning’s merchandise.
torch.nn.DataParallel: no ache, no acquire
DataParallel is single-process, multi-thread, and solely works on a single machine. For every GPU, we use the identical mannequin to do the ahead cross. We scatter the information all through the GPUs and carry out ahead passes in every one among them. Primarily, what occurs is that the batch measurement is split throughout the variety of staff.
On this use case, this performance offered no acquire. That’s as a result of the system that I’m utilizing has a CPU and arduous disk bottleneck. Different machines which have very quick disk and CPU however battle with the GPU pace (GPU bottleneck) could profit from this performance.
In apply, the one change it’s essential do within the code is the next:
internet = torchvision.fashions.resnet50(False)
if torch.cuda.device_count() > 1:
print("Let's use", torch.cuda.device_count(), "GPUs!")
internet = nn.DataParallel(internet)
When utilizing nn.DataParallel, the batch measurement must be divisible by the variety of GPUs.
nn.DataParallelsplits the batch and processes it independently in all of the obtainable GPU’s. In every ahead cross, the module is replicated on every GPU, which is a big overhead. Every reproduction handles a portion of the batch (batch_size / gpus). Throughout the backwards cross, gradients from every reproduction are summed into the unique module.
Extra information on our earlier article on information vs mannequin parallelism.
A great apply when utilizing a number of GPUs is to outline prematurely the GPUs that your script is going to make use of:
import os
os.environ['CUDA_VISIBLE_DEVICES'] = "0,1"
This must be DONE earlier than another import-related to CUDA.
Even from the Pytorch documentation it’s apparent that this can be a very poor technique:
It is strongly recommended to make use of
nn.DistributedDataParallel, as an alternative of this class, to do multi-GPU coaching, even when there may be solely a single node.
The reason being that DistributedDataParallel makes use of one course of per employee (GPU) whereas DataParallel encapsulates all the information communication in a single course of.
In line with the docs, the information will be on any system earlier than they’re handed into the mannequin.
In my experiment, DataParallel was slower than coaching on a single GPU. Even with 4 GPUs. After rising the variety of staff I decreased the time, however nonetheless worse than a single GPU. I measure and report the time required to coach the mannequin for one epoch, that’s 50K 32×32 photos.
Ultimate word: to match the efficiency with a single GPU, I multiplied the batch measurement by the variety of staff, i.e. 4 for 4 GPUs. In any other case, it’s greater than 2X slower.
This brings us to the hardcore matter of Distributed Information-Parallel.
Code is obtainable on GitHub. You may at all times assist our work by social media sharing, making a donation, and shopping for our ebook and e-course.
Pytorch Distributed Information-Parallel
Distributed information parallel is multi-process and works for each single and multi-machine coaching. In pytorch, nn.parallel.DistributedDataParallel parallelizes the module by splitting the enter throughout the required gadgets. This module is appropriate for multi-node,multi-GPU coaching as nicely. Right here, I solely experimented with a single node (1 machine with 4 GPUs).
The primary distinction right here is that every GPU is dealt with by a course of. Parameters are by no means broadcasted between processes, solely gradients.
The module is replicated on every machine and every system. Throughout the ahead cross, every employee (GPU) processes the information and computes its personal gradient regionally. Throughout the backwards cross, gradients from every node are averaged. Lastly, every employee performs a parameter replace and sends to all the opposite nodes the computed parameter replace.
The module performs an all-reduce step on gradients and assumes that they are going to be modified by the optimizer in all processes in the identical manner.
Beneath are the rules for changing your single GPU script to multi-GPU coaching.
Step 1: Initialize the distributed studying processes
def init_distributed():
dist_url = "env://"
rank = int(os.environ["RANK"])
world_size = int(os.environ['WORLD_SIZE'])
local_rank = int(os.environ['LOCAL_RANK'])
dist.init_process_group(
backend="nccl",
init_method=dist_url,
world_size=world_size,
rank=rank)
torch.cuda.set_device(local_rank)
dist.barrier()
This initialization works once we launch our script with torch.distributed.launch (Pytorch 1.7 and 1.8) or torch.run (Pytorch 1.9+) from every node (right here 1).
Step 2: Wrap the mannequin utilizing DDP
internet = torchvision.fashions.resnet50(False).cuda()
internet = nn.SyncBatchNorm.convert_sync_batchnorm(internet)
local_rank = int(os.environ['LOCAL_RANK'])
internet = nn.parallel.DistributedDataParallel(internet, device_ids=[local_rank])
If every course of has the proper native rank, tensor.cuda() or mannequin.cuda() will be known as accurately all through the script.
Step 3: Use a DistributedSampler in your DataLoader
import torch
from torch.utils.information.distributed import DistributedSampler
from torch.utils.information import DataLoader
import torch.nn as nn
def create_data_loader_cifar10():
rework = transforms.Compose(
[
transforms.RandomCrop(32),
transforms.RandomHorizontalFlip(),
transforms.ToTensor(),
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
batch_size = 256
trainset = torchvision.datasets.CIFAR10(root='./information', practice=True,
obtain=True, rework=rework)
train_sampler = DistributedSampler(dataset=trainset, shuffle=True)
trainloader = torch.utils.information.DataLoader(trainset, batch_size=batch_size,
sampler=train_sampler, num_workers=10, pin_memory=True)
testset = torchvision.datasets.CIFAR10(root='./information', practice=False,
obtain=True, rework=rework)
test_sampler =DistributedSampler(dataset=testset, shuffle=True)
testloader = torch.utils.information.DataLoader(testset, batch_size=batch_size,
shuffle=False, sampler=test_sampler, num_workers=10)
return trainloader, testloader
In distributed mode, calling the data_loader.sampler.set_epoch() technique at first of every epoch earlier than creating the DataLoader iterator is critical to make shuffling work correctly throughout a number of epochs. In any other case, the identical ordering will likely be at all times used.
def practice(internet, trainloader):
print("Begin coaching...")
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(internet.parameters(), lr=0.001, momentum=0.9)
epochs = 1
num_of_batches = len(trainloader)
for epoch in vary(epochs):
trainloader.sampler.set_epoch(epoch)
In a extra common type:
for epoch in vary(epochs):
data_loader.sampler.set_epoch(epoch)
train_one_epoch(...)
Good practices for DDP
Any strategies that obtain information must be remoted to the grasp course of. Any strategies that carry out file I/O must be remoted to the grasp course of.
import torch.distributed as dist
import torch
def is_dist_avail_and_initialized():
if not dist.is_available():
return False
if not dist.is_initialized():
return False
return True
def save_on_master(*args, **kwargs):
if is_main_process():
torch.save(*args, **kwargs)
def get_rank():
if not is_dist_avail_and_initialized():
return 0
return dist.get_rank()
def is_main_process():
return get_rank() == 0
Based mostly on this operate you possibly can make sure that some instructions are solely executed from the principle course of:
if is_main_process():
Launch script utilizing torch.distributed.launch or torch.run
$ python -m torch.distributed.launch --nproc_per_node=4 main_script.py
Errors will happen. You’ll want to kill any undesirable distributed coaching course of by:
$ kill $(ps aux | grep main_script.py | grep -v grep | awk '{print $2}')
Exchange main_script.py together with your script title. One other extra easy possibility is $ kill -9 PID. In any other case you possibly can go to extra superior stuff, like killing all CUDA GPU associated processes when not proven in nvidia-smi
lsof /dev/nvidia* | awk '{print $2}' | xargs -I {} kill {}
That is just for the case that you simply can not discover the PID of the method operating within the GPU.
An excellent ebook on distributed coaching is Distributed Machine Studying with Python: Accelerating mannequin coaching and serving with distributed methods by Guanhua Wang.
Blended-precision coaching in Pytorch
Blended precision combines Floating Level (FP) 16 and FP 32 in numerous steps of the coaching. FP16 coaching is often known as half-precision coaching, which comes with inferior efficiency. Computerized mixed-precision is actually the perfect of each worlds: decreased coaching time with comparable efficiency to FP32.
In Blended Precision Coaching, all of the computational operations (ahead cross, backward cross, weight gradients) see the FP16 casted model. To take action, an FP32 copy of the burden is critical, in addition to computing the loss in FP32 after the ahead cross in FP16 to keep away from over and underflows. The load gradients are casted again to FP32 to replace the mannequin’s weights. Furthermore, the loss in FP32 is scaled as much as keep away from gradient underflow earlier than getting casted to FP16 to carry out the backward cross. As compensation, the FP32 weights will likely be scaled down by the identical scalar earlier than the burden replace.
Listed below are the modifications within the practice operate:
fp16_scaler = torch.cuda.amp.GradScaler(enabled=True)
for epoch in vary(epochs):
trainloader.sampler.set_epoch(epoch)
running_loss = 0.0
for i, information in enumerate(trainloader, 0):
inputs, labels = information
photos, labels = inputs.cuda(), labels.cuda()
optimizer.zero_grad()
with torch.cuda.amp.autocast():
outputs = internet(photos)
loss = criterion(outputs, labels)
fp16_scaler.scale(loss).backward()
fp16_scaler.step(optimizer)
fp16_scaler.replace()
Outcomes and Sum up
In a utopian parallel world, N staff would give a speedup of N. Right here you see that you simply want 4 GPUs in DistributedDataParallel mode to get a speedup of 2X. Blended precision coaching usually gives a considerable speedup however the A100 GPU and different Ampere-based GPU architectures have restricted positive factors (so far as I’ve learn on-line).
Outcomes under report the time in seconds for 1 epoch on CIFAR10 with a resnet50 (batch measurement 256, NVidia A100 40GB GPU reminiscence):
| Time in seconds | |
| Single GPU (baseline) | 13.2 |
| DataParallel 4 GPUs | 19.1 |
| DistributedDataParallel 2 GPUs | 9.8 |
| DistributedDataParallel 4 GPUs | 6.1 |
| DistributedDataParallel 4 GPUs + Blended Precision | 6.5 |
A vital word right here is that DistributedDataParallel makes use of an efficient batch measurement of 4*256=1024 so it makes fewer mannequin updates. That’s why I consider it scores a a lot decrease validation accuracy (14% in comparison with 27% within the baseline).
Code is obtainable on GitHub if you wish to mess around. The outcomes will fluctuate based mostly in your {hardware}. There may be at all times the case that I missed one thing in my experiments. Should you discover a flaw please let me know on our Discord server.
These findings would give you a strong begin to coaching your fashions. I hope you discover them helpful. Helps us by social media sharing, making a donation, shopping for our ebook or e-course. Your assist would assist us produce extra free content material and accessible AI content material. As at all times, thanks in your curiosity in our weblog.
* Disclosure: Please word that among the hyperlinks above is likely to be affiliate hyperlinks, and at no extra price to you, we’ll earn a fee for those who determine to make a purchase order after clicking by means of.