A complete Weights and Biases tutorial

Weights and Biases has grow to be one of many AI group favorite libraries. The workforce has completed a superb work making a platform the place the Machine Studying engineer can effortlessly:

• Monitor his/her experiments

• Visualize the coaching course of

• Share the outcomes with the workforce

• Enhance the mannequin’s efficiency

Personally I began utilizing it a few months in the past and rapidly grew to become a person a part of all my initiatives. This text summarizes my expertise with the library and goals to be a self-complete tutorial of its most helpful options. To perform that, we are going to look at how we will combine the wandb library in a brand new venture.

Shall we start?

Conditions

We are going to use a typical Deep Studying mannequin that performs picture recognition on the CIFAR10 dataset. The mannequin doesn’t actually have an effect on our experiments so I assumed to maintain it so simple as potential. The mannequin might be skilled on the dataset from scratch as a way to discover how we will make the most of the wandb library.

Right here is the Pytorch code for our mannequin alongside with the information processing :

import torch
import torch.nn as nn
import torch.nn.purposeful as F
import torchvision
import torchvision.transforms as transforms
rework = transforms.Compose(
    [transforms.ToTensor(),
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
trainset = torchvision.datasets.CIFAR10(root='./knowledge', practice=True,
                                        obtain=True, rework=rework)
trainloader = torch.utils.knowledge.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)
testset = torchvision.datasets.CIFAR10(root='./knowledge', practice=False,
                                       obtain=True, rework=rework)
testloader = torch.utils.knowledge.DataLoader(testset, batch_size=4,
                                         shuffle=False, num_workers=2)
class Web(nn.Module):
    def __init__(self):
        tremendous(Web, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)
    def ahead(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

Step one is to put in the library and create a brand new account.

Set up and initialization

If you happen to haven’t already, you’ll need to create a brand new account so as to have the ability to use Weights and Biases. The library is free for private use however comes with a month-to-month worth for groups. You possibly can go to the web site and join.

When you do this, you need to be capable of set up it utilizing pip or conda. After putting in, you’ll need to authenticate your self. This may be completed with the wandb login command. You may be prompted to repeat paste an authorization key as a way to proceed.

$ conda set up -c conda-forge wandb
$ wandb login

The library will be initialized in our code with the init methodology which receives an elective venture identify and your username, amongst different issues.

import wandb
wandb.init(venture='check', entity='serkar')

Now that we’re all arrange, let’s try to combine the library to our coaching loop.

Experiment monitoring

The primary use of the wandb library is to trace and visualize the completely different machine studying experiments, the coaching course of, the hyperparameters and the fashions. Let’s see some examples.

Monitor metrics

The wonderful factor in regards to the Weights and Biases (W&B) library is how simple it’s to make use of. In lots of instances, it’s actually one line of code:

wandb.log({'epoch': epoch, 'loss': running_loss})

The .log() command will seize all of the arguments and ship them to the W&B occasion. This can permit us to entry and monitor them from the UI. You’ll find the dashboard within the W&B web site beneath your venture.

In our software, a pattern coaching loop can appear like beneath:

for epoch in vary(10):
    running_loss = 0.0
    for i, knowledge in enumerate(trainloader, 0):
        inputs, labels = knowledge[0].to(machine), knowledge[1].to(machine)
        optimizer.zero_grad()
        outputs = internet(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        running_loss += loss.merchandise()
        if i % 2000 == 1999:    
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            wandb.log({'epoch': epoch+1, 'loss': running_loss/2000})
            running_loss = 0.0
print('Completed Coaching')

Did you discover the wandb.log line? That manner, we will examine the coaching course of in actual time. The consequence will appear like this:

Fairly superior, proper?

One other command that will also be used is the wandb.watch, which is able to robotically accumulate the mannequin’s gradients and the mannequin’s topology.

wandb.watch(internet, criterion, log="all")

Apart from the outlined metrics, we will additionally monitor many different helpful issues corresponding to our trainable parameters.

Or the gradients:

One different function that actually impressed me is the system dashboard. There, we will examine our {hardware} and the way the completely different parts behave throughout coaching. For instance, one can look at the CPU, GPU, reminiscence utilization, energy utilization, temperatures and extra.

Every time we execute our coaching script, a brand new “run” is created and appended to the venture’s historical past. Every “run” comprises the logged data with completely different metadata. We will, in fact, discover all of the completely different runs in a single dashboard.

Monitor hyperparameters

Apart from metrics, W&B has one other neat performance that permits us to trace the hyperparameters of our coaching. The wandb.config object is used to avoid wasting the coaching configuration corresponding to hyperparameters. However it’s not restricted to them. We will primarily retailer every bit of data we wish. Examples embrace: dataset names, mannequin sorts and flags.

A config will be initialized like this:

config = wandb.config
config.learning_rate = 0.01
config.momentum = 0.9

For advanced configs, we will additionally use a yaml file or a python dictionary.

All these values can be utilized to investigate the experiments and reproduce the outcomes. Within the following dashboard, we will see 5 “runs” with their hyperparameters. Be aware that we will use the config values to group, filter or type them.

Visualize mannequin

If we use the aforementioned watch command, we will additionally examine the mannequin’s topology within the mannequin dashboard. In our case, the mannequin will appear like this:

Examine logs

The identical is true for the precise logs printed in our native console:

Information and Mannequin Versioning

Apart from experiment monitoring, W&B has a built-in versioning system. Artifacts are the primary entity in the direction of that purpose. Artifacts allow dataset versioning, mannequin versioning and dependencies monitoring.

An artifact is nothing greater than a versioned folder of knowledge. Let’s look at an instance utilizing our venture. With a view to model our dataset, all we have now to do is create an artifact and add it.

cifar10_artifact = wandb.Artifact("cifar10", sort="dataset")
file_path = './knowledge/cifar-10-batches-py'
cifar10_artifact.add_dir(file_path)
run.log_artifact(cifar10_artifact)

	

			
	

			


		
			
			
					
			
		

You possibly can think about that one thing related will be completed for versioning the mannequin or the dependencies. It’s value mentioning that as a substitute of utilizing the entire dataset, we will create an artifact with an exterior reference of the item as beneath:

artifact.add_reference('s3://my-bucket/my_dataset)

Obtain and utilizing an already uploaded artifact in our code can also be easy:

artifact = run.use_artifact('cifar10_artifact')
artifact_dir = artifact.obtain()

Hyperparameter Tuning with Sweeps

Weights & Biases Sweeps is a software to automate hyperparameter optimization and exploration. It eliminates a lot of the boilerplate code and comes with tremendous good visualizations. Let’s discover how we will make the most of Sweeps in our initiatives.

In our use case, we need to tune 4 completely different parameters: the scale of the final linear layer within the mannequin, the batch measurement, the training charge and the optimization algorithm. To realize this, we first have to create a configuration with the completely different choices. Here’s a pattern config:

sweep_config = {
                'methodology': 'random',
                'metric': {'purpose': 'decrease', 'identify': 'loss'},
                'parameters': {
                    'batch_size': {
                        'distribution': 'q_log_uniform',
                        'max': math.log(256),
                        'min': math.log(32),
                        'q': 1
                    },
                    'epochs': {'worth': 5},
                    'fc_layer_size': {'values': [128, 256, 512]},
                    'learning_rate': {'distribution': 'uniform',
                                      'max': 0.1,
                                      'min': 0},
                    'optimizer': {'values': ['adam', 'sgd']}
                }
 }

First we outline the tuning methodology which is the search technique. We’ve got 3 choices: random, grid search and bayes search. The metric is the tip purpose that must be minimized. Lastly the parameters discuss with the hyperparameters to be searched by Sweeps. As you may see we are going to tune the next:

• The batch measurement to be within the [log(32), log(256)] vary. The choice of the batch measurement will comply with the quantized log uniform distribution. Different decisions can be found as effectively.

• The variety of epochs to all the time be equal to five.

• The scale of the final linear layer to be 128, 256 or 512.

• The training charge to be within the [0, 0.1] vary following the uniform distribution

• The optimizer to be SGD or Adam.

Sweeps will strive all completely different mixtures and calculate the loss for every one. Sweeps will be initialized utilizing:

sweep_id = wandb.sweep(sweep_config, venture="check")

The coaching loop ought to then be remodeled to learn from the predefined config. Check out the next code:

def practice(config=None):
    with wandb.init(venture='check', entity='serkar', config=config):
        config = wandb.config
        rework = transforms.Compose(
            [transforms.ToTensor(),
             transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
        trainset = torchvision.datasets.CIFAR10(root='./knowledge', practice=True,
                                                obtain=True, rework=rework)
        trainloader = torch.utils.knowledge.DataLoader(trainset, batch_size=config.batch_size,
                                                  shuffle=True, num_workers=2)
        testset = torchvision.datasets.CIFAR10(root='./knowledge', practice=False,
                                               obtain=True, rework=rework)
        internet = Web(config.fc_layer_size)
        internet.to(machine)
        criterion = nn.CrossEntropyLoss()
        optimizer = optim.SGD(internet.parameters(), lr=config.learning_rate)
        if config.optimizer == "sgd":
            optimizer = optim.SGD(internet.parameters(),
                                  lr=config.learning_rate, momentum=0.9)
        elif optimizer == "adam":
            optimizer = optim.Adam(internet.parameters(),
                                   lr=config.learning_rate)
        wandb.watch(internet, criterion, log="all")
        for epoch in vary(config.epochs):  
            running_loss = 0.0
            for i, knowledge in enumerate(trainloader, 0):
                inputs, labels = knowledge[0].to(machine), knowledge[1].to(machine)
                optimizer.zero_grad()
                outputs = internet(inputs)
                loss = criterion(outputs, labels)
                loss.backward()
                optimizer.step()
                running_loss += loss.merchandise()
            print('[%d, %5d] loss: %.3f' %
                (epoch + 1, i + 1, running_loss / len(trainloader)))
            wandb.log({'epoch': epoch + 1, 'loss': running_loss / len(trainloader)})
        print('Completed Coaching')

Two issues to note right here:

• The coaching loop is being wrapped with with wandb.init(venture='check', entity='serkar', config=config). That is one other approach to bind the W&B library with our code.

• We learn the config utilizing config = wandb.config after which we go every parameter within the coaching code. We’ve got to make it possible for the hyperparameters used come from the config file to ensure that Sweeps to be executed correctly.

Lastly we will execute the tuning with the beneath command.

wandb.agent(sweep_id, perform=practice, rely=5)

This instructs Sweeps to run the practice perform solely 5 instances selecting 5 random mixtures of hyperparameters. The outcomes are illustrated beneath:

Discover that we purchase one of the best outcomes for the next set of hyperparameters:

• Batch measurement = 55

• Linear layer measurement = 256

• Studying charge = 0.02131

• Optimizer = SGD

Utilizing this mix, the loss grew to become equal to 0.003.

One other very cool chart is the next one:

Right here we look at what parameters have the upper influence within the loss and the way. That is referred to as the hyperparameter significance plot. It signifies which hyperparameters had been one of the best predictors of our metrics. The function significance is derived utilizing a random forest mannequin and the correlation utilizing a linear mannequin.

Information visualization

One other function that I’m actually having fun with is knowledge visualization. W&B allow us to outline a desk of knowledge and visualize it on the platform. The desk can consist of just about something: knowledge corresponding to pictures, textual content or audio, gradients, trainable parameters, and so forth. Apart from visualization, we will additionally filter, type, group and on the whole discover the information.

To make that clear, we are going to current a easy instance. Let’s create a small desk with all the photographs from the primary batch of our knowledge and their labels. Making a desk will be completed utilizing the wandb.Desk class. To sync the desk, we should log it.

lessons = ('airplane', 'automotive', 'chook', 'cat',
           'deer', 'canine', 'frog', 'horse', 'ship', 'truck')
columns=['image','label']
knowledge = []
for i, batch in enumerate(trainloader, 0):
    inputs, labels = batch[0], batch[1]
    for j, picture in enumerate(inputs,0):
        knowledge.append([wandb.Image(image),classes[labels[j].merchandise()]])
    break
desk= wandb.Desk(knowledge=knowledge, columns=columns)
run.log({"cifar10_images": desk})

Be aware that we use the built-in knowledge sort wandb.Picture in order that we will preview the picture. As soon as we run the above code, we will examine our desk within the dashboard.

You possibly can think about that utilizing the identical logic, we will visualize virtually something.

Studies

Lastly, I need to shut this tutorial with a function that’s focused extra in the direction of groups. Studies. Studies allow us builders to prepare the completely different visualizations, talk our outcomes and doc our work.

W&B offers a WYSIWYG editor with a plethora of options. It helps markdown and latex for textual content, code snippets, in addition to quite a lot of different charts. Examples embrace: line plots, bar plots, scatter plots and extra. The workforce is working exhausting so as to add extra functionalities corresponding to embedded movies, html, audio and extra.

The experiences will be shared and edited by different individuals enabling full collaboration amongst our friends.

Conclusion

And that concludes our journey within the Weights and Biases library. W&B has grow to be one in all my private favorites and has improved my workflow lots. I extremely advocate you strive it out when you haven’t already. Extra particulars you’ll find of their documentation, which may be very effectively written. Many examples are additionally supplied of their Github repository.

Have enjoyable taking part in round with it. Tell us when you’ve got any questions or in order for you us to cowl W&B in additional element sooner or later. As all the time, please share this text when you discover it helpful. It actually issues for us as a way to preserve writing content material.

* Disclosure: Please notice that a few of the hyperlinks above is perhaps affiliate hyperlinks, and at no extra value to you, we are going to earn a fee when you resolve to make a purchase order after clicking by.