3D Medical image segmentation with transformers tutorial

Transformers are an enormous pattern in laptop imaginative and prescient. I lately gave an outline of some wonderful developments. This time I’ll use my re-implementation of a transformer-based mannequin for 3D segmentation. Specifically, I’ll use the well-known UNETR transformer and attempt to see if it performs on par with a classical UNET. The pocket book is on the market.

UNETR is the primary profitable transformer structure for 3D medical picture segmentation. On this weblog publish, I’ll attempt to match the outcomes of a UNET mannequin on the BRATS dataset, which comprises 3D MRI mind photos. Here’s a high-level overview of UNETR that we’ll prepare on this tutorial:

Supply: UNETR: Transformers for 3D Medical Picture Segmentation, Hatamizadeh et al.

To check my implementation I used an present tutorial on a 3D MRI segmentation dataset. Thus, I’ve to offer credit score to the wonderful open-source library of Nvidia referred to as MONAI for offering the preliminary tutorial that I modified for instructional functions. In case you are into medical imaging you’ll want to take a look at this superior library and its tutorials.

Let’s have a look at the info first!

Replace: E-book launch! Study “Deep studying in manufacturing” to serve your ML fashions to tens of millions of customers.

BRATS dataset

BRATS is a multi-modal large-scale 3D imaging dataset. It comprises 4 3D volumes of MRI photos captured below completely different modalities and setups. Here’s a pattern of the dataset. It is very important see that solely the tumor is annotated. This makes issues similar to segmentation tougher because the mannequin has to localize on the tumor.

Official information teaser picture from the BRATS completion web site

The picture patches depict tumor classes as follows (from left to proper):

1. Edema: The entire tumor (yellow) is often seen in T2-FLAIR MRI picture.

2. Non-enhancing stable core: The tumor core (crimson) seen in T2 MRI.

3. The enhancing tumor constructions (gentle blue). Often seen in T1Gd, surrounding the necrotic core (inexperienced).

4. The segmentations are mixed to generate the ultimate labels of the dataset.

With MONAI, loading a dataset from the medical imaging decathlon competitors turns into trivial.

Information loading with MONAI and transformations

By using the DecathlonDataset class of MONAI library one can load any of the ten accessible datasets from the web site. We are going to use Task01_BrainTumour in our case.

cache_num = 8
from monai.apps import DecathlonDataset
train_ds = DecathlonDataset(
   root_dir=root_dir,
   activity="Task01_BrainTumour",
   rework=train_transform,
   part="coaching",
   obtain=True,
   num_workers=4,
   cache_num=cache_num,
)
train_loader = DataLoader(train_ds, batch_size=2, shuffle=True, num_workers=2)
val_ds = DecathlonDataset(
   root_dir=root_dir,
   activity="Task01_BrainTumour",
   rework=val_transform,
   part="validation",
   obtain=False,
   num_workers=4,
   cache_num=cache_num,
)
val_loader = DataLoader(val_ds, batch_size=2, shuffle=False, num_workers=2)

Imports and supporting capabilities will be discovered within the pocket book. What’s essential right here is the transformation pipeline, which I assure is just not a simple factor in 3D photos. MONAI supplies some capabilities to make a quick pipeline for the aim of this tutorial. Particulars just like the picture orientation are not noted of the tutorial on objective.

Briefly, we’ll resample our photos to a voxel dimension of 1.5, 1.5, and a couple of.0 mm in every dimension. Afterwards, we take random 3D sub-volumes of sizes 128, 128, 64. This after all must be utilized to each the enter picture and the segmentation masks.

Then a few augmentations are utilized similar to randomly flipping the primary axis, and rescaling the depth (jittering).

The category ConvertToMultiChannelBasedOnBratsClassesd brings the labels to the format that we would like.

from monai.transforms import (
   Activations,
   AsChannelFirstd,
   AsDiscrete,
   CenterSpatialCropd,
   Compose,
   LoadImaged,
   MapTransform,
   NormalizeIntensityd,
   Orientationd,
   RandFlipd,
   RandScaleIntensityd,
   RandShiftIntensityd,
   RandSpatialCropd,
   Spacingd,
   ToTensord,
)
roi_size=[128, 128, 64]
pixdim=(1.5, 1.5, 2.0)
class ConvertToMultiChannelBasedOnBratsClassesd(MapTransform):
   """
   Convert labels to multi channels based mostly on brats courses:
   label 1 is the peritumoral edema
   label 2 is the GD-enhancing tumor
   label 3 is the necrotic and non-enhancing tumor core
   The potential courses are TC (Tumor core), WT (Entire tumor)
   and ET (Enhancing tumor).
   """
   def __call__(self, information):
       d = dict(information)
       for key in self.keys:
           end result = []
           end result.append(np.logical_or(d[key] == 2, d[key] == 3))
           end result.append(
               np.logical_or(
                   np.logical_or(d[key] == 2, d[key] == 3), d[key] == 1
               )
           )
           end result.append(d[key] == 2)
           d[key] = np.stack(end result, axis=0).astype(np.float32)
       return d
train_transform = Compose(
   [
       LoadImaged(keys=["image", "label"]),
       AsChannelFirstd(keys="picture"),
       ConvertToMultiChannelBasedOnBratsClassesd(keys="label"),
       Spacingd(
           keys=["image", "label"],
           pixdim=pixdim,
           mode=("bilinear", "nearest"),
       ),
       Orientationd(keys=["image", "label"], axcodes="RAS"),
       RandSpatialCropd(
           keys=["image", "label"], roi_size=roi_size, random_size=False),
       RandFlipd(keys=["image", "label"], prob=0.5, spatial_axis=0),
       NormalizeIntensityd(keys="picture", nonzero=True, channel_wise=True),
       RandScaleIntensityd(keys="picture", elements=0.1, prob=0.5),
       RandShiftIntensityd(keys="picture", offsets=0.1, prob=0.5),
       ToTensord(keys=["image", "label"]),
   ]
)
val_transform = Compose(
   [
       LoadImaged(keys=["image", "label"]),
       AsChannelFirstd(keys="picture"),
       ConvertToMultiChannelBasedOnBratsClassesd(keys="label"),
       Spacingd(
           keys=["image", "label"],
           pixdim=pixdim,
           mode=("bilinear", "nearest"),
       ),
       Orientationd(keys=["image", "label"], axcodes="RAS"),
       CenterSpatialCropd(keys=["image", "label"], roi_size=roi_size),
       NormalizeIntensityd(keys="picture", nonzero=True, channel_wise=True),
       ToTensord(keys=["image", "label"]),
   ]
)

It is at all times higher to see the pipeline in motion, by visualizing some slices from all of the modalities. Under is a pattern of our prepare information:

Supply: Picture from the writer based mostly on the pocket book

It may be noticed that the tumor are not mutually unique. On this regard we count on the enhancing tumor and necrotic cells (rightmost segmentation map) to be probably the most tough to foretell.

The information and transformation pipeline are actually all set. Let’s take a more in-depth have a look at the mannequin’s structure.

Study extra about AI utilized in medical imaging purposes from the well-structured course AI for Medication provided by Coursera.

The UNETR structure

Right here is the mannequin structure that includes transformers into the notorious UNET structure:

Supply: UNETR: Transformers for 3D Medical Picture Segmentation, Hatamizadeh et al.

Curiously, I started to implement this mannequin as within the paper determine depicted above. Afterward, I found that it was already carried out in MONAI. After checking their code I discovered vital particulars lacking. Conclusion: don’t belief the structure photos, they don’t embody all of the story on implement the paper. To see the implementation code, take a look at my implementation within the self-attention-cv library.

Now I can lastly use my implementation of UNETR. I’ve created a small library that implements a number of self-attention blocks for laptop imaginative and prescient and packs them in a pip-installable bundle. So now I solely have to put in my pip bundle that comprises the mannequin and voila:

$ pip set up self-attention-cv==1.2.3

To initialize the mannequin we have to present the amount dimension, the enter imaging modalities, the variety of labels (output_dim) and a number of other issues relating to the imaginative and prescient transformer. Examples embody embedding patch dimension, patch dimension, variety of heads, normalization sort and so on.

from self_attention_cv import UNETR
machine = torch.machine("cuda:0")
num_heads = 10
embed_dim= 512
mannequin = UNETR(img_shape=tuple(roi_size), input_dim=4, output_dim=3,
                embed_dim=embed_dim, patch_size=16, num_heads=num_heads,
                ext_layers=[3, 6, 9, 12], norm='occasion',
                base_filters=16,
                dim_linear_block=2048).to(machine)

I’m nonetheless undecided why occasion normalization works very properly with UNETs and multi-model datasets, but it surely does! The purpose is that we now have our 49.7 million parameter mannequin able to be skilled.

We are going to use the DICE loss mixed with cross-entropy, and make a easy coaching loop:

import torch.nn as nn
from monai.losses import DiceLoss, DiceCELoss
loss_function = DiceCELoss(to_onehot_y=False, sigmoid=True)
optimizer = torch.optim.AdamW(mannequin.parameters(), lr=1e-4, weight_decay=1e-5)
max_epochs = 180
val_interval = 5
best_metric = -1
best_metric_epoch = -1
epoch_loss_values = []
for epoch in vary(max_epochs):
   print(f"epoch {epoch + 1}/{max_epochs}")
   mannequin.prepare()
   epoch_loss = 0
   step = 0
   for batch_data in train_loader:
       step += 1
       inputs, labels = (
           batch_data["image"].to(machine),
           batch_data["label"].to(machine),
       )
       optimizer.zero_grad()
       outputs = mannequin(inputs)
       loss = loss_function(outputs, labels)
       loss.backward()
       optimizer.step()
       epoch_loss += loss.merchandise()
   epoch_loss /= step
   epoch_loss_values.append(epoch_loss)
   print(f"epoch {epoch + 1} common loss: {epoch_loss:.4f}")

Baseline comparability: UNET

Nonetheless, the largest query right here is how good this mannequin can carry out. For that motive, we want a robust baseline! What’s higher than the well-configured UNET that’s used within the preliminary tutorial?

I additionally in contrast my implementation with MONAI’s UNETR implementation. Why? As a result of there could be no that means if I match the efficiency of the UNET baseline and nonetheless carry out inferior to the official implementation. In spite of everything, I modified my code to replicate the architectural modifications of the official code. And certainly I noticed enormous beneficial properties in efficiency in comparison with a simplistic implementation from the paper’s determine.

from monai.networks.nets import UNet
mannequin = UNet(
   dimensions=3,
   in_channels=4,
   out_channels=3,
   channels=(16, 32, 64, 128, 256),
   strides=(2, 2, 2, 2),
   num_res_units=2,
).to(machine)

Let’s have a look at the quantity’s first:

Mannequin	epochs	Imply DICE coeff.
UNET (baseline)	170	76.6 %
UNETR (self-attention-cv)	180	76.9 %
UNETR (MONAI)	180	76.1 %

To trace coaching we measure the coaching loss from each cube loss and cross-entropy. We additionally report the cube coefficients for the three labels (channels), specifically Tumor Core (TC), Entire Tumor (WT), and Enhancing Tumor (EC).

Under you possibly can see these metrics whereas coaching:

Supply: Picture from the writer based mostly on the pocket book

Lastly, one can see the outcomes by evaluating the output segmentation map in comparison with the bottom fact:

Supply: Picture from the writer based mostly on the pocket book

The channel of the necrotic space is omitted as a result of this explicit slice had nearly no occurrences of this label. This illustration is just a center slice of the 3D segmentation map, so it’s definitely not the entire image. Nonetheless, it offers you the sense of how the skilled mannequin supplies a extra smothered model of the unique label, which was annotated by an professional radiologist. As a result of as at all times neural networks love easy optimization areas.

Conclusion and considerations

I’m not but satisfied by the efficiency of transformers in 3D medical imaging. I imagine extra superior strategies and different contributions will comply with up. But I admit that it’s the primary attention-grabbing work that challenges the well-configured UNET architectures, that are the go-to choice in these duties.

From the above evaluation, I discover it essential to spotlight additionally that an important facet to get a great efficiency, right here Cube coefficient, is the info preprocessing and transformation pipelines. That’s precisely why I see restricted innovation within the medical imaging world when it comes to machine studying modelling, and extra promising work on information processing optimization. That alone causes no concern in any respect, but it surely makes me very suspicious when a brand new paper comes out and claims a brand new structure. As a result of the comparisons are sometimes not truthful in area of interest domains I occur to have labored on similar to medical imaging.

As at all times, thanks on your curiosity in AI and keep tuned for extra. We’re proud to share with you our e-book on “Deep studying in manufacturing”, which teaches you put your mannequin in manufacturing and scale it up. Group assist (like social media sharing) is at all times appreciated.

* Disclosure: Please observe that among the hyperlinks above is perhaps affiliate hyperlinks, and at no extra price to you, we’ll earn a fee in case you resolve to make a purchase order after clicking by way of.