fastbook19 A fastai Learner from Scratch
This final chapter (other than the conclusion and the online chapters) is going to look a bit different. It contains far more code and far less prose than the previous chapters. We will introduce new Python keywords and libraries without discussing them. This chapter is meant to be the start of a significant research project for you. You see, we are going to implement many of the key pieces of the fastai and PyTorch APIs from scratch, building on nothing other than the components that we developed in < The end-of-chapter questionnaire is particularly important for this chapter. This is where we will be pointing you in the many interesting directions that you could take, using this chapter as your starting point. We suggest that you follow along with this chapter on your computer, and do lots of experiments, web searches, and whatever else you need to understand what's going on. You've built up the skills and expertise to do this in the rest of this book, so we think you are going to do great! Let's begin by gathering (manually) some data. Have a look at the source to Where does 'FastDownload' come from??? As seen above, fastai or Jeremy has many packages installed with prefixed 'fast'. How should we look into such source code??? It seems that you need to 'import' here too to access the source of the indirectly called functions/classes. To access the image files, we can use Or we could do the same thing using just Python's standard library, with If you look at the source for We can open an image with the Python Imaging Library's That's going to be the basis of our independent variable. For our dependent variable, we can use ...and the reverse mapping, thanks to That's all the pieces we need to put together our A We need a list of training and validation filenames to pass to Now we can try it out: As you see, our dataset is returning the independent and dependent variables as a tuple, which is just what we need. We'll need to be able to collate these into a mini-batch. Generally this is done with Here's a mini-batch with two items, for testing our Now that we have a dataset and a collation function, we're ready to create Let's try it out with our training and validation datasets: This data loader is not much slower than PyTorch's, but it's far simpler. So if you're debugging a complex data loading process, don't be afraid to try doing things manually to help you see exactly what's going on. For normalization, we'll need image statistics. Generally it's fine to calculate these on a single training mini-batch, since precision isn't needed here: Our We always like to test everything we build in a notebook, as soon as we build it: Here PIL uses That's all we need for the data for our model. So now we need the model itself! To create a model, we'll need The implementation here is a bit awkward: we have to define the special Our Now that we have this, we can define The key functionality is in the definition of This means that we can ask any As you see, this is where we use our new Python's Other than these pieces of functionality, our Now we can create our first We're not implementing And that we can call it (which will result in In the same way, we can implement and test if it works: Let's also create a testing module to check that if we include multiple parameters as attributes, they are all correctly registered: Since we have a conv layer and a linear layer, each of which has weights and biases, we'd expect four parameters in total: We should also find that calling We can now use those pieces to create a CNN. As we've seen, a The We can create a simplified That's enough for us to create a CNN! Let's see if our parameters are all being registered correctly: Now we can try adding a hook. Note that we've only left room for one hook in We have data and model. Now we need a loss function. We've already seen how to define "negative log likelihood": Well actually, there's no log here, since we're using the same definition as PyTorch. That means we need to put the log together with softmax: Well actually, there's no log here, since we're using the same definition as PyTorch. That means we need to put the log together with softmax: Combining these gives us our cross-entropy loss: Note that the formula: gives a simplification when we compute the log softmax, which was previously defined as Then, there is a more stable way to compute the log of the sum of exponentials, called the LogSumExp trick. The idea is to use the following formula:
$$\log \left ( \sum_{j=1}^{n} e^{x_{j}} \right ) = \log \left ( e^{a} \sum_{j=1}^{n} e^{x_{j}-a} \right ) = a + \log \left ( \sum_{j=1}^{n} e^{x_{j}-a} \right )$$
where $a$ is the maximum of $x_{j}$. Here's the same thing in code: We'll put that into a function: so we can use it for our Which gives the same result as before: We can use these to create Let's now combine all those pieces together to create a We have data, a model, and a loss function; we only need one more thing before we can fit a model, and that's an optimizer! Here's SGD: As we've seen in this book, life is easier with a Now we're ready to create our This is the largest class we've created in the book, but each method is quite small, so by looking at each in turn you should be able to follow what's going on. The main method we'll be calling is and at each epoch calls Before and after each step, As you can see, In In other words, every callback knows what learner it is used in. This is critical, since otherwise a callback can't get information from the learner, or change things in the learner. Because getting information from the learner is so common, we make that easier by defining For instance, we want to move all model parameters to the GPU automatically at the start of In Before we try our Now we're ready to use our It's quite amazing to realize that we can implement all the key ideas from fastai's If we're going to get good results, we'll want an LR finder and 1cycle training. These are both annealing callbacks—that is, they are gradually changing hyperparameters as we train. Here's This shows how we're using And take a look at the results: Now we can define our We'll try an LR of 0.1: Let's fit for a while and see how it looks (we won't show all the output in the book—try it in the notebook to see the results): Finally, we'll check that the learning rate followed the schedule we defined (as you see, we're not using cosine annealing here): We have explored how the key concepts of the fastai library are implemented by re-implementing them in this chapter. Since it's mostly full of code, you should definitely try to experiment with it by looking at the corresponding notebook on the book's website. Now that you know how it's built, as a next step be sure to check out the intermediate and advanced tutorials in the fastai documentation to learn how to customize every bit of the library.Learner class, and some callbacks—enough to be able to train a model on Imagenette, including examples of each of the key techniques we've studied. On the way to building Learner, we will create our own version of Module, Parameter, and parallel DataLoader so you have a very good idea of what those PyTorch classes do.</p>
untar_data to see how it works. We'll use it here to access the 160-pixel version of Imagenette for use in this chapter:untar_data??
FastDownload?
!ls -d /root/mambaforge/lib/python3.9/site-packages/fast* | egrep -v '\-info'
import fastdownload
fastdownload.FastDownload??
URLs??
path = untar_data(URLs.IMAGENETTE_160)
get_image_files:t = get_image_files(path)
t[0]
glob:glob?
from glob import glob
files = L(glob(f'{path}/**/*.JPEG', recursive=True)).map(Path)
files[0]
get_image_files, you'll see it uses Python's os.walk; this is a faster and more flexible function than glob, so be sure to try it out.get_image_files??
get_files??
os.walk?
L(os.walk('/'))
Image class:Image.open?
im = Image.open(files[0])
im
im_t = tensor(im)
im_t.shape
Path.parent from pathlib. First we'll need our vocab:files[0]
files[0].parent
files[0].parent.name
files.map(lambda x: x.parent.name).unique()
files.map(lambda x: x.parent.name).unique().val2idx??
{v:i for i, v in enumerate(files.map(lambda x: x.parent.name).unique())}
lbls = files.map(Self.parent.name()).unique(); lbls
L.val2idx:v2i = lbls.val2idx(); v2i
Dataset.Dataset in PyTorch can be anything that supports indexing (__getitem__) and len:class Dataset:
def __init__(self, fns): self.fns=fns
def __len__(self): return len(self.fns)
def __getitem__(self, i):
im = Image.open(self.fns[i]).resize((64,64)).convert('RGB')
y = v2i[self.fns[i].parent.name]
return tensor(im).float()/255, tensor(y)
dset = Dataset(get_image_files(path)[:3])
len(dset)
X, y = dset[0]
X.shape, y
Dataset.__init__:set(o.parent.parent.name for o in files)
train_filt = L(o.parent.parent.name=='train' for o in files)
train,valid = files[train_filt],files[~train_filt]
len(train),len(valid)
train_ds,valid_ds = Dataset(train),Dataset(valid)
x,y = train_ds[0]
x.shape,y
show_image(x, title=lbls[y]);
torch.stack, which is what we'll use here:tmp = [(o, i) for i, o in enumerate("A B C D E F".split())]
a, b = zip(*tmp)
a, b
def collate(idxs, ds):
xb,yb = zip(*[ds[i] for i in idxs])
return torch.stack(xb),torch.stack(yb)
class Dataset0:
def __init__(self, l): self.l=l
def __len__(self): return len(self.l)
def __getitem__(self, i):
return torch.tensor(ord(self.l[i][0])), torch.tensor(self.l[i][1])
collate([0,1,2],
Dataset0([(o, i) for i, o in enumerate("a b c d e f".split())]))
collate:x,y = collate([1,2], train_ds)
x.shape,y
DataLoader. We'll add two more things here: an optional shuffle for the training set, and a ProcessPoolExecutor to do our preprocessing in parallel. A parallel data loader is very important, because opening and decoding a JPEG image is a slow process. One CPU core is not enough to decode images fast enough to keep a modern GPU busy. Here's our DataLoader class:L.range(9).shuffle()
class DataLoader:
def __init__(self, ds, bs=128, shuffle=False, n_workers=1):
self.ds,self.bs,self.shuffle,self.n_workers = ds,bs,shuffle,n_workers
def __len__(self): return (len(self.ds)-1)//self.bs+1
def __iter__(self):
idxs = L.range(self.ds)
if self.shuffle: idxs = idxs.shuffle()
chunks = [idxs[n:n+self.bs] for n in range(0, len(self.ds), self.bs)]
with ProcessPoolExecutor(self.n_workers) as ex:
yield from ex.map(collate, chunks, ds=self.ds)
defaults.cpus
n_workers = min(16, defaults.cpus)
%time train_dl = DataLoader(train_ds, bs=128, shuffle=True, n_workers=n_workers)
valid_dl = DataLoader(valid_ds, bs=256, shuffle=False, n_workers=n_workers)
xb,yb = first(train_dl)
xb.shape,yb.shape,len(train_dl)
stats = [xb.mean((0,1,2)),xb.std((0,1,2))]
stats
Normalize class just needs to store these stats and apply them (to see why the to_device is needed, try commenting it out, and see what happens later in this notebook):class Normalize:
def __init__(self, stats): self.stats=stats
def __call__(self, x):
if x.device != self.stats[0].device:
# self.stats = to_device(self.stats, x.device)
print(x.device, self.stats[0].device)
return (x-self.stats[0])/self.stats[1]
torch.arange(2*3*4*5).reshape(2,3,4,5).permute((0,3,1,2)).shape # swap axises
x.shape, x.permute(0,3,1,2).shape # -> (N, C, H, W)
norm = Normalize(stats)
def tfm_x(x): return norm(x).permute((0,3,1,2))
t = tfm_x(x)
t.mean((0,2,3)),t.std((0,2,3))
tfm_x isn't just applying Normalize, but is also permuting the axis order from NHWC to NCHW (see <HWC axis order, which we can't use with PyTorch, hence the need for this permute.Module. To create Module, we'll need Parameter, so let's start there. Recall that in <Parameter class "doesn't actually add any functionality (other than automatically calling requires_grad_ for us). It's only used as a "marker" to show what to include in parameters." Here's a definition which does exactly that:</p>
</div>
</div>
</div>
class Parameter(Tensor):
def __new__(self, x): return Tensor._make_subclass(Parameter, x, True)
def __init__(self, *args, **kwargs): self.requires_grad_()
__new__ Python method and use the internal PyTorch method _make_subclass because, as at the time of writing, PyTorch doesn't otherwise work correctly with this kind of subclassing or provide an officially supported API to do this. This may have been fixed by the time you read this, so look on the book's website to see if there are updated details.Parameter now behaves just like a tensor, as we wanted:Parameter(tensor(3.))
Module:class Module:
def __init__(self):
self.hook,self.params,self.children,self._training = None,[],[],False
def register_parameters(self, *ps): self.params += ps
def register_modules (self, *ms): self.children += ms
@property
def training(self): return self._training
@training.setter
def training(self,v):
self._training = v
for m in self.children: m.training=v
def parameters(self):
return self.params + sum([m.parameters() for m in self.children], [])
def __setattr__(self,k,v):
super().__setattr__(k,v)
if isinstance(v,Parameter): self.register_parameters(v)
if isinstance(v,Module): self.register_modules(v)
def __call__(self, *args, **kwargs):
res = self.forward(*args, **kwargs)
if self.hook is not None: self.hook(res, args)
return res
def cuda(self):
for p in self.parameters(): p.data = p.data.cuda()
parameters:self.params + sum([m.parameters() for m in self.children], [])
Module for its parameters, and it will return them, including all its child modules (recursively). But how does it know what its parameters are? It's thanks to implementing Python's special __setattr__ method, which is called for us any time Python sets an attribute on a class. Our implementation includes this line:if isinstance(v,Parameter): self.register_parameters(v)
Parameter class as a "marker"—anything of this class is added to our params.__call__ allows us to define what happens when our object is treated as a function; we just call forward (which doesn't exist here, so it'll need to be added by subclasses). Before we do, we'll call a hook, if it's defined. Now you can see that PyTorch hooks aren't doing anything fancy at all—they're just calling any hooks that have been registered.Module also provides cuda and training attributes, which we'll use shortly.Module, which is ConvLayer:class ConvLayer(Module):
def __init__(self, ni, nf, stride=1, bias=True, act=True):
super().__init__()
self.w = Parameter(torch.zeros(nf,ni,3,3))
self.b = Parameter(torch.zeros(nf)) if bias else None
self.act,self.stride = act,stride
init = nn.init.kaiming_normal_ if act else nn.init.xavier_normal_
init(self.w)
def forward(self, x):
x = F.conv2d(x, self.w, self.b, stride=self.stride, padding=1)
if self.act: x = F.relu(x)
return x
F.conv2d from scratch, since you should have already done that (using unfold) in the questionnaire in <Module.parameters:</p>
</div>
</div>
</div>
l = ConvLayer(3, 4)
l.parameters()[0].shape, l.parameters()[1].shape
forward being called):xbt = tfm_x(xb)
xb.shape, xbt.shape
r = l(xbt)
r.shape
Linear:class Linear(Module):
def __init__(self, ni, nf):
super().__init__()
self.w = Parameter(torch.zeros(nf,ni))
self.b = Parameter(torch.zeros(nf))
nn.init.xavier_normal_(self.w)
def forward(self, x): return x@self.w.t() + self.b
l = Linear(4,2)
r = l(torch.ones(3,4))
r.shape
class T(Module):
def __init__(self):
super().__init__()
self.c,self.l = ConvLayer(3,4),Linear(4,2)
t = T()
t.children, len(t.children[0].parameters()), len(t.children[1].parameters())
len(t.parameters())
cuda on this class puts all these parameters on the GPU:t.l.w.device
t.cuda()
t.l.w.device
Sequential class makes many architectures easier to implement, so let's make one:class Sequential(Module):
def __init__(self, *layers):
super().__init__()
self.layers = layers
self.register_modules(*layers)
def forward(self, x):
for l in self.layers: x = l(x)
return x
forward method here just calls each layer in turn. Note that we have to use the register_modules method we defined in Module, since otherwise the contents of layers won't appear in parameters.register_modules does, or why it’s needed, have another look at our code for Module to see what we wrote!
AdaptivePool that only handles pooling to a 1×1 output, and flattens it as well, by just using mean:class AdaptivePool(Module):
def forward(self, x): return x.mean((2,3))
def simple_cnn():
return Sequential(
ConvLayer(3 ,16 ,stride=2), #32
ConvLayer(16,32 ,stride=2), #16
ConvLayer(32,64 ,stride=2), # 8
ConvLayer(64,128,stride=2), # 4
AdaptivePool(),
Linear(128, 10)
)
m = simple_cnn()
len(m.parameters())
[len(o.parameters()) for o in m.children]
sum([len(o.parameters()) for o in m.children])
Module; you could make it a list, or use something like Pipeline to run a few as a single function:def print_stats(outp, inp): print (outp.mean().item(),outp.std().item())
for i in range(4): m.layers[i].hook = print_stats
r = m(xbt)
r.shape
m.layers
m.children
m
def print_stats(outp, inp): print (outp.mean().item(),outp.std().item())
for o in m.layers: o.hook = print_stats
r = m(xbt)
r.shape
def print_stats(outp, inp): if self.act: print(act)
for o in m.layers: o.hook = print_stats
r = m(xbt)
r.shape
def nll(input, target): return -input[range(target.shape[0]), target].mean()
batch_size = 5
n_classes = 3
input = torch.randn(batch_size, n_classes).relu()
input
target = torch.randint(0, n_classes, (batch_size,))
target
-input[range(batch_size), target]
-input[range(batch_size), target].mean()
nll(input, target)
def log_softmax(x): return (x.exp()/(x.exp().sum(-1,keepdim=True))).log()
sm = log_softmax(r); sm[0][0]
loss = nll(sm, yb)
loss
(x.exp()/(x.exp().sum(-1))).log():def log_softmax(x): return x - x.exp().sum(-1,keepdim=True).log()
sm = log_softmax(r); sm[0][0]
x = torch.rand(5)
a = x.max()
x.exp().sum().log() == a + (x-a).exp().sum().log()
def logsumexp(x):
m = x.max(-1)[0]
return m + (x-m[:,None]).exp().sum(-1).log()
logsumexp(r)[0]
log_softmax function:def log_softmax(x): return x - x.logsumexp(-1,keepdim=True)
sm = log_softmax(r); sm[0][0]
cross_entropy:def cross_entropy(preds, yb): return nll(log_softmax(preds), yb).mean()
Learner.class SGD:
def __init__(self, params, lr, wd=0.): store_attr()
def step(self):
for p in self.params:
p.data -= (p.grad.data + p.data*self.wd) * self.lr
p.grad.data.zero_()
Learner. The Learner class needs to know our training and validation sets, which means we need DataLoaders to store them. We don't need any other functionality, just a place to store them and access them:class DataLoaders:
def __init__(self, *dls): self.train,self.valid = dls
dls = DataLoaders(train_dl,valid_dl)
Learner class:store_attr??
self?
class Learner:
def __init__(self, model, dls, loss_func, lr, cbs, opt_func=SGD):
store_attr()
for cb in cbs: cb.learner = self
def one_batch(self):
self('before_batch')
xb,yb = self.batch
self.preds = self.model(xb)
self.loss = self.loss_func(self.preds, yb)
if self.model.training:
self.loss.backward()
self.opt.step()
self('after_batch')
def one_epoch(self, train):
self.model.training = train
self('before_epoch')
dl = self.dls.train if train else self.dls.valid
#def progress_bar(dl, leave): return dl
for self.num,self.batch in enumerate(progress_bar(dl, leave=False)):
self.one_batch()
self('after_epoch')
def fit(self, n_epochs):
self('before_fit')
self.opt = self.opt_func(self.model.parameters(), self.lr)
self.n_epochs = n_epochs
try:
for self.epoch in range(n_epochs):
self.one_epoch(True)
self.one_epoch(False)
except CancelFitException: pass
self('after_fit')
def __call__(self,name):
for cb in self.cbs: getattr(cb,name,noop)()
fit. This loops with:for self.epoch in range(n_epochs)
self.one_epoch for each of train=True and then train=False. Then self.one_epoch calls self.one_batch for each batch in dls.train or dls.valid, as appropriate (after wrapping the DataLoader in fastprogress.progress_bar. Finally, self.one_batch follows the usual set of steps to fit one mini-batch that we've seen throughout this book.Learner calls self, which calls __call__ (which is standard Python functionality). __call__ uses getattr(cb,name) on each callback in self.cbs, which is a Python built-in function that returns the attribute (a method, in this case) with the requested name. So, for instance, self('before_fit') will call cb.before_fit() for each callback where that method is defined.Learner is really just using our standard training loop, except that it's also calling callbacks at appropriate times. So let's define some callbacks!Learner.__init__ we have:for cb in cbs: cb.learner = self
Callback as a subclass of GetAttr, with a default attribute of learner:class Callback(GetAttr): _default='learner'
GetAttr is a fastai class that implements Python's standard __getattr__ and __dir__ methods for you, such that any time you try to access an attribute that doesn't exist, it passes the request along to whatever you have defined as _default.fit. We could do this by defining before_fit as self.learner.model.cuda(); however, because learner is the default attribute, and we have SetupLearnerCB inherit from Callback (which inherits from GetAttr), we can remove the .learner and just call self.model.cuda():class SetupLearnerCB(Callback):
def before_batch(self):
xb,yb = to_device(self.batch)
self.learner.batch = tfm_x(xb),yb
def before_fit(self): self.model.cuda()
SetupLearnerCB we also move each mini-batch to the GPU, by calling to_device(self.batch) (we could also have used the longer to_device(self.learner.batch). Note however that in the line self.learner.batch = tfm_x(xb),yb we can't remove .learner, because here we're setting the attribute, not getting it.Learner out, let's create a callback to track and print progress. Otherwise we won't really know if it's working properly:class TrackResults(Callback):
def before_epoch(self): self.accs,self.losses,self.ns = [],[],[]
def after_epoch(self):
n = sum(self.ns)
print(self.epoch, self.model.training,
sum(self.losses).item()/n, sum(self.accs).item()/n)
def after_batch(self):
xb,yb = self.batch
acc = (self.preds.argmax(dim=1)==yb).float().sum()
self.accs.append(acc)
n = len(xb)
self.losses.append(self.loss*n)
self.ns.append(n)
Learner for the first time!cbs = [SetupLearnerCB(),TrackResults()]
learn = Learner(simple_cnn(), dls, cross_entropy, lr=0.1, cbs=cbs)
learn.fit(1)
Learner in so little code! Let's now add some learning rate scheduling.LRFinder:class LRFinder(Callback):
def before_fit(self):
self.losses,self.lrs = [],[]
self.learner.lr = 1e-6
def before_batch(self):
if not self.model.training: return
self.opt.lr *= 1.2
def after_batch(self):
if not self.model.training: return
if self.opt.lr>10 or torch.isnan(self.loss): raise CancelFitException
self.losses.append(self.loss.item())
self.lrs.append(self.opt.lr)
CancelFitException, which is itself an empty class, only used to signify the type of exception. You can see in Learner that this exception is caught. (You should add and test CancelBatchException, CancelEpochException, etc. yourself.) Let's try it out, by adding it to our list of callbacks:lrfind = LRFinder()
learn = Learner(simple_cnn(), dls, cross_entropy, lr=0.1, cbs=cbs+[lrfind])
learn.fit(2)
plt.plot(lrfind.lrs[:-2],lrfind.losses[:-2])
plt.xscale('log')
OneCycle training callback:class OneCycle(Callback):
def __init__(self, base_lr): self.base_lr = base_lr
def before_fit(self): self.lrs = []
def before_batch(self):
if not self.model.training: return
n = len(self.dls.train)
bn = self.epoch*n + self.num
mn = self.n_epochs*n
pct = bn/mn
pct_start,div_start = 0.25,10
if pct<pct_start:
pct /= pct_start
lr = (1-pct)*self.base_lr/div_start + pct*self.base_lr
else:
pct = (pct-pct_start)/(1-pct_start)
lr = (1-pct)*self.base_lr
self.opt.lr = lr
self.lrs.append(lr)
onecyc = OneCycle(0.1)
learn = Learner(simple_cnn(), dls, cross_entropy, lr=0.1, cbs=cbs+[onecyc])
learn.fit(8)
plt.plot(onecyc.lrs);
glob?L.map do?Self do?L.val2idx?Dataset?convert when we open an image from Imagenette?~ do? How is it useful for splitting training and validation sets?~ work with the L or Tensor classes? What about NumPy arrays, Python lists, or pandas DataFrames?ProcessPoolExecutor?L.range(self.ds) work?__iter__?first?permute? Why is it needed?parameters method?super?Module need to override forward instead of defining __call__?ConvLayer, why does init depend on act?Sequential need to call register_modules?log_softmax useful?GetAttr? How is it helpful for callbacks?Callback or GetAttr.Learner.__call__ do?getattr? (Note the case difference to GetAttr!)try block in fit?model.training in one_batch?store_attr?TrackResults.before_epoch?model.cuda do? How does it work?model.training in LRFinder and OneCycle?OneCycle.
resnet18 from scratch (refer to <Learner in this chapter.</li>
resnet18.
nbdev to create and deploy your package.