Learning Rate Finder¶
%matplotlib inline
from fastai.gen_doc.nbdoc import *
from fastai.vision import *
from fastai.callbacks import *
Learning rate finder plots lr vs loss relationship for a Learner. The idea is to reduce the amount of guesswork on picking a good starting learning rate.
Overview:
- First run lr_find
learn.lr_find() - Plot the learning rate vs loss
learn.recorder.plot() - Pick a learning rate before it diverges then start training
Technical Details: (first described by Leslie Smith)
Train
Learnerover a few iterations. Start with a very lowstart_lrand change it at each mini-batch until it reaches a very highend_lr.Recorderwill record the loss at each iteration. Plot those losses against the learning rate to find the optimal value before it diverges.
Choosing a good learning rate¶
For a more intuitive explanation, please check out Sylvain Gugger's post
path = untar_data(URLs.MNIST_SAMPLE)
data = ImageDataBunch.from_folder(path)
def simple_learner(): return Learner(data, simple_cnn((3,16,16,2)), metrics=[accuracy])
learn = simple_learner()
First we run this command to launch the search:
show_doc(Learner.lr_find)
learn.lr_find(stop_div=False, num_it=200)
LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.
Then we plot the loss versus the learning rates. We're interested in finding a good order of magnitude of learning rate, so we plot with a log scale.
learn.recorder.plot()
Then, we choose a value that is approximately in the middle of the sharpest downward slope. This is given as an indication by the LR Finder tool, so let's try 1e-2.
simple_learner().fit(2, 1e-2)
| epoch | train_loss | valid_loss | accuracy | time |
|---|---|---|---|---|
| 1 | 0.127434 | 0.070243 | 0.973013 | 00:02 |
| 2 | 0.050703 | 0.039493 | 0.984789 | 00:02 |
Don't just pick the minimum value from the plot!
learn = simple_learner()
simple_learner().fit(2, 1e-0)
| epoch | train_loss | valid_loss | accuracy | time |
|---|---|---|---|---|
| 1 | 0.727221 | 0.693147 | 0.495584 | 00:02 |
| 2 | 0.693826 | 0.693147 | 0.495584 | 00:02 |
Picking a value before the downward slope results in slow training:
learn = simple_learner()
simple_learner().fit(2, 1e-3)
| epoch | train_loss | valid_loss | accuracy | time |
|---|---|---|---|---|
| 1 | 0.152897 | 0.134366 | 0.950932 | 00:02 |
| 2 | 0.120961 | 0.117550 | 0.960746 | 00:02 |
Suggested LR¶
If you pass suggestion=True in learn.recorder.plot, you will see the point where the gardient is the steepest with a
red dot on the graph. We can use that point as a first guess for an LR.
learn.lr_find(stop_div=False, num_it=200)
LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.
learn.recorder.plot(suggestion=True)
Min numerical gradient: 5.25E-03
You can access the corresponding learning rate like this:
min_grad_lr = learn.recorder.min_grad_lr
min_grad_lr
0.005248074602497722
learn = simple_learner()
simple_learner().fit(2, min_grad_lr)
| epoch | train_loss | valid_loss | accuracy | time |
|---|---|---|---|---|
| 1 | 0.109475 | 0.081607 | 0.970559 | 00:02 |
| 2 | 0.070303 | 0.050977 | 0.982826 | 00:02 |
show_doc(LRFinder)
class LRFinder[source][test]
LRFinder(learn:Learner,start_lr:float=*1e-07,end_lr:float=10,num_it:int=100,stop_div:bool=True*) ::LearnerCallback
No tests found for LRFinder. To contribute a test please refer to this guide and this discussion.
Causes learn to go on a mock training from start_lr to end_lr for num_it iterations.
Callback methods¶
You don't call these yourself - they're called by fastai's Callback system automatically to enable the class's functionality.
show_doc(LRFinder.on_train_begin)
on_train_begin[source][test]
on_train_begin(pbar, ****kwargs**:Any)
No tests found for on_train_begin. To contribute a test please refer to this guide and this discussion.
Initialize optimizer and learner hyperparameters.
show_doc(LRFinder.on_batch_end)
on_batch_end[source][test]
on_batch_end(iteration:int,smooth_loss:TensorOrNumber, ****kwargs**:Any)
No tests found for on_batch_end. To contribute a test please refer to this guide and this discussion.
Determine if loss has runaway and we should stop.
show_doc(LRFinder.on_epoch_end)
on_epoch_end[source][test]
on_epoch_end(****kwargs**:Any)
No tests found for on_epoch_end. To contribute a test please refer to this guide and this discussion.
Called at the end of an epoch.
show_doc(LRFinder.on_train_end)
on_train_end[source][test]
on_train_end(epoch:int,num_batch:int, ****kwargs**:Any)
No tests found for on_train_end. To contribute a test please refer to this guide and this discussion.
Cleanup learn model weights disturbed during LRFinder exploration.