I am using a validation set to determine stop training when the validation loss stops decreasing. I have my validation patience as 3. Assuming that my networks stops training when it has surpassed this patience threshold, during the final 3 validation steps of my network would have been overfitting.
As a result, I want to recover the network state at the step that produced the minimum validation loss. Is there a way to do this?