When does Back Propagation terminate?
Avoid cost at all costs.
Back Propagation is not designed to run in an eternal manner, and its termination is an interesting chapter by itself, which should be paid a respected attention. Generally, deciding whether this powerful and effective algorithm should stop is a matter which can be examined by the use of two parameters:
- Synaptic Weights
- Loss (J), which is usually the most commonly used.
Essentially, J can be used in order to define two basic yet effective criterion
Criterion #1.1
If loss J(n), where n is an epoch, is reduced below a specified threshold, then stop.
As a general rule, calculating J(n) is immediately tied with keeping the weights fixed and calculating the error for each single input p:
where d is the target value and y the network’s output respectively.
Afterwards, the sum of those errors is our requested value, hence
Criterion #1.2
The error between two consecutive epochs, n-1 and n, is not drastically reduced.
where ε is again a predefined threshold by the user.
If puzzled, consider the weights.
There is also an alternative way of manifesting the true colors of the algorithm’s performance, and that’s by making use of the weights themselves.
Criterion #2
The changes conducted on weights at a specific epoch n are of trivial importance.
Now, this requires a bit of your attention. In essence, this criterion is divided in two variations of a single solution, where the common parameters is the weights and the usual user-set threshold, while the difference is that the first one compares the squared weights to the latter,
while the second one takes their absolute value,
Final Thoughts
I’m a firm believer of a good grasp on theory, whether it’s about music, history, or math oriented one. Although I still have lots of ground to cover, I strongly suggest that it’s up to our personal efforts to learn how those algorithms work at their core so that we can evaluate performance and have a more-than-just-gut approach on its shortcomings (or lucky successes), despite many of those procedures having taken a second class seat due to our fully automated libraries and software. Consequently, I hope to have helped you with this short bit of information.
References
K. Diamantaras, Artificial Neural Networks, Publisher: Kleidarithmos, 2007
