How to avoid numerical overflow in Sigmoid function: Numerically stable sigmoid function

Quick answer:

Detailed answer:

Sigmoid can be written in two mathematically equivalent form as below. Each equation takes care of overflow either for large positive or negative values, but not both.

Sigmoid implementation avoiding overflow for negative x values, but having overflow for large positive x values

Sigmoid implementation avoiding overflow for positive x values, but having overflow for large negative x values

Both these forms can be combined to avoid overflow for all values of ‘x’. So, for negative values of ‘x’ we compute equation 1 and for positive values of ‘x’ we compute equation 2

Naive implementation example: let x be -200000000

New implementation example: let x be -200000000

While developing Machine learning/ Deep learning code, this issue is most likely to occur if sigmoid function is used. I came across this error myself and found a great! solution to it in the below reference, therefore decided to share this concept. I am open to feedback!. If any inconsistency in above material, please let me know, I will analyze and will be happy to correct it!.

References:

https://timvieira.github.io/blog/post/2014/02/11/exp-normalize-trick/