There has been a massive growth of neural networks after 2005.
- There were issues with scaling:
- The original back-prop algorithm does not scale well to many layers.
- The original back-prop does not scale well to long time series
(see recurrent networks).
- Enhancements were needed.
- Modern neural networks have many hidden layers (could be something like 20 layers).
- Google Brain
has implemented networks with 1 billion weights.
- Deep learning
A series of new approaches to fixing memory,
credit assignment, weight initialisation,
more zero output nodes,
GPUs for parallel hardware,
and other modifications
led to "deep neural networks".
- Long short-term memory (LSTM).
Modification to back-prop
allows recurrent networks to scale to long sequences.
Breakthrough paper for deep neural networks:
- "A fast learning algorithm for deep belief nets".
Hinton, G. E., Osindero, S., & Teh, Y. W. (2006).
Neural computation, 18(7), 1527-1554.
Weights are initialized by training in a certain way rather than randomly.
Deep learning discovered issues in using the
and other continuous functions
as the activation function.
It has been discovered that a much simpler activation function,
has important properties for deep neural networks.
The Rectifier function is:
f(x) = max(0,x)
The "Rectifier" activation function (blue).
- Slope is 0 below x=0.
- Not differentiable at point x=0
- Slope is constant for x above 0
Impact on neural network learning:
- Using this activation function vastly increases the number of neurons outputting zero for a given input.
Meaning these neurons can be entirely ignored by the next layer
(as opposed to a system where almost all neurons are in play, even if only by a small amount).
This leads to what is called "sparse representations"
and makes very large networks computationally practical.
- It is similar to "separating the inputs".
- It is much faster to calculate than sigmoid.
Very important for scaling.
Clearly a neural network maps perfectly to
It consists almost entirely
of simple calculations that could be done
in parallel, with a CPU at each node.
It is very wasteful to implement a neural network
on serial hardware.
Modern computers already have massively parallel systems for doing simple calculations:
So implementing neural networks on GPUs
became an important part of Deep Learning.
Neural networks in JS on GPU
It allows intensive graphics calculations to be done by the GPU.
- These calculations can be any calculations, e.g. for neural networks not actually graphics.
- There are JS libraries to do neural networks calculations in JS in the browser
using the GPU for speed.
Use the GPU "backend"
to get performance maybe 100 times faster than the plain CPU "backend".
The brain has 100 billion neurons,
each with up to 15,000 connections with other neurons.
(Actually these figures include the entire nervous system,
distributed over the body,
which can be seen as an extension of the brain).
The adult brain of H.Sapiens
is the most complex known object in the universe.
Perhaps the most complex object that has ever existed.
One brain is far more complex than the entire world telephone system / Internet
(which has smaller number of nodes,
and much less connectivity).
If we considered each neuron as
roughly the equivalent of a simple CPU with 100 k of memory,
then we have 100 billion CPUs with
10,000 terabytes of memory,
all working in parallel and massively interconnected
with hundreds of trillions of connections.
It is not surprising that the brain is so complex
and at the same time consciousness and intelligence are mysterious.
What would be surprising would be if the brain
was a simple object.
- 1999 estimate:
- Ray Kurzweil,
The Age of Spiritual Machines, 1999, Ch.6,
argues that all of the world's computers plus the Internet
now match the computational capacity
(in terms of calculations per second)
of a single human brain.
- Internet - say
500 million machines times an
average of say 50 million calculations per second
= 25,000 trillion calculations per second.
- A single brain - 100 billion neurons times an average 1000 connections each,
times 200 calculations per second
= 20,000 trillion calculations per second.
Obviously the Internet's computers are not all wired up
to act as a single machine
- but they could be, in theory.
However, the question of very sparse connectivity still
puts the Internet behind a single brain.
- Still, it is encouraging
what humanity has built.
- 2008 estimate:
Kevin Kelly, Wired, July 2008
the entire planet of computers, phones, cameras, etc. to a single human brain.
Only on connectivity can the brain now compete, according to his numbers.
(Apart from the fact that the world's computers and phones are not all wired up
to act as a single machine.)
- List of animals by number of neurons
- Human - 86 billion neurons
- Elephant - 257 billion neurons
"If the human mind was simple enough to understand, we'd be too simple to understand it."
- IBM scientist Emerson Pugh.