Basic introduction

###Neural Network History

The early period of ANN

In 1943, Warren McCulloch and Walter Pitts proposed a mathematical model for neural networks known as “threshold logic.”
This model paved the way for neural network research to split into two approaches. One approach focused on biological processes in the brain while the other focused on the application of neural networks to artificial intelligence. This work led to work on nerve networks and their link to finite automata.[3]
In 1954, Farley and Clark first used computational machines, then called “calculators”, to simulate a Hebbian network. Other neural network computational machines were created by Rochester, Holland, Habit and Duda in 1956.

The first wave of ANN

In 1958, Rosenblatt created the perceptron, an algorithm for pattern recognition. With mathematical notation, he described circuitry not in the basic perceptron, such as the exclusive-or circuit that could not be processed by neural networks at the time.
In 1959, a biological model proposed by Nobel laureates Hubel and Wiesel was based on their discovery of two types of cells in the primary visual cortex: simple cells and complex cells
Next, in 1965, the first functional networks with many layers were published by Ivakhnenko and Lapa, becoming the group method of data handling.
Neural network research stagnated after machine learning research by Minsky and Papert (1969), who discovered two key issues with the computational machines that processed neural networks.
The first was that basic perceptrons were incapable of processing the exclusive-or circuit.
The second was that computers didn’t have enough processing power to effectively handle the work required by large neural networks. Neural network research slowed until computers achieved far greater processing power.

The second wave of ANN

In 1975, a key trigger for the renewed interest in neural networks and learning was Paul Werbos’s backpropagation algorithm that effectively solved the exclusive-or problem and more generally accelerated the training of multi-layer networks.[8][further explanation needed]
In the mid-1980s, parallel distributed processing became popular under the name connectionism. Rumelhart and McClelland (1986) described the use of connectionism to simulate neural processes.
Much of artificial intelligence had focused on high-level (symbolic) models that are processed by using algorithms, characterized for example by expert systems with knowledge embodied in if-then rules, until in the late 1980s research expanded to low-level (sub-symbolic) machine learning, characterized by knowledge embodied in the parameters of a cognitive model. From the 1990s to 2000s, support vector machines and other much simpler methods, such as linear classifiers, gradually overtook neural networks in machine learning popularity.

The third wave of ANN

Note that in 1992, max-pooling was introduced to help with least shift invariance and tolerance to deformation to aid in 3D object recognition.
The vanishing gradient problem affects many-layered feedforward networks that use backpropagation and also recurrent neural networks. As errors propagate from layer to layer, they shrink exponentially with the number of layers, impeding the tuning of neuron weights that is based on those errors, particularly affecting deep networks. To overcome this problem, Schmidhuber’s multi-level hierarchy of networks (1992) pre-trained one level at a time by unsupervised learning, fine-tuned by backpropagation.

The fourth wave of ANN

In 2003, Behnke relied only on the sign of the gradient (Rprop) on problems such as image reconstruction and face localization.
Then, Hinton et al. (2006) employed learning the distribution of a high-level representation using successive layers of binary or real-valued latent variables with a restricted Boltzmann machine to model each layer. Once sufficiently many layers have been learned, the deep architecture may be used as a generative model by reproducing the data when sampling down the model (an “ancestral pass”) from the top level feature activations.
As of 2011, the state of the art in deep and feedforward networks, particularly convolutional neural networks, alternated convolutional layers and max-pooling layers, topped by several fully or sparsely connected layers followed by a final classification layer. Learning is usually done without unsupervised pre-training.
In 2012, Ng and Dean created a neural network that learned to recognize higher-level concepts, such as cats, only from watching unlabeled images taken from YouTube videos.
Such supervised deep learning methods were the first artificial pattern recognizers to achieve human-competitive performance on certain tasks.

Papers

A history of Bayesian neural networks

intro: by Zoubin Ghahramani, University of Cambridge. NIPS 2016 Bayesian Deep Learning
slides: http://bayesiandeeplearning.org/slides/nips16bayesdeep.pdf

Calculus and Backpropagation

paper: https://github.com/mtomassoli/papers/blob/master/backprop.pdf

A Tutorial about Random Neural Networks in Supervised Learning

intro: Neural Network World
arxiv: https://arxiv.org/abs/1609.04846

A Fast C++ Implementation of Neural Network Backpropagation Training Algorithm: Application to Bayesian Optimal Image Demosaicking

project page: http://www.ipol.im/pub/art/2015/137/
paper: http://www.ipol.im/pub/art/2015/137/article_lr.pdf
code: http://www.ipol.im/pub/art/2015/137/NN.tar.gz

CrAIg: Using Neural Networks to learn Mario

Blogs

Artificial Neural Networks (ANN) Introduction

blog: https://annalyzin.wordpress.com/2016/03/13/how-do-computers-recognise-handwriting-using-artificial-neural-networks/

Getting It Done - What I learnt from finishing the Neural Network Algorithm

Learning How To Code Neural Networks

blog: https://medium.com/learning-new-stuff/how-to-learn-neural-networks-758b78f2736e#.nc2bx42ct

Watch Tiny Neural Nets Learn

http://swanintelligence.com/watch-tiny-neural-nets-learn.html

Rohan & Lenny #1: Neural Networks & The Backpropagation Algorithm, Explained

https://medium.com/a-year-of-artificial-intelligence/rohan-lenny-1-neural-networks-the-backpropagation-algorithm-explained-abf4609d4f9d#.9igtj84gm

The Essence of Artificial Neural Networks

https://medium.com/@ivanliljeqvist/the-essence-of-artificial-neural-networks-5de300c995d6#.yk0kydmf8

Neural Networks with PHP

youtube: https://www.youtube.com/watch?v=5bFxDsoNFzU
mirror: http://pan.baidu.com/s/1boc933p

Neural Networs in MySQL

Neural Networks in JavaScript

blog: http://www.antoniodeluca.info/blog/10-08-2016/neural-networks-in-javascript.html

Multiplication with simple neural nets

blog: http://nbviewer.jupyter.org/gist/fperez/c7b1cb4810f9d0935e893f34c41f0c62

Training Neural Networks with Theano

http://blog.asidatascience.com/training-neural-networks-with-theano/

An Introduction to Machine Learning in Julia

http://juliacomputing.com/blog/2016/09/28/knn-char-recognition.html

A Quick Introduction to Neural Networks

http://www.kdnuggets.com/2016/11/quick-introduction-neural-networks.html

Starting with Neural Networks in Swift 3.0

Visual and Interactive Guide to the Basics of Neural Networks

http://jalammar.github.io/visual-interactive-guide-basics-neural-networks/

Backpropagation

A Step by Step Backpropagation Example

Can you give a visual explanation for the back propagation algorithm for neural networks?

github: https://github.com/rasbt/python-machine-learning-book/blob/master/faq/visual-backpropagation.md Visualize Algorithms based on the Backpropagation
blog: http://neupy.com/2015/07/04/visualize_backpropagation_algorithms.html#gradient-descent

Backpropagation — How Neural Networks Learn Complex Behaviors

blog: https://medium.com/autonomous-agents/backpropagation-how-neural-networks-learn-complex-behaviors-9572ac161670#.ipu4jr2ka

Derivation of Backpropagation

https://www.cs.swarthmore.edu/~meeden/cs81/s10/BackPropDeriv.pdf

Backpropagation

https://en.wikipedia.org/wiki/Backpropagation#Derivation

A Derivation of Backpropagation in Matrix Form

blog: http://sudeepraja.github.io/Neural/

Backpropagation — How Neural Networks Learn Complex Behaviors

blog: https://medium.com/autonomous-agents/backpropagation-how-neural-networks-learn-complex-behaviors-9572ac161670#.dmdy8glec

How to Implement the Backpropagation Algorithm From Scratch In Python

blog: http://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/

Notes on Backpropagation

intro: This document derives backpropagation for some common error functions and describes some other tricks.
paper: https://www.ics.uci.edu/~pjsadows/notes.pdf

Yes you should understand backprop

intro: Andrej Karpathy
blog: https://medium.com/@karpathy/yes-you-should-understand-backprop-e2f06eab496b#.hbbhgjzi9

Back-propagation, an introduction

blog: http://www.offconvex.org/2016/12/20/backprop/

Project

Neural Networks demo in Javascript

demo: http://cs.stanford.edu/people/karpathy/svmjs/demo/demonn.html

Interactive visualization of artificial neural networks

homepage: http://experiments.mostafa.io/public/ffbpann/
github: https://github.com/drdrsh/interactive-bpann

NNX: Neural networks for Excel

intro: “Multilayer perceptrons done the Excel way”
homepage: https://nnx-addin.org/
github: https://github.com/ikhramts/nnx

GNU Gneural Network

homepage: http://www.gnu.org/software/gneuralnetwork/

Generating procedural plants with neural networks

blog: http://www.iotapersei.com/_neural_networks_article.html

NeuroFlow: A lightweight, scala based library for Artificial Neural Networks

github: https://github.com/zenecture/neuroflow

DN2A - Digital Neural Network Architecture

homepage: http://www.dn2a.org/
github: https://github.com/dn2a/dn2a-javascript

visual-neural-net

intro: Interactive visualization of artificial neural networks that leverages deeplearning4j, spring, and angular2
github(server): https://github.com/JavaFXpert/visual-neural-net-server
github(client):https://github.com/JavaFXpert/ng2-spring-websocket-client

NN++: A small and easy to use neural net implementation for C++

intro: It includes the neural net implementation and a Matrix class for basic linear algebra operations
github: https://github.com/stagadish/NNplusplus

Neural networks module for Redis