Should neurons in a neural network be asynchronous?

Question

I am developing a neural network and trying to determine if I should write it in such a way that each neuron is its own “process” in Erlang, or I just need to go with C ++ and start the network in one thread (I will use all the same their cores by running an instance of each network in its thread).

Is there any good reason to abandon C ++ speed for asynchronous neurons that Erlang offers?

Answer 1

I'm not sure I understand what you are trying to do. An artificial neural network is essentially represented by the weight of connections between nodes. The nodes themselves do not exist in isolation; their values ​​are calculated (at least in direct communication networks) using the direct propagation algorithm when it is introduced.

The backpropagation algorithm for updating weights is definitely parallelizable, but it doesn't seem to be what you are describing.

Answer 2

The usefulness of having neurons in a neural network (NN) is to have a multidimensional matrix of what coefficients you want to process (train them, change them, adapt them little by little, so that they fit well with the problem you want to solve). On this matrix, you can apply numerical methods (proven and effective) to find an acceptable solution at an acceptable time.

IMHO, with NN (namely, the reverse propagation method), the goal is to have a matrix that is effective both at run time / forecast time and during training.

I do not understand the meaning of the presence of asynchronous neurons. What would he suggest? what question will he solve?

Maybe you could clearly explain what problem you would pose by putting them asynchronous?

I am really addressing your question: what do you want to get asynchronously with respect to traditional NN methods?

Answer 3

It will depend on your use case: the computational model of the neural network and the runtime. Here is a recent article (2014) written by Plotnikova et al., Which uses “Erlang and the Erlang / OTP platform with a predefined base implementation of actor model functions” and a new model developed by the authors, which they describe as “one neuron-one process,” using "Gravity Search Algorithm" for learning:

http://link.springer.com/chapter/10.1007%2F978-3-319-06764-3_52

We briefly refer to their abstract: “The article developed an asynchronous distributed modification of this algorithm and presents the results of experiments. The proposed architecture shows an increase in performance for distributed systems with different environmental parameters (high-performance cluster and local area network with a slow gateway).

In addition, most of the other answers here refer to a computational model that uses matrix operations for a training and modeling base, for which the authors of this article compare, saying: “this neural network model of this case (i.e. based on matrix operations) becomes completely the mathematical and its original character (from biological prototypes of neural networks) is lost "

The tests were carried out on three types of systems:

• The IBM cluster is represented as 15 virtual machines.
• A distributed system deployed on a local network is presented in the form of 15 physical machines.
• The hybrid system is based on system 2, but each physical machine has four processor cores.

They give the following specific results: "The presented results indicate a good ability to distribute gravity, especially for large networks (801 or more neurons). The acceleration depends on the number of nodes almost linearly. If we use 15 nodes, we can get about eight times the acceleration of the learning process "

Finally, they conclude with respect to their model: “The model includes three levels of abstraction: NNET, MLP and NEURON. This architecture allows you to encapsulate some common functions at common levels and some specific for the considered functions of neural networks at special levels. Asynchronous message passing between levels allow us to differentiate the synchronous and asynchronous parts of the training and modeling algorithms and, as a result, improve the use of resources. "

Answer 4

I think that asynchronous computing for NN may be useful for performance (recognition). In fact, the result may be similar (perhaps less pronounced) using dropout .

But direct implementation of asynchronous NNs will be much slower, because for synchronous NNs you can use linear algebra libraries that make good use of vectorization or GPUs.