What is the difference between deep learning and traditional artificial neural network learning?

Question

Can you offer a brief explanation of the differences between Deep Learning and traditional machine learning that use neural networks? How many levels are needed to create a "deep" neural network? Is this just a marketing ad?

Answer 1

I ask you to distinguish yourself with @Frank Puffer's answer. I don’t understand what he meant by performing an uncontrolled learning process on hidden layers, etc.

Deep learning refers to neural network models with a total of more than 2 or 3 hidden layers. Most DL models have 10 to 100 or more layers.

The recent revolution in Deep Learning models is based on two things:
1. the presence of a large amount of data that is a product of Internet age
2. GPU availability

The algorithm used to optimize DL models is called the backpropagation algorithm (which is mathematically equivalent to the gradient). Backprop has actually been around since at least the 80s - this is not a DL specific thing.

DL models usually require a lot of data due to the complexity and size of the models. They usually have millions of custom weighting options. Optimization requires high computing power due to the size of the training data and the millions of partial derivatives (relative to the weights) that need to be calculated at each iteration.

In essence, Deep Learning is not a marketing ad. This is a large multi-layer neural network model that requires a lot of data and powerful GPUs for training. And once they are trained, they achieve superhuman accuracy in certain tasks.

Answer 2

The standard backpropagation algorithm (gradient descent) becomes serious when the number of layers becomes large. The probability of local minima in the error function increases with each layer. Not only local minima in the mathematical sense cause problems, sometimes there are only flat areas in the error function (changing one or more weights does not lead to a significant change in error), where the gradient descent does not work.

On the other hand, networks with many layers can solve more complex problems, since each layer of cells can also provide an abstraction layer.

Deep learning addresses this particular problem. The basic idea is to perform an uncontrolled learning process on each individual layer in addition to using gradient descent for the entire network. The goal of unsupervised learning is to make each individual layer an extract of characteristic features outside of its input, which can be used by subsequent layers.

Although the term "Deep Learning" is currently used too broadly, it is more than just marketing advertising.

Edit: A few years ago, many people, including myself, believed that uncontrolled pre-training was a major factor in deep learning. Since then, other methods have become popular, which in many cases gives even better results. As mentioned in a comment by @Safak Okzan (below his own answer), they include:

• Residual Networks

• Party normalization

• Straightened linear units

Answer 3

In recent years, models designed to solve various machine learning problems have become much more complex, with a very large number of layers. For example, the Google Inception-v3 model has (I think) 42 layers. Traditional neural networks usually use only a few hidden layers. The term “deep”, used in the context of “deep learning” and “deep convolutional neural networks”, is a nod to the significant number of layers involved.

Answer 4

From the point of view of the application, I would distinguish the neural networks used in DL and ML, according to the following lines:

• Number of layers - Simple neural networks have fewer hidden layers (typically within 10), while deep neural networks can have more hidden layers (as mentioned above from 10 to 100 or higher).
• Purpose - Deep Learning has specialized neural networks (in terms of architecture) that can be used for specific tasks, such as CNN for image analysis, RNN / LSTM for sequences, etc. Simple neural networks can be used for tasks such as linear regression, classification (one / several classes).
• Processing Complexity - Learning a deep learning model requires more data and computational resources than a traditional machine learning neural network.

Answer 5

A traditional artificial neural network deals with fewer hidden layers, while deep learning deals with many hidden layers close to 1000. For example: in both of them you have an input layer that is transmitted to different layers, such as layer 1, 2, etc., In the case of deep study, you will have more hidden layers, and therefore there will be more weights that you will need to calculate and update when performing backtracking, and in ANN the weights will be much less.