Personal tools

Deep Learning

Fleet Week 2014_1000276.jpg
(US Navy Blue Angels, San Francisco Fleet Week 2014 - Jeff M. Wang)

Deep Learning — A Technique for Implementing Machine Learning



- Overview

Deep Learning (DL) is a subset of Machine Learning (ML), which is a subset of Artificial Intelligence AI). DL is just a type of ML, inspired by the structure of a human brain. DL algorithms attempt to draw similar conclusions as humans would by continually analyzing data with a given logical structure. To achieve this, DL uses a multi-layered structure of algorithms called neural networks.

The design of the neural network is based on the structure of the human brain. Just as we use our brains to identify patterns and classify different types of information, neural networks can be taught to perform the same tasks on data.

The individual layers of neural networks can also be thought of as a sort of filter that works from gross to subtle, increasing the likelihood of detecting and outputting a correct result. The human brain works similarly. Whenever we receive new information, the brain tries to compare it with known objects. The same concept is also used by deep neural networks.

Neural networks enable us to perform many tasks, such as clustering, classification or regression. With neural networks, we can group or sort unlabeled data according to similarities among the samples in this data. Or in the case of classification, we can train the network on a labeled dataset in order to classify the samples in this dataset into different categories. Artificial neural networks have unique capabilities that enable deep learning models to solve tasks that machine learning models can never solve.

All recent advances in artificial intelligence in recent years are due to deep learning. Without deep learning, we would not have self-driving cars, chatbots or personal assistants like Alexa and Siri. The Google Translate app would continue to be as primitive as 10 years ago (before Google switched to neural networks for this App), and Netflix or Youtube would have no idea which movies or TV series we like or dislike. Behind all these technologies are neural networks.

- Deep Learning with Big Data on GPUs

The AI boom took off when people realized that they can utilize GPU technology to train deep-learning models much faster than waiting days for a general-purpose CPU to complete one cycle of model training. Since 2016, when GPU manufacturers like NVIDIA, Intel, and others created their first AI-optimized GPUs, most AI development has been related to how to train models to be as accurate and predictive as possible.

Deep learning is a type of machine learning that trains a computer to perform human-like tasks, such as recognizing speech, identifying images or making predictions. Instead of organizing data to run through predefined equations, deep learning sets up basic parameters about the data and trains the computer to learn on its own by recognizing patterns using many layers of processing.

Deep learning is probably one of the hottest tech topics right now. Large corporations and young startups alike are all gold-rushing this fancy field. If you think big data is important, then you should care about deep learning. The Economist says that data is the new oil in the 21st Century. If data is the crude oil, databases and data warehouses are the drilling rigs that digs and pumps the data on the Internet, then think of deep learning as the oil refinery that finally turns crude oil into all the useful and insightful final products. There could be a lot of “fossil fuels” hidden underground, and there are a lot of drills and pumps in the market, but without the right refinery tools, you ain’t gonna get anything valuable. That’s why deep learning is important. It’s part of the data-driven big picture.

The good news is, we are not going to run out of data and our “refinery machine” is getting better and better. Today, just about doing anything online will generate data. In the meantime, as long as the data isn’t garbage-in, then there’s no garbage-out from deep learning. Also, this “oil refinery” is improving on both software and hardware. Deep learning algorithms have improved over the past few decades and developers around the world have contributed to open source frameworks like TensorFlow, Theano, Keras, and Torch, all of which make it easy for people to build deep learning algorithms as if playing with LEGO pieces. And thanks to the demand from gamers around the world, GPUs (graphics processing units) make it possible for us to leverage deep learning algorithms to build and train models with impressive results in a time-efficient manner.


- Deep Learning Accelerators 

Machine learning is widely used in many modern artificial intelligence applications. Various hardware platforms are implemented to support such applications. Among them, graphics processing unit (GPU) is the most widely used one due to its fast computation speed and compatibility with various algorithms. Field programmable gate arrays (FPGA) show better energy efficiency compared with GPU when computing machine learning algorithm at the cost of low speed.  

Deep learning accelerators such as GPUs, FPGAs, and more recently TPUs. More companies have been announcing plans to design their own accelerators, which are widely used in data centers. There is also an opportunity to deploy them at the edge, initially for inference and for limited training over time. This also includes accelerators for very low power devices. The development of these technologies will allow machine learning (or smart devices) to be used in many IoT devices and appliances.


San Diego_California_111420A
[San Diego, California - Civil Engineering Discoveries]

- Deep Learning and Neural Networks

Neural networks are based on the principles of the human brain. A neural network consists of many neurons and connections between them. A neuron can be represented as a function with several inputs and one output. Each neuron takes parameters from inputs (each input may have a different weight, which determines its importance), performs a specific function on them and gives the result to the output. The output of one neuron can be the input for another. Thus, multi-layer neural networks are formed, which are the subject of deep learning.   

Deep learning is machine learning. More specifically, deep learning is considered an evolution of machine learning. It uses a programmable neural network that enables machines to make accurate decisions without help from humans. Deep learning achieves great power and flexibility by learning to represent the world as nested hierarchy of concepts, with each concept defined in relation to simpler concepts, and more abstract representations computed in terms of less abstract ones.

Deep learning software attempts to mimic the activity in layers of neurons in the neocortex, the wrinkly 80 percent of the brain where thinking occurs. The software learns, in a very real sense, to recognize patterns in digital representations of sounds, images, and other data.

The basic idea - that software can simulate the neocortex’s large array of neurons in an artificial “neural network” - is decades old, and it has led to as many disappointments as breakthroughs. But because of improvements in mathematical formulas and increasingly powerful computers, computer scientists can now model many more layers of virtual neurons than ever before.


- Deep Learning Architectures

Deep learning isn’t a single approach but rather a class of algorithms and topologies that you can apply to a broad spectrum of problems. While deep learning is certainly not new, it is experiencing explosive growth because of the intersection of deeply layered neural networks and the use of GPUs to accelerate their execution. Big data has also fed this growth. Because deep learning relies on supervised learning algorithms (those that train neural networks with example data and reward them based on their success), the more data, the better to build these deep learning structures.

There are five of the most popular deep learning architectures:


  • Recurrent neural networks (RNNs)
  • Long short-term memory (LSTM)/gated recurrent unit (GRU)
  • Convolutional neural networks (CNNs)
  • Deep belief networks (DBN)
  • Deep stacking networks (DSNs)


and then, open source software options for deep learning.


[More to come ...]


Document Actions