Artificial Neurons

- Overview

Artificial neurons (or nodes) are the basic computational units of artificial neural networks (ANNs). They mimic biological neurons, which are software programs that simulate the information processing methods of the human brain. They receive multiple inputs, process them through weighted connections and activation functions, and generate outputs. Artificial neurons are also one of the core architectures of deep learning (DL).

An artificial neuron performs mathematical operations, combines inputs, adds biases, and then applies transfer functions to determine whether and how it "fires" or activates, passing signals to other neurons. 

Artificial neurons are the connection points in an ANN. Similar to biological neural networks in the human body, ANNs have a layered architecture in which each network node (connection point) is capable of processing inputs and forwarding outputs to other nodes in the network.

The artificial neuron receives one or more inputs, applies weights to these inputs, and sums them to produce an output. The value is then passed to the activation function to become the neuron's output. 

A neural network is also a mathematical function. The simplest case is a single input node, a weight, and an output node. When multiple layers are added, the neural network becomes a composition of functions as the signal passes from layer to layer. 

An artificial neuron (also referred to as a perceptron) is a mathematical function. It takes one or more inputs that are multiplied by values called “weights” and added together

Please refer to the following for more details:

• Wikipedia: Activation Function

- Understanding Artificial Neurons (Perceptions)

An artificial neuron (or perceptron), is a mathematical function that models a biological neuron. It consists of inputs (x₁, x₂, ..., x_n) which are multiplied by corresponding weights (w₁, w₂, ..., w_n) and summed, then passed through a non-linear activation function to produce the neuron's output. 

A neural network is a collection of these artificial neurons, organized in layers, where the output of one layer serves as the input for the next, creating a composition of functions to solve complex problems. 

Components of an Artificial Neuron:

• Inputs (x₁, x₂, ..., x_n): These are the numerical values that the neuron receives, similar to signals from other neurons in a biological network.
• Weights (w₁, w₂, ..., w_n): Each input is multiplied by a weight, which determines its strength or importance in the neuron's overall calculation.
• Weighted Sum: The inputs are multiplied by their respective weights and then summed together.
• Bias: An additional term, called a bias, is often added to the weighted sum, allowing the activation function to be shifted.
• Activation Function: This non-linear function processes the weighted sum (plus bias) to produce the final output of the neuron. 

- Neural Network Activation Functions

Activation functions are fundamental components of artificial neural networks, determining the output of a node based on its weighted inputs. They introduce non-linearity, enabling neural networks to learn and represent complex relationships within data.

The activation function of a node in an ANN is a function that calculates the output of the node based on its individual inputs and their weights. Neural networks can represent a wide variety of functions with appropriate weights. 

Activation functions can be linear and non-linear, although the most useful ones are non-linear. Non-linear activation functions play a vital role in neural networks and other deep and algorithmic learning models. Nontrivial problems can be solved using only a few nodes if the activation function is nonlinear.  

Key aspects of activation functions: 

• Non-linearity: While linear activation functions exist, non-linear functions are crucial for solving non-trivial problems. Without them, a multi-layered neural network would essentially behave as a single-layer linear model, limiting its ability to learn complex patterns.
• Decision-making: The activation function decides whether a neuron should "fire" or be activated, and how strongly the signal is passed to subsequent layers. This is based on the weighted sum of the inputs to that neuron.
• Enabling learning: Non-linear activation functions allow neural networks to model intricate, non-linear mappings between inputs and outputs, which is essential for tasks like image recognition, natural language processing, and other advanced machine learning applications.
• Common types: Popular non-linear activation functions include the Sigmoid, Tanh (hyperbolic tangent), and ReLU (Rectified Linear Unit), along with variations like Leaky ReLU and GELU. Each has specific characteristics and is suited for different types of problems and network architectures.
• Impact on training: The choice of activation function significantly impacts the training process and the model's ability to converge and generalize. Factors like vanishing/exploding gradients and computational efficiency are considerations when selecting an appropriate activation function.

- Artificial Neuron Example

An artificial neuron is a mathematical function conceived as a model of biological neurons, a neural network. Artificial neurons are elementary units in an artificial neural network. Neural networks are multi-layer networks of neurons (the blue and magenta nodes in the chart above) that we use to classify things, make predictions, etc. 

Above is the diagram of a simple neural network with five inputs, 5 outputs, and two hidden layers of neurons. Starting from the left, we have:

• The input layer of our model in orange.
• Our first hidden layer of neurons in blue.
• Our second hidden layer of neurons in magenta.
• The output layer (a.k.a. the prediction) of our model in green.

The arrows that connect the dots shows how all the neurons are interconnected and how data travels from the input layer all the way through to the output layer. 

Later we will calculate step by step each output value. We will also watch how the neural network learns from its mistake using a process known as backpropagation.

- How an Artificial Neuron Works

• Inputs and Weights: A neuron receives several input signals (X) from other neurons or the data itself. Each input is multiplied by a specific weight (W), which determines the strength of the connection.
• Summation: The weighted inputs are summed together.
• Bias: A bias term is added to this sum, which helps adjust the neuron's activation point and flexibility.
• Activation Function: The result is then passed through a transfer or activation function (F).
• Output: This function decides whether the neuron should "activate" and "fire" its signal (Y) to other neurons, based on whether the combined input exceeds a certain threshold.

- The Role of Artificial Neurons in ANNs

• Interconnected Network: Artificial neurons are arranged in layers and interconnected to form an ANN, creating a complex network that can learn and process information.
• Weighted Connections: The connections between neurons have weights, which are adjusted during the training process to allow the network to learn and make accurate predictions.
• Learning: By adjusting these weights, the ANN can learn to recognize patterns and relationships in data, much like how the brain learns.
• Distributed Logic: Unlike a traditional computer program with explicit instructions, the logic in an ANN is distributed across the thousands of artificial neurons and their weights.

- Artificial Neural Networks and Artificial Neurons

Neural networks are sometimes called artificial neural networks (ANNs) or simulated neural networks (SNNs). They are a subset of machine learning (ML), and at the heart of deep learning (DL) models. ANNs are a type of ML process that uses interconnected nodes to teach computers to process data like the human brain.

ANNs are made up of layers of interconnected nodes, each with a different role in data processing. The structure and name of ANNs is inspired by the human brain, mimicking how biological neurons signal to each other. 

In addition to the living world, in the field of ANNs in computer science, a neuron is a collection of inputs, a set of weights, and an activation function. It converts these inputs into a single output. Another layer of neurons selects this output as input, and so on. In essence, we can say that each neuron is a mathematical function that closely models the function of biological neurons.

ANNs are used to solve problems in artificial intelligence (AI). They model the connections of biological neurons as weights between nodes. A positive weight reflects an excitatory connection, while negative values mean inhibitory connections. All inputs are modified by a weight and summed. 

- Deep Learning and Artificial Neurons

In artificial and biological architectures, nodes are called neurons, and connections are characterized by synaptic weights, which represent the importance of the connection. As new data is received and processed, synaptic weights change, and this is how learning occurs. 

Artificial neurons are modeled after the hierarchical arrangement of neurons in a biological sensory system. For example, in the visual system, light input passes through neurons in successive layers of the retina, then to neurons in the thalamus of the brain, and then to neurons in the visual cortex of the brain. 

ANNs work by passing information through multiple layers of interconnected neurons. The neurons receive input, apply an activation function, and use a threshold to determine if messages are passed along. The network learns from mistakes and improves continuously.

As neurons pass signals through more and more layers, the brain gradually extracts more information until it is confident that it can recognize what a person is seeing. In artificial intelligence AI), this fine-tuning process is called deep learning. 

Deep learning models can recognize patterns in data like text, images, and sounds to make predictions and produce insights. For example, some law enforcement agencies use deep learning to detect crimes by matching faces against digital images.

[Neuron and Synapse - GeeksforGeeks]

- Artificial And Biological Neuron

We have heard of the latest advancements in the field of DL due to the usage of different neural networks. At the most basic level, all such neural networks are made up of artificial neurons that try to mimic the working of biological neurons. Understanding how these artificial neurons compare to the structure of biological neurons in our brains and if possibly this could lead to a way to improve neural networks further. 

An Artificial Neural Network (ANN) is an information processing paradigm that is inspired the brain. ANNs, like people, learn by example. An ANN is configured for a specific application, such as pattern recognition or data classification, through a learning process. Learning largely involves adjustments to the synaptic connections that exist between the neurons.  

In artificial and biological networks, as neurons process the input they receive, they decide whether or not the output should be passed as input to the next layer. The decision of whether to send information is called bias, and it is determined by an activation function built into the system. 

For example, the artificial neuron may only pass the output signal to the next layer if the sum of its inputs (actually voltages) exceeds a certain threshold. Because activation functions can be linear or non-linear, neurons will typically have a wide range of convergence and divergence. Divergence is the ability of one neuron to communicate with many other neurons in the network while convergence is the ability of one neuron to receive input from many other neurons in the network.