Programming/Deep Learning - Revision history

Pysdlb: /* Deep Learning */

2022-11-17T13:39:01Z

‎Deep Learning

Pysdlb: /* Software repositories */

2022-11-16T11:37:05Z

‎Software repositories

Pysdlb at 13:00, 13 October 2022

2022-10-13T13:00:07Z

Pysdlb at 14:47, 9 June 2022

2022-06-09T14:47:34Z

Pysdlb at 12:59, 3 May 2022

2022-05-03T12:59:16Z

Pysdlb at 16:22, 10 May 2021

2021-05-10T16:22:44Z

Pysdlb at 16:22, 10 May 2021

2021-05-10T16:22:17Z

Pysdlb at 16:19, 10 May 2021

2021-05-10T16:19:20Z

Pysdlb at 16:11, 10 May 2021

2021-05-10T16:11:02Z

Pysdlb at 15:53, 10 May 2021

2021-05-10T15:53:02Z

Show changes

@@ Line 53: / Line 53: @@
 * + <span style="color:#009000">Learns features and classifiers automatically</span>
-* + <span style="color:#009000">Accuracy is unlimited</span
+* + <span style="color:#009000">Accuracy is unlimited</span>
 * - <span style="color:#900000">Requires very large data sets</span>
 * - <span style="color:#900000">Computationally intensive / expensive</span>
 ==Introduction to Python and its libraries==

← Older revision		Revision as of 11:37, 16 November 2022
Line 322:		Line 322:


−	~~{\| class="wikitable"~~
−	~~\|style="width:5%; border-width: 0" \| [[File:icon_home.png]]~~
−	~~\|style="width:95%; border-width: 0" \|~~
−	* [[Main_Page\|Home]]
−	* [[Applications\|Application support]]
−	* [[General\|General]]
−	* [[Training\|Training]]
−	* [[Programming\|Programming support]]

−	\|-	+	[[Main Page]] / [[FurtherTopics/FurtherTopics #Programming\| Back to Further Topics]]
−	\|}

@@ Line 24: / Line 24: @@
 There are 6 Types of Artificial Neural Networks Currently Being:
-* Recurrent Neural Network(RNN) – Long Short Term Memory
+* Recurrent Neural Network(RNN) – Long Short-Term Memory
 * Convolutional Neural Network
 * Feedforward Neural Network – Artificial Neuron
 * Radial basis function Neural Network
-* Kohonen Self Organizing Neural Network
+* Kohonen Self-Organizing Neural Network
 * Modular Neural Network
@@ Line 95: / Line 95: @@
 * output layer: the last layer of neurons that produces given outputs for the program.
-An artificial neural network consists of artificial neurons or processing elements and is organised in three interconnected layers: input, hidden that may include more than one layer and output.
+An artificial neural network consists of artificial neurons or processing elements and is organised in three interconnected layers: input, hidden which may include more than one layer and output.
 The input layer contains input neurons that send information to the hidden layer. The hidden layer sends data to the output layer. Every neuron has weighted inputs (synapses), an activation function (defines the output given an input), and one output. Synapses are the adjustable parameters that convert a neural network to a parameter system.
@@ Line 114: / Line 114: @@
-From the programming point of view, gradient descent is an iterative method. We start with some set of values for our model parameters (weights and biases) and improve them slowly.
+From the programming point of view, gradient descent is an iterative method. We start with some sets of values for our model parameters (weights and biases) and improve them slowly.
 To improve a given set of weights, we try to get a sense of the value of the cost function (described below) for weights similar to the current weights (by calculating the gradient). Then we move in the direction which reduces the cost function.
@@ Line 147: / Line 147: @@
 Neural network models learn a mapping from inputs to outputs from examples and the choice of loss function must match the framing of the specific predictive modelling problem, such as classification or regression. Further, the configuration of the output layer must also be appropriate for the chosen loss function.
-As we keep applying training data to our neural network model we need to measure how close we are achieving our goal, with a suitable model we should describe something like the following:
+As we keep applying training data to our neural network model we need to measure how close we are to achieving our goal, with a suitable model we should describe something like the following:
 [[File:acc-loss.jpg]]
@@ Line 160: / Line 160: @@
 Google's TensorFlow Version 2 is a python library, as of version 2 the library Keras is built into the library together with eager execution.
-This library is a great choice for building commercial-grade deep learning applications.
+This library is a great choice for building commercial-grade deep-learning applications.
 TensorFlow grew out of another library DistBelief V2 which was a part of the Google Brain Project. This library aims to extend the portability of machine learning so that research models could be applied to commercial-grade applications.

@@ Line 61: / Line 61: @@
 ==Introduction to Python and its libraries==
-Python is a general-purpose high-level programming language that is widely used in data science and for producing deep learning algorithms. This brief tutorial introduces Python and its libraries like Numpy, skimage, TensorFlow, Keras.
+Python is a general-purpose high-level programming language that is widely used in data science and for producing deep learning algorithms. This brief tutorial introduces Python and its libraries like Numpy, TensorFlow.
 Deep structured learning or hierarchical learning or deep learning in short is part of the family of machine learning methods which are themselves a subset of the broader field of Artificial Intelligence.

@@ Line 80: / Line 80: @@
 ==Introduction to Neural Network==
-A typical neural network has anything from a few dozen to hundreds, thousands or even millions of artificial neurons called units arranged in a series of layers, each of which connects to the layers on either side. Some of them, known as input units, are designed to receive various forms of information from the outside world that the network will attempt to learn about, recognise, or otherwise process. Other units sit on the opposite side of the network and signal how it responds to the information it's learned; those are known as output units. In between the input units and output, units are one or more layers of hidden units, which, together, form most of the artificial brain. Most neural networks are fully connected, which means each hidden unit and each output unit is connected to every unit in the layers on either side. The connections between one unit and another are represented by a number called a weight, which can be either positive (if one unit excites another) or negative (if one unit suppresses or inhibits another). The higher the weight, the more influence one unit has on another. (This corresponds to the way actual brain cells trigger one another across tiny gaps called synapses.)
+A typical neural network has anything from a few dozen to hundreds, thousands or even millions of artificial neurons called units arranged in a series of layers, each of which connects to the layers on either side. Some of them, known as input units, are designed to receive various forms of information from the outside world that the network will attempt to learn about, recognise, or otherwise process. Other units sit on the opposite side of the network and signal how it responds to the information it's learned; those are known as output units. In between the input units and output, units are one or more layers of hidden units, which, together, form most of the artificial brain. Most neural networks are fully connected, which means each hidden unit and each output unit are connected to every unit in the layers on either side. The connections between one unit and another are represented by a number called a weight, which can be either positive (if one unit excites another) or negative (if one unit suppresses or inhibits another). The higher the weight, the more influence one unit has on another. (This corresponds to the way actual brain cells trigger one another across tiny gaps called synapses.)
 Information flows through a neural network in two ways. When it's learning (being trained) or operating normally (after being trained), patterns of information are fed into the network via the input units, which trigger the layers of hidden units, and these, in turn, arrive at the output units. This common design is called a feedforward network. Not all units "fire" all the time. Each unit receives inputs from the units to its left, and the inputs are multiplied by the weights of the connections they travel along. Every unit adds up all the inputs it receives in this way and (in the simplest type of network) if the sum is more than a certain threshold value, the unit "fires" and triggers the units it's connected to (those on its right).
@@ Line 105: / Line 105: @@
-Neural networks learn things in exactly the same way, typically by a feedback process called back-propagation (sometimes abbreviated as "backprop"). This involves comparing the output a network produces with the output it was meant to produce and using the difference between them to modify the weights of the connections between the units in the network, working from the output units through the hidden units to the input units—going backward, in other words. In time, back-propagation causes the network to learn, reducing the difference between actual and intended output to the point where the two exactly coincide, so the network figures things out exactly as it should.
+Neural networks learn things in exactly the same way, typically by a feedback process called back-propagation (sometimes abbreviated as "backprop"). This involves comparing the output a network produces with the output it was meant to produce and using the difference between them to modify the weights of the connections between the units in the network, working from the output units through the hidden units to the input units—going backwards, in other words. In time, back-propagation causes the network to learn, reducing the difference between actual and intended output to the point where the two exactly coincide, so the network figures things out exactly as it should.
 ===Gradient Descent===
@@ Line 114: / Line 114: @@
-From the programming point of view gradient descent is an iterative method. We start with some set of values for our model parameters (weights and biases), and improve them slowly.
+From the programming point of view, gradient descent is an iterative method. We start with some set of values for our model parameters (weights and biases) and improve them slowly.
 To improve a given set of weights, we try to get a sense of the value of the cost function (described below) for weights similar to the current weights (by calculating the gradient). Then we move in the direction which reduces the cost function.
@@ Line 124: / Line 124: @@
 By repeating this step thousands of times, we’ll continually minimise our cost function.
-On gradients and gradient learning algorithms, the main optimisation technique used to fit neural network weights to training data sets.
+On gradients and gradient learning algorithms, the main optimisation technique is used to fit neural network weights to training data sets.
-This includes the important distinction between batch and stochastic gradient descent, and approximations via mini-batch gradient descent, today all simply referred to as stochastic gradient descent.
+This includes the important distinction between batch and stochastic gradient descent, and approximations via mini-batch gradient descent, today all are simply referred to as stochastic gradient descent.
 * Batch Gradient Descent. The gradient is estimated using all examples in the training data set.
@@ Line 162: / Line 162: @@
 This library is a great choice for building commercial-grade deep learning applications.
-TensorFlow grew out of another library DistBelief V2 that was a part of the Google Brain Project. This library aims to extend the portability of machine learning so that research models could be applied to commercial-grade applications.
+TensorFlow grew out of another library DistBelief V2 which was a part of the Google Brain Project. This library aims to extend the portability of machine learning so that research models could be applied to commercial-grade applications.
 Much like the Theano library, TensorFlow is based on computational graphs where a node represents persistent data or math operation and edges represent the flow of data between nodes, which is a multidimensional array or tensor; hence the name TensorFlow
@@ Line 170: / Line 170: @@
 Even though TensorFlow was designed for neural networks, it works well for other nets where computation can be modelled as a data flow graph.
-TensorFlow also uses several features from Theano such as common and sub-expression elimination, auto differentiation, shared and symbolic variables.
+TensorFlow also uses several features from Theano such as common and sub-expression elimination, auto differentiation, and shared and symbolic variables.
 Different types of deep nets can be built using TensorFlow like convolutional nets, Autoencoders, RNTN, RNN, RBM, DBM/MLP and so on.

← Older revision		Revision as of 16:22, 10 May 2021
Line 314:		Line 314:


−	===Software repositories===	+	==Software repositories==

	* https://github.com/Hvass-Labs/TensorFlow-Tutorials		* https://github.com/Hvass-Labs/TensorFlow-Tutorials

@@ Line 35: / Line 35: @@
 ==Introduction to Machine Learning vs Deep Learning==
-Before starting lets have a look at the two different terms which are Machine Learning and Deep Learning which are closely related. The following diagram show pictorially the key difference:
+Before starting let us have a look at the two different terms which are Machine Learning and Deep Learning which are closely related. The following diagram shows pictorially the key difference:
 [[File:mlvsdl.png]]
-With Machine Learning the approach works like the top half of the picture above. You would have to design a feature extraction algorithm which generally involved a lot of heavy mathematics (complex design), wasn’t very efficient, and didn’t perform too well at all (accuracy level just wasn’t suitable for real-world applications). After doing all of that you would also have to design a whole classification model to classify your input given the extracted features.
+With Machine Learning the approach works as the top half of the picture above. You would have to design a feature extraction algorithm that generally involved a lot of heavy mathematics (complex design), wasn’t very efficient, and didn’t perform too well at all (accuracy level just wasn’t suitable for real-world applications). After doing all of that you would also have to design a whole classification model to classify your input given the extracted features.
 With Deep Learning networks we can perform feature extraction and classification in one shot, which means we only have to design one model. This also means that with have a lot more layers (usually) and parameters to refine our model to an optimal point.
@@ Line 61: / Line 61: @@
 ==Introduction to Python and its libraries==
-Python is a general-purpose high level programming language that is widely used in data science and for producing deep learning algorithms. This brief tutorial introduces Python and its libraries like Numpy, skimage, TensorFlow, Keras.
+Python is a general-purpose high-level programming language that is widely used in data science and for producing deep learning algorithms. This brief tutorial introduces Python and its libraries like Numpy, skimage, TensorFlow, Keras.
 Deep structured learning or hierarchical learning or deep learning in short is part of the family of machine learning methods which are themselves a subset of the broader field of Artificial Intelligence.
@@ Line 80: / Line 80: @@
 ==Introduction to Neural Network==
-A typical neural network has anything from a few dozen to hundreds, thousands, or even millions of artificial neurons called units arranged in a series of layers, each of which connects to the layers on either side. Some of them, known as input units, are designed to receive various forms of information from the outside world that the network will attempt to learn about, recognise, or otherwise process. Other units sit on the opposite side of the network and signal how it responds to the information it's learned; those are known as output units. In between the input units and output units are one or more layers of hidden units, which, together, form most of the artificial brain. Most neural networks are fully connected, which means each hidden unit and each output unit is connected to every unit in the layers either side. The connections between one unit and another are represented by a number called a weight, which can be either positive (if one unit excites another) or negative (if one unit suppresses or inhibits another). The higher the weight, the more influence one unit has on another. (This corresponds to the way actual brain cells trigger one another across tiny gaps called synapses.)
+A typical neural network has anything from a few dozen to hundreds, thousands or even millions of artificial neurons called units arranged in a series of layers, each of which connects to the layers on either side. Some of them, known as input units, are designed to receive various forms of information from the outside world that the network will attempt to learn about, recognise, or otherwise process. Other units sit on the opposite side of the network and signal how it responds to the information it's learned; those are known as output units. In between the input units and output, units are one or more layers of hidden units, which, together, form most of the artificial brain. Most neural networks are fully connected, which means each hidden unit and each output unit is connected to every unit in the layers on either side. The connections between one unit and another are represented by a number called a weight, which can be either positive (if one unit excites another) or negative (if one unit suppresses or inhibits another). The higher the weight, the more influence one unit has on another. (This corresponds to the way actual brain cells trigger one another across tiny gaps called synapses.)
-Information flows through a neural network in two ways. When it's learning (being trained) or operating normally (after being trained), patterns of information are fed into the network via the input units, which trigger the layers of hidden units, and these in turn arrive at the output units. This common design is called a feedforward network. Not all units "fire" all the time. Each unit receives inputs from the units to its left, and the inputs are multiplied by the weights of the connections they travel along. Every unit adds up all the inputs it receives in this way and (in the simplest type of network) if the sum is more than a certain threshold value, the unit "fires" and triggers the units it's connected to (those on its right).
+Information flows through a neural network in two ways. When it's learning (being trained) or operating normally (after being trained), patterns of information are fed into the network via the input units, which trigger the layers of hidden units, and these, in turn, arrive at the output units. This common design is called a feedforward network. Not all units "fire" all the time. Each unit receives inputs from the units to its left, and the inputs are multiplied by the weights of the connections they travel along. Every unit adds up all the inputs it receives in this way and (in the simplest type of network) if the sum is more than a certain threshold value, the unit "fires" and triggers the units it's connected to (those on its right).
-For a neural network to learn, there has to be an element of feedback involved—just as children learn by being told what they're doing right or wrong. In fact, we all use feedback, all the time. Think back to when you first learned to play a game like ten-pin bowling. As you picked up the heavy ball and rolled it down the alley, your brain watched how quickly the ball moved and the line it followed, and noted how close you came to knocking down the skittles. Next time it was your turn, you remembered what you'd done wrong before, modified your movements accordingly, and hopefully threw the ball a bit better. So you used feedback to compare the outcome you wanted with what actually happened, figured out the difference between the two, and used that to change what you did next time ("I need to throw it harder," "I need to roll slightly more to the left," "I need to let go later," and so on). The bigger the difference between the intended and actual outcome, the more radically you would have altered your moves.
+For a neural network to learn, there has to be an element of feedback involved—just as children learn by being told what they're doing right or wrong. In fact, we all user feedback, all the time. Think back to when you first learned to play a game like ten-pin bowling. As you picked up the heavy ball and rolled it down the alley, your brain watched how quickly the ball moved and the line it followed and noted how close you came to knocking down the skittles. Next time it was your turn, you remembered what you'd done wrong before, modified your movements accordingly, and hopefully threw the ball a bit better. So you used feedback to compare the outcome you wanted with what actually happened, figured out the difference between the two, and used that to change what you did next time ("I need to throw it harder," "I need to roll slightly more to the left," "I need to let go later," and so on). The bigger the difference between the intended and actual outcome, the more radically you would have altered your moves.
-Below shows a neural network example with inputs on the left, a hidden layer, and an output layer. In code we will try to mimic this theory.
+Below shows a neural network example with inputs on the left, a hidden layer, and an output layer. In code, we will try to mimic this theory.
 [[File:NeuralNetwork.jpg]]
@@ Line 95: / Line 95: @@
 * output layer: the last layer of neurons that produces given outputs for the program.
-An artificial neural network consists of artificial neurons or processing elements and is organised in three interconnected layers: input, hidden that may include more than one layer, and output.
+An artificial neural network consists of artificial neurons or processing elements and is organised in three interconnected layers: input, hidden that may include more than one layer and output.
 The input layer contains input neurons that send information to the hidden layer. The hidden layer sends data to the output layer. Every neuron has weighted inputs (synapses), an activation function (defines the output given an input), and one output. Synapses are the adjustable parameters that convert a neural network to a parameter system.
@@ Line 109: / Line 109: @@
 ===Gradient Descent===
-Gradient descent has an analogy of looking for the easiest way down a mountain side, you're going to look for the most gentle way down. In reality it is a highly mathematical function but for most programmers this is hidden.
+Gradient descent has an analogy of looking for the easiest way down a mountainside, you're going to look for the most gentle way down. In reality, it is a highly mathematical function but for most programmers this is hidden.
 [[File:grad-descent2.jpg]]
@@ Line 124: / Line 124: @@
 By repeating this step thousands of times, we’ll continually minimise our cost function.
-On gradients and gradient learning algorithms, the main optimisation technique used to fit neural network weights to training data-sets.
+On gradients and gradient learning algorithms, the main optimisation technique used to fit neural network weights to training data sets.
 This includes the important distinction between batch and stochastic gradient descent, and approximations via mini-batch gradient descent, today all simply referred to as stochastic gradient descent.
-* Batch Gradient Descent. Gradient is estimated using all examples in the training data set.
+* Batch Gradient Descent. The gradient is estimated using all examples in the training data set.
-* Stochastic (Online) Gradient Descent. Gradient is estimated using subsets of samples in the training data set.
+* Stochastic (Online) Gradient Descent. The gradient is estimated using subsets of samples in the training data set.
-* Mini-Batch Gradient Descent. Gradient is estimated using each single pattern in the training data set.
+* Mini-Batch Gradient Descent. The gradient is estimated using every single pattern in the training data set.
 The mini-batch variant is offered as a way to achieve the speed of convergence offered by stochastic gradient descent with the improved estimate of the error gradient offered by batch gradient descent.
@@ Line 176: / Line 176: @@
 ===MNIST Dataset Overview===
-This example is using MNIST handwritten digits. The dataset contains 60,000 examples for training and 10,000 examples for testing. The digits have been size-normalized and centered in a fixed-size image (28x28 pixels) with values from 0 to 255.
+This example is using MNIST handwritten digits. The dataset contains 60,000 examples for training and 10,000 examples for testing. The digits have been size-normalized and centred in a fixed-size image (28x28 pixels) with values from 0 to 255.
 In this example, each image will be converted to float32, normalized to [0, 1] and flattened to a 1-D array of 784 features (28*28).
@@ Line 316: / Line 316: @@
 ===Software repositories===
-* https://github.com/Hvass-Labs/TensorFlow-Tutorials https://github.com/aymericdamien/TensorFlow-Examples
+* https://github.com/Hvass-Labs/TensorFlow-Tutorials
 * https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples/tutorials

← Older revision		Revision as of 16:19, 10 May 2021
Line 173:		Line 173:

	Different types of deep nets can be built using TensorFlow like convolutional nets, Autoencoders, RNTN, RNN, RBM, DBM/MLP and so on.		Different types of deep nets can be built using TensorFlow like convolutional nets, Autoencoders, RNTN, RNN, RBM, DBM/MLP and so on.
		+
		+	===MNIST Dataset Overview===
		+
		+	This example is using MNIST handwritten digits. The dataset contains 60,000 examples for training and 10,000 examples for testing. The digits have been size-normalized and centered in a fixed-size image (28x28 pixels) with values from 0 to 255.
		+
		+	In this example, each image will be converted to float32, normalized to [0, 1] and flattened to a 1-D array of 784 features (28*28).

@@ Line 155: / Line 155: @@
 ===Tensorflow===
-* [https://www.tensorflow.org/tutorials/ https://www.tensorflow.org/tutorials/]
+* [https://www.tensorflow.org/ https://www.tensorflow.org/]
-Google's TensorFlow is a python library. This library is a great choice for building commercial-grade deep learning applications.
+Google's TensorFlow Version 2 is a python library, as of version 2 the library Keras is built into the library together with eager execution.
 TensorFlow grew out of another library DistBelief V2 that was a part of the Google Brain Project. This library aims to extend the portability of machine learning so that research models could be applied to commercial-grade applications.
@@ Line 172: / Line 174: @@
 Different types of deep nets can be built using TensorFlow like convolutional nets, Autoencoders, RNTN, RNN, RBM, DBM/MLP and so on.