https://m.facebook.com/groups/107107546348803?view=permalink&id=532112330514987

TensorFlow Speech Recognition - Kaggle competition is going on. I wrote a basic tutorial on speech (word) recognition using some of the datasets from the competition.
.
Hope it will be helpful for some of you. Thanks in advance for reading!
.

https://blog.manash.me/building-a-dead-simple-word-recognition-engine-using-convnet-in-keras-25e72c19c12b

Kaggle-knowhow/README.md at master · zzsza/Kaggle-knowhow · GitHub
https://github.com/zzsza/Kaggle-knowhow/blob/master/README.md

Probabilistic Graphical Models Tutorial — Part 2 – Stats and Bots
https://blog.statsbot.co/probabilistic-graphical-models-tutorial-d855ba0107d1


Homepage
Stats and Bots
Get started
HOMEDATA SCIENCEANALYTICSSTARTUPSBOTSDESIGNSUBSCRIBE TRY STATSBOT FREE
Go to the profile of Prasoon Goyal
Prasoon Goyal
PhD candidate at UT Austin. For more content on machine learning by me, check my Quora profile (https://www.quora.com/profile/Prasoon-Goyal).
Nov 23
Probabilistic Graphical Models Tutorial — Part 2
Parameter estimation and inference algorithms

In the previous part of this probabilistic graphical models tutorial for the Statsbot team, we looked at the two types of graphical models, namely Bayesian networks and Markov networks. We also explored the problem setting, conditional independences, and an application to the Monty Hall problem. In this post, we will cover parameter estimation and inference, and look at another application.

Parameter Estimation
Bayesian networks
Estimating the numbers in the CPD tables of a Bayesian network simply amounts to counting how many times that event occurred in our training data. That is, to estimate p(SAT=s1 | Intelligence = i1), we simply count the fraction of data points where SAT=s1 and Intelligence = i1, out of the total data points where Intelligence = i1. While this approach may appear ad hoc, it turns out that the parameters so obtained maximize the likelihood of the observed data.
Markov networks
For Markov networks, unfortunately, the above counting approach does not have a statistical justification (and will therefore lead to suboptimal parameters). So, we need to use more sophisticated techniques. The basic idea behind most of these techniques is gradient descent — we define parameters that describe the probability distribution, and then use gradient descent to find values for these parameters that maximize the likelihood of the observed data.
Finally, now that we have the parameters of our model, we want to use them on new data, to perform inference!
Inference
The bulk of the literature in probabilistic graphical models focuses on inference. The reasons are two-fold:
Inference is why we came up with this entire framework — being able to make predictions from what we already know.
Inference is computationally hard! In some specific kinds of graphs, we can perform inference fairly efficiently, but on general graphs, it is intractable. So we need to use approximate algorithms that trade off accuracy for efficiency.
There are several questions we can answer with inference:
Marginal inference: Finding the probability distribution of a specific variable. For instance, given a graph with variables A, B, C, and D, where A takes values 1, 2, and 3, find p(A=1), p(A=2) and p(A=3).
Posterior inference: Given some observed variables v_E (E for evidence) that take values e, finding the posterior distribution p(v_H | v_E=e) for some hidden variables v_H.
Maximum-a-posteriori (MAP) inference: Given some observed variables v_E that take values e, finding the setting of other variables v_H that have the highest probability.
Answers to these questions may be useful by themselves, or may need to be used as part of larger tasks.
In what follows, we are going to look at some of the popular algorithms for answering these questions, both exact and approximate. All these algorithms are applicable on both Bayesian networks and Markov networks.
Variable Elimination
Using the definition of conditional probability, we can write the posterior distribution as:

Let’s see how we can compute the numerator and the denominator above, using a simple example. Consider a network with three variables, and the joint distribution defined as follows:

Let’s say we want to compute p(A | B=1). Note that this means that we want to compute the values p(A=0 | B=1)and p(A=1 | B=1), which should sum to one. Using the above equation, we can write

The numerator is the probability that A = 0 and B = 1. We don’t care about the values of C. So we would sum over all the values of C. (This comes from basic probability — p(A=0, B=1, C=0) and p(A=0, B=1, C=1) are mutually exclusive events, so their union p(A = 0, B=1) is just the sum of the individual probabilities.)
So we add rows 3 and 4 to get p(A=0, B=1) = 0.15. Similarly, adding rows 7 and 8 gives usp(A=1, B=1) = 0.40. Also, we can compute the denominator by summing over all rows that contain B=1, that is, rows 3, 4, 7, and 8, to get p(B=1) = 0.55. This gives us the following:
p(A = 0 | B = 1) = 0.15 / 0.55 = 0.27
p(A = 1 | B = 1) = 0.40 / 0.55 = 0.73
If you look at the above computation closely, you would notice that we did some repeated computations — adding rows 3 & 4, and 7 & 8 twice. A more efficient way to compute p(B=1)would have been to simply add the values p(A=0, B=1) and p(A=1, B=1). This is the basic idea of variable elimination.
In general, when you have a lot of variables, not only can you use the values of the numerator to compute the denominator, but the numerator by itself will contain repeated computations, if evaluated naively. You can use dynamic programming to use precomputed values efficiently.
Because we are summing over one variable at a time, thereby eliminating it, the process of summing out multiple variables amounts to eliminating these variables one at a time. Hence, the name “variable elimination.”
It is straightforward to extend the above process to solve the marginal inference or MAP inference problems as well. Similarly, it is easy to generalize the above idea to apply it to Markov networks too.
The time complexity of variable elimination depends on the graph structure, and the order in which you eliminate the variables. In the worst case, it has exponential time complexity.
Belief Propagation
The VE algorithm that we just saw gives us only one final distribution. Suppose we want to find the marginal distributions for all variables. Instead of running variable elimination multiple times, we can do something smarter.
Suppose you have a graph structure. To compute a marginal, you need to sum the joint distribution over all other variables, which amounts to aggregating information from the entire graph. Here’s an alternate way of aggregating information from the entire graph — each node looks at its neighbors, and approximates the distribution of variables locally.
Then, every pair of neighboring nodes send “messages” to each other where the messages contain the local distributions. Now, every node looks at the messages it receives, and aggregates them to update its probability distributions of variables.

In the figure above, C aggregates information from its neighbors A and B, and sends a message to D. Then, D aggregates this message with the information from E and F.
The advantage of this approach is that if you save the messages that you are sending at every node, one forward pass of messages followed by one backward pass gives all nodes information about all other nodes. That information can then be used to compute all the marginals, which was not possible in variable elimination.
If the graph does not contain cycles, then this process converges after a forward and a backward pass. If the graph contains cycles, then this process may or may not converge, but it can often be used to get an approximate answer.
Approximate inference
Because exact inference may be prohibitively time consuming for large graphical models, numerous approximate inference algorithms have been developed for graphical models, most of which fall into one of the following two categories:
Sampling-based
These algorithms estimate the desired probability using sampling. As a simple example, consider the following scenario — given a coin, how you would determine the probability of getting heads when the coin is tossed? The simplest thing is to flip the coin, say, 100 times, and find out the fraction of tosses in which you get heads.
This is a sampling-based algorithm to estimate the probability of heads. For more complex questions in probabilistic graphical models, you can use a similar procedure. Sampling-based algorithms can further be divided into two classes. In the first one, the samples are independent of each other, as in the coin toss example above. These algorithms are called Monte Carlo methods.
For problems with many variables, generating good quality independent samples is difficult, and therefore, we generate dependent samples, that is, each new sample is random, but close to the last sample. Such algorithms are called Markov Chain Monte Carlo (MCMC) methods, because the samples form a “Markov chain.” Once we have the samples, we can use them to answer various inference questions.
Variational methods
Instead of using sampling, variational methods try to approximate the required distribution analytically. Suppose you write out the expression for computing the distribution of interest — marginal probability distribution or posterior probability distribution.
Often, these expressions have summations or integrals in them that are computationally expensive to evaluate exactly. A good way to approximate these expressions is to then solve for an alternate expression, and somehow ensure that this alternate expression is close to the original expression. This is the basic idea behind variational methods.
When we are trying to estimate a complex probability distribution p_complex, we define a separate set of probability distributions P_simple, which are easier to work with, and then find the probability distribution p_approx from P_simple that is closest to p_complex.
Application: Image denoising
Let us now use some of the ideas we just discussed on a real problem. Let’s say you have the following image:

Now suppose that it got corrupted by random noise, so that your noisy image looks as follows:

The goal is to recover the original image. Let’s see how we can use probabilistic graphical models to do this.
The first step is to think about what our observed and unobserved variables are, and how we can connect them to form a graph. Let us define each pixel in the noisy image as an observed random variable, and each pixel in the ground truth image as an unobserved variable. So, if the image is M x N, then there are MN observed variables and MN unobserved variables. Let us denote observed variables as X_ij and unobserved variables as Y_ij. Each variable takes values +1 and -1 (corresponding to black and white pixels, respectively). Given the observed variables, we want to find the most likely values of the unobserved variables. This corresponds to MAP inference.
Now, let us use some domain knowledge to build the graph structure. Clearly, the observed variable at position (i, j) in the noisy image depends on the unobserved variable at position (i, j) in the ground truth image. This is because most of the time, they are identical.
What more can we say? For ground truth images, the neighboring pixels usually have the same values — this is not true at the boundaries of color change, but inside a single-colored region, this property holds. Therefore, we connect Y_ij and Y_kl if they are neighboring pixels.
So, our graph structure looks as follows:

Here, the white nodes denote the unobserved variables Y_ij and the grey nodes denote observed variables X_ij. Each X_ij is connected to the corresponding Y_ij, and each Y_ij is connected to its neighbors.
Note that this is a Markov network, because there is no cause-effect relation between pixels of an image, and therefore, defining directions of arrows in Bayesian networks is unnatural here.
Our MAP inference problem can be mathematically written as follows:

Here, we used some standard simplification techniques common in maximum log likelihood computation. We will use X and Y(without subscripts) to denote the collection of all X_ij and Y_ij values, respectively.
Now, we need to define our joint distribution P(X, Y) based on our graph structure. Let’s assume that P(X, Y) consists of two kinds of factors — ϕ(X_ij, Y_ij) and ϕ(Y_ij,Y_kl), corresponding to the two kinds of edges in our graph. Next, we define the factors as follows:
ϕ(X_ij, Y_ij) = exp(w_e X_ij Y_ij), where w_e is a parameter greater than zero. This factor takes large values when X_ij and Y_ij are identical, and takes small values when X_ij and Y_ij are different.
ϕ(Y_ij, Y_kl) = exp(w_s Y_ij Y_kl), where w_s is a parameter greater than zero, as before. This factor favors identical values of Y_ij and Y_kl.
Therefore, our joint distribution is given by:

where (i, j) and (k, l) in the second product are adjacent pixels, and Z is a normalization constant.
Plugging this into our MAP inference equation gives:

Note that we have dropped the term containing Zsince it does not affect the solution.
The values of w_e and w_s are obtained using parameter estimation techniques from pairs of ground truth and noisy images. This process is fairly mathematically involved (although, at the end of the day, it is just gradient descent on a complicated function), and therefore, we shall not delve into it here. We will assume that we have obtained the following values of these parameters — w_e = 8 and w_s = 10.
The main focus of this example will be inference. Given these parameters, we want to solve the MAP inference problem above. We can use a variant of belief propagation to do this, but it turns out that there is a much simpler algorithm called Iterated conditional modes (ICM) for graphs with this specific structure.
The basic idea is that at each step, you choose one node, Y_ij, look at the value of the MAP inference expression for both Y_ij = -1 and Y_ij = 1, and pick the one with the higher value. Repeating this process for a fixed number of iterations or until convergence usually works reasonably well.
You can use this Python code to do this for our model.
This is the denoised image returned by the algorithm:

Pretty good, isn’t it? Of course, you can use more fancy techniques, both within graphical models, and outside, to generate something better, but the takeaway from this example is that a simple Markov network with a simple inference algorithm already gives you reasonably good results.
Quantitatively, the noisy image has about 10% of the pixels that are different from the original image, while the denoised image produced by our algorithm has about 0.6% of the pixels that are different from the original image.
It is important to note that the graph that we used is fairly large — the image size is about 440 x 300, so the total number of nodes is close to 264,000. Therefore, exact inference in such models is essentially infeasible, and what we get out of most algorithms, including ICM, is a local optimum.
Let’s recap
In this section, let us briefly review the key concepts we covered in this two-part series:
Graphical models: A graphical model consists of a graph structure where nodes represent random variables and edges represent dependencies between variables.
Bayesian networks: These are directed graphical models, with a conditional probability distribution table associated with each node.
Markov networks: These are undirected graphical models, with a potential function associated with each clique.
Conditional independences: Based on how the nodes in the graph are connected, we can write conditional independence statements of the form “X is independent of Y given Z.”
Parameter estimation: Given some data and the graph structure, we want to fill the CPD tables or compute the potential functions.
Inference: Given a graphical model, we want to answer questions about unobserved variables. These questions are usually one of the following — Marginal inference, posterior inference, and MAP inference.
Inference on general graphical models is computationally intractable. We can divide inference algorithms into two broad categories — exact and approximate. Variable elimination and belief propagation in acyclic graphs are examples of exact inference algorithms. Approximate inference algorithms are necessary for large-scale graphs, and usually fall into sampling-based methods or variational methods.
Conclusions
We looked at some of the core ideas in probabilistic graphical models in this two-part tutorial. As you should be able to appreciate at this point, graphical models provide an interpretable way to model many real-world tasks, where there are dependencies. Using graphical models gives us a way to work on such tasks in a principled manner.
Before we close, it is important to point out that this tutorial, by no means, is complete — many details have been skipped to keep the content intuitive and simple. The standard textbook on probabilistic graphical models is over a thousand pages! This tutorial is meant to serve as a starting point, to get you interested in the field, so that you can look up more rigorous resources.
Here are some additional resources that you can use to dig deeper into the field:
Graphical Models in a Nutshell
Graphical Models textbook
You should also be able to find a few chapters on graphical models in standard machine learning textbooks.

YOU’D ALSO LIKE:
Probabilistic Graphical Models Tutorial — Part 1

Basic terminology and the problem setting
blog.statsbot.co
Machine Learning Algorithms: Which One to Choose for Your Problem

Intuition of using different kinds of algorithms in different tasks
blog.statsbot.co
Neural networks for beginners: popular types and applications

An introduction to neural networks learning
blog.statsbot.co
Machine LearningData ScienceAlgorithmsBayesian StatisticsMarkov Chains
One clap, two clap, three clap, forty?
By clapping more or less, you can signal to us which stories really stand out.


265

Follow
Go to the profile of Prasoon Goyal
Prasoon Goyal
PhD candidate at UT Austin. For more content on machine learning by me, check my Quora profile (https://www.quora.com/profile/Prasoon-Goyal).
Follow
Stats and Bots
Stats and Bots
Data stories on machine learning and analytics. From Statsbot’s makers.
More from Stats and Bots
Neural networks for beginners: popular types and applications
Go to the profile of Jay Shah
Jay Shah

287

Also tagged Markov Chains
Implementing Markov Chain in Swift. Generating Texts.
Go to the profile of Swift The Sorrow
Swift The Sorrow

41

More on Algorithms from Stats and Bots
Data Structures Related to Machine Learning Algorithms
Go to the profile of Peter Mills
Peter Mills

275

An Introduction to different Types of Convolutions in Deep Learning
https://towardsdatascience.com/types-of-convolutions-in-deep-learning-717013397f4d

Homepage
Towards Data Science
Get started
HOMEDATA SCIENCEMACHINE LEARNINGPROGRAMMINGVISUALIZATIONEVENTSLETTERSCONTRIBUTE
Go to the profile of Paul-Louis Pröve
Paul-Louis Pröve
Artificial Intelligence @ PwC
Jul 22
An Introduction to different Types of Convolutions in Deep Learning

Let me give you a quick overview of different types of convolutions and what their benefits are. For the sake of simplicity, I’m focussing on 2D convolutions only.
Convolutions
First we need to agree on a few parameters that define a convolutional layer.

2D convolution using a kernel size of 3, stride of 1 and padding
Kernel Size: The kernel size defines the field of view of the convolution. A common choice for 2D is 3 — that is 3x3 pixels.
Stride: The stride defines the step size of the kernel when traversing the image. While its default is usually 1, we can use a stride of 2 for downsampling an image similar to MaxPooling.
Padding: The padding defines how the border of a sample is handled. A (half) padded convolution will keep the spatial output dimensions equal to the input, whereas unpadded convolutions will crop away some of the borders if the kernel is larger than 1.
Input & Output Channels: A convolutional layer takes a certain number of input channels (I) and calculates a specific number of output channels (O). The needed parameters for such a layer can be calculated by I*O*K, where K equals the number of values in the kernel.
Dilated Convolutions
(a.k.a. atrous convolutions)

2D convolution using a 3 kernel with a dilation rate of 2 and no padding
Dilated convolutions introduce another parameter to convolutional layers called the dilation rate. This defines a spacing between the values in a kernel. A 3x3 kernel with a dilation rate of 2 will have the same field of view as a 5x5 kernel, while only using 9 parameters. Imagine taking a 5x5 kernel and deleting every second column and row.
This delivers a wider field of view at the same computational cost. Dilated convolutions are particularly popular in the field of real-time segmentation. Use them if you need a wide field of view and cannot afford multiple convolutions or larger kernels.
Transposed Convolutions
(a.k.a. deconvolutions or fractionally strided convolutions)
Some sources use the name deconvolution, which is inappropriate because it’s not a deconvolution. To make things worse deconvolutions do exists, but they’re not common in the field of deep learning. An actual deconvolution reverts the process of a convolution. Imagine inputting an image into a single convolutional layer. Now take the output, throw it into a black box and out comes your original image again. This black box does a deconvolution. It is the mathematical inverse of what a convolutional layer does.
A transposed convolution is somewhat similar because it produces the same spatial resolution a hypothetical deconvolutional layer would. However, the actual mathematical operation that’s being performed on the values is different. A transposed convolutional layer carries out a regular convolution but reverts its spatial transformation.

2D convolution with no padding, stride of 2 and kernel of 3
At this point you should be pretty confused, so let’s look at a concrete example. An image of 5x5 is fed into a convolutional layer. The stride is set to 2, the padding is deactivated and the kernel is 3x3. This results in a 2x2 image.
If we wanted to reverse this process, we’d need the inverse mathematical operation so that 9 values are generated from each pixel we input. Afterward, we traverse the output image with a stride of 2. This would be a deconvolution.

Transposed 2D convolution with no padding, stride of 2 and kernel of 3
A transposed convolution does not do that. The only thing in common is it guarantees that the output will be a 5x5 image as well, while still performing a normal convolution operation. To achieve this, we need to perform some fancy padding on the input.
As you can imagine now, this step will not reverse the process from above. At least not concerning the numeric values.
It merely reconstructs the spatial resolution from before and performs a convolution. This may not be the mathematical inverse, but for Encoder-Decoder architectures, it’s still very helpful. This way we can combine the upscaling of an image with a convolution, instead of doing two separate processes.
Separable Convolutions
In a separable convolution, we can split the kernel operation into multiple steps. Let’s express a convolution as y = conv(x, k) where y is the output image, x is the input image, and k is the kernel. Easy. Next, let’s assume k can be calculated by: k = k1.dot(k2). This would make it a separable convolution because instead of doing a 2D convolution with k, we could get to the same result by doing 2 1D convolutions with k1 and k2.

Sobel X and Y filters
Take the Sobel kernel for example, which is often used in image processing. You could get the same kernel by multiplying the vector [1, 0, -1] and [1,2,1].T. This would require 6 instead of 9 parameters while doing the same operation.
The example above shows what’s called a spatial separable convolution, which to my knowledge isn’t used in deep learning. I just wanted to make sure you don’t get confused when stumbling upon those. In neural networks, we commonly use something called a depthwise separable convolution.
This will perform a spatial convolution while keeping the channels separate and then follow with a depthwise convolution. In my opinion, it can be best understood with an example.
Let’s say we have a 3x3 convolutional layer on 16 input channels and 32 output channels. What happens in detail is that every of the 16 channels is traversed by 32 3x3 kernels resulting in 512 (16x32) feature maps. Next, we merge 1 feature map out of every input channel by adding them up. Since we can do that 32 times, we get the 32 output channels we wanted.
For a depthwise separable convolution on the same example, we traverse the 16 channels with 1 3x3 kernel each, giving us 16 feature maps. Now, before merging anything, we traverse these 16 feature maps with 32 1x1 convolutions each and only then start to them add together. This results in 656 (16x3x3 + 16x32x1x1) parameters opposed to the 4608 (16x32x3x3) parameters from above.
The example is a specific implementation of a depthwise separable convolution where the so called depth multiplier is 1. This is by far the most common setup for such layers.
We do this because of the hypothesis that spatial and depthwise information can be decoupled. Looking at the performance of the Xception model this theory seems to work. Depthwise separable convolutions are also used for mobile devices because of their efficient use of parameters.
Questions?
This concludes our little tour through different types of convolutions. I hope it helped to get a brief overview of the matter. Drop a comment if you have any remaining questions and check out this GitHub page for more convolution animations.
Machine LearningConvolutionalCnnNeural NetworksDeep Learning
One clap, two clap, three clap, forty?
By clapping more or less, you can signal to us which stories really stand out.


572
5
Follow
Go to the profile of Paul-Louis Pröve
Paul-Louis Pröve
Medium member since Oct 2017
Artificial Intelligence @ PwC
Follow
Towards Data Science
Towards Data Science
Sharing concepts, ideas, and codes.
More from Towards Data Science
Making Your Own Spotify Discover Weekly Playlist
Go to the profile of Nick Behrens
Nick Behrens

1.4K

Also tagged Neural Networks
Yes you should understand backprop
Go to the profile of Andrej Karpathy
Andrej Karpathy

4.4K

Related reads
Using GANS for semi-supervised learning
In supervised learning, we have a training set of inputs x and class labels y. We train a model that takes x as input and gives y as output…
Go to the profile of Manish Chablani
Manish Chablani

34

Responses
Conversation between Krishna Teja and Paul-Louis Pröve.
Go to the profile of Krishna Teja
Krishna Teja
Sep 18
Hi Paul,
It’s a great post. I would like to add a bit to your explanation on usage of separable convolutions in neural networks.
Note: In neural networks a 2D convolution has 3 dimensions such as height, width and depth where the depth is always equivalent to the number of input channels. For example…
Read more…

8
1 response
Go to the profile of Paul-Louis Pröve
Paul-Louis Pröve
Oct 16
Krishna, thank you so much for taking the time and showing the details of the actual tensor transformations. When you use high level frameworks such as Keras you never touch this functional level. I have a couple of questions:
Are you aware of any papers using spatial separable convolutions in deep learning? It sounds like a…

역시나 래블업에서 다른 작업중에 파생된 결과물을 공개합니다.

구글에서 공개한 머신러닝 용어집 (https://developers.google.com/machine-learning/glossary/) 을 번역한 글입니다. TensorFlow 및 케라스 온라인 강의/실습을 준비하며 용어 통일 및 참조용으로 만든 글인데, 고칠 부분에 대해 피드백을 받을 겸 먼저 공개해 봅니다.

많은 도움이 되셨으면 합니다.

덧) 고수님들께 피드백도 많이 부탁 드립니다!
덧2) 가능하면 연말까지는 하루에 하나씩 backend.AI 및 코드온웹의 파생 결과물들을 공개해 보겠습니다!

https://www.codeonweb.com/@mookiekim/ml-glossary

Top 10 Videos on Deep Learning in Python
https://www.kdnuggets.com/2017/11/top-10-videos-deep-learning-python.html?utm_content=buffer765d8&utm_medium=social&utm_source=facebook.com&utm_campaign=buffer


This ‘Top 10’ list has been created on the basis of best content, and not exactly the number of views. To help you choose an appropriate framework, we first start with a video that compares few of the popular Python DL libraries. I have included the highlights and my views on the pros and cons of each of these 10 items, so you can choose one that best suits your needs. I have saved the best for last- the most comprehensive yet free YouTube course on DL ☺. Let’s begin!

1. Overview: Deep Learning Frameworks compared (96K views) - 5 minutes

Before I actually list the best DL in Python videos, it is important that one understands the differences between the 5 most popular deep learning frameworks -SciKit Learn, TensorFlow, Theano, Keras, and Caffe. This 5 minute video by Siraj Raval gives you the best possible comparison between the pros and cons of each framework and even presents the structure of code samples to help you better decide. Start with this.

2. Playlist: TensorFlow tutorial by Sentdex (114 K views) - 4.5 hours

This playlist of 14 videos by Sentdex is the most well-organized, thoroughly explained ,concise yet easy to follow tutorial on Deep Learning in Python. It includes TensorFlow implementation of a Recurrent Neural Network and Convolutional Neural Network with the MNIST dataset.

3. Individual tutorial: TensorFlow tutorial 02: Convolutional Neural Network (69.7 K views) - 36 minutes

This tutorial by Magnus Pedersen on the YouTube channel Hvass Laboratories, is worth its weight in gold- excellent comments in the code; plus, the instructor speaks without interruption. Watch this video to understand scripts in TensorFlow. Thank me later☺

4. Overview : How to predict stock prices easily (210 K views) - 9 minutes

In this video, Siraj Raval uses a special type of recurrent neural network called an LSTM network. He uses the Keras library with a TensorFlow backend. He explains the reason behind using recurrent nets for time series data and later, uses it to predict the daily closing price of the S&P 500 based on training data for 16 years. The link to the Github code is given in its description box.

5. Tutorial: Introduction to Deep Learning with Python and the Theano library (201 K views) - 52 minutes

If you want a talk on Python with the Theano library in under an hour, targeted towards beginners, then you can refer to this talk by Alec Radford. Unlike most other talks on this topic, this one compares the features of an ‘old’ net versus a ‘modern’ net, ie nets prior to 2000 versus nets post-2012.

6. Playlist: PyTorch Zero to All (3 K views) - 2 hours 15 minutes

In this series of 11 videos, Sung Kim teaches you PyTorch from the ground up. A highlight of this series is Lecture 10, where he teaches you to build a basic CNN with detailed emphasis of understanding the concept of CNN’s using his detailed diagrams.

7. Individual tutorial: TensorFlow tutorial (43.9 K views) - 49 minutes

This single tutorial by Edureka implements DL using TensorFlow. It is a very good tutorial for beginners in TensorFlow. It teaches TensorFlow basics and data structures. It also includes a usecase for using DL as a Naval Mine identifier- to identify whether an underwater obstacle is a rock or a mine.

8. Playlist: Deep Learning with Python (1.8K views) - 83 minutes

The YouTube channel ‘Machine Learning TV‘ has published a series of 15 videos totaling 83 minutes using Theano and Keras to use DL for automatic image captioning. It shows you how to train your first deep neural net for classifying digits from the MNIST dataset. It also has a good explanation on loading and reusing pre-trained models in Theano.

9. Playlist: Deep Learning with Keras- Python (30.3 K views) - 85 minutes

The YouTube channel ‘The SemiColon‘ has published a series of 11 videos on tutorials using Theano and Keras to implement a chatbot using DL. It includes explanations on Convolutional Neural Network, Recurrent Neural Network in Theano with Keras, Neural Networks and Backpropagation in scikit-learn library on the handwriting recognition (MNIST) dataset.

The speaking is punctuated by ‘umms’ and ‘ahhs’, but there is a good explanation on Word2Vec used to build chatbots.

10. Free online course: Deep Learning by Andrew Ng (Full course) (28 K views) - 4 week course

As in my previous Top 10 videos post on ML in Finance, I have saved the best for last☺ .If you want to learn Deep Learning as an online course from arguably the most famous ML instructor- Andrew Ng, then this playlist is for you. Intended as a 4-week course covering 98 videos, this course teaches you DL, Neural Networks, binary classification, derivatives, gradient descent, activation function, backpropagation, regularization, RMSprop, tuning, dropout, training and testing on different distributions, among others, using Python code in a Jupyter notebook.

 

https://m.facebook.com/groups/255834461424286?view=permalink&id=556808447993551

안녕하세요. 유령회원이 오랜만에 글을 적습니다. ;;;
올해도 다 지나갔네요....ㅠㅠ

Pattern Recognition and Machine Learning 책 2장 식(2.117)까지 정리한 노트입니다.

딥러닝 공부하면서 통계학 지식이 너무 없어서 혼자서 책보고 정리한 자료인데 원서라서 저같이 영알못 수알못에다 기억력까지 좋지 못한 경우는 다시보면 정말 첨보는 듯 한 느낌....다시 첨부터 읽어야하는듯한 자괴감을 막기위해 조금씩 주피터로 정리를 했습니다.

처음 관련 내용보았을 때 식이 너무 복잡해서 와 이거 뭐지..먼소리하는지 모르겠는데 싶었는데 어찌어찌 읽기는했네요.

식2.117까지는 꼭알아야겠다 싶어서 정리를 했는데 혹시나 보시고 계신분들 계시면 도움이 되면 좋겠습니다. 좀 어이없을 정도로 식을 풀어적어서 읽기 짜증나실 수 도 있습니다. 그냥 참고삼아..... ㅠㅠ

혹시 오류있으면 지적해주세요. 감사합니다.

http://nbviewer.jupyter.org/github/metamath1/ml-simple-works/blob/master/PRML/prml-chap2.ipynb

NumPy for MATLAB users – Mathesaurus

 

 

http://mathesaurus.sourceforge.net/matlab-numpy.html

 

 

 

NumPy for MATLAB users

Time-stamp: "2007-11-09T16:46:36 vidar"
©2006 Vidar Bronken Gundersen, /mathesaurus.sf.net
Permission is granted to copy, distribute and/or modify this document as long as the above attribution is retained.

Machine Learning · Artificial Inteligence
https://leonardoaraujosantos.gitbooks.io/artificial-inteligence/content/machine_learning.html

How a 22 year old from Shanghai won a global deep learning challenge
https://blog.getnexar.com/how-a-22-year-old-from-shanghai-won-a-global-deep-learning-challenge-76f2299446a1

MNIST Kaggle submission with CNN Keras Swish activation
https://medium.com/@shahariarrabby/mnist-kaggle-submission-with-cnn-keras-switch-activation-62108f9463df

Keras Tutorial: The Ultimate Beginner's Guide to Deep Learning in Python
https://elitedatascience.com/keras-tutorial-deep-learning-in-python?utm_content=bufferbce2c&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer

Introduction to Numpy -1 : An absolute beginners guide to Machine Learning and Data science.

 

https://hackernoon.com/introduction-to-numpy-1-an-absolute-beginners-guide-to-machine-learning-and-data-science-5d87f13f0d51

 

 

Lets get started quickly. Numpy is a math library for python. It enables us to do computation efficiently and effectively. It is better than regular python because of it’s amazing capabilities.

In this article I’m just going to introduce you to the basics of what is mostly required for machine learning and datascience. I’m not going to cover everything that’s possible with numpy library. This is the part one of numpy tutorial series.

The first thing I want to introduce you to is the way you import it.

import numpy as np

Okay, now we’re telling python that “np” is the official reference to numpy from further on.

Let’s create python array and np array.

# python array
a = [1,2,3,4,5,6,7,8,9]

# numpy array
A = np.array([1,2,3,4,5,6,7,8,9])

If I were to print them, I wouldn’t see much difference.

print(a)
print(A)
====================================================================[1, 2, 3, 4, 5, 6, 7, 8, 9]
[1 2 3 4 5 6 7 8 9]

Okay, but why do I have to use an np array instead of a regular array?

The answer is that np arrays are better interms of faster computation and ease of manipulation.

More on those details here, if you’re interested:

Let’s proceed further with more cool stuff. Wait, there was nothing cool we saw yet! Okay, here’s something:

np.arange()

np.arange(0,10,2)
====================================================================array([0, 2, 4, 6, 8])

What arange([start],stop,[step]) does is that it arranges numbers from starting to stop, in steps of step. Here is what it means for np.arange(0,10,2):

return an np list starting from 0 all the way upto 10 but don’t include 10 and increment numbers by 2 each time.

So, that’s how we get :

array([0, 2, 4, 6, 8])

important thing remember here is that the stopping number is not going to be included in the list.

another example:

np.arange(2,29,5)
====================================================================
array([ 2,  7, 12, 17, 22, 27])

Before I proceed further, I’ll have to warn you that this “array” is interchangeably called “matrix” or also “vector”. So don’t get panicked when I say for example “Matrix shape is 2 X 3”. All it means is that array looks something like this:

array([ 2,  7, 12,], 
      [17, 22, 27])

Now, Let’s talk about the shape of a default np array.

Shape is an attribute for np array. When a default array, say for example A is called with shape, here is how it looks.

A = [1, 2, 3, 4, 5, 6, 7, 8, 9] 
A.shape
====================================================================
(9,)

This is a rank 1 matrix(array), where it just has 9 elements in a row. 
Ideally it should be a 1 X 9 matrix right?

I agree with you, so that’s where reshape() comes into play. It is a method that changes the dimensions of your original matrix into your desired dimension.

Let’s look at reshape in action. You can pass a tuple of whatever dimension you want as long as the reshaped matrix and original matrix have the same number of elements.

A = [1, 2, 3, 4, 5, 6, 7, 8, 9]
A.reshape(1,9)
====================================================================
array([[1, 2, 3, 4, 5, 6, 7, 8, 9]])

Notice that reshape returns a multi-dim matrix. Two square brackets in the beginning indicate that. [[1, 2, 3, 4, 5, 6, 7, 8, 9]] is a potentially multi-dim matrix as opposed to [1, 2, 3, 4, 5, 6, 7, 8, 9].

Another example:

B = [1, 2, 3, 4, 5, 6, 7, 8, 9] 
B.reshape(3,3)
====================================================================
array([[1, 2, 3],
       [4, 5, 6],
       [7, 8, 9]])

If I look at B’s shape, it’s going to be (3,3):

B.shape
====================================================================
(3,3)

Perfect. Let’s proceed to np.zeros().

This time it’s your job to tell me what happens looking at this code:

np.zeros((4,3))
====================================================================
???????????

Good, if you thought it’s going to print a 4 X 3 matrix filled with zeros. Here’s the output:

np.zeros((4,3))
====================================================================
array([[ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.]])

np.zeros((n,m)) returns an n x m matrix that contains zeros. It’s as simple as that.

Let’s guess again here: what does is np.eye() do?

Hint: eye() stands for Identity.

np.eye(5)
====================================================================
array([[ 1.,  0.,  0.,  0.,  0.],
       [ 0.,  1.,  0.,  0.,  0.],
       [ 0.,  0.,  1.,  0.,  0.],
       [ 0.,  0.,  0.,  1.,  0.],
       [ 0.,  0.,  0.,  0.,  1.]])

np.eye() returns an identity matrix with the specified dimensions.

What if we have to multiply 2 matrices?

No problem, we have np.dot().

np.dot() performs matrix multiplication, provided both the matrices are “multiply-able”. It just means that the number of columns of the first matrix must match the number of rows in second matrix.

ex: A = (2,3) & B=(3,2). Here number of cols in A= 3. Number of rows in B = 3. Since they match, multiplication is possible.

Let’s illustrate multiplication via np code:

# generate an identity matrix of (3 x 3)
I = np.eye(3)
I
====================================================================
array([[ 1.,  0.,  0.],
       [ 0.,  1.,  0.],
       [ 0.,  0.,  1.]])

# generate another (3 x 3) matrix to be multiplied.
D = np.arange(1,10).reshape(3,3)
D
====================================================================
array([[1, 2, 3],
       [4, 5, 6],
       [7, 8, 9]])

We now prepared both the matrices to be multiplied. Let’s see them in action.

# perform actual dot product.
M = np.dot(D,I)
M
====================================================================
array([[ 1.,  2.,  3.],
       [ 4.,  5.,  6.],
       [ 7.,  8.,  9.]])

Great! Now you know how easy and possible it is to multiply matrices! Also, notice that the entire array is now float type.

What about adding Elements of the matrix?

# add all the elements of matrix.
sum_val = np.sum(M)
sum_val
====================================================================
45.0

np.sum() adds all the elements of the matrix.

However are 2 variants.

1. Sum along the rows.

# sum along the rows
np.sum(M,axis=1)
====================================================================
array([  6.,  15.,  24.])

6 is the sum of 1st row (1, 2, 3).

15 is the sum of 2nd row (4, 5, 6).

24 is the sum of 3rd row (7, 8, 9).

2. Sum along the columns.

# sum along the cols
np.sum(M,axis=0)
====================================================================
array([ 12.,  15.,  18.])

12 is the sum of 1st col (1, 4, 7).

15 is the sum of 2nd col (2, 5, 8).

18 is the sum of 3rd col (3, 6, 9).

Here is the follow up tutorial — part 2 . That’s it at this point.

Here’s a video tutorial explaining everything that I did if you’re interested to consume via video.

If you liked this article, a clap/recommendation would be really appreciated. It helps me to write more such articles.

 

https://m.facebook.com/groups/869457839786067?view=permalink&id=1478681038863741

기계학습을 공부한 산업공학과의 입장에서 공부할만한 한글로된 자료(강의자료 + 영상)을 모아봤습니다. 아래의 자료는 코드 실습이 필요한 내용일 경우는 언어는 전부 Python입니다. (통계학개론강의만 제외) 딥러닝에 관한 한글로된 영상 및 자료는 김성훈 교수님의 "모두의 딥러닝"이 있습니다. (정말 감사드립니다.) 영어로된 좋은 강의도 많지만 (cs231n, cs224d , RL course by David Silver, Neural Network course by Hugo Larochelle), 영어로 본격적으로 딥러닝을 공부하기전에 빠르게 익숙한 언어로 기계학습을 공부해보실 분들은 참고하셔도 좋을 것 같습니다.

cf. 웹프로그래밍은 사심으로 넣어봤습니다.

[k-mooc]
미적분학1 (성균관대 채영도 교수님)
- http://www.kmooc.kr/courses/course-v1:SKKUk+SKKU_EXGB506.01K+2017_SKKU22/about

미적분학2 (성균관대 채영도 교수님)
- http://www.kmooc.kr/courses/course-v1:SKKUk+SKKU_2017_05-01+2017_SKKU01/about

선형대수학 (성균관대 이상구 교수님)
- http://www.kmooc.kr/courses/course-v1:SKKUk+SKKU_2017_01+2017_SKKU01/about

R을 활용한 통계학개론 (부산대 김충락 교수님)
- http://www.kmooc.kr/courses/course-v1:PNUk+RS_C01+2017_KM_009/about

인공지능과 기계학습 (카이스트 오혜연 교수님)
- http://www.kmooc.kr/courses/course-v1:KAISTk+KCS470+2017_K0202/about

[kooc]
파이썬 자료구조 (카이스트 문일철 교수님)
- http://kooc.kaist.ac.kr/datastructure-2017f

영상이해를 위한 최적화 (카이스트 김창익 교수님)
- http://kooc.kaist.ac.kr/optimization2017/lecture/10543

인공지능 및 기계학습 개론 1 (카이스트 문일철 교수님)
- http://kooc.kaist.ac.kr/machinelearning1_17/lecture/10574
- http://seslab.kaist.ac.kr/xe2/page_GBex27

인공지능 및 기계학습 개론 2 (카이스트 문일철 교수님)
- http://kooc.kaist.ac.kr/machinelearning2__17/lecture/10573
- http://seslab.kaist.ac.kr/xe2/page_Dogc43
 
인공지능 및 기계학습 심화 (카이스트 문일철 교수님)
- http://kooc.kaist.ac.kr/machinelearning3
- http://seslab.kaist.ac.kr/xe2/page_lMmY25

[TeamLab]
데이터과학을 위한 파이썬 입문 (가천대 최성철 교수님)
- https://github.com/TeamLab/Gachon_CS50_Python_KMOOC

밑바닥부터 기계학습 (가천대 최성철 교수님)
- https://github.com/TeamLab/machine_learning_from_scratch_with_python

경영과학(가천대 최성철 교수님)
- https://github.com/TeamLab/Gachon_CS50_OR_KMOOC

웹프로그래밍 (가천대 최성철 교수님)
- https://github.com/TeamLab/cs50_web_programming

https://m.facebook.com/story.php?story_fbid=383487222087279&id=303538826748786

Naive Bayes Classification
우리는 베이즈 정리(Bayes Theorem)라고 알려진 사후확률 (posterior probability)에 관한 몇가지 수학을 논의할 것입니다. 이것은 Naive Bayes Classifier의 핵심 부분입니다. 그리고, Python의 sklearn 라이브러리를 탐색하고 논의할 문제에 대해 Python의 Naive Bayes Classifier의 코드를 작성합니다.
이 글은 두부분으로 나누어져 있습니다 . 파트1 에서는 naive bayes classier가 어떻게 작동하는지 설명합니다. 파트2 에서는 Python에서 Naive Bayes Classifier를 제공하는 sklearn 라이브러리를 사용한 프로그래밍 연습으로 구성됩니다. 그리고 우리가 학습시키는 프로그램의 정확성에 대해 논의합니다.

원문
https://medium.com/machine-learning-101/chapter-1-supervised-learning-and-naive-bayes-classification-part-1-theory-8b9e361897d5

Machine Learning  |  Google Developers
https://developers.google.com/machine-learning/glossary/



Products   Machine Learning   Glossary
목차
A
B
C
D
E
F
G
H
I
K
L
M
N
O
P
Q
R
S
T
U
V
W

A


accuracy

The fraction of predictions that a classification model got right. In multi-class classification, accuracy is defined as follows:

Accuracy=CorrectPredictionsTotalNumberOfExamples
In binary classification, accuracy has the following definition:

Accuracy=TruePositives+TrueNegativesTotalNumberOfExamples
See true positive and true negative.


activation function

A function (for example, ReLU or sigmoid) that takes in the weighted sum of all of the inputs from the previous layer and then generates and passes an output value (typically nonlinear) to the next layer.


AdaGrad

A sophisticated gradient descent algorithm that rescales the gradients of each parameter, effectively giving each parameter an independent learning rate. For a full explanation, see this paper.


AUC (Area under the ROC Curve)

An evaluation metric that considers all possible classification thresholds.

The Area Under the ROC curve is the probability that a classifier will be more confident that a randomly chosen positive example is actually positive than that a randomly chosen negative example is positive.

B


backpropagation

The primary algorithm for performing gradient descent on neural networks. First, the output values of each node are calculated (and cached) in a forward pass. Then, the partial derivative of the error with respect to each parameter is calculated in a backward pass through the graph.


baseline

A simple model or heuristic used as reference point for comparing how well a model is performing. A baseline helps model developers quantify the minimal, expected performance on a particular problem.


batch

The set of examples used in one iteration (that is, one gradient update) of model training.


batch size

The number of examples in a batch. For example, the batch size of SGD is 1, while the batch size of a mini-batch is usually between 10 and 1000. Batch size is usually fixed during training and inference; however, TensorFlow does permit dynamic batch sizes.


bias

An intercept or offset from an origin. Bias (also known as the bias term) is referred to as b or w0 in machine learning models. For example, bias is the b in the following formula:

y′=b+w1x1+w2x2+…wnxn
Not to be confused with prediction bias.


binary classification

A type of classification task that outputs one of two mutually exclusive classes. For example, a machine learning model that evaluates email messages and outputs either "spam" or "not spam" is a binary classifier.


binning

See bucketing.


bucketing

Converting a (usually continuous) feature into multiple binary features called buckets or bins, typically based on value range. For example, instead of representing temperature as a single continuous floating-point feature, you could chop ranges of temperatures into discrete bins. Given temperature data sensitive to a tenth of a degree, all temperatures between 0.0 and 15.0 degrees could be put into one bin, 15.1 to 30.0 degrees could be a second bin, and 30.1 to 50.0 degrees could be a third bin.

C


calibration layer

A post-prediction adjustment, typically to account for prediction bias. The adjusted predictions and probabilities should match the distribution of an observed set of labels.


candidate sampling

A training-time optimization in which a probability is calculated for all the positive labels, using, for example, softmax, but only for a random sample of negative labels. For example, if we have an example labeled beagle and dog candidate sampling computes the predicted probabilities and corresponding loss terms for the beagle and dog class outputs in addition to a random subset of the remaining classes (cat, lollipop, fence). The idea is that the negative classes can learn from less frequent negative reinforcement as long as positive classes always get proper positive reinforcement, and this is indeed observed empirically. The motivation for candidate sampling is a computational efficiency win from not computing predictions for all negatives.


checkpoint

Data that captures the state of the variables of a model at a particular time. Checkpoints enable exporting model weights, as well as performing training across multiple sessions. Checkpoints also enable training to continue past errors (for example, job preemption). Note that the graph itself is not included in a checkpoint.


class

One of a set of enumerated target values for a label. For example, in a binary classification model that detects spam, the two classes are spam and not spam. In a multi-class classification model that identifies dog breeds, the classes would be poodle, beagle, pug, and so on.


class-imbalanced data set

A binary classification problem in which the labels for the two classes have significantly different frequencies. For example, a disease data set in which 0.0001 of examples have positive labels and 0.9999 have negative labels is a class-imbalanced problem, but a football game predictor in which 0.51 of examples label one team winning and 0.49 label the other team winning is not a class-imbalanced problem.


classification model

A type of machine learning model for distinguishing among two or more discrete classes. For example, a natural language processing classification model could determine whether an input sentence was in French, Spanish, or Italian. Compare with regression model.


classification threshold

A scalar-value criterion that is applied to a model's predicted score in order to separate the positive class from the negative class. Used when mapping logistic regression results to binary classification. For example, consider a logistic regression model that determines the probability of a given email message being spam. If the classification threshold is 0.9, then logistic regression values above 0.9 are classified as spam and those below 0.9 are classified as not spam.


confusion matrix

An NxN table that summarizes how successful a classification model's predictions were; that is, the correlation between the label and the model's classification. One axis of a confusion matrix is the label that the model predicted, and the other axis is the actual label. N represents the number of classes. In a binary classification problem, N=2. For example, here is a sample confusion matrix for a binary classification problem:

Tumor (predicted) Non-Tumor (predicted)
Tumor (actual) 18 1
Non-Tumor (actual) 6 452
The preceding confusion matrix shows that of the 19 samples that actually had tumors, the model correctly classified 18 as having tumors (18 true positives), and incorrectly classified 1 as not having a tumor (1 false negative). Similarly, of 458 samples that actually did not have tumors, 452 were correctly classified (452 true negatives) and 6 were incorrectly classified (6 false positives).

The confusion matrix of a multi-class confusion matrix can help you determine mistake patterns. For example, a confusion matrix could reveal that a model trained to recognize handwritten digits tends to mistakenly predict 9 instead of 4, or 1 instead of 7. The confusion matrix contains sufficient information to calculate a variety of performance metrics, including precision and recall.


continuous feature

A floating-point feature with an infinite range of possible values. Contrast with discrete feature.


convergence

Informally, often refers to a state reached during training in which training loss and validation loss change very little or not at all with each iteration after a certain number of iterations. In other words, a model reaches convergence when additional training on the current data will not improve the model. In deep learning, loss values sometimes stay constant or nearly so for many iterations before finally descending, temporarily producing a false sense of convergence.

See also early stopping.

See also Convex Optimization by Boyd and Vandenberghe.


convex function

A function typically shaped approximately like the letter U or a bowl. However, in degenerate cases, a convex function is shaped like a line. For example, the following are all convex functions:

L2 loss
Log Loss
L1 regularization
L2 regularization
Convex functions are popular loss functions. That's because when a minimum value exists (as is often the case), many variations of gradient descent are guaranteed to find a point close to the minimum point of the function. Similarly, many variations of stochastic gradient descent have a high probability (though, not a guarantee) of finding a point close to the minimum.

The sum of two convex functions (for example, L2 loss + L1 regularization) is a convex function.

Deep models are usually not convex functions. Remarkably, algorithms designed for convex optimization tend to work reasonably well on deep networks anyway, even though they rarely find a minimum.


cost

Synonym for loss.


cross-entropy

A generalization of Log Loss to multi-class classification problems. Cross-entropy quantifies the difference between two probability distributions. See also perplexity.

D


data set

A collection of examples.


decision boundary

The separator between classes learned by a model in a binary class or multi-class classification problems. For example, in the following image representing a binary classification problem, the decision boundary is the frontier between the orange class and the blue class:

A
well-defined boundary between one class and another.


deep model

A type of neural network containing multiple hidden layers. Deep models rely on trainable nonlinearities.

Contrast with wide model.


dense feature

A feature in which most values are non-zero, typically a Tensor of floating-point values. Contrast with sparse feature.


derived feature

Synonym for synthetic feature.


discrete feature

A feature with a finite set of possible values. For example, a feature whose values may only be animal, vegetable, or mineral is a discrete (or categorical) feature. Contrast with continuous feature.


dropout regularization

A form of regularization useful in training neural networks. Dropout regularization works by removing a random selection of a fixed number of the units in a network layer for a single gradient step. The more units dropped out, the stronger the regularization. This is analogous to training the network to emulate an exponentially large ensemble of smaller networks. For full details, see Dropout: A Simple Way to Prevent Neural Networks from Overfitting.


dynamic model

A model that is trained online in a continuously updating fashion. That is, data is continuously entering the model.

E


early stopping

A method for regularization that involves ending model training before training loss finishes decreasing. In early stopping, you end model training when the loss on a validation data set starts to increase, that is, when generalization performance worsens.


embeddings

A categorical feature represented as a continuous-valued feature. Typically, an embedding is a translation of a high-dimensional vector into a low-dimensional space. For example, you can represent the words in an English sentence in either of the following two ways:

As a million-element (high-dimensional) sparse vector in which all elements are integers. Each cell in the vector represents a separate English word; the value in a cell represents the number of times that word appears in a sentence. Since a single English sentence is unlikely to contain more than 50 words, nearly every cell in the vector will contain a 0. The few cells that aren't 0 will contain a low integer (usually 1) representing the number of times that word appeared in the sentence.
As a several-hundred-element (low-dimensional) dense vector in which each element holds a floating-point value between 0 and 1.
In TensorFlow, embeddings are trained by backpropagating loss just like any other parameter in a neural network.


empirical risk minimization (ERM)

Choosing the model function that minimizes loss on the training set. Contrast with structural risk minimization.


ensemble

A merger of the predictions of multiple models. You can create an ensemble via one or more of the following:

different initializations
different hyperparameters
different overall structure
Deep and wide models are a kind of ensemble.


Estimator

An instance of the tf.Estimator class, which encapsulates logic that builds a TensorFlow graph and runs a TensorFlow session. You may create your own Estimators (as described here) or instantiate pre-made Estimators created by others.


example

One row of a data set. An example contains one or more features and possibly a label. See also labeled example and unlabeled example.

F


false negative (FN)

An example in which the model mistakenly predicted the negative class. For example, the model inferred that a particular email message was not spam (the negative class), but that email message actually was spam.


false positive (FP)

An example in which the model mistakenly predicted the positive class. For example, the model inferred that a particular email message was spam (the positive class), but that email message was actually not spam.


false positive rate (FP rate)

The x-axis in an ROC curve. The FP rate is defined as follows:

FalsePositiveRate=FalsePositivesFalsePositives+TrueNegatives

feature

An input variable used in making predictions.


feature columns (FeatureColumns)

A set of related features, such as the set of all possible countries in which users might live. An example may have one or more features present in a feature column.

Feature columns in TensorFlow also encapsulate metadata such as:

the feature's data type
whether a feature is fixed length or should be converted to an embedding
A feature column can contain a single feature.

"Feature column" is Google-specific terminology. A feature column is referred to as a "namespace" in the VW system (at Yahoo/Microsoft), or a field.


feature cross

A synthetic feature formed by crossing (multiplying or taking a Cartesian product of) individual features. Feature crosses help represent nonlinear relationships.


feature engineering

The process of determining which features might be useful in training a model, and then converting raw data from log files and other sources into said features. In TensorFlow, feature engineering often means converting raw log file entries to tf.Example protocol buffers. See also tf.Transform.

Feature engineering is sometimes called feature extraction.


feature set

The group of feature your machine learning model trains on. For example, postal code, property size, and property condition might comprise a simple feature set for a model that predicts housing prices.


feature spec

Describes the information required to extract features data from the tf.Example protocol buffer. Because the tf.Example protocol buffer is just a container for data, you must specify the following:

the data to extract (that is, the keys for the features)
the data type (for example, float or int)
The length (fixed or variable)
The Estimator API provides facilities for producing a feature spec from a list of FeatureColumns.


full softmax

See softmax. Contrast with candidate sampling.

G


generalization

Refers to your model's ability to make correct predictions on new, previously unseen data as opposed to the data used to train the model.


generalized linear model

A generalization of least squares regression models, which are based on Gaussian noise, to other types of models based on other types of noise, such as Poisson noise or categorical noise. Examples of generalized linear models include:

logistic regression
multi-class regression
least squares regression
The parameters of a generalized linear model can be found through convex optimization.

Generalized linear models exhibit the following properties:

The average prediction of the optimal least squares regression model is equal to the average label on the training data.
The average probability predicted by the optimal logistic regression model is equal to the average label on the training data.
The power of a generalized linear model is limited by its features. Unlike a deep model, a generalized linear model cannot "learn new features."


gradient

The vector of partial derivatives with respect to all of the independent variables. In machine learning, the gradient is the the vector of partial derivatives of the model function. The gradient points in the direction of steepest ascent.


gradient clipping

Capping gradient values before applying them. Gradient clipping helps ensure numerical stability and prevents exploding gradients.


gradient descent

A technique to minimize loss by computing the gradients of loss with respect to the model's parameters, conditioned on training data. Informally, gradient descent iteratively adjusts parameters, gradually finding the best combination of weights and bias to minimize loss.


graph

In TensorFlow, a computation specification. Nodes in the graph represent operations. Edges are directed and represent passing the result of an operation (a Tensor) as an operand to another operation. Use TensorBoard to visualize a graph.

H


heuristic

A practical and nonoptimal solution to a problem, which is sufficient for making progress or for learning from.


hidden layer

A synthetic layer in a neural network between the input layer (that is, the features) and the output layer (the prediction). A neural network contains one or more hidden layers.


hinge loss

A family of loss functions for classification designed to find the decision boundary as distant as possible from each training example, thus maximizing the margin between examples and the boundary. KSVMs use hinge loss (or a related function, such as squared hinge loss). For binary classification, the hinge loss function is defined as follows:

loss=max(0,1−(y′∗y))
where y' is the raw output of the classifier model:

y′=b+w1x1+w2x2+…wnxn
and y is the true label, either -1 or +1.

Consequently, a plot of hinge loss vs. (y * y') looks as follows:

A
plot of hinge loss vs raw classifier score shows a distinct hinge at the
coordinate (1,0).


holdout data

Examples intentionally not used ("held out") during training. The validation data set and test data set are examples of holdout data. Holdout data helps evaluate your model's ability to generalize to data other than the data it was trained on. The loss on the holdout set provides a better estimate of the loss on an unseen data set than does the loss on the training set.


hyperparameter

The "knobs" that you tweak during successive runs of training a model. For example, learning rate is a hyperparameter.

Contrast with parameter.

I


independently and identically distributed (i.i.d)

Data drawn from a distribution that doesn't change, and where each value drawn doesn't depend on values that have been drawn previously. An i.i.d. is the ideal gas of machine learning—a useful mathematical construct but almost never exactly found in the real world. For example, the distribution of visitors to a web page may be i.i.d. over a brief window of time; that is, the distribution doesn't change during that brief window and one person's visit is generally independent of another's visit. However, if you expand that window of time, seasonal differences in the web page's visitors may appear.


inference

In machine learning, often refers to the process of making predictions by applying the trained model to unlabeled examples. In statistics, inference refers to the process of fitting the parameters of a distribution conditioned on some observed data. (See the Wikipedia article on statistical inference.)


input layer

The first layer (the one that receives the input data) in a neural network.


instance

Synonym for example.


inter-rater agreement

A measurement of how often human raters agree when doing a task. If raters disagree, the task instructions may need to be improved. Also sometimes called inter-annotator agreement or inter-rater reliability. See also Cohen's kappa, which is one of the most popular inter-rater agreement measurements.

K


Kernel Support Vector Machines (KSVMs)

A classification algorithm that seeks to maximize the margin between positive and negative classes by mapping input data vectors to a higher dimensional space. For example, consider a classification problem in which the input data set consists of a hundred features. In order to maximize the margin between positive and negative classes, KSVMs could internally map those features into a million-dimension space. KSVMs uses a loss function called hinge loss.

L


L1 loss

Loss function based on the absolute value of the difference between the values that a model is predicting and the actual values of the labels. L1 loss is less sensitive to outliers than L2 loss.


L1 regularization

A type of regularization that penalizes weights in proportion to the sum of the absolute values of the weights. In models relying on sparse features, L1 regularization helps drive the weights of irrelevant or barely relevant features to exactly 0, which removes those features from the model. Contrast with L2 regularization.


L2 loss

See squared loss.


L2 regularization

A type of regularization that penalizes weights in proportion to the sum of the squares of the weights. L2 regularization helps drive outlier weights (those with high positive or low negative values) closer to 0 but not quite to 0. (Contrast with L1 regularization.) L2 regularization always improves generalization in linear models.


label

In supervised learning, the "answer" or "result" portion of an example. Each example in a labeled data set consists of one or more features and a label. For instance, in a housing data set, the features might include the number of bedrooms, the number of bathrooms, and the age of the house, while the label might be the house's price. in a spam detection dataset, the features might include the subject line, the sender, and the email message itself, while the label would probably be either "spam" or "not spam."


labeled example

An example that contains features and a label. In supervised training, models learn from labeled examples.


lambda

Synonym for regularization rate.

(This is an overloaded term. Here we're focusing on the term's definition within regularization.)


layer

A set of neurons in a neural network that process a set of input features, or the output of those neurons.

Also, an abstraction in TensorFlow. Layers are Python functions that take Tensors and configuration options as input and produce other tensors as output. Once the necessary Tensors have been composed, the user can convert the result into an Estimator via a model function.


learning rate

A scalar used to train a model via gradient descent. During each iteration, the gradient descent algorithm multiplies the learning rate by the gradient. The resulting product is called the gradient step.

Learning rate is a key hyperparameter.


least squares regression

A linear regression model trained by minimizing L2 Loss.


linear regression

A type of regression model that outputs a continuous value from a linear combination of input features.


logistic regression

A model that generates a probability for each possible discrete label value in classification problems by applying a sigmoid function to a linear prediction. Although logistic regression is often used in binary classification problems, it can also be used in multi-class classification problems (where it becomes called multi-class logistic regression or multinomial regression).


Log Loss

The loss function used in binary logistic regression.


loss

A measure of how far a model's predictions are from its label. Or, to phrase it more pessimistically, a measure of how bad the model is. To determine this value, a model must define a loss function. For example, linear regression models typically use mean squared error for a loss function, while logistic regression models use Log Loss.

M


machine learning

A program or system that builds (trains) a predictive model from input data. The system uses the learned model to make useful predictions from new (never-before-seen) data drawn from the same distribution as the one used to train the model. Machine learning also refers to the field of study concerned with these programs or systems.


Mean Squared Error (MSE)

The average squared loss per example. MSE is calculated by dividing the squared loss by the number of examples. The values that TensorFlow Playground displays for "Training loss" and "Test loss" are MSE.


metric

A number that you care about. May or may not be directly optimized in a machine-learning system. A metric that your system tries to optimize is called an objective.


mini-batch

A small, randomly selected subset of the entire batch of examples run together in a single iteration of training or inference. The batch size of a mini-batch is usually between 10 and 1,000. It is much more efficient to calculate the loss on a mini-batch than on the full training data.


mini-batch stochastic gradient descent (SGD)

A gradient descent algorithm that uses mini-batches. In other words, mini-batch SGD estimates the gradient based on a small subset of the training data. Vanilla SGD uses a mini-batch of size 1.


ML

Abbreviation for machine learning.


model

The representation of what an ML system has learned from the training data. This is an overloaded term, which can have either of the following two related meanings:

The TensorFlow graph that expresses the structure of how a prediction will be computed.
The particular weights and biases of that TensorFlow graph, which are determined by training.

model training

The process of determining the best model.


Momentum

A sophisticated gradient descent algorithm in which a learning step depends not only on the derivative in the current step, but also on the derivatives in the step(s) that immediately preceded it. Momentum involves computing an exponentially weighted moving average of the gradients over time, analogous to momentum in physics. Momentum sometimes prevents learning from getting stuck in local minima.


multi-class

Classification problems that distinguish among more than two classes. For example, there are approximately 128 species of maple trees, so a model that categorized maple tree species would be multi-class. Conversely, a model that divided emails into only two categories (spam and not spam) would be a binary classification model.

N


NaN trap

When one number in your model becomes a NaN during training, which causes many or all other numbers in your model to eventually become a NaN.

NaN is an abbreviation for "Not a Number."


negative class

In binary classification, one class is termed positive and the other is termed negative. The positive class is the thing we're looking for and the negative class is the other possibility. For example, the negative class in a medical test might be "not tumor." The negative class in an email classifier might be "not spam." See also positive class.


neural network

A model that, taking inspiration from the brain, is composed of layers (at least one of which is hidden) consisting of simple connected units or neurons followed by nonlinearities.


neuron

A node in a neural network, typically taking in multiple input values and generating one output value. The neuron calculates the output value by applying an activation function (nonlinear transformation) to a weighted sum of input values.


normalization

The process of converting an actual range of values into a standard range of values, typically -1 to +1 or 0 to 1. For example, suppose the natural range of a certain feature is 800 to 6,000. Through subtraction and division, you can normalize those values into the range -1 to +1.

See also scaling.


numpy

An open-source math library that provides efficient array operations in Python. pandas is built on numpy.

O


objective

A metric that your algorithm is trying to optimize.


offline inference

Generating a group of predictions, storing those predictions, and then retrieving those predictions on demand. Contrast with online inference.


one-hot encoding

A sparse vector in which:

One element is set to 1.
All other elements are set to 0.
One-hot encoding is commonly used to represent strings or identifiers that have a finite set of possible values. For example, suppose a given botany data set chronicles 15,000 different species, each denoted with a unique string identifier. As part of feature engineering, you'll probably encode those string identifiers as one-hot vectors in which the vector has a size of 15,000.


one-vs.-all

Given a classification problem with N possible solutions, a one-vs.-all solution consists of N separate binary classifiers—one binary classifier for each possible outcome. For example, given a model that classifies examples as animal, vegetable, or mineral, a one-vs.-all solution would provide the following three separate binary classifiers:

animal vs. not animal
vegetable vs. not vegetable
mineral vs. not mineral

online inference

Generating predictions on demand. Contrast with offline inference.


Operation (op)

A node in the TensorFlow graph. In TensorFlow, any procedure that creates, manipulates, or destroys a Tensor is an operation. For example, a matrix multiply is an operation that takes two Tensors as input and generates one Tensor as output.


optimizer

A specific implementation of the gradient descent algorithm. TensorFlow's base class for optimizers is tf.train.Optimizer. Different optimizers (subclasses of tf.train.Optimizer) account for concepts such as:

momentum (Momentum)
update frequency (AdaGrad = ADAptive GRADient descent; Adam = ADAptive with Momentum; RMSProp)
sparsity/regularization (Ftrl)
more complex math (Proximal, and others)
You might even imagine an NN-driven optimizer.


outliers

Values distant from most other values. In machine learning, any of the following are outliers:

Weights with high absolute values.
Predicted values relatively far away from the actual values.
Input data whose values are more than roughly 3 standard deviations from the mean.
Outliers often cause problems in model training.


output layer

The "final" layer of a neural network. The layer containing the answer(s).


overfitting

Creating a model that matches the training data so closely that the model fails to make correct predictions on new data.

P


pandas

A column-oriented data analysis API. Many ML frameworks, including TensorFlow, support pandas data structures as input. See pandas documentation.


parameter

A variable of a model that the ML system trains on its own. For example, weights are parameters whose values the ML system gradually learns through successive training iterations. Contrast with hyperparameter.


Parameter Server (PS)

A job that keeps track of a model's parameters in a distributed setting.


parameter update

The operation of adjusting a model's parameters during training, typically within a single iteration of gradient descent.


partial derivative

A derivative in which all but one of the variables is considered a constant. For example, the partial derivative of f(x, y) with respect to x is the derivative of f considered as a function of x alone (that is, keeping y constant). The partial derivative of f with respect to x focuses only on how x is changing and ignores all other variables in the equation.


partitioning strategy

The algorithm by which variables are divided across parameter servers.


performance

Overloaded term with the following meanings:

The traditional meaning within software engineering. Namely: How fast (or efficiently) does this piece of software run?
The meaning within ML. Here, performance answers the following question: How correct is this model? That is, how good are the model's predictions?

perplexity

One measure of how well a model is accomplishing its task. For example, suppose your task is to read the first few letters of a word a user is typing on a smartphone keyboard, and to offer a list of possible completion words. Perplexity, P, for this task is approximately the number of guesses you need to offer in order for your list to contain the actual word the user is trying to type.

Perplexity is related to cross-entropy as follows:

P=2−crossentropy

pipeline

The infrastructure surrounding a machine learning algorithm. A pipeline includes gathering the data, putting the data into training data files, training one or more models, and exporting the models to production.


positive class

In binary classification, the two possible classes are labeled as positive and negative. The positive outcome is the thing we're testing for. (Admittedly, we're simultaneously testing for both outcomes, but play along.) For example, the positive class in a medical test might be "tumor." The positive class in an email classifier might be "spam."

Contrast with negative class.


precision

A metric for classification models. Precision identifies the frequency with which a model was correct when predicting the positive class. That is:

Precision=TruePositivesTruePositives+FalsePositives

prediction

A model's output when provided with an input example.


prediction bias

A value indicating how far apart the average of predictions is from the average of labels in the data set.


pre-made Estimator

An Estimator that someone has already built. TensorFlow provides several pre-made Estimators, including DNNClassifier, DNNRegressor, and LinearClassifier. You may build your own pre-made Estimators by following these instructions.


pre-trained model

Models or model components (such as embeddings) that have been already been trained. Sometimes, you'll feed pre-trained embeddings into a neural network. Other times, your model will train the embeddings itself rather than rely on the pre-trained embeddings.


prior belief

What you believe about the data before you begin training on it. For example, L2 regularization relies on a prior belief that weights should be small and normally distributed around zero.

Q


queue

A TensorFlow Operation that implements a queue data structure. Typically used in I/O.

R


rank

Overloaded term in ML that can mean either of the following:

The number of dimensions in a Tensor. For instance, a scalar has rank 0, a vector has rank 1, and a matrix has rank 2.
The ordinal position of a class in an ML problem that categorizes classes from highest to lowest. For example, a behavior ranking system could rank a dog's rewards from highest (a steak) to lowest (wilted kale).

rater

A human who provides labels in examples. Sometimes called an "annotator."


recall

A metric for classification models that answers the following question: Out of all the possible positive labels, how many did the model correctly identify? That is:

Recall=TruePositivesTruePositives+FalseNegatives

Rectified Linear Unit (ReLU)

An activation function with the following rules:

If input is negative or zero, output is 0.
If input is positive, output is equal to input.

regression model

A type of model that outputs continuous (typically, floating-point) values. Compare with classification models, which output discrete values, such as "day lily" or "tiger lily."


regularization

The penalty on a model's complexity. Regularization helps prevent overfitting. Different kinds of regularization include:

L1 regularization
L2 regularization
dropout regularization
early stopping (this is not a formal regularization method, but can effectively limit overfitting)

regularization rate

A scalar value, represented as lambda, specifying the relative importance of the regularization function. The following simplified loss equation shows the regularization rate's influence:

minimize(loss function + λ(regularization function))
Raising the regularization rate reduces overfitting but may make the model less accurate.


representation

The process of mapping data to useful features.


ROC (receiver operating characteristic) Curve

A curve of true positive rate vs. false positive rate at different classification thresholds. See also AUC.


root directory

The directory you specify for hosting subdirectories of the TensorFlow checkpoint and events files of multiple models.


Root Mean Squared Error (RMSE)

The square root of the Mean Squared Error.

S


Saver

A TensorFlow object responsible for saving model checkpoints.


scaling

A commonly used practice in feature engineering to tame a feature's range of values to match the range of other features in the data set. For example, suppose that you want all floating-point features in the data set to have a range of 0 to 1. Given a particular feature's range of 0 to 500, you could scale that feature by dividing each value by 500.

See also normalization.


scikit-learn

A popular open-source ML platform. See www.scikit-learn.org.


sequence model

A model whose inputs have a sequential dependence. For example, predicting the next video watched from a sequence of previously watched videos.


session

Maintains state (for example, variables) within a TensorFlow program.


sigmoid function

A function that maps logistic or multinomial regression output (log odds) to probabilities, returning a value between 0 and 1. The sigmoid function has the following formula:

y=11+e−σ
where σ in logistic regression problems is simply:

σ=b+w1x1+w2x2+…wnxn
In other words, the sigmoid function converts σ into a probability between 0 and 1.

In some neural networks, the sigmoid function acts as the activation function.


softmax

A function that provides probabilities for each possible class in a multi-class classification model. The probabilities add up to exactly 1.0. For example, softmax might determine that the probability of a particular image being a dog at 0.9, a cat at 0.08, and a horse at 0.02. (Also called full softmax.)

Contrast with candidate sampling.


sparse feature

Feature vector whose values are predominately zero or empty. For example, a vector containing a single 1 value and a million 0 values is sparse. As another example, words in a search query could also be a sparse feature—there are many possible words in a given language, but only a few of them occur in a given query.

Contrast with dense feature.


squared loss

The loss function used in linear regression. (Also known as L2 Loss.) This function calculates the squares of the difference between a model's predicted value for a labeled example and the actual value of the label. Due to squaring, this loss function amplifies the influence of bad predictions. That is, squared loss reacts more strongly to outliers than L1 loss.


static model

A model that is trained offline.


stationarity

A property of data in a data set, in which the data distribution stays constant across one or more dimensions. Most commonly, that dimension is time, meaning that data exhibiting stationarity doesn't change over time. For example, data that exhibits stationarity doesn't change from September to December.


step

A forward and backward evaluation of one batch.


step size

Synonym for learning rate.


stochastic gradient descent (SGD)

A gradient descent algorithm in which the batch size is one. In other words, SGD relies on a single example chosen uniformly at random from a data set to calculate an estimate of the gradient at each step.


structural risk minimization (SRM)

An algorithm that balances two goals:

The desire to build the most predictive model (for example, lowest loss).
The desire to keep the model as simple as possible (for example, strong regularization).
For example, a model function that minimizes loss+regularization on the training set is a structural risk minimization algorithm.

For more information, see http://www.svms.org/srm/.

Contrast with empirical risk minimization.


summary

In TensorFlow, a value or set of values calculated at a particular step, usually used for tracking model metrics during training.


supervised machine learning

Training a model from input data and its corresponding labels. Supervised machine learning is analogous to a student learning a subject by studying a set of questions and their corresponding answers. After mastering the mapping between questions and answers, the student can then provide answers to new (never-before-seen) questions on the same topic. Compare with unsupervised machine learning.


synthetic feature

A feature that is not present among the input features, but is derived from one or more of them. Kinds of synthetic features include the following:

Multiplying one feature by itself or by other feature(s). (These are termed feature crosses.)
Dividing one feature by a second feature.
Bucketing a continuous feature into range bins.
Features created by normalizing or scaling alone are not considered synthetic features.

T


target

Synonym for label.


Tensor

The primary data structure in TensorFlow programs. Tensors are N-dimensional (where N could be very large) data structures, most commonly scalars, vectors, or matrices. The elements of a Tensor can hold integer, floating-point, or string values.


Tensor Processing Unit (TPU)

An ASIC (application-specific integrated circuit) that optimizes the performance of TensorFlow programs.


Tensor rank

See rank.


Tensor shape

The number of elements a Tensor contains in various dimensions. For example, a [5, 10] Tensor has a shape of 5 in one dimension and 10 in another.


Tensor size

The total number of scalars a Tensor contains. For example, a [5, 10] Tensor has a size of 50.


TensorBoard

The dashboard that displays the summaries saved during the execution of one or more TensorFlow programs.


TensorFlow

A large-scale, distributed, machine learning platform. The term also refers to the base API layer in the TensorFlow stack, which supports general computation on dataflow graphs.

Although TensorFlow is primarily used for machine learning, you may also use TensorFlow for non-ML tasks that require numerical computation using dataflow graphs.


TensorFlow Playground

A program that visualizes how different hyperparameters influence model (primarily neural network) training. Go to http://playground.tensorflow.org to experiment with TensorFlow Playground.


TensorFlow Serving

A platform to deploy trained models in production.


test set

The subset of the data set that you use to test your model after the model has gone through initial vetting by the validation set.

Contrast with training set and validation set.


tf.Example

A standard protocol buffer for describing input data for machine learning model training or inference.


training

The process of determining the ideal parameters comprising a model.


training set

The subset of the data set used to train a model.

Contrast with validation set and test set.


true negative (TN)

An example in which the model correctly predicted the negative class. For example, the model inferred that a particular email message was not spam, and that email message really was not spam.


true positive (TP)

An example in which the model correctly predicted the positive class. For example, the model inferred that a particular email message was spam, and that email message really was spam.


true positive rate (TP rate)

Synonym for recall. That is:

TruePositiveRate=TruePositivesTruePositives+FalseNegatives
True positive rate is the y-axis in an ROC curve.

U


unlabeled example

An example that contains features but no label. Unlabeled examples are the input to inference. In semi-supervised and unsupervised learning, unlabeled examples are used during training.


unsupervised machine learning

Training a model to find patterns in a data set, typically an unlabeled data set.

The most common use of unsupervised machine learning is to cluster data into groups of similar examples. For example, an unsupervised machine learning algorithm can cluster songs together based on various properties of the music. The resulting clusters can become an input to other machine learning algorithms (for example, to a music recommendation service). Clustering can be helpful in domains where true labels are hard to obtain. For example, in domains such as anti-abuse and fraud, clusters can help humans better understand the data.

Another example of unsupervised machine learning is principal component analysis (PCA). For example, applying PCA on a data set containing the contents of millions of shopping carts might reveal that shopping carts containing lemons frequently also contain antacids.

Compare with supervised machine learning.

V


validation set

A subset of the data set—disjunct from the training set—that you use to adjust hyperparameters.

Contrast with training set and test set.

W


weight

A coefficient for a feature in a linear model, or an edge in a deep network. The goal of training a linear model is to determine the ideal weight for each feature. If a weight is 0, then its corresponding feature does not contribute to the model.


wide model

A linear model that typically has many sparse input features. We refer to it as "wide" since such a model is a special type of neural network with a large number of inputs that connect directly to the output node. Wide models are often easier to debug and inspect than deep models. Although wide models cannot express nonlinearities through hidden layers, they can use transformations such as feature crossing and bucketization to model nonlinearities in different ways.

Contrast with deep model.

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 3.0 License, and code samples are licensed under the Apache 2.0 License. For details, see our Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 9월 19, 2017.
Connect

Blog
Facebook
Google+
Medium
Twitter
YouTube
Programs

Women Techmakers
Agency Program
GDG
Google Developers Experts
Startup Launchpad
Developer Consoles

Google API Console
Google Cloud Platform Console
Google Play Console
Firebase Console
Cast SDK Developer Console
Chrome Web Store Dashboard

Android
Chrome
Firebase
Google Cloud Platform
모든 제품
한국어
Terms Privacy
Sign up for the Google Developers newsletter
구독하기