phi(z) = 1 / (1 + e^-(alpha + beta*z)) 일 때 다양한 alpha, beta에 따른 logistic function plot.

import matplotlib.pyplot as plt
import numpy as np

def sigmoid1(x, alpha, beta):
    return 1.0 / (1.0 + np.exp(-(alpha + beta*x)))
z = np.arange(-7, 7, 0.1)
# phi_z = sigmoid1(alpha=3, beta=2, z)
c0 = [sigmoid1(x, alpha=0, beta=1) for x in z]
plt.plot(z, c0, label='alpha=0, beta=1', linestyle='--')
c1 = [sigmoid1(x, alpha=1, beta=1) for x in z]
plt.plot(z, c1, label='alpha=1, beta=1')
c2 = [sigmoid1(x, alpha=1, beta=2) for x in z]
plt.plot(z, c2, label='alpha=1, beta=2')
c3 = [sigmoid1(x, alpha=0, beta=2) for x in z]
plt.plot(z, c3, label='alpha=0, beta=2')
plt.axvline(0.0, color='k')
plt.ylim(-0.1, 1.1)
plt.xlabel('z')
plt.ylabel('$\phi (z)$')
# y축의 눈금과 격자선
plt.yticks([0.0, 0.5, 1.0])
ax = plt.gca()
ax.yaxis.grid(True)
plt.legend(loc='best')
plt.tight_layout()
plt.show()

https://www.facebook.com/groups/DeepAI/permalink/2577160382575423/?sfnsn=mo

Having an unbalanced dataset is not an uncommon problem in real world. Interested in knowing how to handle Imbalanced Classification problem? Checkout this brief introduction from team at Robofied.

#AI #ML #artificialintelligence #machinelearning #DL #deeplearning #neuralnetworks #dataanalytics #datascience #data #bigdata #technology #automation #datascientist #programming #code #coding #developer #tech #love #geek #ecommerce #design #startup #contentmarketing #branding #business #robofied

PR(Precision Recall) curve VS ROC curve에 대한 근본적인 이해를 돕는 글들이 있다. 

                       | predicted negative | predicted positive |
|-----------------|---------------------|---------------------|
| actual negative | True Negative (TN) | False Positive (FP) |
| actual positive  | False Negative (FN) | True Positive (TP) |

[1]과 [2]에서 설명하는 내용은 비슷하다. ROC 커브는 Recall(=Sensitivity)과  False Positive Rate(= 1-Specificity)인데, 이 둘은 모두 actual class가 1인 것중에서 1로 분류한 비율과 0인 것 중에서 1로 분류한 것의 비율을 보는 것인데, PR 커브는 Precision과 Recall(=Sensitivity)을 사용하는데, Recall은 같은데 Precision은 1로 분류된 것 중에서 실제 1의 비율을 보는 것이다. Precision은 실제클래스가 1인 샘플들의 base probability 또는 prior probability가 달라짐에 따라서 Precision이 달라진다. 반면, Recall(=Sensitivity)과 FPR은 base probability 또는 prior probability가 항상 같다. [3]에 PR, ROC를 잘 설명한 그림이 있어 복사해왔다. 

위 그림을 예로들어 설명하면 (분홍색 선과 청록색 선은 threshold를 나타냄), 

Recall(=Sensitivity) : 전체 노란 샘플개수 중 분홍색 선 위쪽에 존재하는 노란 샘플개수 비율.

False Positive Rate(= 1-Specificity) : 전체 파란 샘플개수 중 분홍색 선 위쪽에 존재하는 파란 샘플개수 비율.

Precision : 분홍색 선 위쪽에 존재하는 노란색 샘플과 파란색 샘플의 합 중에서 분홍색 선 위쪽에 존재하는 노란 샘플개수 비율.

Recall(=Sensitivity)과 FPR은 threshold가 청록cyan색에서 분홍색 라인으로 올라가도 분자는 바뀌어도 분모(base probability 또는 prior probability)는 그대로이다. 반면, Precision은 threshold가 바뀌면 분모(base probability 또는 prior probability)가 바뀌고 분자도 바뀐다. 위와 같이 imbalanced data인 경우 threshold가 위로 올라가면서 Precision의 분모가 급격히 바뀌게 된다. 위 그림에서 클래스 1인 노란색 샘플이 파란색 샘플만큼 많은 경우를 상상해보자. 즉 파란색 샘플들은 그대로인데, 노란색 샘플들이 prediction score가 대략 [-0.3, 1] 범위에 노란 샘플들이 많이 존재하는 경우다. 그러면 threshold가 올라가면서 Precision의 분모가 급격히 변하지 않고 smooth하게 변할 것이다. 윗부분이 노란색 샘플들이 많으므로. 그러면 threshold가 위로 갈수록 분모와 분자는 거의 비슷한 값을 가질 것이다. threshold가 변함에 따라 Precision이 급격하게 변하지 않는 것이다. 따라서 Precision은 1클래스가 rare한 경우에 민감하여 분류 performance를 잘 반영해준다. 

[1] stats.stackexchange.com/questions/7207/roc-vs-precision-and-recall-curves

The key difference is that ROC curves will be the same no matter what the baseline probability is, but PR curves may be more useful in practice for needle-in-haystack type problems or problems where the "positive" class is more interesting than the negative class.

To show this, first let's start with a very nice way to define precision, recall and specificity. Assume you have a "positive" class called 1 and a "negative" class called 0. Y^Y^ is your estimate of the true class label YY. Then:

Precision            =P(Y=1|Y^=1)

Recall=Sensitivity =P(Y^=1|Y=1)

Specificity           =P(Y^=0|Y=0)

The key thing to note is that sensitivity/recall and specificity, which make up the ROC curve, are probabilities conditioned on the true class label. Therefore, they will be the same regardless of what P(Y=1)P(Y=1) is. Precision is a probability conditioned on your estimate of the class label and will thus vary if you try your classifier in different populations with different baseline P(Y=1). However, it may be more useful in practice if you only care about one population with known background probability and the "positive" class is much more interesting than the "negative" class. (IIRC precision is popular in the document retrieval field, where this is the case.) This is because it directly answers the question, "What is the probability that this is a real hit given my classifier says it is?".

Interestingly, by Bayes' theorem you can work out cases where specificity can be very high and precision very low simultaneously. All you have to do is assume P(Y=1) is very close to zero. In practice I've developed several classifiers with this performance characteristic when searching for needles in DNA sequence haystacks.

IMHO when writing a paper you should provide whichever curve answers the question you want answered (or whichever one is more favorable to your method, if you're cynical). If your question is: "How meaningful is a positive result from my classifier given the baseline probabilities of my problem?", use a PR curve. If your question is, "How well can this classifier be expected to perform in general, at a variety of different baseline probabilities?", go with a ROC curve.

[2] www.quora.com/What-is-the-difference-between-a-ROC-curve-and-a-precision-recall-curve-When-should-I-use-each

There is a very important difference between what a ROC curve represents vs that of a PRECISION vs RECALL curve.

Remember, a ROC curve represents a relation between sensitivity (RECALL) and False Positive Rate (NOT PRECISION). Sensitivity is the other name for recall but the False Positive Rate is not PRECISION.

Recall/Sensitivity is the measure of the probability that your estimate is 1 given all the samples whose true class label is 1. It is a measure of how many of the positive samples have been identified as being positive.

Specificity is the measure of the probability that your estimate is 0 given all the samples whose true class label is 0. It is a measure of how many of the negative samples have been identified as being negative.

PRECISION on the other hand is different. It is a measure of the probability that a sample is a true positive class given that your classifier said it is positive. It is a measure of how many of the samples predicted by the classifier as positive is indeed positive. Note here that this changes when the base probability or prior probability of the positive class changes. Which means PRECISION depends on how rare is the positive class. In other words, it is used when positive class is more interesting than the negative class.

So, if your problem involves kind of searching a needle in the haystack when for ex: the positive class samples are very rare compared to the negative classes, use a precision recall curve. Othwerwise use a ROC curve because a ROC curve remains the same regardless of the baseline prior probability of your positive class (the important rare class).

[3] towardsdatascience.com/on-roc-and-precision-recall-curves-c23e9b63820c

For the benefit of new folks in the field, here's a list of some well known and prestigious free online courses/books on Data Science and machine learning (for certificate you might have to pay though in some of these). Do add if you know any good free courses/resources in comments:

1) Machine Learning (Stanford University, Andrew Ng Course, Coursera)

This is a very respected course and widely accepted in machine learning industry.

https://www.coursera.org/learn/machine-learning

Do the assignments of this course in python as the course is in Octave language which isn't used that much in industry. Assignment solutions in Python.

https://github.com/dibgerge/ml-coursera-python-assignments

https://github.com/jdwittenauer/ipython-notebooks

2) Python Data Science Handbook (Jake VanderPlas)

Famous Python Data Science handbook topic-wise:

https://jakevdp.github.io/PythonDataScienceHandbook/

3) Fast.ai ML course (Jeremy Howard)

http://course18.fast.ai/ml

This course is designed by Kaggle 2 time competition winner Jeremy Howard and he is also chief scientist at Kaggle.

4) ML Course by IBM (edX)

https://www.edx.org/course/machine-learning-with-python-a-practical-introduct

5) Machine Learning Crash Course by Google

https://developers.google.com/machine-learning/crash-course/ml-intro

6) The Analytics Edge - Massachusetts Institute of Technology (MIT)

https://courses.edx.org/courses/course-v1:MITx+15.071x+1T2020/course/

7) Deep Learning (Andrew Ng, Coursera)

https://www.coursera.org/specializations/deep-learning

【자기주도온라인평생학습 】기초 탈출 파이썬 판다스(Python Pandas) 무료 온라인 강의 리스트 – 8개 교육과정

컴퓨터 프로그래밍에서 pandas는 데이터 조작 및 분석을 위해 Python 프로그래밍 언어로 작성된 소프트웨어 라이브러리입니다. 특히, 숫자 테이블 및 시계열 조작을위한 데이터 구조 및 조작을 제공합니다. 3 절 BSD 라이센스에 따라 공개 된 무료 소프트웨어입니다. 위키백과(영어)

Pandas 튜토리얼 1 – 데이콘 (YouTube)
: http://bitly.kr/gxleF15yLk

Pandas 팬더스 강의 기초 실습 – Minsuk Heo 허민석 (YouTube)
: http://bitly.kr/CUDIQ7q3z0E

토크ON 44차. Pandas로 하는 시계열 데이터분석 | T아카데미 – SKplanet Tacademy (YouTube)
: http://bitly.kr/Pjg0hLuDTJ

파이썬 판다스로 데이터 분석하고 엑셀로 뽑기 – Kyeongrok Kim (YouTube)
: http://bitly.kr/tzB9dPbQOJB

데이터분석/판다스 강의 – NeoWizard (YouTube)
: http://bitly.kr/EIPbWyAadFK

Python을 활용한 데이터분석 기초 | T아카데미 – SKplanet Tacademy (YouTube)
: http://bitly.kr/XXjLP2C0n6

판다스 – Sungchul Lee (YouTube)
: http://bitly.kr/aPAoUoVOPg

빅데이터 프로그래밍 - 건국대학교 이정환교수 (KOCW)
: http://bitly.kr/tmI1VuiDMA

출처 : http://bitly.kr/XMGZ6Ntol3

【자기주도온라인평생학습】 http://withmooc.com/courses/

https://towardsdatascience.com/clustering-metrics-better-than-the-elbow-method-6926e1f723a6

Clustering metrics better than the elbow-method

We show what metric to use for visualizing and determining an optimal number of clusters much better than the usual practice — elbow method.

Tirthajyoti Sarkar

Follow

Sep 7, 2019 · 7 min read

Introduction

Clustering is an important part of the machine learning pipeline for business or scientific enterprises utilizing data science. As the name suggests, it helps to identify congregations of closely related (by some measure of distance) data points in a blob of data, which, otherwise, would be difficult to make sense of.

However, mostly, the process of clustering falls under the realm of unsupervised machine learning. And unsupervised ML is a messy business.

There is no known answers or labels to guide the optimization process or measure our success against. We are in the uncharted territory.

Machine Learning for Humans, Part 3: Unsupervised Learning

Clustering and dimensionality reduction: k-means clustering, hierarchical clustering, PCA, SVD.

medium.com

It is, therefore, no surprise, that a popular method like k-means clustering does not seem to provide a completely satisfactory answer when we ask the basic question:

“How would we know the actual number of clusters, to begin with?”

This question is critically important because of the fact that the process of clustering is often a precursor to further processing of the individual cluster data and therefore, the amount of computational resource may depend on this measurement.

In the case of a business analytics problem, repercussion could be worse. Clustering is often done for such analytics with the goal of market segmentation. It is, therefore, easily conceivable that, depending on the number of clusters, appropriate marketing personnel will be allocated to the problem. Consequently, a wrong assessment of the number of clusters can lead to sub-optimum allocation of precious resources.

Source: https://www.singlegrain.com/digital-marketing/strategists-guide-marketing-segmentation/

The elbow method

For the k-means clustering method, the most common approach for answering this question is the so-called elbow method. It involves running the algorithm multiple times over a loop, with an increasing number of cluster choice and then plotting a clustering score as a function of the number of clusters.

What is the score or metric which is being plotted for the elbow method? Why is it called the ‘elbow’ method?

A typical plot looks like following,

The score is, in general, a measure of the input data on the k-means objective function i.e. some form of intra-cluster distance relative to inner-cluster distance.

For example, in Scikit-learn’s k-means estimator, a score method is readily available for this purpose.

But look at the plot again. It can get confusing sometimes. Is it 4, 5, or 6, that we should take as the optimal number of clusters here?

Not so obvious always, is it?

Silhouette coefficient — a better metric

The Silhouette Coefficient is calculated using the mean intra-cluster distance (a) and the mean nearest-cluster distance (b) for each sample. The Silhouette Coefficient for a sample is (b - a) / max(a, b). To clarify, b is the distance between a sample and the nearest cluster that the sample is not a part of. We can compute the mean Silhouette Coefficient over all samples and use this as a metric to judge the number of clusters.

Here is a video from Orange on this topic,

For illustration, we generated random data points using Scikit-learn’s make_blob function over 4 feature dimensions and 5 cluster centers. So, the ground truth of the problem is that the data is generated around 5 cluster centers. However, the k-means algorithm has no way of knowing this.

The clusters can be plotted (pairwise features) as follows,

Next, we run k-means algorithm with a choice of k=2 to k=12 and calculate the default k-means score and the mean silhouette coefficient for each run and plot them side by side.

The difference could not be starker. The mean silhouette coefficient increases up to the point when k=5 and then sharply decreases for higher values of k i.e. it exhibits a clear peak at k=5, which is the number of clusters the original dataset was generated with.

Silhouette coefficient exhibits a peak characteristic as compared to the gentle bend in the elbow method. This is easier to visualize and reason with.

If we increase the Gaussian noise in the data generation process, the clusters look more overlapped.

In this case, the default k-means score with elbow method produces even more ambiguous result. In the elbow plot below, it is difficult to pick a suitable point where the real bend occurs. Is it 4, 5, 6, or 7?

But the silhouette coefficient plot still manages to maintain a peak characteristic around 4 or 5 cluster centers and make our life easier.

In fact, if you look back at the overlapped clusters, you will see that mostly there are 4 clusters visible — although the data was generated using 5 cluster centers, due to high variance, only 4 clusters are structurally showing up. Silhouette coefficient picks up this behavior easily and shows the optimal number of clusters somewhere between 4 and 5.

BIC score with a Gaussian Mixture Model

There are other excellent metrics for determining the true count of the clusters such as Bayesian Information Criterion (BIC) but they can be applied only if we are willing to extend the clustering algorithm beyond k-means to the more generalized version — Gaussian Mixture Model (GMM).

Basically, a GMM treats a blob of data as a superimposition of multiple Gaussian datasets with separate mean and variances. Then it applies the Expectation-Maximization (EM) algorithm to determine these mean and variances approximately.

Gaussian Mixture Models Explained

In the world of Machine Learning, we can distinguish two main areas: Supervised and unsupervised learning. The main…

towardsdatascience.com

The idea of BIC as regularization

You may recognize the term BIC from statistical analysis or your previous interaction with linear regression. BIC and AIC (Akaike Information Criterion) are used as regularization techniques in linear regression for the variable selection process.

BIC/AIC is used for regularization of linear regression model.

The idea is applied in a similar manner here for BIC. In theory, extremely complex clusters of data can also be modeled as a superimposition of a large number of Gaussian datasets. There is no restriction on how many Gaussians to use for this purpose.

But this is similar to increasing model complexity in linear regression, where a large number of features can be used to fit any arbitrarily complex data, only to lose the generalization power, as the overly complex model fits the noise instead of the true pattern.

The BIC method penalizes a large number of Gaussians and tries to keep the model simple enough to explain the given data pattern.

The BIC method penalizes a large number of Gaussians i.e. an overly complex model.

Consequently, we can run the GMM algorithm for a range of cluster centers, and the BIC score will increase up to a point, but after that will start decreasing as the penalty term grows.

Summary

Here is the Jupyter notebook for this article. Feel free to fork and play with it.

We discussed a couple of alternative options to the often-used elbow method for picking up the right number of clusters in an unsupervised learning setting using the k-means algorithm.

We showed that Silhouette coefficient and BIC score (from the GMM extension of k-means) are better alternatives to the elbow method for visually discerning the optimal number of clusters.

Ifyou have any questions or ideas to share, please contact the author at tirthajyoti[AT]gmail.com. Also, you can check the author’s GitHub repositories for other fun code snippets in Python, R, and machine learning resources. If you are, like me, passionate about machine learning/data science, please feel free to add me on LinkedIn or follow me on Twitter.