0 Voturi pozitive0 Voturi negative

0 (de) vizualizări41 paginiMar 20, 2019

Lesson_4.1_-_Unsupervised_Learning_Partitioning_Methods.pdf

© © All Rights Reserved

PDF, TXT sau citiți online pe Scribd

© All Rights Reserved

0 (de) vizualizări

Lesson_4.1_-_Unsupervised_Learning_Partitioning_Methods.pdf

© All Rights Reserved

- Discovery of Patterns and evaluation of Clustering Algorithms in SocialNetwork Data (Face book 100 Universities) through Data Mining Techniques and Methods
- Plant Recognition using Hog and Artificial Neural Network
- LOG-BACKUP UTILITY: A SECURE LOG MINING SYSTEM
- bahan
- Sml
- Adaptive Neuro-Fuzzy Inference System based Fractal Image Compression
- Neural Nw K-means
- qmat97_iwanowski_skoneczny_szostakowski.pdf
- Ubicc Paper 419
- NWU-EECS-07-05
- Thermal image sgmentation
- IJETR031830
- 08540811.pdf
- Document Clustering in Correlation Similarity
- Knn
- Detecting the Number of Clusters During Expectation-Maximization (ICMLC 2010)
- G Unit en Ml Introduction
- Clustering Algorithms For High Dimensional Data.pdf
- ML Topic1A
- 108.pdf

Sunteți pe pagina 1din 41

Anis ur Rahman

Department of Computing

NUST-SEECS

Islamabad

December 3, 2018

1 / 41

Unsupervised Learning: Nearest-Neighbor Classification

Roadmap

1 Basic Concepts

2 K-Means

3 K-Medoids

2 / 41

Unsupervised Learning: Nearest-Neighbor Classification

Cluster Analysis

Group data to form new categories (i.e., clusters), e.g., cluster

houses to find distribution patterns

Principle

Maximizing intra-class similarity & minimizing interclass similarity

Typical Applications

WWW, Social networks, Marketing, Biology, Library, etc.

3 / 41

Unsupervised Learning: Nearest-Neighbor Classification

Partitioning Methods

Given

A data set of n objects

K the number of clusters to form

Organize the objects into k partitions (k ≤ n) where each partition

represents a cluster

The clusters are formed to optimize an objective partitioning

criterion

Objects within a cluster are similar

Objects of different clusters are dissimilar

4 / 41

Unsupervised Learning: Nearest-Neighbor Classification

1 Eager learning

Given a set of training set, constructs a classification model before

receiving new (e.g., test) data to classify

e.g. decision tree induction, Bayesian classification, rule-based

classification

2 Lazy learning

Simply stores training data (or only minor processing) and waits

until it is given a new instance

Lazy learners take less time in training but more time in predicting

e.g., k-nearest-neighbor classifiers, case-based reasoning classifiers

5 / 41

Unsupervised Learning: Nearest-Neighbor Classification

We are given a new image and want to find the most similar

images in the database

Represent faces by (relatively) invariant values, e.g., ratio of nose

width to eye width

Each image represented by a large number of numerical features

Problem: given the features of a new face, find those in the DB that

are close in at least ¾ (say) of the features

6 / 41

Unsupervised Learning: Nearest-Neighbor Classification

Lazy Learning

1 k-nearest neighbor approach

Instances represented as points in a Euclidean space.

2 Case-based reasoning

Uses symbolic representations and knowledge-based inference

3 Locally weighted regression

Constructs local approximation

7 / 41

Unsupervised Learning: Nearest-Neighbor Classification

Roadmap

1 Basic Concepts

2 K-Means

3 K-Medoids

8 / 41

Unsupervised Learning: Nearest-Neighbor Classification

K-Means

Goal

Create 3 clusters (partitions)

2 Assign each object to the closest

centroid to form Clusters

9 / 41

Unsupervised Learning: Nearest-Neighbor Classification

K-Means

Goal

Create 3 clusters (partitions)

4 Recompute Clusters

10 / 41

Unsupervised Learning: Nearest-Neighbor Classification

K-Means Algorithm

Input

K : the number of clusters

D : a data set containing n objects

Output: A set of k clusters

Method:

1 Arbitrary choose k objects from D as in initial cluster centers

2 Repeat

3 Reassign each object to the most similar cluster based on the mean

value of the objects in the cluster

4 Update the cluster means

5 Until no change

11 / 41

Unsupervised Learning: Nearest-Neighbor Classification

K-Means Properties

square-error function

k X

X

E= |p − mi |2

i =1 p∈Ci

E : the sum of the squared error for all objects in the dataset

P : the data point in the space representing an object

mi : is the mean of cluster Ci

It works well when the clusters are compact clouds that are rather

well separated from one another

12 / 41

Unsupervised Learning: Nearest-Neighbor Classification

K-Means: Advantages

data sets

The computational complexity of the algorithm is O (nkt)

n: the total number of objects

k : the number of clusters

t: the number of iterations

Normally: k << n and t << n

13 / 41

Unsupervised Learning: Nearest-Neighbor Classification

K-Means: Disadvantages

Users need to specify k

K-means is not suitable for discovering clusters with nonconvex

shapes or clusters of very different size

It is sensitive to noise and outlier data points (can influence the

mean value)

14 / 41

Unsupervised Learning: Nearest-Neighbor Classification

K-Means demo

Demo

15 / 41

Unsupervised Learning: Nearest-Neighbor Classification

Variations of K-Means

Selection of the initial k means

Dissimilarity calculations

Strategies to calculate cluster means

How can we change K-Means to deal with categorical data?

Handling categorical data: k-modes (Huang’98)

Replacing means of clusters with modes

Using new dissimilarity measures to deal with categorical objects

Using a frequency-based method to update modes of clusters

A mixture of categorical and numerical data

16 / 41

Unsupervised Learning: Nearest-Neighbor Classification

k-Nearest-Neighbor Classifiers

It has since been widely used in the area of pattern recognition

The training instances are described by n attributes

Each instance represents a point in an n-dimensional space

A k-nearest-neighbor classifier searches the pattern space for the

k training instances that are closest to the unknown instance

17 / 41

Unsupervised Learning: Nearest-Neighbor Classification

k-Nearest-Neighbor Classifiers

Example:

We are interested in classifying the type of drug a patient should

be prescribed

Based on the age of the patient and the patient’s

sodium/potassium ratio (Na/K)

Dataset includes 200 patients

18 / 41

Unsupervised Learning: Nearest-Neighbor Classification

Scatter plot

19 / 41

Unsupervised Learning: Nearest-Neighbor Classification

Main questions:

How many neighbors should we consider? That is, what is k?

How do we measure distance?

Should all points be weighted equally, or should some points have

more influence than others?

20 / 41

Unsupervised Learning: Nearest-Neighbor Classification

k-Nearest-Neighbor Classifiers

distance, dist(X1 , X2 )

The Euclidean distance between two points or instances, say,

X1 = (x11 , x12 , · · · , x1n ) and X2 = (x21 , x22 , · · · , x2n ), is:

v

t n

X

dist(X1 , X2 ) = (x1n − x2n )2

i =1

Refer to cluster analysis for more distance metrics

21 / 41

Unsupervised Learning: Nearest-Neighbor Classification

k-Nearest-Neighbor Classifiers

This helps prevent attributes with initially large ranges (such as

income) from outweighing attributes with initially smaller ranges

(such as binary attributes).

v − minA

v0 =

maxA − minA

Min-max normalization:

all attribute values lie between 0 and 1

For more information on normalization methods refer to data

preprocessing section

22 / 41

Unsupervised Learning: Nearest-Neighbor Classification

k-Nearest-Neighbor Classifiers

assigned the most common class among its k nearest neighbors

When k = 1, the unknown instance is assigned the class of the

training instance that is closest to it in pattern space

Nearest-neighbor classifiers can also be used for prediction, that

is, to return a real-valued prediction for a given unknown instance

In this case, the classifier returns the average value of the

real-valued labels associated with the k nearest neighbors of the

unknown instance

23 / 41

Unsupervised Learning: Nearest-Neighbor Classification

k-Nearest-Neighbor Classifiers

A simple method is to compare the corresponding value of the

attribute in instance X1 with that in instance X2

If the two are identical (e.g., instances X1 and X2 both have the

color blue), then the difference between the two is taken as 0,

otherwise 1

Other methods may incorporate more sophisticated schemes for

differential grading (e.g., where a difference score is assigned, say,

for blue and white than for blue and black)

24 / 41

Unsupervised Learning: Nearest-Neighbor Classification

k-Nearest-Neighbor Classifiers

In general, if the value of a given attribute A is missing in instance

X1 and/or in instance X2 , we assume the maximum possible

difference

For categorical attributes, we take the difference value to be 1 if

either one or both of the corresponding values of A are missing

If A is numeric and missing from both instances X1 and X2 , then

the difference is also taken to be 1

If only one value is missing and the other (which we’ll call v 0 ) is

present and normalized, then we can take the difference to be either

|1 − v 0 | or |0 − v 0 |, whichever is greater

25 / 41

Unsupervised Learning: Nearest-Neighbor Classification

k-Nearest-Neighbor Classifiers

k can be determined experimentally.

Starting with k = 1, we use a test set to estimate the error rate of

the classifier.

This process can be repeated each time by incrementing k to allow

for one more neighbor.

The k value that gives the minimum error rate may be selected.

In general, the larger the number of training instances is, the

larger the value of k will be

26 / 41

Unsupervised Learning: Nearest-Neighbor Classification

Classification takes time proportional to the product of the number

of instances in training and test sets

Nearest-neighbor search can be done more efficiently using

appropriate methods

kD-trees (k-dimensional trees) represent training data in a tree

structure

27 / 41

Unsupervised Learning: Nearest-Neighbor Classification

Roadmap

1 Basic Concepts

2 K-Means

3 K-Medoids

28 / 41

Unsupervised Learning: Nearest-Neighbor Classification

K-Medoids Method

Pick actual objects to represent clusters instead of mean values

Each remaining object is clustered with the representative object

(Medoid) to which is the most similar

The algorithm minimizes the sum of the dissimilarities between

each object and its corresponding representative object

k X

X

E= |p − oi |

i −1 p∈Ci

E : the sum of absolute error for all objects in the data set

P : the data point in the space representing an object

Oi : is the representative object of cluster Ci

29 / 41

Unsupervised Learning: Nearest-Neighbor Classification

The iterative process of replacing representative objects by no

representative objects continues as long as the quality of the

clustering is improved

For each representative Object O

For each non-representative object R , swap O and R

Choose the configuration with the lowest cost

Cost function is the difference in absolute error-value if a current

representative object is replaced by a non-representative object

30 / 41

Unsupervised Learning: Nearest-Neighbor Classification

Data Objects

A1 A2

O1 2 6

O2 3 4

O3 3 8

O4 4 7

O5 6 2

O6 6 4

O7 7 3

O8 7 4

O9 8 5

O10 7 6 Goal: create two clusters

Choose randmly two medoids

O2 = (3, 4)

O8 = (7, 4)

31 / 41

Unsupervised Learning: Nearest-Neighbor Classification

Data Objects

A1 A2

O1 2 6

O2 3 4

O3 3 8

O4 4 7

O5 6 2

O6 6 4

O7 7 3

O8 7 4

O9 8 5 Assign each object to the closest representative object

O10 7 6

Using L1 Metric (Manhattan), we form the following

clusters

Cluster1 = {O1 , O2 , O3 , O4 }

Cluster2 = {O5 , O6 , O7 , O8 , O9 , O10 }

32 / 41

Unsupervised Learning: Nearest-Neighbor Classification

Data Objects

A1 A2

O1 2 6

O2 3 4

O3 3 8

O4 4 7

O5 6 2

O6 6 4 Compute the absolute error criterion [for the set of Medoids

O7 7 3 (O2 ,O8 )]

O8 7 4

O9 8 5 k X

X

O10 7 6 E= |p − oi |

i =1 p∈Ci

= |O1 − O2 | + |O3 − O2 | + |O4 − O2 | + |O5 − O8 |+

|O6 − O8 | + |O7 − O8 | + |O9 − O8 | + |O10 − O8 |

= (3 + 4 + 4) + (3 + 1 + 1 + 2 + 2) = 20

33 / 41

Unsupervised Learning: Nearest-Neighbor Classification

Data Objects

A1 A2

O1 2 6

O2 3 4

O3 3 8

O4 4 7

O5 6 2

O6 6 4

O7 7 3

O8 7 4

O9 8 5

O10 7 6 Choose a random object O7

Swap O8 and O7

Compute the absolute error criterion [for the set of

Medoids (O2 , O7 )]

34 / 41

Unsupervised Learning: Nearest-Neighbor Classification

Data Objects

A1 A2

O1 2 6

O2 3 4

O3 3 8

O4 4 7

O5 6 2

O6 6 4

O7 7 3

O8 7 4

O9 8 5 Compute the cost function

O10 7 6 Absolute error [for O2 ,O7 ] - Absolute error [O2 ,O8 ]

S = 22 − 20

35 / 41

Unsupervised Learning: Nearest-Neighbor Classification

Data Objects

A1 A2

O1 2 6

O2 3 4

O3 3 8

O4 4 7

O5 6 2

O6 6 4

O7 7 3

O8 7 4

O9 8 5

O10 7 6 In this example, changing the medoid of cluster 2 did

not change the assignments of objects to clusters.

What are the possible cases when we replace a medoid

by another object?

36 / 41

Unsupervised Learning: Nearest-Neighbor Classification

K-Medoids

First case

Currently P assigned to A

The assignment of P to A does not

change

Second case

Currently P assigned to B

P is reassigned to A

37 / 41

Unsupervised Learning: Nearest-Neighbor Classification

K-Medoids

Third case

Currently P assigned to B

P is reassigned to the new B

Fourth case

Currently P assigned to A

P is reassigned to A

38 / 41

Unsupervised Learning: Nearest-Neighbor Classification

Input

K : the number of clusters

D : a data set containing n objects

Output: A set of k clusters

Method:

1 Arbitrary choose k objects from D as representative objects (seeds)

2 Repeat

3 Assign each remaining object to the cluster with the nearest

representative object

4 For each representative object Oj

5 Randomly select a non representative object Orandom

6 Compute the total cost S of swapping representative object Oj with

Orandom

7 if S < 0 then replace Oj with Orandom

8 Until no change

39 / 41

Unsupervised Learning: Nearest-Neighbor Classification

PAM Properties

For large values of n and k , such computation becomes very costly

Advantages

K-Medoids method is more robust than k-Means in the presence of

noise and outliers

Disadvantages

K-Medoids is more costly than k-Means

Like k-means, k-medoids requires the user to specify k

It does not scale well for large data sets

40 / 41

Unsupervised Learning: Nearest-Neighbor Classification

References

Inc. (2006). (Chapter 6)

I. H. Witten and E. Frank, Data Mining: Practical Machine Learning

Tools and Techniques, 2nd Edition, Elsevier Inc., 2005. (Chapter 6)

41 / 41

- Discovery of Patterns and evaluation of Clustering Algorithms in SocialNetwork Data (Face book 100 Universities) through Data Mining Techniques and MethodsÎncărcat deLewis Torres
- Plant Recognition using Hog and Artificial Neural NetworkÎncărcat deEditor IJRITCC
- LOG-BACKUP UTILITY: A SECURE LOG MINING SYSTEMÎncărcat deIJIERT-International Journal of Innovations in Engineering Research and Technology
- bahanÎncărcat deAbinaya Prasaja
- SmlÎncărcat deEricsson Marin
- Adaptive Neuro-Fuzzy Inference System based Fractal Image CompressionÎncărcat deIDES
- Neural Nw K-meansÎncărcat deIr Dennis N Mwighusa
- qmat97_iwanowski_skoneczny_szostakowski.pdfÎncărcat dehub23
- Ubicc Paper 419Încărcat deUbiquitous Computing and Communication Journal
- NWU-EECS-07-05Încărcat deeecs.northwestern.edu
- Thermal image sgmentationÎncărcat deeshwari2000
- IJETR031830Încărcat deerpublication
- 08540811.pdfÎncărcat deisaac
- Document Clustering in Correlation SimilarityÎncărcat deshtl2901
- KnnÎncărcat deAnonymous Ys17x7
- Detecting the Number of Clusters During Expectation-Maximization (ICMLC 2010)Încărcat deambatvinay
- G Unit en Ml IntroductionÎncărcat deShantam
- Clustering Algorithms For High Dimensional Data.pdfÎncărcat deKonala Teja Saisandeep Reddy
- ML Topic1AÎncărcat deAbegunde Omolayo
- 108.pdfÎncărcat deInternational Journal of Scientific Research in Science, Engineering and Technology ( IJSRSET )
- SeminarÎncărcat deGlobal info - tech Kumbakonam
- Data Mining Mcq'sÎncărcat debhanu prakash
- Data MiningÎncărcat demofoel
- 2003_apr_357-367Încărcat deRazzougui Sarah
- IJETR2226Încărcat deanil kasot
- A k-nearest ML KNNÎncărcat dedoctorsur7
- Algorithms for Predictive Maintenance Efficiently Developed With MatlabÎncărcat deHanif Andi
- [doi 10.1109_COMPSAC.2016.227] Song, Wenai; Wang, Ping; Zhang, Xudong; Wang, Qing -- [IEEE 2016 IEEE 40th Annual Computer Software and Applications Conference (COMPSAC) - Atlanta, GA, USA (2016.6.10.pdfÎncărcat deWayang Indonesia Palsu
- 915an Integration of Genetic Algorithm and Projected Clustering for Optimization of Content Based Image Retrieval System PDFÎncărcat dedannyjosep
- Naive Bayes..and OtherÎncărcat deManushi

- Lesson_3.6_-_Supervised_Learning_Neural_Networks (1).pdfÎncărcat deTayyaba Faisal
- Lesson 4.1 - Unsupervised Learning Partitioning MethodsÎncărcat deTayyaba Faisal
- Lesson_2.1_-_Know_Your_Data.pdfÎncărcat deTayyaba Faisal
- Lesson 3.6 - Supervised Learning Neural NetworksÎncărcat deTayyaba Faisal
- Lesson_3.2_-_Supervised_Learning_Evaluation.pdfÎncărcat deTayyaba Faisal
- Lesson 2.2 - Frequent Pattern AnalysisÎncărcat deTayyaba Faisal
- Lesson 3.3 - Supervised Learning Rule Based ClassificationÎncărcat deTayyaba Faisal
- Lesson 3.1 - Supervised Learning Decision TreesÎncărcat deTayyaba Faisal
- Lesson 3.2 - Supervised Learning EvaluationÎncărcat deTayyaba Faisal

- Speech emotion recognitionÎncărcat deHugo Leonardo Rufiner
- Theft AnalysisÎncărcat deGaurav Kumar
- perbandingan metode kmeans dan dbscanÎncărcat deRisa
- Dimensionality Reduction Lecture SlideÎncărcat deHalil Durmuş
- Deep LearningÎncărcat deanilgs1803
- BackPropagation Through TimeÎncărcat demilan gemersky
- NeuralNetworks.CheungCannonNotesÎncărcat deKshirod Kumar Dash
- Backprop DerivationÎncărcat dejoeyb379
- Implementacion entrenamiento redes CNNÎncărcat deandyy4e
- PCA using JMPÎncărcat deprakhar8
- Convolutional Neural NetworkÎncărcat dePhan-128282
- Combined Use of Association Rules Mining and Clustering Methods to Find Relevant Links Between Binary Rare Attributes in a Large Data SetÎncărcat deInformatika Universitas Malikussaleh
- Journey of Machine LearningÎncărcat delinkranjit
- Lecture 18 - Kohonen SOMÎncărcat deDanh Bui Cong
- Super Cheatsheet Deep LearningÎncărcat decidsant
- LSTM PaperÎncărcat devarun3dec1
- Keras_Cheat_Sheet_Python.pdfÎncărcat deokhtaya
- Design of C-slotted Microstrip Antenna UsingÎncărcat deInternational Journal of Research in Engineering and Technology
- A Step by Step Backpropagation Example – Matt MazurÎncărcat dejeevan gangavarapu
- NNFL CBCGS SyllabusÎncărcat deSelvin Furtado
- 4 Simple ANNÎncărcat defauzansaadon
- Data Mining-Model Based ClusteringÎncărcat deRaj Endran
- IEEE Fuzzy_PCA-Guided_Robust_k-Means_Clustering-J1t.pdfÎncărcat deDexter Kamal
- Quality of Cluster Index Based on Study of Decision TreeÎncărcat deWhite Globe Publications (IJORCS)
- Exercises ClusteringÎncărcat deThyago
- DM BITSÎncărcat deSumanth_Yedoti
- 03-Back Propagation NetworkÎncărcat demdhakar
- LSTMÎncărcat de潘 韋志
- Email PrioritizationÎncărcat deVidul Ap
- r7411009-Artificial Neural NetworksÎncărcat desivabharathamurthy

## Mult mai mult decât documente.

Descoperiți tot ce are Scribd de oferit, inclusiv cărți și cărți audio de la editori majori.

Anulați oricând.