How to decide the cluster number automatically?

Clustering is a popular tool for exploratory data analysis. One of the major problems in cluster analysis is the determination of the number of clusters in unlabeled data, which is a basic input for most clustering algorithms. In this paper we investigate a new method called DBE (Dark Block Extraction) for automatically estimating the number of clusters in unlabeled data sets, which is based on an existing algorithm for Visual Assessment of cluster Tendency (VAT) of a data set, using several common image and signal processing techniques. Basic steps include: 1) Generating a VAT image of an input dissimilarity matrix; 2) Performing image segmentation on the VAT image to obtain a binary image, followed by directional morphological filtering; 3) Applying a distance transform to the filtered binary image and projecting the pixel values onto the main diagonal axis of the image to form a projection signal; 4) Smoothing the projection signal, computing its first-order derivative, and then detecting major peaks and valleys in the resulting signal to decide the number of clusters. Our new DBE method is nearly "automatic", depending on just one easy-to-set parameter. Several numerical and real-world examples are presented to illustrate the effectiveness of DBE.

Activity Partners

This is an area of active research, and there are a number of interesting methods. Google "gap method" or "l-method" for a couple of more recent ideas.

Best,
John