Abstract
The clustering of large and high-dimensional data is still a problem because of the problems of scalability, memory, and quality evaluation of the cluster. This paper also suggests a Hybrid Scalable Clustering Framework (HSCF) which is a combination of hierarchical and partition-based clustering algorithms to provide efficient and effective behavior at different scales of data. The framework uses the BIRCH clustering algorithm of memory efficient, incremental clustering coupled with fast and scalable centroid-based optimization with MiniBatchKMeans clustering algorithm. Elbow Method is used to have the optimal clustering configuration, to make the number of clusters manually chosen and thus minimize human involvement and enhance reproducibility. The quality of clustering is computationally assessed by the Silhouette Score which gives information about the cohesiveness within a cluster and the distance between clusters. Moreover, PCA is applied to dimensionality reduction and visualization, with the help of which it becomes possible to produce useful visualizations of the structure of clustering in reduced feature space. Experimental evidence shows that MiniBatchKMeans is more efficient in its computational speed with large datasets whereas BIRCH is more efficient in terms of memory usage and hierarchical representation. The proposed framework is the most appropriate in the application of the real-world large-scale data analysis. The combination of multi-scale assessment, automatic optimization and multi-scale clustering is the main innovation of this article with the ability to provide a complete solution to the contemporary clustering problems.