This article demonstrates how to construct a text clustering pipeline by combining large language model embeddings with HDBSCAN, a density-based algorithm. By transforming raw, unstructured text into semantic mathematical representations, you can automatically discover hidden topics in unlabeled datasets without requiring prior labels.
Key stages of the process include:
* Generating text embeddings using pre-trained sentence-transformers models
* Reducing embedding dimensionality with UMAP to prepare data for clustering
* Applying HDBSCAN to identify clusters and visualize results through scatterplots
Key stages of the process include:
* Generating text embeddings using pre-trained sentence-transformers models
* Reducing embedding dimensionality with UMAP to prepare data for clustering
* Applying HDBSCAN to identify clusters and visualize results through scatterplots
This article provides a beginner-friendly introduction to HDBSCAN, a powerful hierarchical clustering algorithm that extends the capabilities of DBSCAN by handling varying densities more effectively. It compares HDBSCAN to DBSCAN and KMeans, highlighting the advantages of HDBSCAN in handling clusters of different shapes and sizes.
Discusses reasons why clustering in data science might not produce desired results and how to address these issues.
Leverage LLM-enhanced natural language processing and traditional machine learning techniques are used to extract structure and to build a knowledge graph from unstructured corpus.
