This article demonstrates how to construct a text clustering pipeline by combining large language model embeddings with HDBSCAN, a density-based algorithm. By transforming raw, unstructured text into semantic mathematical representations, you can automatically discover hidden topics in unlabeled datasets without requiring prior labels.
Key stages of the process include:
* Generating text embeddings using pre-trained sentence-transformers models
* Reducing embedding dimensionality with UMAP to prepare data for clustering
* Applying HDBSCAN to identify clusters and visualize results through scatterplots