Unlock advanced customer segmentation techniques using LLMs, and improve your clustering models with advanced techniques
tokenizing and stemming each synopsis
transforming the corpus into vector space using tf-idf
calculating cosine distance between each document as a measure of similarity
clustering the documents using the k-means algorithm
using multidimensional scaling to reduce dimensionality within the corpus
plotting the clustering output using matplotlib and mpld3
conducting a hierarchical clustering on the corpus using Ward clustering
plotting a Ward dendrogram
topic modeling using Latent Dirichlet Allocation (LDA)
transforming the corpus into vector space using tf-idf
calculating cosine distance between each document as a measure of similarity
clustering the documents using the k-means algorithm
using multidimensional scaling to reduce dimensionality within the corpus
plotting the clustering output using matplotlib and mpld3
conducting a hierarchical clustering on the corpus using Ward clustering
plotting a Ward dendrogram
topic modeling using Latent Dirichlet Allocation (LDA)
ow can you learn about the underlying structure of documents in a way that is informative and intuitive? This basic motivating question led me on a journey to visualize and cluster documents in a two-dimensional space. What you see above is an output of an analytical pipeline that begin by gathering synopses on the top 100 films of all time and ended by analyzing the latent topics within each document. In between I ran significant manipulations on these synopses (tokenization, stemming), transformed them into a vector space model (tf-idf), and clustered them into groups (k-means). You can learn all about how I did this with my detailed guide to Document Clustering with Python. But first, what did I learn?
Word embeddings are suitable for use with neural network language models (as will be discussed later); they can also be used to enhance conventional (MEMM, CRF) models. The best ways to incorporate embeddings into such feature-based language models are still being explored. The simplest approach involves the direct use of the vector components as features (Turian et al 2010, Word Representations: A Simple and General Method for Semi-Supervised Learning, ACL 2010; Nguyen and Grishman, ACL 2014). Less direct approaches include building clusters from the embeddings and then using the clusters as features, or selecting prototypical examples of each type and then using similarity to these prototypes (based on embedding similarity) as features. Early results on NE tagging indicate a small advantage for the indirect methods (Guo et al., Revisiting embedding features for simple semi-supervised learning, EMNLP 2014). Models based on word embeddings are producing the best performance on named entity recognition (A. Passos et al, Lexicon Infused Phrase Embeddings for Named Entity Resolution, CoNLL 2014) and are effective for chunking (Turian et al ACL 2010).
