Exploratory data analysis (EDA) is a powerful technique to understand the structure of word embeddings, the basis of large language models. In this article, we'll apply EDA to GloVe word embeddings and find some interesting insights.
Alternative to t-SNE and PCA
In your example if you use PCA to initialize your t-SNE you get widely spaced centroids; if you use random initialization you'll get tiny centroids and an uninteresting picture.
I used a Python t-SNE library to reduce the 200 feature dimensions for each word to 2 dimensions and plotted them in matplotlib. I saved out the x/y coordinates for each word in the book, so that I can show those words on the graph as you mouse over the replaced (blue) words.
