You can fix the problem on Mac OS X Mavericks and Yosemite by doing the following:
cd ~/.matplotlib/
fc-list # Should take a couple minutes.
After fc-list finishes running you will be able to import the package using import matplotlib.pyplot as plt.
On Linux and macOS you can find the user base binary directory by running python -m site --user-base and adding bin to the end. F