Levy & Goldberg

2014 paper observing that early word embeddings can be seen as matrix factorizations of log-likelihood matrices, paralleling the bigram behavior of zero-layer transformers