The Predictability-Diversity Trade-Off in Language Modeling: A Comparative Analysis of Memory-Based and Transformer-Based Next-Word Prediction