About this document

Effects of Scaling on LLMs by John Rawai is a document available to read on EtoBox.

This document investigates the effects of scaling down large language models (LLMs) on their core capabilities, specifically fact recall and in-context learning (ICL). The study finds that reducing model size by over 30% significantly impairs fact recall, while ICL remains largely unaffected even with a 60-70% reduction in model size. The results suggest that scaling has distinct impacts on these two abilities, highlighting the need for nuanced evaluations of model performance beyond standard metrics.

Author

John Rawai

Language

EN