Can I read Steel-LLM:From Scratch to Open Source -- A Personal Journey in Building a Chinese-Centric LLM on EtoBox?

Steel-LLM:From Scratch to Open Source -- A Personal Journey in Building a Chinese-Centric LLM by Gu, Qingshui; Li, Shu; Zheng, Tianyu; Zhang, Zhaoxiang is a scholarly article available to read on EtoBox.

What is Steel-LLM:From Scratch to Open Source -- A Personal Journey in Building a Chinese-Centric LLM about?

Steel-LLM is a Chinese-centric language model developed from scratch with the goal of creating a high-quality, open-source model despite limited computational resources. Launched in March 2024, the project aimed to train a 1-billion-parameter model on a large-scale dataset, prioritizing transparency and the sharing of practical insights to assist others in the community. The training process primarily focused on Chinese data, with a small proportion of English data included, addressing gaps in existing open-source LLMs by providing a more detailed and practical account of the model-building journey. Steel-LLM has demonstrated competitive performance on benchmarks such as CEVAL and CMMLU, outperforming early models from larger institutions. This paper provides a comprehensive summary of the project's key contributions, including data collection, model design, training methodologies, and the challenges encountered along the way, offering a valuable resource for researchers and practitioners looking to develop their own LLMs. The model checkpoints and training script are available at https://github.com/zhanshijinwat/Steel-LLM.

Author

Gu, Qingshui; Li, Shu; Zheng, Tianyu; Zhang, Zhaoxiang

Published

2025

Language

EN