About this document

Tokens With Meaning: A Hybrid Tokenization Approach For NLP by jo Ge is a document available to read on EtoBox.

This paper presents a hybrid tokenization framework that combines rule-based morphological analysis with statistical subword segmentation to improve tokenization for morphologically rich languages like Turkish. The proposed method enhances semantic coherence and interpretability by addressing the limitations of existing subword-based techniques, achieving the highest Turkish Token Percentage and Pure Token Percentage on the TR-MMLU benchmark. While focused on Turkish, the methodology is adaptable to other l

Author

jo Ge

Language

EN