About this document
Token Normalization Techniques Explained by Hassan Khan is a document available to read on EtoBox.
Token normalization is the process of canonicalizing tokens so that matches occur despite superficial differences. It includes techniques such as case folding, truecasing, handling alternate spellings, internationalized names and transliterations, and normalizing numbers and dates. Stemming chops off the ends of words while lemmatization uses a dictionary to return words to their dictionary form.
- Author
- Hassan Khan
- Language
- EN