Word Parsing

A Dependency Parser extracts a dependency graph from a sentence. In the graph the grammatical structure, like subject and object, and relationships between words represented by Dependency tags.

The syntactic function of a word, like Noun or Verb, is defined by the Part-of-Speech (POS tags) and is based on the context.

Assigning additional morphological information clarifies the grammatical meaning of a word, additionally to the syntax.

The goal of tokenization is a vocabulary. Word-based tokenizers have larger vocabularies with more Out-Of-Vocabulary (OOV) words than sub-word vocabularies.

Many NLP tasks make use of intensive matrix calculations, for which word id’s are used, rather than words. For this, raw text is split up into tokens that represent (sub)words.