In written text, orthographic noise is a common concern for NLP, especially when operating social network comments and raw documents. This is mainly due to its orthographic conventions and morphological ambiguity. We propose to automatically normalize the social media dialect corpora by following CODA-TUN, the Conventional Orthography for Tunisian Arabic (TA). The existing system developed for TA is not able to handle all forms of TA. Therefore, we propose to extend its rules and lexicons to address the peculiarities of social media dialect. In certain words, the COTA Orthography 1.0 system provides the user with several correction possibilities. Therefore, in the new version, we incorporated a trigram language model to automatically select the right correction. Our results show that the system can reduce transcription errors by 95.72%.
Key words: Orthographic normalization, Tunisian Arabic, COTA Orthography system, CODA-TUN
|
