Home

Awesome

LINSPECTOR

Language Inspector (LINSPECTOR) is an effort to interpret the multilingual black box NLP models. With this work, we release an easy-to-use framework to help researchers,especially the ones that are interested in world languages, understand their word representations better.

Cite

Please use the following citation:

@article{DBLP:journals/corr/abs-1903-09442,
  author    = {G{\"{o}}zde G{\"{u}}l {\C{S}}ahin and
               Clara Vania and
               Ilia Kuznetsov and
               Iryna Gurevych},
  title     = {{LINSPECTOR:} Multilingual Probing Tasks for Word Representations},
  journal   = {CoRR},
  volume    = {abs/1903.09442},
  year      = {2019},
  url       = {http://arxiv.org/abs/1903.09442},
  archivePrefix = {arXiv},
  eprint    = {1903.09442},
  timestamp = {Mon, 01 Apr 2019 14:07:37 +0200},
  biburl    = {https://dblp.org/rec/bib/journals/corr/abs-1903-09442},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

Abstract: Despite an ever growing number of word representation models introduced for a large number of languages, there is a lack of a standardized technique to provide insights into what is captured by these models. Such insights would help the community to get an estimate of the downstream task performance, as well as to design more informed neural architectures, while avoiding extensive experimentation which requires substantial computational resources not all researchers have access to. A recent development in NLP is to use simple classification tasks, also called probing tasks, that test for a single linguistic feature such as part-of-speech. Existing studies mostly focus on exploring the information encoded by the sentence-level representations for English. However, from a typological perspective the morphologically poor English is rather an outlier: the information encoded by the word order and function words in English is often stored on a subword, morphological level in other languages. To address this, we introduce 15 word-level probing tasks such as case marking, possession, word length, morphological tag count and pseudoword identification for 24 languages. We present experiments on several state of the art word embedding models, in which we relate the probing task performance for a diverse set of languages to a range of classic NLP tasks such as semantic role labeling and natural language inference. We find that a number of probing tests have significantly high positive correlation to the downstream tasks, especially for morphologically rich languages. We show that our tests can be used to explore word embeddings or black-box neural models for linguistic cues in a multilingual setting.

Contact person:

Gözde Gül Şahin, sahin@ukp.informatik.tu-darmstadt.de

https://www.ukp.tu-darmstadt.de/

Please send us an e-mail or report an issue, if something is broken or if you have further questions.