Awesome

NYTK-NerKor+Cars-OntoNotes++

This corpus was derived from NYTK-NerKor, a Hungarian gold standard named entity annotated corpus containing about 1 million tokens. It includes a small addition of 12k tokens of text (individual sentences) concerning motor vehicles (cars, buses, motorcycles) from the news archive of hvg.hu. While the annotation in NYTK-NerKor followed the CoNLL2002 labelling standard with just four NE categories (PER, LOC, MISC, ORG), this version of the corpus features over 30 entity types, including all entity types used in the [OntoNotes 5.0] English NER annotation. The new annotation elaborates on subtypes of the LOC and MISC entity types, and includes annotation for non-names like times and dates, quantities, languages and nationalities or religious or political groups. The tags are in the IOB2 format: a B- prefix denotes the first item of a NE phrase and an I- prefix any non-initial word. Non-entities are marked by an O label.

The new annotation was created by applying zero-shot cross-lingual transfer using three models (Flair/ner-english-ontonotes-large, DeepPavlov/ner_ontonotes_bert_mult and NameTag2 trained on CNEC2) the outputs of which were automatically merged with the original annotation and then corrected using regex-based automatic correction patterns and checked manually. Manual correction also concerned certain types of orthographic/tokenization/segmentation errors.

Note, however, that manual checking of the corpus was not exhaustive, thus annotation in the current version needs further curation to be considered gold standard. Nevertheless, we release this version of the corpus hoping that it proves to be a useful resource for the NLP community.

Tags derived from the OntoNotes 5.0 annotation

Names are annotated according to the following set of types:


`PER`	= PERSON People, including fictional
`FAC`	= FACILITY Buildings, airports, highways, bridges, etc.
`ORG`	= ORGANIZATION Companies, agencies, institutions, etc.
`GPE`	Geopolitical entites: countries, cities, states
`LOC`	= LOCATION Non-GPE locations, mountain ranges, bodies of water
`PROD`	= PRODUCT Vehicles, weapons, foods, etc. (Not services)
`EVENT`	Named hurricanes, battles, wars, sports events, etc.
`WORK_OF_ART`	Titles of books, songs, etc.
`LAW`	Named documents made into laws

The following are also annotated in a style similar to names:


`NORP`	Nationalities or religious or political groups
`LANGUAGE`	Any named language
`DATE`	Absolute or relative dates or periods
`TIME`	Times smaller than a day
`PERCENT`	Percentage (including "%")
`MONEY`	Monetary values, including unit
`QUANTITY`	Measurements, as of weight or distance
`ORDINAL`	"first", "second"
`CARDINAL`	Numerals that do not fall under another type

Additional tags

Further subtypes of names of type MISC:


`AWARD`	Awards and prizes
`CAR`	Cars and trucks
`MEDIA`	Media outlets, TV channels, news portals
`SMEDIA`	Social media platforms
`ORG-GPE`	Geopolitical entities as agents: they would (justly) simply be marked as GPE in the OntoNotes annotation scheme
`PROJ`	Projects and initiatives
`MISC`	Unresolved subtypes of MISC entities
`MISC-ORG`	Organization-like unresolved subtypes of MISC entities
`MISC-PER`	Metonymic use of person's names

Further non-name entities:


`DUR`	Time duration
`ID`	identifier

License and usage

The corpus is available under the license CC-BY-SA 4.0 following the license of the original NYTK-NerKor corpus. If you use this corpus, please cite the papers below.

Data

Names of the corpus files reflect the names of corresponding files in the NerKor dataset, however, names of subfolders in NerKor (including reference to the train/dev/test split) are part of the file names in this version.

See further details about the corpus and the annotation guidelines of the original NerKor dataset in the NYTK-NerKor repo.

Data format

The format of data files are CoNLL-U Plus with the standard .conllup file extension. The first line in each file is: # global.columns = FORM ONPP:NER, where:

FORM: the token itself; ONPP:NER: OntoNotes++ NE annotation.

Morphological annotation in NerKor was discarded in this version.

Citation

If you use this resource or any part of its documentation, please cite:

Novák Attila, Novák Borbála. NerKor 1.41e. In: XVIII. Magyar Számítógépes Nyelvészeti Konferencia.

@inproceedings{NerKorCarsONPP,
  author    = {Nov{\'a}k Attila and
               Nov{\'a}k Borb{\'a}la,
  title     = {{NerKor} 1.41e},
  booktitle = {XVIII. Magyar Számítógépes Nyelvészeti Konferencia},
  year      = {2022},
}

and the publication presenting the original corpus:

Simon, Eszter; Vadász, Noémi. (2021) Introducing NYTK-NerKor, A Gold Standard Hungarian Named Entity Annotated Corpus. In: Ekštein K., Pártl F., Konopík M. (eds) Text, Speech, and Dialogue. TSD 2021. Lecture Notes in Computer Science, vol 12848. Springer, Cham. https://doi.org/10.1007/978-3-030-83527-9_19

@inproceedings{DBLP:conf/tsd/SimonV21,
  author    = {Eszter Simon and
               No{\'{e}}mi Vad{\'{a}}sz},
  editor    = {Kamil Ekstein and
               Frantisek P{\'{a}}rtl and
               Miloslav Konop{\'{\i}}k},
  title     = {Introducing {NYTK-NerKor}, A Gold Standard {Hungarian} Named Entity
               Annotated Corpus},
  booktitle = {Text, Speech, and Dialogue - 24th International Conference, {TSD}
               2021, Olomouc, Czech Republic, September 6-9, 2021, Proceedings},
  series    = {Lecture Notes in Computer Science},
  volume    = {12848},
  pages     = {222--234},
  publisher = {Springer},
  year      = {2021},
  doi       = {10.1007/978-3-030-83527-9\_19},
}