Home

Awesome

Awesome AI Sign Language Papers & Popularization of Professional Knowledge [UPDATING]

This repository is all you need to open the door about AI sign languages!

Keywords: Sign Language, Sign Language Translation (SLT), Sign Language Recognition (SLR), Sign Language Production (SLP), Sign Language Retrieval, Sign Language Linguistics

<details><summary><b>What's New</b></summary> `Update 2023-12-23:` More concise layout & updated contents & Add the introduction of sign language knowledge. </details> <details open><summary><b name="table-of-content">Table of contents</b></summary> </details>

Introduction <a name="introduction"></a>

This repository is for those interested in the field of AI sign language (SL). The collected papers have been categorized according to different criteria (btime, type, institution, etc.) for ease of searching.

If useful, please Star this repo, we are keeping it updated.

NOTE: There is overlap between the different categories. Please feel free to submit your Pull Requests for any updates.

Popularization of Professional Sign Language Knowledge <a name="sl_knowledge"></a>

- Why AI Sign Language Research? <a name="why_sl"></a>

<details open><summary><b name="table-of-content">Unfolding Details</b></summary> <!-- ![](./src/fig1.jpg) --> <center> <img style="border-radius: 0.3125em; box-shadow: 0 2px 4px 0 rgba(34,36,38,.12),0 2px 10px 0 rgba(34,36,38,.08);width:60%" src="./src/fig1.jpg"> <br> <div style="color:orange; border-bottom: 1px solid #d9d9d9; display: inline-block; color: #999; padding: 2px;">Figure 1. Communication gap between the hearing and the deaf. </div> </center>

Sign languages are the primary language of the deaf community. However, most hearing people find it difficult to understand sign languages. With the development of AI, researchers are trying to help people understand sign languages using AI techniques that are designed to convert sign languages into spoken languages in textual form.

</details>

- Do you know the differences between SLT and SLR? <a name="slt_vs_slr"></a>

<details open><summary><b name="table-of-content">Unfolding Details</b></summary> <!-- ![](./src/fig2.jpg) --> <center> <img style="border-radius: 0.3125em; box-shadow: 0 2px 4px 0 rgba(34,36,38,.12),0 2px 10px 0 rgba(34,36,38,.08);" src="./src/fig2.jpg"> <br> <div style="color:orange; border-bottom: 1px solid #d9d9d9; display: inline-block; color: #999; padding: 2px;">Figure 2. SLR (Sign Language Recognition) vs. SLT (Sign Language Translation). </div> </center>

At the very beggining, I wanna explain the difference between SLT and SLR, as shown in Fig. 2. I'm sure this is very important for most of you!

In early efforts, researchers explored this problem (sign languages -> text-form glosses) as a recognition problem (i.e., SLR), which converts sign languages to glosses word by word according to the sign languages sequentially. Although glosses are in textual form, they do not provide meaningful interpretations of what a signer is saying because sign languages and glosses have their own specific linguistic rules, which are quite different from spoken languages.

As a result, researchers find it terrible to ignore the linguistic properties of sign language. Contrary to SLR, sign language translaton (SLT) systems aim to translate sign language videos into spoken sentences directly.

As far as AI technology is concerned, SLR belongs to the field of Computer Vision + Text Recognition, while SLT belongs to the field of Computer Vision + Text Translation.

</details>

- More details you must know about SLR tasks? <a name="slr_details"></a>

<details open><summary><b name="table-of-content">Unfolding Details</b></summary>

By default, we refer to the current Continuous SLR as SLR because this type of SLR is the mainstream of the deep learning era.

In fact, it is important to note that SLR tasks are generally divided into three categories: finger-spelling recognition, isolated word recognition and continuous sign language recognition. In the early days (before about 2018), SLR works mainly focused on lexical-level tasks, such as finger-spelling recognition and isolated word recognition. Nowadays, the more practical continuous SLR has become mainstream sign language research

For more categorization and details, we recommend you read this survey/review paper for a detailed look at AI Sign history.

</details>

SURVEY/REVIEW PAPERS <a name="survey"></a>

[Back to TOP]

AI Sign Language in Timeline [Papers] <a name='sl_paper_timeline'></a>

2024 <a name='sl_paper_2024'></a>

[Back to TOP]

2023 <a name='sl_paper_2023'></a>

[Back to TOP]

2022 <a name='sl_paper_2022'></a>

[Back to TOP]

2021 <a name='sl_paper_2021'></a>

[Back to TOP]

2020 <a name='sl_paper_2020'></a>

[Back to TOP]

2019 <a name='sl_paper_2019'></a>

[Back to TOP]

2018 <a name='sl_paper_2018'></a>

[Back to TOP]

2017 <a name='sl_paper_2017'></a>

[Back to TOP]

Earlier <a name='sl_paper_earlier'></a>

[Back to TOP]

AI Sign Language in Well-Known Institutions [Papers] <a name="sl_paper_institution"></a>

XMU for AI Sign Language <a name='sl_paper_xmu'></a>

[Back to TOP]

USTC for AI Sign Language <a name='sl_paper_ustc'></a>

[Back to TOP]

ZJU for AI Sign Language <a name='sl_paper_zju'></a>

[Back to TOP]

THU for AI Sign Language <a name='sl_paper_thu'></a>

[Back to TOP]

Germany-UK for AI Sign Language <a name='sl_paper_germany-uk'></a>

[Back to TOP]

Datasets <a name="datasets"></a>

[Back to TOP]

DatasetLanguageClassesSamplesData TypeLanguage Level
ASLG-PC12paperAmerican-87,709GLOSS&Sentencesisolated
CSL Dataset IChinese500125,000Videos&Depth from Kinectisolated
CSL Dataset IIChinese10025,000Videos&Depth from Kinectcontinuous
RWTH-PHOENIX-Weather 2014German1,0816,841Videoscontinuous
RWTH-PHOENIX-Weather 2014 TGerman1,0668,257Videoscontinuous
ASLLVDAmerican3,3009,800Videos(multiple angles)isolated
ASLLVD-SkeletonAmerican3,3009,800Skeletonisolated
SIGNUMGerman45033,210Videoscontinuous
DGS Kinect 40German403,000Videos(multiple angles)isolated
DEVISIGN-GChinese36432Videosisolated
DEVISIGN-DChinese5006,000Videosisolated
DEVISIGN-LChinese200024,000Videosisolated
LSA64Argentinian643,200Videosisolated
GSL isol.Greek31040,785Videos&Depth from RealSenseisolated
GSL SDGreek31010,290Videos&Depth from RealSensecontinuous
GSL SIGreek31010,290Videos&Depth from RealSensecontinuous
IIITA -ROBITAIndian23605Videosisolated
PSL KinectPolish30300Videos&Depth from Kinectisolated
PSL ToFPolish841,680Videos&Depth from ToF cameraisolated
BUHMAP-DBTurkish8440Videosisolated
LSE-SignSpanish2,4002,400Videosisolated
Purdue RVL-SLLLAmerican39546Videosisolated
RWTH-BOSTON-50American50483Videos(multiple angles)isolated
RWTH-BOSTON-104American104201Videos(multiple angles)continuous
RWTH-BOSTON-400American400843Videoscontinuous
WLASLAmerican2,00021,083Videosisolated

AI Sign Language Related Fields <a name="related_fields"></a>

[Back to TOP]

The related fields, such as Large Pretrained Language Model,Corss-modal/Multi-modality, Action Recognition, Machine Translation, Speech Recognition, Video Captioning, can help you gain more inspiration and knowledge