← All data sources← All Language Processing observatories on IDLI

data source

85 language directories; presence of tokenizer_exceptions and a lemmatizer distinguishes a real pipeline from a stub.

tokenizationmorphological-analysis
Languages with a directory
78
Max pipeline modules
6
Unmatched codes
1
not in the crosswalk

Languages in spaCy

78 of 78 shown

Detail
Catalan
cat
6detail →
Dutch
nld
6detail →
English
eng
6detail →
French
fra
6detail →
Haitian
hat
6detail →
Modern Greek (1453-)
ell
6detail →
Spanish
spa
6detail →
Danish
dan
5detail →
Finnish
fin
5detail →
Indonesian
ind
5detail →
Italian
ita
5detail →
Malay (macrolanguage)
msa
5detail →
Persian
fas
5detail →
Portuguese
por
5detail →
Swedish
swe
5detail →
Amharic
amh
4detail →
Ancient Greek (to 1453)
grc
4detail →
Arabic
ara
4detail →
German
deu
4detail →
Kirghiz
kir
4detail →
Latin
lat
4detail →
Lithuanian
lit
4detail →
Luxembourgish
ltz
4detail →
Macedonian
mkd
4detail →
Norwegian Bokmål
nob
4detail →
Polish
pol
4detail →
Romanian
ron
4detail →
Russian
rus
4detail →
Serbian
srp
4detail →
Slovenian
slv
4detail →
Tatar
tat
4detail →
Tigrinya
tir
4detail →
Turkish
tur
4detail →
Ukrainian
ukr
4detail →
Basque
eus
3detail →
Bengali
ben
3detail →
Bulgarian
bul
3detail →
Ganda
lug
3detail →
Hungarian
hun
3detail →
Irish
gle
3detail →
Korean
kor
3detail →
Ligurian
lij
3detail →
Tagalog
tgl
3detail →
Thai
tha
3detail →
Tswana
tsn
3detail →
Upper Sorbian
hsb
3detail →
Urdu
urd
3detail →
Armenian
hye
2detail →
Azerbaijani
aze
2detail →
Chinese
zho
2detail →
Czech
ces
2detail →
Hebrew
heb
2detail →
Hindi
hin
2detail →
Japanese
jpn
2detail →
Lower Sorbian
dsb
2detail →
Malayalam
mal
2detail →
Nepali (macrolanguage)
nep
2detail →
Northern Kurdish
kmr
2detail →
Norwegian Nynorsk
nno
2detail →
Sanskrit
san
2detail →
Scottish Gaelic
gla
2detail →
Sinhala
sin
2detail →
Slovak
slk
2detail →
Tamil
tam
2detail →
Telugu
tel
2detail →
Tibetan
bod
2detail →
Vietnamese
vie
2detail →
Yoruba
yor
2detail →
Afrikaans
afr
1detail →
Albanian
sqi
1detail →
Croatian
hrv
1detail →
Estonian
est
1detail →
Faroese
fao
1detail →
Gujarati
guj
1detail →
Icelandic
isl
1detail →
Kannada
kan
1detail →
Latvian
lav
1detail →
Marathi
mar
1detail →

Community portal

Opportunities, events, and experts across the digital language inclusion ecosystem.