Various languages have additional features and filters enabled that are not particularly language-specific. Unpacked analyzers get automatic ICU normalization upgrades, too. This can reveal inconsistencies in language analysis across wikis. For example, CamelCase is split by English Wikipedia, but not French Wikipedia (see T219108).
So, let's compare the top 20+ analyzers and see how they treat a mixed-language sample of documents (from, say. those Wikipedias plus a handful of docs from each of the top 100 Wikipedias) and look for ways to increase consistency. (It may not be necessary to do a 20-way comparison; we may be able to take a survey of components, remove the language-specific ones, and see what differences the remainder cause for different kinds of text.)
Likely necessary steps for eventual harmonization:
- T272606 Unpack all current monolithic analyzers. (With care, Greek wasn't trivial, for example.)
- Investigate and file tickets or patches upstream for non-Elastic analyzers that cannot be unpacked.
- T315118 Handle apostrophe-like characters better
- T170625 Figure out what to do with word_break_helper.
- Find or create a plugin to identify acronyms (N.A.S.A.) and de-periodize them (NASA); compare tokenizing wikipedia.org
- T219108 Figure out whether aggressive_splitting makes sense everywhere.
- T180387 Look into enabling hiragana/katakana mapping everywhere.
- T332337 Put back together some multi-script tokens split by the icu_tokenizer (e.g., NGi, И, X → NGiИX or Ko, Я, n → KoЯn)
- T332342 See if it makes sense to standardize ASCII folding/ICU folding; some languages have ASCII folding disabled, some have it enabled, some have it enabled with the option to preserve the unfolded original, some upgrade ASCII folding (with or without preserving the original) to ICU folding.
- Refactor existing analysis configs to use AnalyzerBuilder where possible (some may happen incidentally as part of the above), possibly including for the default config.
- T358495 Enable dotted_I_fix (almost) everywhere and investigate enabling Turkish lowercase for languages that distinguish I/ı and İ/i.
This may also require coming up with a clever way of configuring all of these options, since they may not make sense for all languages—for example, the hiragana/katakana mapping is probably undesirable on Japanese-language wikis—or may require custom ordering with respect to other analysis components. Hard-coded config for each language with language-specific components is possible, but not desirable.
The new(ish) AnalyzerBuilder will make some of this much more orderly and understandable, while also making it easy to update the defaults for almost every language in one go.