Highlights

• Behavior of the dictionary printer and builder are changed (#234)
  • DictionaryPrinter now prints word references in the (Surface, POS, Reading) triple format, instead of the line number format.
  • DictionaryBuilder now allows the dictionary form to be written in the triple format, not only the line number format.

Added

• Benchmark scripts are added (#235)

Fi...

Highlights

• Add Tokenizer.lazyTokenizeSentences(SplitMode mode, Readable input), that performs analysis lazily and saves memory usage (#231)
  • Tokenizer.tokenizeSentences(SplitMode mode, Reader input) is marked as deprecated.

Fixed

• Do not segfault on tokenizing with closed dictionary (#217)
• The default config sudachi.json sets non-existent property joinKanjiNumeric in Joi...

This is a support release for Elasticsearch/OpenSearch integration 3.1.0 release.

Highlights

• Added Config.fromResource method for reading Configs vial PathAnchor. (#212)

Internals

• Plugin classloading is done by PathAnchor and support multiple classloaders (#210, #209)

Notes about v0.7.2

Release v0.7.2 contains subset of the functionality of this release but did not c...

This is a maintenance release

Highlights

• Fixed analysis truncation when using analysis with sentence splitting and the input does not contain data which can be treated as splittable sentences
• Fixed O(N^2) performance in sentence splitting when underlying reader does not fill buffer fully at once
• Stop calling into reader with full buffer

This is a maintenance release

Highlights

• Fixed analysis truncation when using analysis with sentence splitting and the input does not contain data which can be treated as splittable sentences
• Fixed O(N^2) performance in sentence splitting when underlying reader does not fill buffer fully at once
• Stop calling into reader with full buffer