Highlights

• Add Tokenizer.lazyTokenizeSentences(SplitMode mode, Readable input), that performs analysis lazily and saves memory usage (#231)
  • Tokenizer.tokenizeSentences(SplitMode mode, Reader input) is marked as deprecated.

Fixed

• Do not segfault on tokenizing with closed dictionary (#217)
• The default config sudachi.json sets non-existent property joinKanjiNumeric in JoinNumericPlugin (#221)
• fix incorrect size calculation when expand (#227)
• Update tutorial.md (#226)