Fixes

Fixed a bug that Groonga couldn't find records that were added after index construction when centroid_column was used

If this bug occurs, Groonga doesn't return records that added as below.

plugin_register language_model/knn

table_create Data TABLE_NO_KEY
column_create Data text COLUMN_SCALAR ShortText
column_create Data rabitq_code COLUMN_SCALAR ShortBinary

load --table Data
[
{"text": "I am a boy."}
]

table_create RaBitQ TABLE_HASH_KEY UInt32 \
  --default_tokenizer \
    'TokenLanguageModelKNN("model", "hf:///groonga/multilingual-e5-base-Q4_K_M-GGUF", \
                           "centroid_column", "centroid", \
                           "code_column", "rabitq_code", \
                           "passage_prefix", "passage: ", \
                           "query_prefix", "query: ")'
column_create RaBitQ centroid COLUMN_VECTOR Float32
column_create RaBitQ data_text COLUMN_INDEX Data text

load --table Data
[
{"text": "This is an apple."}
]

select Data \
  --filter 'language_model_knn(text, "fruit")' \
  --output_columns text
[
  [
    0,
    0.0,
    0.0
  ],
  [
    [
      [
        1
      ],
      [
        [
          "text",
          "ShortText"
        ]
      ],
      [
        "I am a boy."
      ]
    ]
  ]
]

Experimental features

These features are still experimental and unstable. We must not use these features in production.

Bundled OpenZL

OpenZL is a compression framework. It provides high compression ratios and high speed.

We can specify COMPRESS_OPENZL in column_create, but if we specify COMPRESS_OPENZL, it doesn't work yet. Because we haven't implemented the feature for compression and decompression.

Added support for JSON type

We can load and dump against some data types. However, we don't implement other features yet.

Contributors

$ git shortlog -sn v15.2.1..
    23	Sutou Kouhei
     7	Horimoto Yasuhiro
     7	takuya kodama
     3	dependabot[bot]
     1	Michael Cho

Translations

• Japanese