Improvements

[column_create] Added support for progress log of index construction

You can view the progress log during offline index construction.

The default log level is debug. This means you must enable debug level logging to view progress logs. If you set the log level to notice, you can view progress logs without other debug logs.

[token_language_model_knn] Added centroid_column option

This option is for large embeddings with more than 1025 dimensions (more than 4100 bytes). Groonga table keys must be 4 KiB or smaller. Embeddings larger than 4 KiB cannot be stored as keys.

You can use this option to store large embeddings in a column instead of in the table key. Specify the column for storing embeddings and add it to the table for the index as below.

table_create LargeCentroids TABLE_HASH_KEY UInt32 \
  --default_tokenizer \
      'TokenLanguageModelKNN("model", "hf:///groonga/multilingual-e5-base-Q4_K_M-GGUF", \
                             "centroid_column", "centroid", \
                             "code_column", "embedding_code", \
                             "passage_prefix", "passage: ", \
                             "query_prefix", "query: ")'
column_create LargeCentroids centroid COLUMN_VECTOR Float32
column_create LargeCentroids data_content COLUMN_INDEX Memos content

[token_language_model_knn] Added n_gpu_layers option

Specify the number of GPU layers to use for language model. If not specified, Groonga uses GPU as much as possible.

In most cases, you don't need to set this option explicitly.

To disable GPU usage, set n_gpu_layers to 0.

[language_model_knn] Added k option

Specify the number of records to return. If you don't set this option, the upper limit of the number of returned records is 10.

You can specify a negative value. It means: (number of matched records) + k + 1. For example, "{ "k" : -1 }" outputs all records. It’s a very useful value to show all records.

Fixes

Fixed a bug that causes overflow in the key of TABLE_PAT_KEY tables.

This issue is found in PGroonga by pgroonga/pgroonga#901. If you execute offline index construction against a column that has a large size value (When the size of a token is 4096 bytes after tokenization), the index may be broken.

Reported by lilydjwg.

Thanks

• lilydjwg

Contributors

$ git shortlog -sn v15.2.0..
     9	Sutou Kouhei
     7	Horimoto Yasuhiro
     6	Abe Tomoaki
     5	takuya kodama
     1	dependabot[bot]

Translations

• Japanese