better memory defragmentation for json objects: #6240 #6276
What's Changed
actions: Add a volume mount to container by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6048
chore: remove RecvProvided from dragonfly connection by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/6050
fix(rdb_load): Store streamed items per rdb lodaer by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6049
fix(search): Result set cutoff by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/5906
fix: remove duplicate expired/evicted metric header by @romange in https://github.com/dragonflydb/dragonfly/pull/6058
fix(test): ssl test to work with pytest 3.12 by @romange in https://github.com/dragonflydb/dragonfly/pull/6059
feat(hset): Load tiered values in read-only by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/5996
chore(tiering): Add throttling metric + more tests by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6040
v1.36.0 - dragonfly Release Notes | AnnounceHQ
fix: macOS build by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6065
server: Parse JSON on destination shard by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6061
feat: add CSS option for FT.CREATE by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6057
chore(facade): Handle Socket::Recv() returning 0 explicitly by @romange in https://github.com/dragonflydb/dragonfly/pull/6066
fix: Remove index docs on eviction and cluster migration by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/6062
fix(tests): Disable huge val streaming by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6074
fix(replication): Add detailed error message for REPLTAKEOVER failures by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6071
fix(tiering): Fix upload strategy by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6067
fix(replication): use correct replica pointer in takeover catchup loop by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6068
fix(replication): add socket timeout to REPLTAKEOVER command by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6070
fix(takeover): pick correct replica to reconcile slots by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/6034
test: fix flaky test_takeover_bug_wrong_replica_checked_in_logs by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6075
feat(helm): adds common label support to the helm chart by @gaganhr94 in https://github.com/dragonflydb/dragonfly/pull/6042
Revert "feat(helm): adds common label support to the helm chart (#6042)" by @romange in https://github.com/dragonflydb/dragonfly/pull/6077
fix(server): Move LpGetView to core target by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6076
feat: adds common label support to the helm chart by @gaganhr94 in https://github.com/dragonflydb/dragonfly/pull/6080
docs(build-from-source): update LTO instructions to use USE_MOLD option by @glevkovich in https://github.com/dragonflydb/dragonfly/pull/6081
Update build-from-source.md by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6087
feat: add ftsearch result parsing for css by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6090
chore: pull the latest helio by @romange in https://github.com/dragonflydb/dragonfly/pull/6096
fix: suppress BUSYGROUP warning logs by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6095
feat(tiering): Basic metrics by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6086
chore: change the resolution of reply latency to measurement to ns by @romange in https://github.com/dragonflydb/dragonfly/pull/6092
fix: test_reply_count regression by @romange in https://github.com/dragonflydb/dragonfly/pull/6099
chore(hooks): add commit-msg hook to enforce Signed-off-by line in commit messages by @glevkovich in https://github.com/dragonflydb/dragonfly/pull/6082
fix(server): handle INFO command with multiple sections correctly by @glevkovich in https://github.com/dragonflydb/dragonfly/pull/6093
chore: pull helio by @romange in https://github.com/dragonflydb/dragonfly/pull/6100
fix: smashing stack on Huffman test by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6102
tools: Add scripts and supporting files to generate os packge repo by @abhijat in https://github.com/dragonflydb/dragonfly/pull/5923
chore(deps): bump the actions group across 1 directory with 4 updates by @dependabot[bot] in https://github.com/dragonflydb/dragonfly/pull/6107
chore(core): Refactoring to prepare for using stateless allocator by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6111
tools: Download ARM deb package, disable fedora ARM test by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6110
fix: replica of self by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/6097
chore: relax pytest requirements by @romange in https://github.com/dragonflydb/dragonfly/pull/6113
feat(JSON): Use stateless allocator by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6060
fix(dfly_bench): fix parsing for memcache by @romange in https://github.com/dragonflydb/dragonfly/pull/6116
chore: update vector-bench README by @romange in https://github.com/dragonflydb/dragonfly/pull/6117
chore: add mi_malloc memory test by @romange in https://github.com/dragonflydb/dragonfly/pull/6118
docs: add copilot-instructions.md for AI coding agent onboarding by @Copilot in https://github.com/dragonflydb/dragonfly/pull/6112
fix: Handle django-cache script so it could run in dragonfly by @romange in https://github.com/dragonflydb/dragonfly/pull/6123
chore: replace PMR_NS with StatelessAllocator in dense_set by @romange in https://github.com/dragonflydb/dragonfly/pull/6121
chore: implement background self-identification of huffman encoding by @romange in https://github.com/dragonflydb/dragonfly/pull/6106
chore: gcc14 for fedora 30 by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6124
chore: expose more information when takeover fails on master. by @romange in https://github.com/dragonflydb/dragonfly/pull/6130
feat(search): Global HNSW vector index by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/6103
fix(search): Support unicode for tags by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6127
chore: use our own vector for hnsw distance computation by @romange in https://github.com/dragonflydb/dragonfly/pull/6132
chore: more fixes for django-cacheops compatibility by @romange in https://github.com/dragonflydb/dragonfly/pull/6129
chore: refactor hnsw index into separate files by @romange in https://github.com/dragonflydb/dragonfly/pull/6134
chore: move HierarchicalNSW class to our codebase by @romange in https://github.com/dragonflydb/dragonfly/pull/6136
feat(tiering): Serialize hashes by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6015
fix(stream): Allow XINFO in script by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6128
fix(core): Remove async delete flag by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6139
feat: implement DenseSet::Shrink by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6094
chore(core): Move page_usage related code to its own library by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6146
chore: quietly ignore LATENCY HISTOGRAM command by @romange in https://github.com/dragonflydb/dragonfly/pull/6151
fix(tiering): missing comma in dashboard json by @goochs in https://github.com/dragonflydb/dragonfly/pull/6150
chore: add a tool that creates charts based on memtier metrics data by @romange in https://github.com/dragonflydb/dragonfly/pull/6154
chore: add latency graph to local dashboard by @romange in https://github.com/dragonflydb/dragonfly/pull/6153
fix: incorrect stalled heartbeat warning by @romange in https://github.com/dragonflydb/dragonfly/pull/6155
feat(lua): add legacy-float flag for Lua 5.1 number compatibility by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6149
chore(dfly_bench): properly handle 0 from Recv by @romange in https://github.com/dragonflydb/dragonfly/pull/6159
chore: minor cleanups around the eval flow by @romange in https://github.com/dragonflydb/dragonfly/pull/6152
tests: Use AWS public docker registry by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6161
fix(search): Fix numeric tree splits by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6143
fix(tiering): Small test fixes by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6133
chore: introduce common directory by @romange in https://github.com/dragonflydb/dragonfly/pull/6170
chore: introduce ParsedArguments class by @romange in https://github.com/dragonflydb/dragonfly/pull/6169
chore: rename WRITE to JOURNALED by @romange in https://github.com/dragonflydb/dragonfly/pull/6172
chore: fix data corruption bug by @romange in https://github.com/dragonflydb/dragonfly/pull/6174
chore: add ability for ParsedArgs to reference BackedArguments by @romange in https://github.com/dragonflydb/dragonfly/pull/6171
chore: use BackedArguments for implementing a pipeline message by @romange in https://github.com/dragonflydb/dragonfly/pull/6167
feat(search): Defragment search indices by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6144
chore(core): Refactor SmallString by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6138
chore: clean up sanitizers checks by @romange in https://github.com/dragonflydb/dragonfly/pull/6173
feat(fuzz): improve fuzzing effectiveness, better timeouts, and comprehensive seeds, reproduction tooling by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6166
chore: use BackedArguments for memcache_parser by @romange in https://github.com/dragonflydb/dragonfly/pull/6176
CI: download docker for correct architecture by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6180
fix(search): Fix numeric tree finalization by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6188
chore: improve backed_args by allowing random access to its arguments by @romange in https://github.com/dragonflydb/dragonfly/pull/6189
Add coordinator connection reusing by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6179
chore(server): Don't decode size for empty blob by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/6194
chore: hold memcache value in BackedArguments by @romange in https://github.com/dragonflydb/dragonfly/pull/6193
chore(core): Remove some unused CompactObj code by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6141
feat(search): Simple multi-reader multi-writer mutex for hnsw index by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/6156
chore: asynchronous IO for connection fiber by @kostasrim in https://github.com/dragonflydb/dragonfly/pull/6069
fix: macOS test by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6195
Integrate hiredis by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6198
fix(search): Use thread lock for hnsw multi reader/multi writer mutex by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/6205
chore: to move some of the command-specific state into CommandContext by @romange in https://github.com/dragonflydb/dragonfly/pull/6199
hset_family: Preserve ttl on increment command by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6186
Pass CommandContext as a mutable pointer to all the commands. by @romange in https://github.com/dragonflydb/dragonfly/pull/6206
fix: macOS compilation error by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6208
chore(rax): add move ctor to rax tree, cleanup defragmap by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6178
docs: add antigravity agent instructions by @romange in https://github.com/dragonflydb/dragonfly/pull/6209
chore: reduce boilerplate by removing redundant function declarations by @romange in https://github.com/dragonflydb/dragonfly/pull/6207
chore: move some SendError calls to CommandContext by @romange in https://github.com/dragonflydb/dragonfly/pull/6211
chore(server): polish CommandContext data structure by @romange in https://github.com/dragonflydb/dragonfly/pull/6214
fix: regression tests by @romange in https://github.com/dragonflydb/dragonfly/pull/6212
feat: add WITHSORTKEYS option to FT.SEARCH by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6210
chore(tiering): Optimize serialization by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6164
chore: Introduce ParsedCommand by @romange in https://github.com/dragonflydb/dragonfly/pull/6219
fix(search): remove invalid DCHECK in FT.INFO for concurrent index operations by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6204
fix(server): Fix EXPIRE bugs, support non-exclusive options by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6203
chore: pass ParsedCommand to DispatchMC by @romange in https://github.com/dragonflydb/dragonfly/pull/6221
fix(server): Rename with tiering by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6217
feat(search): Yield when rebuilding index by @mkaruza in https://github.com/dragonflydb/dragonfly/pull/6224
fix: load search index from old snapshots with TAG SORTABLE SEPARATOR order by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6220
chore: switch dev docker to GCC 14 by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6225
fix: fix build when search is off by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6229
chore: further clean ups around ParsedCommand. by @romange in https://github.com/dragonflydb/dragonfly/pull/6223
docs: create unified AI agent instruction system with AGENTS.md by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6231
fix(core): Account for size change on json object defragment by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6053
feat: create a wrapper for hiredis by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6233
chore: further replace SendError calls with CommandContext by @romange in https://github.com/dragonflydb/dragonfly/pull/6236
chore: simplify memcache pipelining flow by @romange in https://github.com/dragonflydb/dragonfly/pull/6230
chore: reduce reply builder interface by @romange in https://github.com/dragonflydb/dragonfly/pull/6237
chore: swap SendError calls in json_family by @romange in https://github.com/dragonflydb/dragonfly/pull/6238
feat: add open addressing hash set by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/4705
fix: macOS build by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6239
chore: reduce compilation dependencies between server and facade by @romange in https://github.com/dragonflydb/dragonfly/pull/6242
chore: switch SendError calls to CommandContext by @romange in https://github.com/dragonflydb/dragonfly/pull/6245
chore: minor fixes around async execution by @romange in https://github.com/dragonflydb/dragonfly/pull/6241
chore: Split CompactObj into Key&Value by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6218
chore(core): remove unused num_used_buckets_ from DenseSet by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6250
fix: cross-shard search tests by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6244
chore: refactor ParsedCommand implementation into a separate file by @romange in https://github.com/dragonflydb/dragonfly/pull/6251
fix: reset obj_malloc_used_ in DenseSet::Clear by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6254
chore: simplify memcache_parser_test by @romange in https://github.com/dragonflydb/dragonfly/pull/6259
chore(server): Remove more family header definitions by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6255
chore: MemcacheParser now consumes all the input. by @romange in https://github.com/dragonflydb/dragonfly/pull/6260
chore: an issue of destruction of in-flight parsed commands by @romange in https://github.com/dragonflydb/dragonfly/pull/6258
chore: switch SendError calls in zset_family by @romange in https://github.com/dragonflydb/dragonfly/pull/6248
feat(json): Improve json defragmentation by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6240
chore: align parse errors with deferred replies by @romange in https://github.com/dragonflydb/dragonfly/pull/6261
chore(server): Split tiering into own cmake target by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6252
chore(server): Remove some manual acl by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6256
chore(common): Include cstdint to fix build error on newer fedora versions by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6268
chore(server): Multi consistency test by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6215
chore: move memcache reply rendering into MCRender by @romange in https://github.com/dragonflydb/dragonfly/pull/6267
chore: reduce compilation dependencies between facade and server by @romange in https://github.com/dragonflydb/dragonfly/pull/6269
chore(deps): bump the actions group with 4 updates by @dependabot[bot] in https://github.com/dragonflydb/dragonfly/pull/6222
chore: get rid of command specific flags in MCReplyBuilder by @romange in https://github.com/dragonflydb/dragonfly/pull/6273
feat(transaction): Blocking unlock multi by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6246
fix: macOS build by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6275
Add alloc size to OAHSet by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6272
chore: switch SendError calls to CommandContext API by @romange in https://github.com/dragonflydb/dragonfly/pull/6270
chore: implement async dispatching for memcache SET command by @romange in https://github.com/dragonflydb/dragonfly/pull/6247
chore(helio): bump for improved TLS EOF handling by @glevkovich in https://github.com/dragonflydb/dragonfly/pull/6274
chore(server): Unite family headers by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6266
chore(server): Use cycle based quota for defragmentation by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6262
chore: swap ParsedEntry::CmdData with BackedArguments by @romange in https://github.com/dragonflydb/dragonfly/pull/6278
chore(json): Use new json defragmentation by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6276
fix: loading of a snapshot from a cloud folder by @romange in https://github.com/dragonflydb/dragonfly/pull/6280
chore: fix build error by @romange in https://github.com/dragonflydb/dragonfly/pull/6289
chore: simplify GAT command arguments by @romange in https://github.com/dragonflydb/dragonfly/pull/6288
chore: combine GAT and MGET code together by @romange in https://github.com/dragonflydb/dragonfly/pull/6290
fix: allow Cm_Cache_Backend_Redis Lua scripts by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6292
fix: CONFIG GET returns numeric bytes for memory configs by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6291
chore: remove Stored and Deleted interfaces from ParsedCommand by @romange in https://github.com/dragonflydb/dragonfly/pull/6287
fix: macOS build by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6293
chore: change DispatchCommand interface by @romange in https://github.com/dragonflydb/dragonfly/pull/6283
chore: implement support for deferred mget by @romange in https://github.com/dragonflydb/dragonfly/pull/6286
chore: update .gitignore and clarify TRAFFIC help by @glevkovich in https://github.com/dragonflydb/dragonfly/pull/6294
chore: optimize experimental_io_loop_v2 flow by @romange in https://github.com/dragonflydb/dragonfly/pull/6295
feat(build): rename cxx_test to helio_cxx_test by @glevkovich in https://github.com/dragonflydb/dragonfly/pull/6297
chore(server): Remove list family header file by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6296
Sorting for cluster search by @BorysTheDev in https://github.com/dragonflydb/dragonfly/pull/6301
feat(server): Allow SELECT inside global/non-atomic exec/eval by @dranikpg in https://github.com/dragonflydb/dragonfly/pull/6302
chore: introduce RespSrvParser by @romange in https://github.com/dragonflydb/dragonfly/pull/6300
chore: move more reply calls to CommandContext API by @romange in https://github.com/dragonflydb/dragonfly/pull/6303
fix: Handle expired hash fields during RENAME without crashing by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6306
chore: refresh release settings for dragonfly by @romange in https://github.com/dragonflydb/dragonfly/pull/6304
chore: fix release action by @romange in https://github.com/dragonflydb/dragonfly/pull/6308
test(geo): relax fp checks for Helio Haswell upgrade by @glevkovich in https://github.com/dragonflydb/dragonfly/pull/6307
Fix fiber stack overflow and improve JSON safety by @romange in https://github.com/dragonflydb/dragonfly/pull/6316
CI: sync valkey tests by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6319
chore: Add debug information to investigate memcache regtest failure. by @romange in https://github.com/dragonflydb/dragonfly/pull/6322
fix: handle control messages during SquashPipeline iteration by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6313
chore(defrag): Add defragmentation of search indices by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6281
CI: Extract common step for syncing valkey tests into action by @abhijat in https://github.com/dragonflydb/dragonfly/pull/6333
tests: make valkey_search integration tests optional by @vyavdoshenko in https://github.com/dragonflydb/dragonfly/pull/6332
chore: Reused parsed messages for memcache workloads by @romange in https://github.com/dragonflydb/dragonfly/pull/6324
fix: memcache parser CRLF handling by @romange in https://github.com/dragonflydb/dragonfly/pull/6330
Huge thanks to all the contributors! ❤️
New Contributors
@gaganhr94 made their first contribution in https://github.com/dragonflydb/dragonfly/pull/6042
@glevkovich made their first contribution in https://github.com/dragonflydb/dragonfly/pull/6081
@Copilot made their first contribution in https://github.com/dragonflydb/dragonfly/pull/6112
@goochs made their first contribution in https://github.com/dragonflydb/dragonfly/pull/6150
Full Changelog: https://github.com/dragonflydb/dragonfly/compare/v1.35.0...v1.36.0