* feat: add huffman coding to compact object
Read path and tests will follow in the next PR.
Partially addresses #4880
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* Update src/core/compact_object.cc
Co-authored-by: Kostas Kyrimis <kostas@dragonflydb.io>
Signed-off-by: Roman Gershman <romange@gmail.com>
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Signed-off-by: Roman Gershman <romange@gmail.com>
Co-authored-by: Kostas Kyrimis <kostas@dragonflydb.io>
Specifically get rit of MaskEnum and replace it with explicit bits aliasing the mask.
Reorganize the encoding bits to be able to store 4 states that include huffman encoding.
Solves the first part of #4880
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Allow export/import of huffman tables via
`DEBUG COMPRESSION EXPORT` or `DEBUG COMPRESSION IMPORT <bintable>`
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore(zset_family/score_map): Remove usages of sds
The usages of WrapSds are removed from zset_family calls.
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
The bug: during the override of the existing external string, we called
`TieredStorage::Delete` to delete the external reference. This function
called CompactObj::Reset that cleared all the attributes on the value, including
expiry.
The fix: preserve the mask but clear the external state from the object.
Fixes#4672
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Latest helio fixes the bug in DnsResolve - fixes#4244
Also, add some comments and perform minor clean ups in tiered codebase.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. rdb loader big string loading in chunks
2. snapshot compression logic is disabled in case of big buffer
Signed-off-by: adi_holden <adi@dragonflydb.io>
* chore: introduce background deletions of DenseSet objects
We currently implement them only for sets but the same approach can work for zset, hashes, lists as well.
1. Use introsive::list for CoolQueue.
2. Make sure that we ignore cool memory usage when computing average object size to
prevent evictions during dashtable growth attempts.
3. Remove items from the cool storage before evicting them from the dash table.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore: reenable evictions upon insertion to avoid OOM rejections
Before: when running dragonfly with --cache_mode we could get OOM rejections
even though the eviction policy allowed to evict items to free memory.
Ideally, dragonfly in cache mode should not respond with the OOM error.
This PR reuses the same Eviction step we have in the Heartbeat and conditionally applies it
during the insertion. In my test the OOM errors went from 500K to 0 and the server
still respected memory limit.
Also, remove the old heuristics that has never been used.
Test:
./dfly_bench --key_prefix=bar: -d 1024 --ratio=1:0 --qps=200 -n 3000
./dragonfly --dbfilename= --proactor_threads=2 --maxmemory=600M --cache_mode
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Also, add the according API to compact object.
Now external objects can be in two states: Cool and Offloaded.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
It's more efficient to offload raw blobs kept in CompactObject than decoded strings.
Unfortunately, it's also more complicated. The complexity arises around Read/Modify operations
that must convert a raw blob into a processed string. Moreover, Modify decodes the raw string, therefore
subsequent callbacks already see the decoded string.
Finally, this PR fixes the logic in NotifyFetched flow:
before it could skip uploading the modified value back to memory, which breaks correctness.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore: Add CompactObj Raw methods
Currently, compact object does decoding/encoding of its internal string blobs.
For tiering usecase, it makes sense to introduce the direct access to the blobs and avoid
this unnecessary translation.
This PR introduces such access.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Signed-off-by: Roman Gershman <romange@gmail.com>
Co-authored-by: Shahar Mike <chakaz@users.noreply.github.com>
* chore: implement path mutation for JsonFlat
This is done by converting the flat blob into mutable c++ json
and then building a new flat blob.
Also, add JsonFlat encoding to CompactObject class.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Fix linker dependencies - now dfly_core depends on jsonpath.
jsonpath does not depend on CompactObject anymore.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* feat(server): Return per-type memory breakdown from INFO and /metrics
* OBJ_TYPE_MAX
* Move AddTypeMemoryUsage, use it from TieredStorage
* Remove strval_memory_usage
* Remove redundant strval_bytes
---------
Signed-off-by: Shahar Mike <chakaz@users.noreply.github.com>
* chore: Refactor SortedMap
1. intoduce variant of Redis and Dragonfly implementation and
wrap each operation with a dedicated dispatcher operator.
The dragonfly implementation path is a noop and not implemented.
2. Introduce a very basic sorted_map_test around Add and move the unrelated old test
from sorted_map_test to bptree_set_test.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Consolidate skiplist based zset functionality into a dedicated class
called SortedMap. The code is adapted from t_zset.c
The old code in t_zset.c is deleted.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
This is pure refactoring PR that does not change any functionality besides
prohibiting using AsRobj/SyncRobj functions for compact objects of type
OBJ_ZSET. This is needed in case we decide in the future to implement our own
zset type.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. Use experimental/memory_resource because clang on freebsd does not support yet std::memory_resource
2. Pull latest helio with all the compatibility fixes.
3. Replace linux only SOL_TCP constant with IPPROTO_TCP (same value).
4. Fix build files to work on FreeBsd system.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. Support tiered deletion.
2. Add notion of tiered entity in "DEBUG OBJECT" output.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. Allow offloading blobs larger than 2KB.
2. Totally redesign the offloading algorithm for blobs smaller than 2KB.
3. Fix bugs around IO request cancelations.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
