* feat: add huffman coding to compact object
Read path and tests will follow in the next PR.
Partially addresses #4880
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* Update src/core/compact_object.cc
Co-authored-by: Kostas Kyrimis <kostas@dragonflydb.io>
Signed-off-by: Roman Gershman <romange@gmail.com>
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Signed-off-by: Roman Gershman <romange@gmail.com>
Co-authored-by: Kostas Kyrimis <kostas@dragonflydb.io>
Specifically get rit of MaskEnum and replace it with explicit bits aliasing the mask.
Reorganize the encoding bits to be able to store 4 states that include huffman encoding.
Solves the first part of #4880
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Allow export/import of huffman tables via
`DEBUG COMPRESSION EXPORT` or `DEBUG COMPRESSION IMPORT <bintable>`
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore(zset_family/score_map): Remove usages of sds
The usages of WrapSds are removed from zset_family calls.
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
The bug: during the override of the existing external string, we called
`TieredStorage::Delete` to delete the external reference. This function
called CompactObj::Reset that cleared all the attributes on the value, including
expiry.
The fix: preserve the mask but clear the external state from the object.
Fixes#4672
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. rdb loader big string loading in chunks
2. snapshot compression logic is disabled in case of big buffer
Signed-off-by: adi_holden <adi@dragonflydb.io>
chore: refactorings around deletions
Done as a preparation to introduce asynchronous deletions for sets/zsets/hmaps.
1. Restrict the interface around DbSlice::Del. Now it requires for the iterator to be valid and the checks should
be explicit before the call. Most callers already provides a valid iterator.
2. Some minor refactoring in compact_object_test.
3. Expose DenseSet::ClearStep to allow iterative deletions of elements.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
The problem: apparently, jsoncons uses strtod by default when parsing doubles.
On some platforms (alpine/musl) this function uses lots of stack, which potentially can lead to stack corruption.
This PR configures jsoncons to use std::from_chars that is more efficient and less stack hungry.
The single include point to consume jsoncons/json.hpp should be "core/json_object.h"
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
This PR syncs some of the improvements that were introduced in streams in Redis 7.2.3 OSS.
1. verify xsetid against max deleted id in the stream
2. Implement precise memory measurement of streams for "memory usage" command.
3. Always compact nodes in stream listpacks after creating new nodes.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Also, add the according API to compact object.
Now external objects can be in two states: Cool and Offloaded.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
It's more efficient to offload raw blobs kept in CompactObject than decoded strings.
Unfortunately, it's also more complicated. The complexity arises around Read/Modify operations
that must convert a raw blob into a processed string. Moreover, Modify decodes the raw string, therefore
subsequent callbacks already see the decoded string.
Finally, this PR fixes the logic in NotifyFetched flow:
before it could skip uploading the modified value back to memory, which breaks correctness.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore: Add CompactObj Raw methods
Currently, compact object does decoding/encoding of its internal string blobs.
For tiering usecase, it makes sense to introduce the direct access to the blobs and avoid
this unnecessary translation.
This PR introduces such access.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Signed-off-by: Roman Gershman <romange@gmail.com>
Co-authored-by: Shahar Mike <chakaz@users.noreply.github.com>
* chore: implement path mutation for JsonFlat
This is done by converting the flat blob into mutable c++ json
and then building a new flat blob.
Also, add JsonFlat encoding to CompactObject class.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* feat(server): Return per-type memory breakdown from INFO and /metrics
* OBJ_TYPE_MAX
* Move AddTypeMemoryUsage, use it from TieredStorage
* Remove strval_memory_usage
* Remove redundant strval_bytes
---------
Signed-off-by: Shahar Mike <chakaz@users.noreply.github.com>
Then use the right version (hopefully) in the right places.
Specifically, this fixes a serialization bug, where we could send
malformed responses when using `UpperBoundSize()` to write array length.
chore: reduce double encoding for listpacks
Following memory improvements by Redis 7, use double convrsion library to represent
double values with less space for listpacks.
The change is to use double conversion library instead of plain sprintf inside
zzlInsertAt. This requires to move zzlInsertAt to Dragonfly codebase.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. Cherry-pick changes from Redis 7 that encode integer scores more efficiently
2. Introduces optimization that first checks if the new element should be the last
for listpack sorted sets.
3. Consolidates listpack related constants and tightens usage for listpack.
4. Introduce MEMORY USAGE command.
5. Introduce a small delay before decommitting memory pages back to OS.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
