Commit graph

3387 commits

Author SHA1 Message Date
Roman Gershman
ad5aa66350
chore: Add HuffmanDecoder class (#5078)
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-05-07 19:58:43 +03:00
Borys
8c125a23e3
feat: add migrated keys statistic (#5043) 2025-05-07 15:48:22 +03:00
Roman Gershman
54328fd00e
chore: reorganize compact object mask bits (#5077)
Specifically get rit of MaskEnum and replace it with explicit bits aliasing the mask.
Reorganize the encoding bits to be able to store 4 states that include huffman encoding.

Solves the first part of #4880

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-05-07 15:08:22 +03:00
Kostas Kyrimis
05d99769e1
chore: add password flag to dfly_bench (#5074)
* add password flag to dfly_bench

---------

Signed-off-by: kostas <kostas@dragonflydb.io>
2025-05-07 07:32:13 +00:00
Roman Gershman
3977f0f60a
chore: futher dash table clean ups (#5072)
Added types PhysicalBid and LogicalBid to help showing the meaning/context behind
a bucket id.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-05-07 10:30:49 +03:00
Kostas Kyrimis
843a40dba9
chore(metrics): add rdb_bgsave_in_progress and rdb_last_bgsave_status (#5061)
* add metric rdb_bgsave_in_progress
* add metric rdb_last_bgsave_status
2025-05-07 10:15:14 +03:00
Roman Gershman
3f3d232211
chore: introduce huffman encoder class (#5076)
Move the code in debugcmd.cc into HuffmanEncoder.

One of the building blocks for #4883

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-05-07 10:02:40 +03:00
Stepan Bagritsevich
3d79664a19
fix(json_family): Fix memory tracking for the JSON.SET command. THIRD PR (#5069)
* fix(json_family): Fix memory tracking for the JSON.SET command

fixes dragonflydb#5054

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* refactor: address comments

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

---------

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>
2025-05-07 08:56:43 +02:00
Roman Gershman
b3e0bcfb31
chore: further extend the compression analysis (#5065)
Allow export/import of huffman tables via
`DEBUG COMPRESSION EXPORT` or `DEBUG COMPRESSION IMPORT <bintable>`

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-05-06 13:15:54 +03:00
Roman Gershman
4d07d7d053
chore: dash table clean ups (#5064)
Remove stash template parameter because we only use dashtable with a single configuration
of STASH_CNT=4.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-05-06 09:33:00 +00:00
Stepan Bagritsevich
3da7e49712
feat(memory_cmd): Add WITHOUTKEY option for the MEMORY USAGE command. SECOND PR (#5068)
feat(memory_cmd): Add WITHOUTKEY option for the MEMORY USAGE command

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>
2025-05-06 10:44:16 +02:00
Kostas Kyrimis
a8b19c9b88
chore: skip test_bug_in_json_memory_tracking (#5066)
Signed-off-by: kostas <kostas@dragonflydb.io>
2025-05-06 07:38:02 +00:00
Roman Gershman
d1e5049f64
feat(dfly_bench): introduce connect only flow (#5052)
In this flow dfly_bench only connects to the server, reproducing high connection rate scenario.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-05-06 08:36:33 +03:00
adiholden
6a84ad0208
bug(server): fix lns mismatch in replication (#4967)
Signed-off-by: adi_holden <adi@dragonflydb.io>
2025-05-05 10:50:58 +00:00
Volodymyr Yavdoshenko
f7a40f66d6
fix: json.set recursive processing crash (#5040)
fixed: https://github.com/dragonflydb/dragonfly/issues/5038
2025-05-04 19:06:18 +03:00
Roman Gershman
f09df995a6
feat: introduce 'debug iostats' command (#5051)
The command shows verious io stats per thread

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-05-04 18:34:19 +03:00
Volodymyr Yavdoshenko
5a901b9bc7
fix: send response error for long sequence instead of command crash (#5048)
fixed: https://github.com/dragonflydb/dragonfly/issues/5047
2025-05-04 13:18:03 +00:00
Stepan Bagritsevich
0a33f28bd4
chore(json_family): Clean up code. FIRST PR (#5049)
* chore(json_family): Clean up code

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* refactor: address comments

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* chore(json_family): Remove const ref for mutate operations

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* refactor: address comments 2

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* chore(json_family): Remove GetJson method

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* refactor: revert const modifiers removing

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

---------

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>
2025-05-04 13:09:59 +00:00
Borys
9f7b11628a
refactor: reduce number of cluster logs (#5044) 2025-05-04 13:51:49 +03:00
Borys
d623304c39
refactor: add WARNING regarding max_client_iobuf_len flag usage (#5029) 2025-05-04 11:06:37 +03:00
Kostas Kyrimis
ad946d8e06
chore: add dfly_bench to release workflow (#5045)
* upload dfly_bench binaries on every release
* fix macos failing build

Signed-off-by: kostas <kostas@dragonflydb.io>
2025-05-02 22:34:37 +03:00
Kostas Kyrimis
65f94e5d82
chore: skip failing test_migration_rebalance_node (#5042)
Signed-off-by: kostas <kostas@dragonflydb.io>
2025-05-02 12:08:05 +00:00
Stepan Bagritsevich
74524415c3
fix(stream_family): Fix stream memory tracking issues (#5024) 2025-05-02 11:29:52 +02:00
Kostas Kyrimis
291b2622c6
fix: make FreeMemWithEvictionStep atomic (#4885)
Signed-off-by: kostas <kostas@dragonflydb.io>
2025-05-02 10:31:55 +03:00
Roman Gershman
f4f4668c7a
chore: bump up max_busy_read_usec in tests (#5039)
cycle clock like most clocks is affected by thread context switching, and maybe this is what causes
test failures. Try to increase the parameter to even further.

Also, add benchmarks for the clock as well as fix some compiler warnings around parser.cc

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-05-01 00:17:29 +03:00
Harshit Gupta
1082724500
fix: Support for additional flags in expireat, pexpire and pexpireat commands (#5007)
* fix: Support for additional flags in expireat, pexpire and pexpireat commands.

* Replaced actual clock with Test clock in generic family tests.
2025-04-30 18:19:07 +03:00
Roman Gershman
befb36d477
chore: get rid of possible recursion when unwinding structured reply (#5012) 2025-04-30 18:17:38 +03:00
Roman Gershman
dced0371d3
chore: show moving P99 latency in dfly_bench (#5027)
As opposted to the cumulatative one that is shown at the end.
The reason for this: if P99 went up temporarily - it does not go back easily and stays up.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-04-30 18:13:36 +03:00
Volodymyr Yavdoshenko
ae4615b470
fix: json depth limit has been changed to 64 (#5036)
fixed: https://github.com/dragonflydb/dragonfly/issues/5028
2025-04-30 17:40:03 +03:00
Borys
2661fe16b4
feat: add copy cmd (#5032)
* add copy command
* add tests
2025-04-30 16:25:21 +03:00
Roman Gershman
10cd22375e
chore: more performance improvements around MGET and pipelining (#5022)
1. Remove one vector (affects allocation and data locality) in squashing.
2. stop deduplicating MGET keys by default, but keep it as a run-time flag.

Also, finally fix TieredStorageTest.FlushAll test.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-04-30 08:15:55 +03:00
Volodymyr Yavdoshenko
84456a2442
fix: JSON.ARRPOP command crash fixed (#5026)
fixed: https://github.com/dragonflydb/dragonfly/issues/5025
2025-04-30 08:00:01 +03:00
mkaruza
13d8062bb9
feat(pytest): Generate command with huge value strings in seeder (#4957)
Generate command with long strings. We will first create number of
random samples that will be used to match keys for which we are generating huge 
strings. Total length of huge value string will be divided equally between elements in
generated command.  

Closes #4430

Signed-off-by: mkaruza <mario@dragonflydb.io>
2025-04-29 21:07:13 +02:00
adiholden
880cd8d1a5
feat(server): support partial sync from last master (#5015)
Signed-off-by: adiholden <adi@dragonflydb.io>
2025-04-29 12:12:23 +03:00
Volodymyr Yavdoshenko
23d310b607
chore: polishing. put basic usage info to stdout using one call to a void messing lines (#5019)
chore: polishing. put basic usage info to stdout using one call to avoid messing with lines
2025-04-28 17:45:20 +03:00
Borys
70be62c6a1
chore: add DCHECK to easier tests debugging (#5018) 2025-04-28 11:15:05 +00:00
Volodymyr Yavdoshenko
2128eb31c9
fix: HNSW schema deleting document crash (#4987)
fixed: https://github.com/dragonflydb/dragonfly/issues/4979
2025-04-28 11:01:04 +00:00
Borys
5ea3070abf
refactor: update contributing doc (#5016) 2025-04-28 13:20:44 +03:00
mkaruza
98fb8fb38f
fix(glob): Correctly match longer strings with '*' or '**' pattern (#5001)
* fix(glob): Correctly match longer strings with '*' or '**' pattern

If string is longer than 16 chars we are using reflex library for
matching. When used pattern is '*' or '**' we are going to remove
trailing and leading star and have empty pattern. We should, in this,
edge case set manually star pattern.

Fixes #4948

Signed-off-by: mkaruza <mario@dragonflydb.io>

* Core matching test

---------

Signed-off-by: mkaruza <mario@dragonflydb.io>
2025-04-28 12:54:54 +03:00
Roman Gershman
23e6db900d
fix: regtest failures (#5013)
1. Fix a crash bug in RESETSTAT when number of shards is less than number of threads.
2. Tune regtests parameters for pipelining tests to pass.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-04-28 12:49:44 +03:00
Volodymyr Yavdoshenko
aa7e8abcb6
fix: RENAME in cluster mode crash (#4995)
fixed: https://github.com/dragonflydb/dragonfly/issues/4993
2025-04-28 08:45:05 +00:00
Roman Gershman
0f415acb81
chore: StoredCmd to support both owned and external arguments (#5010)
Before: StoredCmd always copied the backing buffer of the commands.
this of course sub-optimal if the bucking buffer exists during the life-time
of StoredCmd. This is exactly the case in `Service::DispatchManyCommands`.

This PR:
1. Adds support for both owned and non-owned arguments.
2. Improves the interfaces around StoredCmd and removes some code duplication.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-04-28 10:55:14 +03:00
Roman Gershman
d7a7591a46
chore: clean ups around command squasher (#5011)
1. Eliminate replies reverse call - will allow to unite replies and cmd vectors into one.
2. Introduce SwitchTxCmd function to avoid code duplication.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-04-28 10:44:42 +03:00
Roman Gershman
d37318711c
chore: fix some clang-tidy warnings (#5009) 2025-04-28 09:57:19 +03:00
Volodymyr Yavdoshenko
59a347df99
fix: bitops_family crash fixed (#4989)
fixed: https://github.com/dragonflydb/dragonfly/issues/4988
2025-04-28 09:41:46 +03:00
Roman Gershman
6d30baa20b
chore: Pipelining fixes (#4994)
Fixes #4998.
1. Reduces agressive yielding when reading multiple requests since it humpers pipeline efficiency.
   Now we yield consistently based on cpu time spend since the last resume point (via flag with sane defaults).
2. Increases socket read buffer size effectively allowing processing more requests in bulk.

`./dragonfly  --cluster_mode=emulated`
latencies (usec) for pipeline sizes 80-199:
p50: 1887, p75: 2367, p90: 2897, p99: 6266

`./dragonfly  --cluster_mode=emulated --experimental_cluster_shard_by_slot`
latencies (usec) for pipeline sizes 80-199:
p50: 813, p75: 976, p90: 1216, p99: 3528

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-04-27 20:48:02 +03:00
Roman Gershman
ff7d9b79c6
fix: potential crash with multi-sharded pfmerge (#5008)
Fixes #5004

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-04-27 14:12:55 +03:00
Roman Gershman
d5c375235f
feat: allow sharding by cluster slot id (#5006)
This is relevant only for cluster-enabled configurations.
Also, inline the cluster config getter functions, as they are on critical path for 100% of requests.

Finally, skip a test that triggers a check-fail bug filed in #5004

Fixes #5005

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-04-26 22:45:17 +03:00
Kostas Kyrimis
71dd189ebd
chore: split unit test steps in ci workflow (#5002)
Signed-off-by: kostas <kostas@dragonflydb.io>
2025-04-25 20:14:04 +03:00
Phuc-Thanh Nguyen
473e002c84
fix: potential vulnerable cloned functions (#4996)
Fix potential vulnerable cloned functions: Security: update Lua struct package for security.
2025-04-25 19:15:42 +03:00