Commit graph

1985 commits

Author SHA1 Message Date
adiholden
5337711976
fix server: fix deadlock blmove does not conclude on error (#4561)
* fix deadlock: fix blmove does not conclude on error

---------

Signed-off-by: adi_holden <adi@dragonflydb.io>
2025-02-07 11:09:29 +02:00
Borys
33584b9e96
refactor: rename Context into ExecutionState (#4562) 2025-02-06 08:47:20 +02:00
Stepan Bagritsevich
4ba1142cb3
fix(stream_family): Fix journaling in the XADD and XTRIM commands (#4448)
* fix(stream_family): Fix journaling in the XADD and XTRIM commands

fixes dragonflydb#4202

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* fix(replication_test): Fix a typo

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* feat(replication_test): Add test_stream_approximate_trimming test

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* refactor: address comments

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* refactor: address comments 2

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* fix(stream_family): Fix stream id journaling in the XADD command

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

---------

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>
2025-02-05 14:43:26 +00:00
Kostas Kyrimis
a40b5063e2
fix: consume parser error on dfly load (#4556)
* fix dcheck crash on dfly load
* fix dfly load with more than required arguments

Signed-off-by: kostas <kostas@dragonflydb.io>
2025-02-05 10:11:04 +02:00
Roman Gershman
bff9cf6923
chore: move QList::Node definition into dragonfly codebase (#4547)
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-02-02 19:25:01 +00:00
Roman Gershman
8c937ebf37
chore: clean up of deprecated flags (#4545)
Also fix sentinel test by using a precise redis-server version.
Finally, add pytest warnings filter to reduce noise

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-02-02 20:03:23 +02:00
Roman Gershman
9d303f8abe
chore: refactoring around glob matching (#4540)
1. Make GlobMatcher non-copyable. It's something I will need in the next PR.
2. Move low-level benchmarking testing code into dfly_core_test.

No functional changes.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-01-31 09:54:37 +02:00
Stepan Bagritsevich
83569aca28
fix(search_family): Add options test for the FT.SEARCH command (#4478)
Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>
2025-01-30 09:43:01 +00:00
Roman Gershman
fc94b2c265
chore: add more logs to the op_manager code (#4527)
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-01-29 19:08:41 +02:00
Shahar Mike
ffce1cb796
chore: Add Lua force atomicity flag (#4523)
* chore: Add Lua force atomicity flag

We accidentally instructed Sidekiq to add `disable-atomicity` to their
script, despite them needing to run atomically.

This hack-ish PR adds a `--lua_force_atomicity_shas` flag to allow
specifying which SHAs are forced to run in an atomic fashion, even if
they are marked as non-atomic.

Fixes #4522

* fix build on clang
2025-01-29 16:35:39 +02:00
Roman Gershman
bb9819464f
chore: introduce GlobMatcher (#4521)
Right now it's just a wrapper around stringmatchlen, so no functional changes are expected.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-01-29 10:58:17 +02:00
Roman Gershman
f2309f4e7b
chore: add reflex matcher to the benchmarks (#4520)
Add a test covering stringmatchlen.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-01-28 15:43:41 +00:00
Volodymyr Yavdoshenko
efb7861cee
fix: LMPOP implementation was added (#4504) 2025-01-28 08:52:38 +00:00
Roman Gershman
44506ad2a9
chore: remove unused code (#4516)
Move PrimeBumpPolicy definition to cc file.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-01-28 10:41:11 +02:00
Borys
e434f62336
fix: crash for delconsumer during stream reading (#4513)
fix: crash for delconsumer during reading stream
2025-01-27 18:32:53 +02:00
Roman Gershman
2d85f59a74
chore: add benchmarking of ScanOpts::Matches (#4511)
Also, improve robustness of StickyEviction that was failing for me.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-01-27 18:14:55 +02:00
adiholden
d88bdb8bc3
fix(test): fix rdb random fail on OOM (#4509)
fix rdb test random fail on OOM

Signed-off-by: adi_holden <adi@dragonflydb.io>
2025-01-27 11:59:04 +02:00
adiholden
b4ef0a06e2
fix cluster: migration crash fix (#4508)
* fix cluster: migration crash fix

Signed-off-by: adi_holden <adi@dragonflydb.io>
2025-01-27 10:39:11 +02:00
Shahar Mike
3fdd9877f4
fix: Do not bump elements during RDB load (#4507)
fix: Do not bump elements during RDB load #4507

The Issue

Before this PR, when loading an RDB, we modified fetched_items_ as part of the loading process. This has little effect, unless the next issued command calls FLUSHALL / FLUSHDB (could happen in DFLY LOAD, REPLICAOF or just calling FLUSHALL directly). In such a case, a CHECK() fails.

The Fix

While load is not run as a command (in a transaction), it still uses APIs that assume that they are called in the context of a command. As such, it indirectly used DbSlice::FindInternal(), which bumps elements when called.

This PR adds another sub-mode to DbSlice, named load_in_progress_. When true, we treat DbSlice as if it is not in cache mode, ignoring cache_mode_.

BTW this PR also renames caching_mode_ to cache_mode_ as we generally use the term cache mode and not caching mode, including in the --cache_mode flag.

Fixes #4497
2025-01-26 23:18:28 +02:00
Roman Gershman
bafb427a09
fix: rpm package setup (#4506)
Also, fix the deadlock problem on shutdown on Oracle Linux 5.15
Fixes #4505

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-01-26 12:40:33 +02:00
Roman Gershman
23af41ca07
fix: several bugs (#4500)
1. Fixes #4466 (we did not check validity of the scan cursor)
   We fix it by quietly returning "end of scan" result, i.e. no error are returned.
2. Add 34b1048274c8e50a0cc587a3ed9c383a82bb78c5 sha to the lua_undeclared_keys_shas list.
3. Pull helio that fixes bugs with WaitQueue and IdleTasks: 2ab4412b78
2025-01-24 13:53:09 +02:00
adiholden
d6adedb066
fix(cluster): crash in cluster migration (#4495)
fix crash in cluster migration

Signed-off-by: adi_holden <adi@dragonflydb.io>
2025-01-22 14:26:53 +02:00
Roman Gershman
451da72c41
chore: improve test_timeout robustness (#4494)
1. use assert_eventually
2. add more logs
3. unrelated - add a stats event to track timeout shutdowns.
2025-01-22 12:24:29 +00:00
Roman Gershman
4b8fa90a67
feat: add 'testing_time' limit option to dfly_bench (#4487)
* feat: add 'testing_time' limit option to dfly_bench
---------

Signed-off-by: Roman Gershman <romange@gmail.com>
Co-authored-by: Shahar Mike <chakaz@users.noreply.github.com>
2025-01-22 10:26:22 +02:00
Roman Gershman
20bc3188fe
feat: implement connection timeout functionality (#4407)
* feat: implement connection timeout functionality

`timeout` argument shuts down idle connections after the specified time.

Fixes #1677
---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-01-21 13:46:57 +02:00
adiholden
986ef7c0e2
fix (stream): XRANGE incorrectly interprets the end parameter (#4443)
Signed-off-by: adi_holden <adi@dragonflydb.io>
2025-01-20 21:07:49 +02:00
Stepan Bagritsevich
5d3e3146d3
feat(json_family): Add json_nesting_depth_limit flag (#4444)
* feat: Add json_nesting_depth_limit flag

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* refactor: address comments

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

---------

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>
2025-01-20 16:49:04 +00:00
Roman Gershman
99c4ab8abe
feat: add support for noreply option (#4485)
feat: add support for noreply and greeat options

noreply option is applicable for memcached.
great option ensure the tcp connection is indeed established before proceeding with loadtest.
2025-01-20 13:59:47 +00:00
Borys
010166525e
feat: allow finish and start migration in the same config (#4486) 2025-01-20 13:52:38 +00:00
Mykhailo Faraponov
91435bc6af
Fix typo in outgoing_slot_migration.cc (#4484)
Signed-off-by: Mykhailo Faraponov <11322032+moredure@users.noreply.github.com>
2025-01-20 12:38:02 +00:00
Roman Gershman
7d0530547b
chore: remove pubsub semantics from Connection::WeakRef (#4483)
Make it generic and move pubsub related logic into channel_store.
2025-01-20 12:32:17 +00:00
Roman Gershman
4acafa3d47
chore: OpManagerTest fixes (#4480) 2025-01-20 11:38:26 +00:00
Roman Gershman
d2209d9eea
chore: make per-thread QueueBackpressure objects global (#4482)
Before this PR, QueueBackpressure objects are in fact referenced from other threads in an awkward way via
Connection::WeakRef::EnsureMemoryBudget().

This PR removes the complexities of accessing these objects from foreigh threads.
2025-01-20 12:53:36 +02:00
Kostas Kyrimis
b017cdd1a0
fix: hiredis requires df to report version >7.4 (#4474)
* bump up redis version in info command
* add compatibility test
* bump up py dependencies
* fix warnings and deprecated functions

---------

Signed-off-by: kostas <kostas@dragonflydb.io>
2025-01-20 10:38:37 +02:00
Shahar Mike
6f3c6e3d57
chore: Fix all clang build warnings (#4475)
* chore: Fix all clang build warnings

Also add `-Werror` to clang build in CI.

Fixes #4449

* all build targets

* fix search test
2025-01-20 10:24:07 +02:00
Kostas Kyrimis
c759eb8ce6
chore: replace deprecated HELIO_STACK_CHECK (#4465)
* remove HELIO_STACK_CHECK
* use FLAGS_fiber_safety_margin

---------

Signed-off-by: kostas <kostas@dragonflydb.io>
2025-01-20 10:17:24 +02:00
Roman Gershman
ac8e1c80f0
chore: dfly_bench now accepts __data__ and __score__ template names (#4477)
* chore: dfly_bench now accepts __data__ and __score__ template names

---------

Signed-off-by: Roman Gershman <romange@gmail.com>
Co-authored-by: Shahar Mike <chakaz@users.noreply.github.com>
2025-01-20 07:59:03 +00:00
Stepan Bagritsevich
99f52642c9
fix(sanitizers): Set stack size to 64 KB during tests for Sanitizers (#4454)
* fix(rdb_test): Fix sanitizers for RdbTest

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* Revert "fix(rdb_test): Fix sanitizers for RdbTest"

This reverts commit 39ae4cf9958c517e70c666caee5a7fdd6beeba0d.

* feat: Add default fiber stack size

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* fix: Increase default stack size

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* fix: Specify default stack size for sanitizers

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* fix: Add SANITIZERS to dfly_test_lib

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

---------

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>
2025-01-16 10:00:42 +01:00
Borys
0e116b1535
fix: slot calculation during transaction squashing (#4460) 2025-01-15 11:43:52 +00:00
Shahar Mike
5ba608b58d
chore: Add stats print for slot migrations (#4456)
* chore: Add stats print for slot migrations

Fixes #4415
2025-01-15 11:06:09 +00:00
Stepan Bagritsevich
f6441df57a
chore: Remove template Callbacks (#4431)
Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>
2025-01-13 15:33:17 +02:00
Roman Gershman
29cde99ca5
chore: pull latest helio (#4446)
This also pulls the latest abseil library 20240722.0
2025-01-13 15:23:08 +02:00
Roman Gershman
e39e68276e
chore: dragonfly connection refactorings (#4434)
1. Move socket read code into a dedicated function.
   Remove std:: prefix in the code.
2. Add an optional iouring bufring registration. Currently not being used and is disabled by default.
2025-01-09 13:55:00 +00:00
Roman Gershman
f291ae27cb
fix: show managed stuff on privileged port (#4433)
* fix: show managed stuff on privileged port

* chore: comments
2025-01-09 13:56:14 +02:00
Roman Gershman
c77e7cc09f
fix: improve error propagation with RESTORE commands (#4428)
* fix: improve error propagation with RESTORE commands

Also, provide better logs if AddOrNew function fails adding a new entry
2025-01-09 10:22:47 +00:00
Roman Gershman
f3426bd559
chore: update jsoncons version to 0.178 (#4368)
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-01-08 20:05:50 +02:00
Borys
933c9f0b1c
refactor: remove transaction lib on cluster code dependency (#4417) 2025-01-08 09:38:13 +00:00
Kostas Kyrimis
cb752d90cd
chore: split zset and geo family tests (#4425)
* split zset and geo family tests

---------

Signed-off-by: kostas <kostas@dragonflydb.io>
2025-01-08 09:34:31 +00:00
Roman Gershman
0a4008385d
fix: switch to SHUTTING_DOWN state unconditionally (#4408)
* fix: switch to SHUTTING_DOWN state unconditionally

During the shutdown sequence always switch to SHUTTING_DOWN.
Make sure that the rest of the code does not break if it can not switch to the desired
global state + some clean ups around state transitions.

Finally, reduce the amount of data in test_replicaof_reject_on_load
---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-01-08 11:28:36 +02:00
Kostas Kyrimis
739bee5c83
chore: split geo and zset families (#4421)
* split geo family from zset family

---------

Signed-off-by: kostas <kostas@dragonflydb.io>
2025-01-08 10:46:25 +02:00