Commit graph

3019 commits

Author SHA1 Message Date
Shahar Mike
ec6ac23ebc
chore: fix build on FreeBSD (#4412)
* chore: build on FreeBSD

* docs

* older helio
2025-01-07 15:05:50 +02:00
Kostas Kyrimis
e8f43d6858
fix: ExtentTree use of invalidated iterator (#4416)
Fixes accessing and invalidated iterator within ExtendTree
2025-01-07 14:38:32 +02:00
Borys
aaf33d5a1f
test: add test migration in cache mode (#4413) 2025-01-07 13:15:59 +02:00
Roman Gershman
a520ce0195
fix: invalid iterator case in Qlist::Erase (#4414)
The problem was with reverse iterator that was not set properly
when the last node is deleted.

Also, move PushSentinel code into Push.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-01-07 12:56:23 +02:00
Roman Gershman
a4d243b96f
chore: Introduce small buffer in redis parser (#4076)
* chore: Introduce small buffer in redis parser

This is needed in order to eliminate cases where we return INPUT_PENDING but do not consume the whole string by rejecting just several bytes.
This should simplify buffer management for the caller, so that if they pass a string that
did not result in complete parsed request, at least the whole string is consumed and can be discarded.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>

* chore: comments fixes

Signed-off-by: Roman Gershman <roman@dragonflydb.io>

---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-01-07 11:46:14 +02:00
Shahar Mike
6d03afaa76
chore: Increase migration wait time in test_cluster_migration_huge_container (#4409) 2025-01-07 11:34:43 +02:00
Borys
ed1436bb29
test: move ReplicaofRejectOnLoad test from pytest into unit tests (#4410) 2025-01-07 10:20:36 +02:00
Roman Gershman
21fcf5808e
feat: allow querying of json objects stored as strings (#4399)
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-01-06 21:20:52 +02:00
adiholden
1c0f22f5fa
chore(pytest): add check for rss grow in replicaiton big values (#4406)
Signed-off-by: adi_holden <adi@dragonflydb.io>
2025-01-06 08:39:32 +02:00
Shahar Mike
7860a169d9
feat: Yield inside huge values migration serialization (#4197)
* feat: Yield inside huge values migration serialization

With #4144 we break huge values slot migration into multiple commands.
This PR now adds yield between those commands.
It also adds a test that checks that modifying huge values while doing a
migration works well, and that RSS doesn't grow too much.

Fixes #4100
2025-01-05 14:28:45 +00:00
Roman Gershman
ff4add0c9e
chore: deprecate unneeded runtime flags (#4405)
Deprecate multi_exec_mode and track_exec_frequencies.
Remove duplicated call to EnterLameDuck (done also in Service::Shutdown()).

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-01-05 13:31:59 +00:00
Borys
f663f8e841
fix: provide resp3 option to CapturingReplyBuilder (#4400) 2025-01-05 14:16:02 +02:00
adiholden
0f5eb33e6a
chore(pytest): add timeout per test (#4404)
regression action add 5 minutes timeout per test

Signed-off-by: adi_holden <adi@dragonflydb.io>
2025-01-05 13:19:13 +02:00
adiholden
92c3749c8c
fix(tests): check cluster big snapshot in unit test (#4403)
fix tests: check cluster big snapshot in unit test

Signed-off-by: adi_holden <adi@dragonflydb.io>
2025-01-05 13:18:13 +02:00
Roman Gershman
4f09fe036c
feat: support deletions with meta protocol (#4398)
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-01-03 13:06:24 +00:00
Roman Gershman
6e9409c65c
fix: properly clear tiered stashes upon expirations (#4395)
Fixes #4387

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-01-02 13:49:13 +00:00
Borys
0832d23f13
fix: allow cluster node load snapshot bigger than maxmemory (#4394) 2025-01-02 13:50:21 +02:00
adiholden
a3ef239ac7
feat(server): refactor allow preempt on journal record (#4393)
* feat server: refactor allow preempt on journal record

Signed-off-by: adi_holden <adi@dragonflydb.io>
2025-01-02 12:16:21 +02:00
Roman Gershman
7a68528022
chore: minor refactorings around dense_set deletions (#4390)
chore: refactorings around deletions

Done as a preparation to introduce asynchronous deletions for sets/zsets/hmaps.
1. Restrict the interface around DbSlice::Del. Now it requires for the iterator to be valid and the checks should
be explicit before the call. Most callers already provides a valid iterator.

2. Some minor refactoring in compact_object_test.
3. Expose DenseSet::ClearStep to allow iterative deletions of elements.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2025-01-02 11:35:55 +02:00
adiholden
3b082e42b8
fix(replication): do not log to journal on callback fail (#4392)
fix replication: do not log to journal on callback fail

Signed-off-by: adi_holden <adi@dragonflydb.io>
2025-01-02 09:39:31 +02:00
guyzilla
413ec0a1cf
feat:Adding support for ZMPOP command (#4385)
Signed-off-by: Guy Flysher <guyzila@gmail.com>
2025-01-01 11:34:53 +02:00
Borys
c5b3584dfd
refactor: slot_set don't use stack memory anymore (#4386) 2025-01-01 10:51:38 +02:00
adiholden
810af83074
fix(server): debug populate consume less memory (#4384)
* fix server: debug populate consume less memory

Signed-off-by: adi_holden <adi@dragonflydb.io>
2025-01-01 10:49:17 +02:00
Roman Gershman
2abe2c0ac2
fix: ExternalAllocator::Free with large sizes (#4388)
ExternalAllocator allocates large sizes directly from the extent tree bypassing segment data structure.
Unfortunately, we forgot to align Free() the same way. This PR:

1. Make sure that we add back the allocated range to the extent tree in Free.
2. Rewrite and simplify ExtentTree::Add implementation.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-12-31 18:41:14 +02:00
Stepan Bagritsevich
5e7acbb396
fix(snapshot_test): Fix test_big_value_serialization_memory_limit after adding streams support (#4383)
* fix(snapshot_test): Fix test_big_value_serialization_memory_limit after adding streams support

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* feat: Temporary run test 30 times

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* fix: Remove repeat

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* Revert "fix: Remove repeat"

This reverts commit b7f1ed90b2.

* Revert "feat: Temporary run test 30 times"

This reverts commit 78bfb0fc26.

* Revert "fix(snapshot_test): Fix test_big_value_serialization_memory_limit after adding streams support"

This reverts commit f33036db2a.

* refactor: address comments

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

---------

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>
2024-12-30 17:38:57 +04:00
Borys
b7532538fe
test: fix test_network_disconnect_during_migration (#4378) 2024-12-30 14:02:12 +02:00
adiholden
22994cf3b7
chore(server): cleanup replication shard sync execution (#4374)
chore server: cleanup replication shard sync execution

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-12-30 11:52:58 +02:00
Borys
c4be62edc8
fix: return value for DflyMigrateAck (#4379) 2024-12-29 09:37:08 +02:00
Borys
5b9c7e415a
fix: stream memory counting during snapshot loading (#4346)
* fix: stream memory counting during snapshot loading
2024-12-27 09:02:47 +02:00
Stepan Bagritsevich
9fbb301988
chore(snapshot): Small cleanup in Snapshot code (#4377)
* chore(snapshot): Small cleanup in Snapshot code

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* refactor: address comments

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

---------

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>
2024-12-26 16:02:20 +00:00
Stepan Bagritsevich
0065c27f27
feat(rdb_saver): Support big value serialization for stream (#4376)
fixes dragonflydb#4317

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>
2024-12-26 15:15:35 +00:00
Stepan Bagritsevich
fb8234ce43
chore(json_family): Remove redundant JSON value copying (#4375)
Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>
2024-12-26 19:00:10 +04:00
adiholden
14fda354cc
fix(pytest): fix failing test test_replication_timeout_on_full_sync (#4370)
fix: increase timeout
fix: remove skip

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-12-25 14:23:33 +02:00
Roman Gershman
966a1a46fd
feat: support memcache meta responses (#4366)
Fixes #4348 and #3071

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-12-25 07:46:50 +00:00
dependabot[bot]
c88c707341
chore(deps): bump the actions group across 1 directory with 2 updates (#4361)
Bumps the actions group with 2 updates in the / directory: [mozilla-actions/sccache-action](https://github.com/mozilla-actions/sccache-action) and [github/codeql-action](https://github.com/github/codeql-action).


Updates `mozilla-actions/sccache-action` from 0.0.6 to 0.0.7
- [Release notes](https://github.com/mozilla-actions/sccache-action/releases)
- [Commits](https://github.com/mozilla-actions/sccache-action/compare/v0.0.6...v0.0.7)

Updates `github/codeql-action` from 3.27.5 to 3.28.0
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](f09c1c0a94...48ab28a6f5)

---
updated-dependencies:
- dependency-name: mozilla-actions/sccache-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Roman Gershman <roman@dragonflydb.io>
2024-12-25 05:30:59 +02:00
adiholden
937f1bc058
chore(test): disable failing test untill fixed (#4367)
chore test: disable failing test untill fixed

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-12-25 05:27:00 +02:00
Roman Gershman
a27cce81b1
feat: add support for meta memcache commands (#4362)
This is a stripped down version of supporting the memcache meta requests.

a. Not all meta flags are supported, but TTL, flags, arithmetics are supported.
b. does not include reply support.
c. does not include new semantics that are not part of the older, ascii protocol.

The parser interface has not changed significantly, and the meta commands are emulated
using the old, high level commands like ADD,REPLACE, INCR etc.

See https://raw.githubusercontent.com/memcached/memcached/refs/heads/master/doc/protocol.txt for more details
regarding the meta commands spec.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-12-24 15:33:24 +02:00
Stepan Bagritsevich
6946820e56
fix(json_family): Remove std::endl during logging in ParseJsonPath method (#4363)
Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>
2024-12-24 09:04:20 +00:00
Borys
dc81594cd5
test: skip test_network_disconnect_during_migration (#4359) 2024-12-24 10:21:03 +02:00
Stepan Bagritsevich
aeeb625393
fix(search_family): Support multiple fields in SORTBY option in the FT.AGGREGATE command. SECOND PR (#4232)
fix(search_family): Support multiple fields in SORTBY option in the FT.AGGREGATE command

fixes dragonfly#3631

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>
2024-12-24 11:59:48 +04:00
Stepan Bagritsevich
3c7e31240f
fix(search_family): Support boolean and nullable types in indexes (#4314)
* fix(search_family): Support boolean and nullable types in indexes

fixes dragonflydb#4107, dragonflydb#4129

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* refactor: address comments

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

---------

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>
2024-12-24 10:52:39 +04:00
Roman Gershman
01f24da2b6
fix: configure jsonconst double parser to use std::from_chars (#4360)
The problem: apparently, jsoncons uses strtod by default when parsing doubles.
On some platforms (alpine/musl) this function uses lots of stack, which potentially can lead to stack corruption.
This PR configures jsoncons to use std::from_chars that is more efficient and less stack hungry.
The single include point to consume jsoncons/json.hpp should be "core/json_object.h"

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-12-24 04:02:50 +02:00
Roman Gershman
95cd9dfb4c
chore: update helio and improve our stack overflow resiliency (#4349)
1. Run CI/Regression tests with HELIO_STACK_CHECK=4096.
   This will crash if a fiber stack usage goes below this limit.
2. Increase shard queue stack size to 64KB
3. Increase fiber stack size to 40KB on Debug builds.
4. Updated helio has some changes around the TLS socket code.
   In addition we add a helper script to generate self-signed certificates helpful for local development work.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-12-23 08:13:45 +00:00
Roman Gershman
28848d0be2
fix: avoid on stack allocation of lz4 compression context (#4322)
Fixes #4245

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-12-23 09:54:20 +02:00
Stepan Bagritsevich
1fa9a47a86
refactor(search_family): Add Aggregator class (#4290)
* refactor(search_family): Add Aggregator class

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* fix(aggregator_test): Fix tests failing

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* refactor: address comments

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* refactor: Restore the previous comment

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* refactor: address comments 2

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* refactor: address comments 3

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* fix(aggregator): Simplify comparator for the case when one of the values is not present

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

---------

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>
2024-12-23 08:43:48 +04:00
Stepan Bagritsevich
8d66c25bc6
chore(rax_tree): Introduce raxFreeWithCallback call in RaxTreeMap destructor (#4255)
Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>
2024-12-23 08:42:47 +04:00
Stepan Bagritsevich
612d50df3b
refactor(rdb_saver): Add SnapshotDataConsumer to SliceSnapshot (#4287)
* refactor(rdb_saver): Add SnapshotDataConsumer to SliceSnapshot

fixes #4218

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* refactor: address comments

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

---------

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>
2024-12-23 08:42:13 +04:00
romange
d16209461c chore(helm-chart): update to v1.26.0 2024-12-22 12:53:18 +00:00
Borys
76000b9672
fix: test_network_disconnect_during_migration (#4345) 2024-12-22 13:02:53 +02:00
Stepan Bagritsevich
c5ef553ffc
fix(search_family): Fix logging in ParseFieldWithAtSign (#4343)
Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>
2024-12-19 14:11:13 +00:00