Commit graph

1878 commits

Author SHA1 Message Date
Tarun Pothulapati
21e725774c
feat(release): Also bundle dragonfly-debug builds (#2424)
* feat(release): Also bundle dragonfly-debug builds
2024-02-12 18:25:23 +02:00
Roman Gershman
b3b9080901
chore: fix fiber types in the codebase (#2574)
Reduce reliance on core/fibers

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-12 16:29:28 +02:00
Roman Gershman
4000adf57f
fix: do not migrate during connection close (#2570)
* fix: do not migrate during connection close

Fixes #2569
Before the change we had a corner case where Dragonfly would call
OnPreMigrateThread but would not call CancelOnErrorCb because OnBreakCb has already been called
(it resets break_cb_engaged_)

On the other hand in OnPostMigrateThread we called RegisterOnErrorCb if breaker_cb_ which resulted in double registration.
This change simplifies the logic by removing break_cb_engaged_ flag since CancelOnErrorCb is safe to call if nothing is registered.
Moreover, we now skip Migrate flow if a socket is being closed.

---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-12 16:03:34 +02:00
Shahar Mike
6d11f86091
test(cluster-migration): Fix some bugs and add cluster migration fuzzy tests (#2572)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
Co-authored-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-12 13:47:34 +02:00
Roman Gershman
6cd2f05a22
chore: provide plumbing for jsonpath error propagation (#2567)
* chore: provide plumbing for jsonpath error propagation

Also update re/flex library to the latest version.
Finally, introduce very basic parser test.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>

* feat: construct Path object

Parse jsonpath and partially fill it with path segments.
---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-12 13:24:31 +02:00
Roman Gershman
9802f2c489
chore: use helio MPSCIntrusiveQueue queue (#2573) 2024-02-12 10:40:37 +00:00
adiholden
e66e670a6e
chore(server): replica not accumulate multi commands untill exec (#2557)
* chore(server): replica not accumulate multi commands untill exec

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-02-12 10:33:33 +02:00
Roman Gershman
24fcf8d883
chore: Recognize exit status in regression tests (#2571)
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-12 09:03:56 +02:00
Shahar Mike
8ead569b2f
test(memory): Unaccounted memory test + add DEBUG POPULATE TYPE <type> (#2561)
* test(memory): Test memory accounting for all types

* slightly faster

* WIP

* working

* Document

* Update test to use DEBUG POPULATE

* Nothing much

* Working

* fix

* yaml

* explicit capture

* fix ci?

* stub tx
2024-02-12 08:09:48 +02:00
Vladislav
963023f07c
chore(transaction): Simplify armed state (#2508)
* chore(transaction): Simplify armed state

Remove atomic is_armed variable and turn it into a regular local state flag. This is now possible because we have clearly defined phases with the phased barrier and baton barrier for blocking commands

---------

Signed-off-by: Vladislav <vlad@dragonflydb.io>
2024-02-11 12:06:36 +03:00
Kostas Kyrimis
795d00021d
fix: asan errors on unit tests (#2564)
* fix: asan errors on unit tests
2024-02-10 19:48:00 +02:00
Vladislav
815976c9dc
chore: Remove manual memory management from stats (#2550)
Simplift stats memory management

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-10 10:30:30 +03:00
Roman Gershman
908efff7bd
fix: make sure SCRIPT FLUSH concludes (#2565)
InterpreterManager::Reset creates now a new storage for interpreters,
 and waits for the old ones to be returned.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-09 18:55:14 +02:00
Vladislav
881edb501e
feat(pytest): Gen2 seeder, part 1 (#2556)
* feat(pytest): Gen2 seeder

Implement new seeder that uses lua scripts to improve performance

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-09 19:20:25 +03:00
Roman Gershman
06d88ddc88
chore: parse a simple jsonpath (#2566)
Also, add an ubuntu-20 reusable devcontainer.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-09 15:41:22 +02:00
adiholden
c2ee348d2c
fix(server): do not print warning on replica stop/ takeover (#2560)
* fix(server): do not print warning on replica stop/ takeover

---------

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-02-09 08:10:41 +02:00
Roman Gershman
4860c8eac2
chore: enable ContendExpire (#2563)
chore: enable ContendExpire unit test

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-08 15:30:54 +00:00
Shahar Mike
9912df09ae
fix(server): Init tx time for all multi/lua transactions (#2562)
* fix(server): Return correct `TIME` under unscheduled tx

Fixes #2555

* Init tx time in all multi / lua cases

* init ctor
2024-02-08 14:47:07 +02:00
Borys
72f651d527
feat(cluster): add slots blocking when we finalize migration (#2484)
* feat(cluster): add slots blocking when we finalize the migration
2024-02-07 15:19:35 +02:00
Vladislav
9537cbdb0b
fix(pytest): Disable flaky interpreter test monitoring (#2558) 2024-02-07 12:51:24 +03:00
romange
5b96e269a3 chore(helm-chart): update to v1.14.3 2024-02-07 06:19:10 +00:00
Roman Gershman
4b4ff4688b
feat: introduce a skeleton for jsonpath parser (#2521)
Not finished yet, mostly a boilerplate plus minimal bindings for the test.
Motivation: as long as we rely on jsoncons jsonpath implementation we won't be able
to change json implementation as well. jsoncons path is quite qood but it will only
work with jsoncons json library. By implementing this ourselves we will gain the freedom
to change json implementation in the future.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-06 19:21:40 +02:00
Vladislav
ed59a439d1
fix(tests): increase interpreter test load, lower notice limit, include 2553 (#2554)
* fix: increase load + lower notice limit

* chore: include 2553

* fix: lower metric EVEN more

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-06 16:57:32 +02:00
Vladislav
83a12b99c6
fix: fix interpreter acquisition with MULTI (#2549)
* fix: fix interpreter acquisition with MULTI

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-06 12:54:14 +02:00
Shahar Mike
bc9b214ae4
fix(server): Do not yield in journal if not allowed (#2540)
* fix(server): Do not yield in journal if not allowed

* Add pytest

* Compare keys

* check_all_replicas_finished
2024-02-06 12:35:00 +02:00
Roman Gershman
336d6ff181
Update helio (#2538)
chore: UpdateHelio dependency

Add support for asan/ubsan checkers in our dev environment.
Remove more clang warnings.

Once we fix all the problems we will enable them in our CI as well.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-06 11:57:26 +02:00
Kostas Kyrimis
4c0055ddf0
feat: move lint test chart to seperate action and trigger it on reg tests (#2536)
* move lint test chart to a separate action
* use it in regression tests
2024-02-06 10:27:10 +02:00
Kostas Kyrimis
7b4591cc7d
fix: wrongly printing timedout emoji on test failures (#2546)
* fix wrongly printing timedout emoji on non timedout test failures
* add pytest-timeout as dependency in tests/dragonfly/requirements
2024-02-06 10:25:12 +02:00
Michael Primeaux
951b0897e6
bug: DFLY_PASSWORD environment variable deprecation (#2539)
* If the image.tag version is equal to 14.0.0 or greater then specify the DFLY_requirepass environment variable. Otherwise use the DFLY_PASSWORD environment variable.

Signed-off-by: Michael Primeaux <michael.primeaux@mac.com>

* Corrected version logic and added golden test

Signed-off-by: Michael Primeaux <michael.primeaux@mac.com>

* Updated golden file for TLS + image.tag

Signed-off-by: Michael Primeaux <michael.primeaux@mac.com>

* fix CI for chart linting test

* rename new test specific to password

* use v1.13.0

* update golden chart

---------

Signed-off-by: Michael Primeaux <michael.primeaux@mac.com>
Signed-off-by: Michael Primeaux <mprimeaux@users.noreply.github.com>
Co-authored-by: Tarun Pothulapati <tarun@dragonflydb.io>
2024-02-06 13:42:05 +05:30
Borys
eaca66b02f
fix(cluster): fix #2532 tests redis client close errors (#2542)
* fix(cluster): fix #2532 tests redis client close errors

* refactor: address comments
2024-02-05 22:42:03 +02:00
Kostas Kyrimis
f707f6d821
fix: add emoji when regression tests timeout (#2543)
* add emoji when regression tests timeout
2024-02-05 16:33:37 +02:00
romange
97982eef2c chore(helm-chart): update to v1.14.2 2024-02-05 12:21:23 +00:00
Roman Gershman
5c0029978e
chore: remove redis sorted set implementation (#2522)
Also remove unused code.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-05 10:29:11 +02:00
Vladislav
e0f86697f9
fix: fix script replication (#2531)
* fix: fix script replication

Single key script replication was previously broken because the EXEC entry wasn't sent. Send it manually

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-04 20:28:44 +03:00
Andy Dunstall
562aa151a7
fix(zset): ignore cancelled op status in BZPopMinMax (#2527)
fix(zset): ignore cancelled op status
2024-02-03 09:35:05 +00:00
Vladislav
40d08a3c67
fix(transaction): Add special barrier for blocking tx (#2512)
Refactor blocking transaction code. Introduce BatonBarrier for managing atomic and exclusive wakeup notifications that don't conflict with neither expiration nor cancelling

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-01 23:37:36 +03:00
s-shiraki
7e875bdafe
feat(server): Implement CONFIG HELP command (#2510)
* feat(server): Implement CONFIG HELP command

* fix
2024-02-01 21:35:31 +02:00
Shahar Mike
b71d0f6383
feat(server): Add MEMORY TRACK ADDRESS command (#2513)
* feat(server): Add `MEMORY TRACK ADDRESS` command

Given a hex address, it replies with whether or not this address is
currently allocated by mimalloc's backing heap.

* init!

* Add HELP message
2024-02-01 21:31:08 +02:00
Vladislav
c987e64332
fix: handle null length strings in stored commands (#2518)
* fix: handle null length strings in stored commands

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-01 19:25:44 +03:00
Roman Gershman
b5b5093165
fix: fix BLOCKING/REVERSE_MAPPING flags for some commands (#2516)
* fix: BLOCKING/REVERSE_MAPPING flags for some commands

Also, simplify interfaces around REVERSE_MAPPING in the internal tx code.
---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-01 15:39:14 +00:00
Borys
5189dae118
feat(cluster): add migration finalization (#2507)
* feat(cluster): add migration finalization
2024-02-01 17:24:54 +02:00
Roman Gershman
adeac6bd27
Pr1 (#2517)
* fix: Remove a stale reference to blocking watch queue

1. Remove the duplicated FinalizeWatched function
2. Identify the case where we delete the watched queue while we may still have awakedened_keys pointing to it.
3. Add a test reproducing the issue of having in awakened_keys an untangled key.

Properly fixes #2514

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-01 14:19:08 +02:00
Roman Gershman
2b0310db32
fix: do not crash with inconsistent watch queue (#2515)
Output more info about the state of things.
Skip the non existent key and continue the execution.
Fixes #2514

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-01 09:48:42 +02:00
s-shiraki
73fe5a4eb2
fix(server): Implement SCRIPT FLUSH command (#2493)
* fix(server): Implement SCRIPT FLUSH command
2024-01-31 08:31:06 +02:00
Vladislav
90a9f05e36
chore(transaction): Use PhasedBarrier for easier synchronization (#2455)
chore(transaction): Use PhasedBarrier for easier synchronization

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-01-30 19:43:06 +03:00
adiholden
503891b1fa
fix(server): update post updater iterator in tiering (#2497)
* fix(server): update post updater iterator in tiering

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-01-30 11:46:00 +00:00
Shahar Mike
b2bdb0f683
feat(server): Memory tracker (#2501)
* feat(server): Memory tracker

* PR comments
2024-01-30 12:44:07 +02:00
Shahar Mike
ad90602bc2
test: Disable flaky test test_policy_based_eviction_propagation (#2505) 2024-01-30 10:28:00 +02:00
Shahar Mike
87fb07abe3
fix(server): Fix crash in RENAME (#2503)
Crash was a `CHECK()` failure when renaming to an existing key within
the same shard.

Specifically, the problem was that we ran post updater after deletion,
which is illegal as it could invalidate iterators.

Fixes #2502
2024-01-30 10:24:49 +02:00
Borys
1dee082f86
fix(cluster): fix incorrect version checking and resource double free (#2499)
* fix(cluster): fix incorrect version checking and resource double free
2024-01-29 14:14:21 +02:00