Commit graph

1898 commits

Author SHA1 Message Date
Vladislav
1b51e82e55
chore(transaction): Add debug stats for fail printing (#2600)
* chore(transaction): Add debug stats for per shard data

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-18 15:36:14 +03:00
Vladislav
4d4fed6fec
chore(transaction): Untie scheduling from multi status (#2590)
* chore(transaction): Untie scheduling from multi status

Idea: We decide whether we have to schedule not based on our multi status (atomic multi), but solely based on the fact if COORD_SCHED is set

Goal: Being able to use ScheduleSingleHop()/Schedule() for multi transactions, and thus later allow single hop multi transactions
---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-18 12:25:57 +03:00
Roman Gershman
d802a2181a
chore: consolidate json code under core/json (#2609)
Fix linker dependencies - now dfly_core depends on jsonpath.
jsonpath does not depend on CompactObject anymore.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-18 09:22:34 +00:00
Roman Gershman
1ba59e9179
chore: remove ImportRObj routine and reduce reliance on object.c (#2607)
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-18 10:07:00 +02:00
Roman Gershman
6aafe3dc5e
chore: cleanup of redis implementation of sets/hashmaps (#2605)
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-17 18:35:28 +02:00
Roman Gershman
3ec56452f9
chore: refactor DfsItem to allow mutability semantics (#2602)
No functional changes are made.
This change is needed to allow traversal functionality that will mutate json object during the traversal.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-16 10:13:20 +02:00
Roman Gershman
888a437bf5
wire json::Path into json_family (#2587)
* feat: wire json::Path into the server

The feature is disabled under a flag and it covers read operations for now.
---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-15 17:05:53 +00:00
Roman Gershman
1e06c63727
feat: Add dbfilename to config registry (#2598)
Also make "dir" read-only parameter.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-15 18:00:36 +02:00
Roman Gershman
f847a3f4cd
feat: support descent path operator (#2594)
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-15 15:03:07 +02:00
Vladislav
bbbcddfdd6
chore(transaction): Copy poll flags (#2596)
* chore(transaction): Copy poll flags

Copying poll flags prevents concurrent data access to PerShardData::local_mask when dispatching poll tasks
---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-15 11:55:02 +03:00
Shahar Mike
28800df071
fix(test): Use less memory for STRING and HASH memory tests (#2593)
While at it, also register the opt_only mark
2024-02-15 10:08:45 +02:00
Roman Gershman
ded3341b3d
chore: add jsonpath evaluate (#2586)
* chore: add jsonpath evaluate

---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-14 16:51:42 +02:00
Shahar Mike
f921050298
fix(test): Unflake GenericFamilyTest.Time*Keys (#2592) 2024-02-14 15:24:42 +02:00
adiholden
32e8d49123
feat(tiering): add background offload step (#2504)
* feat(tiering): add background offload step

Signed-off-by: adi_holden <adi@dragonflydb.io
2024-02-14 14:28:41 +02:00
Shahar Mike
b18fe8c0a8
test(cluster): Fix and uncomment counter sanity checks (#2591) 2024-02-14 14:06:34 +02:00
Roman Gershman
ca23d72b08
feat: add descent operator to jsonpath parser (#2588)
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-14 13:23:57 +02:00
Shahar Mike
165adc8f87
fix(test): Apply pytest filter to regression tests as well (#2589) 2024-02-14 12:27:29 +02:00
Vladislav
4e3be726c5
chore(pytest): Refactor snapshot test (#2583)
* chore(pytest): Refactor snapshot test

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-14 12:13:52 +03:00
adiholden
7b61b4a8fe
fix(tiering tests): introduce wait until tieting entries num EQ/GT (#2559)
Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-02-14 09:59:11 +02:00
Roman Gershman
4afe278487
feat: jsonpath supports index/wildcard expressions (#2578)
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-12 18:38:05 +00:00
Tarun Pothulapati
21e725774c
feat(release): Also bundle dragonfly-debug builds (#2424)
* feat(release): Also bundle dragonfly-debug builds
2024-02-12 18:25:23 +02:00
Roman Gershman
b3b9080901
chore: fix fiber types in the codebase (#2574)
Reduce reliance on core/fibers

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-12 16:29:28 +02:00
Roman Gershman
4000adf57f
fix: do not migrate during connection close (#2570)
* fix: do not migrate during connection close

Fixes #2569
Before the change we had a corner case where Dragonfly would call
OnPreMigrateThread but would not call CancelOnErrorCb because OnBreakCb has already been called
(it resets break_cb_engaged_)

On the other hand in OnPostMigrateThread we called RegisterOnErrorCb if breaker_cb_ which resulted in double registration.
This change simplifies the logic by removing break_cb_engaged_ flag since CancelOnErrorCb is safe to call if nothing is registered.
Moreover, we now skip Migrate flow if a socket is being closed.

---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-12 16:03:34 +02:00
Shahar Mike
6d11f86091
test(cluster-migration): Fix some bugs and add cluster migration fuzzy tests (#2572)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
Co-authored-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-12 13:47:34 +02:00
Roman Gershman
6cd2f05a22
chore: provide plumbing for jsonpath error propagation (#2567)
* chore: provide plumbing for jsonpath error propagation

Also update re/flex library to the latest version.
Finally, introduce very basic parser test.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>

* feat: construct Path object

Parse jsonpath and partially fill it with path segments.
---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-12 13:24:31 +02:00
Roman Gershman
9802f2c489
chore: use helio MPSCIntrusiveQueue queue (#2573) 2024-02-12 10:40:37 +00:00
adiholden
e66e670a6e
chore(server): replica not accumulate multi commands untill exec (#2557)
* chore(server): replica not accumulate multi commands untill exec

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-02-12 10:33:33 +02:00
Roman Gershman
24fcf8d883
chore: Recognize exit status in regression tests (#2571)
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-12 09:03:56 +02:00
Shahar Mike
8ead569b2f
test(memory): Unaccounted memory test + add DEBUG POPULATE TYPE <type> (#2561)
* test(memory): Test memory accounting for all types

* slightly faster

* WIP

* working

* Document

* Update test to use DEBUG POPULATE

* Nothing much

* Working

* fix

* yaml

* explicit capture

* fix ci?

* stub tx
2024-02-12 08:09:48 +02:00
Vladislav
963023f07c
chore(transaction): Simplify armed state (#2508)
* chore(transaction): Simplify armed state

Remove atomic is_armed variable and turn it into a regular local state flag. This is now possible because we have clearly defined phases with the phased barrier and baton barrier for blocking commands

---------

Signed-off-by: Vladislav <vlad@dragonflydb.io>
2024-02-11 12:06:36 +03:00
Kostas Kyrimis
795d00021d
fix: asan errors on unit tests (#2564)
* fix: asan errors on unit tests
2024-02-10 19:48:00 +02:00
Vladislav
815976c9dc
chore: Remove manual memory management from stats (#2550)
Simplift stats memory management

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-10 10:30:30 +03:00
Roman Gershman
908efff7bd
fix: make sure SCRIPT FLUSH concludes (#2565)
InterpreterManager::Reset creates now a new storage for interpreters,
 and waits for the old ones to be returned.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-09 18:55:14 +02:00
Vladislav
881edb501e
feat(pytest): Gen2 seeder, part 1 (#2556)
* feat(pytest): Gen2 seeder

Implement new seeder that uses lua scripts to improve performance

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-09 19:20:25 +03:00
Roman Gershman
06d88ddc88
chore: parse a simple jsonpath (#2566)
Also, add an ubuntu-20 reusable devcontainer.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-09 15:41:22 +02:00
adiholden
c2ee348d2c
fix(server): do not print warning on replica stop/ takeover (#2560)
* fix(server): do not print warning on replica stop/ takeover

---------

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-02-09 08:10:41 +02:00
Roman Gershman
4860c8eac2
chore: enable ContendExpire (#2563)
chore: enable ContendExpire unit test

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-08 15:30:54 +00:00
Shahar Mike
9912df09ae
fix(server): Init tx time for all multi/lua transactions (#2562)
* fix(server): Return correct `TIME` under unscheduled tx

Fixes #2555

* Init tx time in all multi / lua cases

* init ctor
2024-02-08 14:47:07 +02:00
Borys
72f651d527
feat(cluster): add slots blocking when we finalize migration (#2484)
* feat(cluster): add slots blocking when we finalize the migration
2024-02-07 15:19:35 +02:00
Vladislav
9537cbdb0b
fix(pytest): Disable flaky interpreter test monitoring (#2558) 2024-02-07 12:51:24 +03:00
romange
5b96e269a3 chore(helm-chart): update to v1.14.3 2024-02-07 06:19:10 +00:00
Roman Gershman
4b4ff4688b
feat: introduce a skeleton for jsonpath parser (#2521)
Not finished yet, mostly a boilerplate plus minimal bindings for the test.
Motivation: as long as we rely on jsoncons jsonpath implementation we won't be able
to change json implementation as well. jsoncons path is quite qood but it will only
work with jsoncons json library. By implementing this ourselves we will gain the freedom
to change json implementation in the future.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-06 19:21:40 +02:00
Vladislav
ed59a439d1
fix(tests): increase interpreter test load, lower notice limit, include 2553 (#2554)
* fix: increase load + lower notice limit

* chore: include 2553

* fix: lower metric EVEN more

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-06 16:57:32 +02:00
Vladislav
83a12b99c6
fix: fix interpreter acquisition with MULTI (#2549)
* fix: fix interpreter acquisition with MULTI

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-06 12:54:14 +02:00
Shahar Mike
bc9b214ae4
fix(server): Do not yield in journal if not allowed (#2540)
* fix(server): Do not yield in journal if not allowed

* Add pytest

* Compare keys

* check_all_replicas_finished
2024-02-06 12:35:00 +02:00
Roman Gershman
336d6ff181
Update helio (#2538)
chore: UpdateHelio dependency

Add support for asan/ubsan checkers in our dev environment.
Remove more clang warnings.

Once we fix all the problems we will enable them in our CI as well.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-06 11:57:26 +02:00
Kostas Kyrimis
4c0055ddf0
feat: move lint test chart to seperate action and trigger it on reg tests (#2536)
* move lint test chart to a separate action
* use it in regression tests
2024-02-06 10:27:10 +02:00
Kostas Kyrimis
7b4591cc7d
fix: wrongly printing timedout emoji on test failures (#2546)
* fix wrongly printing timedout emoji on non timedout test failures
* add pytest-timeout as dependency in tests/dragonfly/requirements
2024-02-06 10:25:12 +02:00
Michael Primeaux
951b0897e6
bug: DFLY_PASSWORD environment variable deprecation (#2539)
* If the image.tag version is equal to 14.0.0 or greater then specify the DFLY_requirepass environment variable. Otherwise use the DFLY_PASSWORD environment variable.

Signed-off-by: Michael Primeaux <michael.primeaux@mac.com>

* Corrected version logic and added golden test

Signed-off-by: Michael Primeaux <michael.primeaux@mac.com>

* Updated golden file for TLS + image.tag

Signed-off-by: Michael Primeaux <michael.primeaux@mac.com>

* fix CI for chart linting test

* rename new test specific to password

* use v1.13.0

* update golden chart

---------

Signed-off-by: Michael Primeaux <michael.primeaux@mac.com>
Signed-off-by: Michael Primeaux <mprimeaux@users.noreply.github.com>
Co-authored-by: Tarun Pothulapati <tarun@dragonflydb.io>
2024-02-06 13:42:05 +05:30
Borys
eaca66b02f
fix(cluster): fix #2532 tests redis client close errors (#2542)
* fix(cluster): fix #2532 tests redis client close errors

* refactor: address comments
2024-02-05 22:42:03 +02:00