* chore: Add Lua force atomicity flag
We accidentally instructed Sidekiq to add `disable-atomicity` to their
script, despite them needing to run atomically.
This hack-ish PR adds a `--lua_force_atomicity_shas` flag to allow
specifying which SHAs are forced to run in an atomic fashion, even if
they are marked as non-atomic.
Fixes#4522
* fix build on clang
Deprecate multi_exec_mode and track_exec_frequencies.
Remove duplicated call to EnterLameDuck (done also in Service::Shutdown()).
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Background
We tried to be compatible with Valkey in their support of Lua flags, but we generally failed:
We are not really compatible with Valkey because our flags are different (we reject unknown flags, and Valkey flags are unknown to us)
The #!lua syntax doesn't work with Lua (# is not a comment), so scripts written for older versions of Redis can't be used with Dragonfly (i.e. they can't add Dragonfly flags and remain compatible with older Redis versions)
Changes
Instead of the previous syntax:
#!lua flags=allow-undeclared-keys,disable-atomicity
We now use this syntax:
--!df flags=allow-undeclared-keys,disable-atomicity
It is not backwards compatible (with older versions of Dragonfly), but it should be very easy to adapt to, and doesn't suffer from the disadvantages above.
Related to #3512
* feat: Allow pre-declaring Lua SHAs to run with undeclared keys
By using `--lua_undeclared_keys_shas=SHA,SHA,SHA` users can now specify
which scripts should run globally (undeclared keys) without explicit
support from the scripts themselves.
Fixes#2442
* feat(namespaces): Initial support for multi-tenant #3050
This PR introduces a way to create multiple, separate and isolated
namespaces in Dragonfly. Each user can be associated with a single
namespace, and will not be able to interact with other namespaces.
This is still experimental, and lacks some important features, such as:
* Replication and RDB saving completely ignores non-default namespaces
* Defrag and statistics either use the default namespace or all
namespaces without separation
To associate a user with a namespace, use the `ACL` command with the
`TENANT:<namespace>` flag:
```
ACL SETUSER user TENANT:namespace1 ON >user_pass +@all ~*
```
For more examples and up to date info check
`tests/dragonfly/acl_family_test.py` - specifically the
`test_namespaces` function.
This commit generalizes the machanism of running transaction callbacks during scheduling, removing the need for specialized ScheduleUniqueShard/RunQuickie. Instead, transactions can be run now during ScheduleInShard - called "immediate" runs - if the transaction is concluding and either only a single shard is active or the operation can be safely repeated if scheduling failed (idempotent commands, like MGET).
Updates transaction stats to mirror the new changes more closely.
---------
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
chore: UpdateHelio dependency
Add support for asan/ubsan checkers in our dev environment.
Remove more clang warnings.
Once we fix all the problems we will enable them in our CI as well.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Previously, transactions would run out of order only when all shards determined that the keys locks were free. With this change, each shard might decide to run out of order independently if the locks are free. COORD_OOO is now deprecated and the OUT_OF_ORDER per-shard flag should is used to indicate it
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
chore: simplify transaction multi-locking
Also, add the ananlysis routine that determines whether the schewduled transaction is contended with other transaction in a
shard thread.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. How many transactions we processed by type
2. How many transactions we processed by width (number of unique shards).
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
the bug: when returning error on a command in multi tx we do not cleanup
the multi cntx
the fix: cleanup the cntx when error is returned
Signed-off-by: adi_holden <adi@dragonflydb.io>
* feat(server): support multi eval in lock ahead mode
1. remove validation to allow multi eval only in global script mode
2. send error if there is a mode conflict when running eval inside multi
3. reset uniqe_keys_ when transaction finishes
* test(server): test transaction locked keys
1. add test utility class that will add suspension to transaction
execution.
2. add test for locked keys in transaction
Signed-off-by: adi_holden <adi@dragonflydb.io>
* bug(server): multi atomicity fix
The bug: when multi transaction run OOO we removed it from trasaction
queue, causing non atomic execution.
The fix: When we run multi transaction unless it is the head in txq we
remove it inside unlock multi from txq.
Signed-off-by: adi_holden <adi@dragonflydb.io>
The problem was with the logic inside Exec(...) that unconditinally
labeled multi transactions as global even if they did not contain any
EVAL calls.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* The bug - if all commands inside multi trasaction are eval commands
and global scripts mode is on, we did ignored the trasaction
and run each eval separately.
*Fix - run all evals under multi inside the global lock
* Change multi eval run only if scripts are in global mode and multi
mode is not non atomic
* Fix test flags setup
* Skip test ContendedList as it fails
* change default exec mode for txs to lock ahead
Enables execution of global lua scripts inside multi/exec transactions if the defualt script config enables global execution for scripts. This change is only a fix and does not provide any safeguards against other execution scenarios (namely enabling globality with script flags). In the future, the proper execution mode should be determined more carefully by inspecting the scripts to be executed
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
Co-authored-by: Kostas Kyrimis <kostaskyrim@gmail.com>
feat: ignore MULTI/EXEC if the transaction consists of EVAL commands.
Together with `default_lua_config` solves #781.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
feat: increase the flexibility of how to assign threads in DF.
Specifically, introduce `conn_io_threads` and `conn_io_thread_start` flags that choose
which threads can handle I/O. In addition, introduce `num_shards` flag that may override
how many database shards exist in dragonfly process.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
