* chore: change Namespaces to be a global pointer
Before the namespaces object was defined globally.
However it has non-trivial d'tor that is being called after main exits.
It's quite dangerous to have global non-POD objects being defined globally.
For example, if we used LOG(INFO) inside the Clear function , that would crash dragonfly on exit.
Ths PR changes it to be a global pointer.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* feat(namespaces): Initial support for multi-tenant #3050
This PR introduces a way to create multiple, separate and isolated
namespaces in Dragonfly. Each user can be associated with a single
namespace, and will not be able to interact with other namespaces.
This is still experimental, and lacks some important features, such as:
* Replication and RDB saving completely ignores non-default namespaces
* Defrag and statistics either use the default namespace or all
namespaces without separation
To associate a user with a namespace, use the `ACL` command with the
`TENANT:<namespace>` flag:
```
ACL SETUSER user TENANT:namespace1 ON >user_pass +@all ~*
```
For more examples and up to date info check
`tests/dragonfly/acl_family_test.py` - specifically the
`test_namespaces` function.
* remove helper.h and helper.cc and move them to AclFamily class
* remove global tables from acl_commands_def and move them to AclFamily
Signed-off-by: kostas <kostas@dragonflydb.io>
* chore: get rid of lock keys
1. Introduce LockTag a type representing the part of the key that is used for locking.
2. Hash keys once in each transaction.
3. Expose swap_memory_bytes metric.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
The main change here is introduction of the strong type LockTag
that differentiates from a string_view key.
Also, some testing improvements to improve the footprint of the next PR.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* feat(server): Allow configuration of hashtag extraction
Before this PR: Hashtags, if enabled, meant the text between `{` and `}`
in the key (if exists and non-empty).
After this PR:
* Hashtags can _still_ be enabled / disabled
* Hashtag open / close char can be specified (and can be the same), like `:` popular with BullMQ
* Hashtag can include `N` closing tags to skip, like `{a}b}c}d` with `2` will return `a}b}c`.
This will allow some existing systems to migrate without code changes in
client code.
1. How many transactions we processed by type
2. How many transactions we processed by width (number of unique shards).
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
fixes https://github.com/dragonflydb/dragonfly/issues/2139
This is part two that implements the logic which notifies tracking clients by sending invalidation messages:
- The client tracking state is set by CLIENT TRACKING subcommand as well
as upon client disconnection.
- Track the keys of a readonly command by maintaining mapping that maps
keys to the sets of tracking clients.
- Send invalidation messages to clients when their tracked keys are
updated.
- Make PerformDeletion a member function of DbSlice, and send
invalidation message within the function.
- Mock the function for sending invalidation message to avoid test
crash due to lack of real listener in the testing framework.
- Add functional (some) tests for client tracking based on the mocked interfaces.
---------
Signed-off-by: Yue Li <61070669+theyueli@users.noreply.github.com>
* Update Http auth with username default instead of user
* skip auth for /metrics page
* add/improve tests
* fix a bug with admin port requiring auth on http even if nopass was set
* update helio ref
* update listener class to contain its respective Role
* fix http init to only include admin and main listener
* feat(server): support multi eval in lock ahead mode
1. remove validation to allow multi eval only in global script mode
2. send error if there is a mode conflict when running eval inside multi
3. reset uniqe_keys_ when transaction finishes
* test(server): test transaction locked keys
1. add test utility class that will add suspension to transaction
execution.
2. add test for locked keys in transaction
Signed-off-by: adi_holden <adi@dragonflydb.io>
* feat(server): Add `--lock_on_hashtags` mode.
This new mode effectively locks hashtags (i.e. strings within {curly
braces}) instead of the full keys being used.
This can allow scripts to access undeclared keys if they all use a
common hashtag, like for the case of BullMQ.
To make sure this mode is tested, I added a way to specify flags via env
variables, and modified `ci.yml` to run all tests using this mode as well.
While at it, I also added `--cluster_mode=emulated` mode to CI.
Now `SUBSCRIBE` will respond synchronously. The change is here so we:
1. Maintain the order in pipelined requests
2. Don't have a "race condition": subscribe needs to update channel store pointers on all threads. While it awaits for all threads to complete the callback, some of them might have done it earlier, so they can already start sending messages before the initial ack is sent
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
This change removes most mentions of boost::fibers or util::fibers_ext.
Instead it introduces "core/fibers.h" file that incorporates most of
the primitives under dfly namespace. This is done in preparation to
switching from Boost.Fibers to helio native fibers.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
The issue happens when SendMsgVecAsync is called with PubMessage that has
string_view objects referencing objects in stack. We replace string_view
with either string or shared_ptr<string>
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. Support tiered deletion.
2. Add notion of tiered entity in "DEBUG OBJECT" output.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Partially implements #6.
Before, each shard lazily updated its clock used for the expiry evaluation.
Now, the clock value is set during the transaction scheduling phase and is assigned
to each transaction. From now on DbSlice methods use this value when checking whether
the entry is expired via passed DbContext argument.
Also, implemented transactionally consistent TIME command and
verify that time is the same during the transaction. See
https://ably.com/blog/redis-keys-do-not-expire-atomically for motivation.
Still have not implemented any lamport style updates for background processes
(not sure if it's the right way to proceed).
1. Fix SendStringArr logic. Consolidate both string_view and string variants to using the same code.
2. Tighten the test framework so that it will test that the parser consumed the whole response.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* Fix `SubscribeMap`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* Fix `subscriber_arr`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* Fix `CopySubscribers`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* Fix `SubscriberMessagesLen`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* Fix remaining `subscribe` variants
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* Add Ryan Russell to Contributors
Signed-off-by: Ryan Russell <git@ryanrussell.org>
1. Make EngineShardSet to be a process singleton instead of a variable passed everywhere.
2. Implement a heuristic of updating free memory per shard.
3. Make sure that SET command returns OOM when a database reaches its limits.
1. Added support of loading of compressed strings.
2. Verified we load expiry info.
3. Extended supported expiry period to 4 years (previously I set 1 year).
