* feat(namespaces): Initial support for multi-tenant #3050
This PR introduces a way to create multiple, separate and isolated
namespaces in Dragonfly. Each user can be associated with a single
namespace, and will not be able to interact with other namespaces.
This is still experimental, and lacks some important features, such as:
* Replication and RDB saving completely ignores non-default namespaces
* Defrag and statistics either use the default namespace or all
namespaces without separation
To associate a user with a namespace, use the `ACL` command with the
`TENANT:<namespace>` flag:
```
ACL SETUSER user TENANT:namespace1 ON >user_pass +@all ~*
```
For more examples and up to date info check
`tests/dragonfly/acl_family_test.py` - specifically the
`test_namespaces` function.
Before this change we first printed the value, which could be long
resulting in the TTL not appearing in the log due to max line length
If the TTL is invalid, we should print it first to see what value it has
* feat(json): Deserialize ReJSON format
This PR adds support for Redis-based JSON RDB format deserialization.
Since Redis uses ReJSON as a module, serialization is slightly different
from other types, but overall it's not a big change once we know where
all bits should be.
While this change knows how to _read_ Redis-based JSON keys, it does not
_save_ them in Redis format. That will be in a different PR.
This PR also ignores unknown (non-keys) module data instead of failing the load.
Fixes#2718
* Cleanup
* Add tests
* Skip unsupported modules
* Small refactor
* chore: improve compatibility of EXPIRE functions with Redis
Also, provide a module name if stumbled upon module data that can not be loaded
by dragonfly.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Fix linker dependencies - now dfly_core depends on jsonpath.
jsonpath does not depend on CompactObject anymore.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
chore: UpdateHelio dependency
Add support for asan/ubsan checkers in our dev environment.
Remove more clang warnings.
Once we fix all the problems we will enable them in our CI as well.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore: switch json object to pmr allocator
1. Move json object creation code to the shard-thread inside rdb_load
2. Now json_family never references "json" type, always dfly::JsonType
3. Switch JsonType to pmr::json.
4. Make sure we pass the correct memory_resource when creating json object from string.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore: fix our release pipeline
Also remove alpine prod.wip file that has not been used and unlikely will be for prod.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. Cherry-pick changes from Redis 7 that encode integer scores more efficiently
2. Introduces optimization that first checks if the new element should be the last
for listpack sorted sets.
3. Consolidates listpack related constants and tightens usage for listpack.
4. Introduce MEMORY USAGE command.
5. Introduce a small delay before decommitting memory pages back to OS.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* feat(replication): Use a ring buffer with messages to serve replication.
* Fix libraries dep graph
* Address PR feedback
* nits
* add a comment
* Lower the default log length
* fix(server): Dont apply memory limit when loading/replicating
When loading a snapshot created by the same server configuration (memory and
number of shards) we will create a different dash table segment directory tree, because the
tree shape is related to the order of entries insertion. Therefore when loading data from
snapshot or from replication the conservative memory checks might fail as the new tree might
have more segments. Because we dont want to fail loading a snapshot from the same server
configuration we disable this checks on loading and replication.
Signed-off-by: adi_holden <adi@dragonflydb.io>
While loading rdb snapshot, if oom is reached a bad alloc exception is thrown. Now we
catch it and write warning to log and fali loader.
Signed-off-by: adi_holden <adi@dragonflydb.io>
* chore: several improvements around sorted map
1. Pass memory_resource to sorted_map.
2. Get rid of GetDict leaky accessor in SortedMap and introduce a proper
Scan method.
3. Introduce correct BPTree type inside SortedMap::DFImpl.
4. Added a test for bptree_test that covers sds comparison
(apparently, sdscmp can return values outside of [-1, 1] range).
Fixed bptree code to support a proper spec for three-way comparison.
5. Expose pointers to internal objects allocated by score_map so we could insert them
into bptree.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore: fix comments
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* Bump up RDB_VERSION to 11
* Update RDB_JSON value to 30
* Fix HT being serialized to the wrong type
* Serialize HT as LIST_PACK
* Add support for deserializing SET_LISTPACK
Consolidate skiplist based zset functionality into a dedicated class
called SortedMap. The code is adapted from t_zset.c
The old code in t_zset.c is deleted.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
This is pure refactoring PR that does not change any functionality besides
prohibiting using AsRobj/SyncRobj functions for compact objects of type
OBJ_ZSET. This is needed in case we decide in the future to implement our own
zset type.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
fix: fix crash when inserting to listpack empty value.
We can not pass null pointers to listpack.
Fixes#1305 and probably fixes#1290
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* feat: Use journal LSNs for absolute replication offsets
* 1 - Address small CR comments
2 - Simplify the offset accounting so that we send the correct offset
in `SliceSnapshot::Stop` instead of counting in RdbLoader. This
allows us to revert the changes to slice journaling of EXEC
commands, for example.
* Store int with absl::little_endian
* Document the offset management
1. Pull newest helio that fixes 32-bit overflow in IoBuf.
2. Allow responsiveness when loading huge sets and maps.
3. Break array of blobs into segments of limited size so that
we won't allocate billion byte arrays when handling large (h)sets.
It reduces pressure on allocator when loading millions of items.
Fixes#1076.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Co-authored-by: Roy Jacobson <roy@dragonflydb.io>
This change removes most mentions of boost::fibers or util::fibers_ext.
Instead it introduces "core/fibers.h" file that incorporates most of
the primitives under dfly namespace. This is done in preparation to
switching from Boost.Fibers to helio native fibers.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
