* refactor(server): Reuse `PreUpdate()` internally
This actually fixes a bug where we did not handle tiering upon
expiration if found via `AddOrFind2()`.
Related to #2252
* Simplify code, remove optimization
* Don't do double lookups
* stats
* PreUpdate()--
* fixes
fixes#2224
the bug: when updateing entry pre update will resets the PrimeValue if
this is an external entries leading to crash if we insert entries with expire
time.
the fix: reserve the expire mask in PrimeValue on pre update
Signed-off-by: adi_holden <adi@dragonflydb.io>
* WIP: Auto `PostUpdate()`
* More `Find()` uses
* Final touches
* Fixes
* Fix bug and allow reassigning
* Rename to AutoUpdater
* Fix and add DCHECK
* Also check deletion count
* Use ccache instead of sccache
* Try to upgrade Helio
* off64_t
* off64_t
* Revert changes to CI
The client tracking state is set by CLIENT TRACKING subcommand as well
as upon client disconnection.
Track the keys of a readonly command by maintaining mapping that maps
keys to the sets of tracking clients.
* feat(server): Return per-type memory breakdown from INFO and /metrics
* OBJ_TYPE_MAX
* Move AddTypeMemoryUsage, use it from TieredStorage
* Remove strval_memory_usage
* Remove redundant strval_bytes
---------
Signed-off-by: Shahar Mike <chakaz@users.noreply.github.com>
* fix(teiring): fix crash when item was deleted before offloaded
The bug: On insert key is inserted to bin peinding entries, if this key
was deleted we would check fail on finding this item when tring to flush
bin entries.
The fix: On every insert to bin pending entries erase pending entreis
which were delted or expired
Signed-off-by: adi_holden <adi@dragonflydb.io>
This caused a UBSan warning to be printed in test `DflyEngineTest.Bug207`
```
/usr/include/c++/11/bits/stl_vector.h:1046:34: runtime error: reference binding to null pointer of type 'struct value_type'
```
* fixes#1936
Eviction Implementation
This patch provides a very simple eviction implementation for the interface mentioned above. In my opinion, the eviction algorithm approximates an LRU policy given that normal buckets always store the most recently accessed data while stash buckets are holding less active data.
The algorithm first selects a small set of segments as eviction targets. Starting from the last slot of the last stash bucket in each of the segments, we walk backward to evict key-value pairs stored in each visited slot. The eviction stopped either when a target memory release goal or the max number of evicted key-value pairs is reached. Therefore, we can upper bound the eviction time through the following two parameters that can be set when DF starts. Note that these two parameters could be retrieved and changed by user through CONFIG GET and CONFIG SET commands.
---------
Signed-off-by: Yue Li <61070669+theyueli@users.noreply.github.com>
The issue was that the Heartbeat would run every 1ms and this is problematic because the for-loop would take less than 1ms to finish. Therefore, the memory pool would not adjust and items would not be evicted from the store. By doubling the amount of elements created in the for-loop, we give enough time for the first heartbeat to run and adjust the memory available (which will cause the evictions to happen).
1. Cherry-pick changes from Redis 7 that encode integer scores more efficiently
2. Introduces optimization that first checks if the new element should be the last
for listpack sorted sets.
3. Consolidates listpack related constants and tightens usage for listpack.
4. Introduce MEMORY USAGE command.
5. Introduce a small delay before decommitting memory pages back to OS.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* fix(server): Dont apply memory limit when loading/replicating
When loading a snapshot created by the same server configuration (memory and
number of shards) we will create a different dash table segment directory tree, because the
tree shape is related to the order of entries insertion. Therefore when loading data from
snapshot or from replication the conservative memory checks might fail as the new tree might
have more segments. Because we dont want to fail loading a snapshot from the same server
configuration we disable this checks on loading and replication.
Signed-off-by: adi_holden <adi@dragonflydb.io>
* feat(server): support multi eval in lock ahead mode
1. remove validation to allow multi eval only in global script mode
2. send error if there is a mode conflict when running eval inside multi
3. reset uniqe_keys_ when transaction finishes
Partially addresses #1730.
Beforehand we removed entries from mcflag only inside DbSlice::Del function
but we delete entries in other cases too (like expiration) and in this case we left
dangling entries in mcflag table. The fix adds the deletion of mcflag entries into PerformDeletion function.
It by itself does not explain the crash but
dangling entries cause UB (because the keys are references to non-existent PrimeKeys)
so maybe this somehow causes inconsistency where a key is not present in mcflags.
I am not sure I fixed the original issue but I surely fixed a corruption bug.
In addition, I removed the redundant check fail and added error logs to follow up on this later.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* feat(server): Add `--lock_on_hashtags` mode.
This new mode effectively locks hashtags (i.e. strings within {curly
braces}) instead of the full keys being used.
This can allow scripts to access undeclared keys if they all use a
common hashtag, like for the case of BullMQ.
To make sure this mode is tested, I added a way to specify flags via env
variables, and modified `ci.yml` to run all tests using this mode as well.
While at it, I also added `--cluster_mode=emulated` mode to CI.
This PR would have been 1 line change instead of 11 files, but it
required some plumbing and refactoring:
* Now ClusterConfig is aware of emulated cluster mode
* As a result, this API was moved from ClusterFamily
* And so was the flag & its parsing
* ClusterFamily doesn't need is_emulated_cluster_ member
* ServerFamily no longer needs ClusterFamily* member (because the API is
static)
* I also changed `ClusterConfig::IsClusterEnabled()` to
`ClusterConfig::IsEnabled()` to be shorter
* fix(server): Initialize ServerFamily with all listeners.
- Add a test for CLIENT LIST which is the visible result of this.
* use std move
* feat: Implement replicas take over
* Basic test
* Address CR comments
* Write a better test. Sadly it fails
* chore: Expose AwaitDispatches for reuse in takeover
* Ensure that no commands can execute during or after a takeover
* CR progress
* Actually disable the expiration
* Improve tests coverage
* Fix the dispatch waiting code
* Improve testing coverage and fix a shutdown snaphot bug
* don't replicate a replica
Remove Boost.Fibers mentions and remove fibers_ext mentions.
Done in preparation to switch to helio-native fb2 implementation.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
TopKeys uses a custom implementation of HeavyKeeper to track top (hot)
keys usage for debugging purposes.
This commit also integrates TopKeys (default off) into DbTable and counts
usage of (present) key lookups.
Signed-off-by: chakaz <chakaz@chakaz>
feat(server): Basic multi transaction modes
This commit adds the notion of multi transaction modes that allow controlling the execution and
locking behaviour of multi transactions.
In general, there are four modes:
- GLOBAL: all commands run within a global transaction. There is no need for recording locks. Lua scripts can theoretically run with undeclared keys.
- LOCK_AHEAD: the transaction locks all keys ahead likewise to a regular transaction and schedules itself.
- LOCK_INCREMENTAL: the transaction determines what shards it has keys in and schedules itself on those shards, but locks only when accessing a new key. This allows other transactions to run ooo alonside with a big multi-transaction that accesses a contended key only at its very end.
- NON_ATOMIC: all commands run separately, no atomicity is provided, likewise to a pipeline
This commit only adds support for the first 3 modes to EXEC commands.
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
