chore: simplify transaction multi-locking
Also, add the ananlysis routine that determines whether the schewduled transaction is contended with other transaction in a
shard thread.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* refactor(server): Privatize `PreUpdate()` and `PostUpdate()`
While at it:
* Make `PreUpdate()` not decrease object size
* Remove redundant leftover call to `PreUpdate()` outside `DbSlice`
* Add pytest
* Test delete leads to 0 counters
* Improve test
* fixes
* comments
* feat: add keyspace_mutations metric
Currently we expose hits/misses for read only commands only (compatible with redis).
`keyyspace_mutations` complement this providing number of key operations for write commands.
It's interesting because now we can learn the number of key_ops vs API ops, where
key_ops = misses + hits + mutations
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore: address fixes
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
fixes https://github.com/dragonflydb/dragonfly/issues/2139
This is part two that implements the logic which notifies tracking clients by sending invalidation messages:
- The client tracking state is set by CLIENT TRACKING subcommand as well
as upon client disconnection.
- Track the keys of a readonly command by maintaining mapping that maps
keys to the sets of tracking clients.
- Send invalidation messages to clients when their tracked keys are
updated.
- Make PerformDeletion a member function of DbSlice, and send
invalidation message within the function.
- Mock the function for sending invalidation message to avoid test
crash due to lack of real listener in the testing framework.
- Add functional (some) tests for client tracking based on the mocked interfaces.
---------
Signed-off-by: Yue Li <61070669+theyueli@users.noreply.github.com>
The bug:
When running dragonfly in cache mode we bump up items on dash table when we find them. If we access few items on the callback that reside next to each other we will invalidate the first found iterator.
The fix:
After we bump up entry we insert the prime table ref to bump set. When checking if we can bump down an item we check the item is not in this set. Once we finish running the transaction callback we clear the set.
Signed-off-by: adi_holden <adi@dragonflydb.io>
* refactor(server): Reuse `PreUpdate()` internally
This actually fixes a bug where we did not handle tiering upon
expiration if found via `AddOrFind2()`.
Related to #2252
* Simplify code, remove optimization
* Don't do double lookups
* stats
* PreUpdate()--
* fixes
* WIP: Auto `PostUpdate()`
* More `Find()` uses
* Final touches
* Fixes
* Fix bug and allow reassigning
* Rename to AutoUpdater
* Fix and add DCHECK
* Also check deletion count
* Use ccache instead of sccache
* Try to upgrade Helio
* off64_t
* off64_t
* Revert changes to CI
The client tracking state is set by CLIENT TRACKING subcommand as well
as upon client disconnection.
Track the keys of a readonly command by maintaining mapping that maps
keys to the sets of tracking clients.
* fix(teiring): fix crash when item was deleted before offloaded
The bug: On insert key is inserted to bin peinding entries, if this key
was deleted we would check fail on finding this item when tring to flush
bin entries.
The fix: On every insert to bin pending entries erase pending entreis
which were delted or expired
Signed-off-by: adi_holden <adi@dragonflydb.io>
* fixes#1936
Eviction Implementation
This patch provides a very simple eviction implementation for the interface mentioned above. In my opinion, the eviction algorithm approximates an LRU policy given that normal buckets always store the most recently accessed data while stash buckets are holding less active data.
The algorithm first selects a small set of segments as eviction targets. Starting from the last slot of the last stash bucket in each of the segments, we walk backward to evict key-value pairs stored in each visited slot. The eviction stopped either when a target memory release goal or the max number of evicted key-value pairs is reached. Therefore, we can upper bound the eviction time through the following two parameters that can be set when DF starts. Note that these two parameters could be retrieved and changed by user through CONFIG GET and CONFIG SET commands.
---------
Signed-off-by: Yue Li <61070669+theyueli@users.noreply.github.com>
* feat(server): Add `--lock_on_hashtags` mode.
This new mode effectively locks hashtags (i.e. strings within {curly
braces}) instead of the full keys being used.
This can allow scripts to access undeclared keys if they all use a
common hashtag, like for the case of BullMQ.
To make sure this mode is tested, I added a way to specify flags via env
variables, and modified `ci.yml` to run all tests using this mode as well.
While at it, I also added `--cluster_mode=emulated` mode to CI.
* fix(server): Initialize ServerFamily with all listeners.
- Add a test for CLIENT LIST which is the visible result of this.
* use std move
* feat: Implement replicas take over
* Basic test
* Address CR comments
* Write a better test. Sadly it fails
* chore: Expose AwaitDispatches for reuse in takeover
* Ensure that no commands can execute during or after a takeover
* CR progress
* Actually disable the expiration
* Improve tests coverage
* Fix the dispatch waiting code
* Improve testing coverage and fix a shutdown snaphot bug
* don't replicate a replica
Remove Boost.Fibers mentions and remove fibers_ext mentions.
Done in preparation to switch to helio-native fb2 implementation.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. Support tiered deletion.
2. Add notion of tiered entity in "DEBUG OBJECT" output.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Partially implements #6.
Before, each shard lazily updated its clock used for the expiry evaluation.
Now, the clock value is set during the transaction scheduling phase and is assigned
to each transaction. From now on DbSlice methods use this value when checking whether
the entry is expired via passed DbContext argument.
Also, implemented transactionally consistent TIME command and
verify that time is the same during the transaction. See
https://ably.com/blog/redis-keys-do-not-expire-atomically for motivation.
Still have not implemented any lamport style updates for background processes
(not sure if it's the right way to proceed).
Before this change Dragonfly evicted items only when it was low on memory and had to grow its main dictionary.
It is not enough because in many cases Dragonfly can grow in memory even when the main dictionary does not grow.
For example, when its dictionary is less than 90% utilized, but the newly added objects require lots of memory.
In addition, the dashtable adds additional segments, when other segments have enough available slots to fill the rest of the free memory.
This change adds another layer of defense that allows evicting items even when dictionary segments are not full.
The eviction is still local with respect to a segment. On my tests it seems that it's much harder to cross maxmemory limit than before.
In addition, we tightened the heuristic that allowes the dashtable to grow. Now it takes into account the average bytes per item
in order to project how much memory the full table takes before adding to it new segments.
This really improves dashtable utilization.
There are still things to improve:
1. the eviction behavior is rough. Once an insert does the eviction it tries to free enough objects to bring memory back.
This may result in very spiky insertion/eviction patterns.
2. The eviction procedure, even though it's limited to a single segment, is quite heavy because it goes over all buckets
in the segment. This may result in weird artifacts where we evict just enough to be under the limit, then add and evict
again and so on.
3. Therefore, we may need a periodic eviction that will compliment this emergency eviction step.
Fixes#224 and partially addresses #256
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
This statistic helps understanding how much Dragonfly memory grows via updating the existing value vs the new ones.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. Fix a bug in dash table related to snapshotting.
2. Rewrite GlobalState code and make state transitions atomic and well defined.
3. Fix Save/Flush semantics by capturing snapshotted tables together with the snapshot.
