* remove DbSlice mutex
* add ConditionFlag in SliceSnapshot
* disable compression when big value serialization is on
* add metrics
---------
Signed-off-by: kostas <kostas@dragonflydb.io>
* chore: remove ToUpper calls in main_service
Also, test for IsPaused() first to avoid doing more checks for common-case.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore: reduce pipelining latency by reusing existing shard fibers
To prove the benefits, run `./dfly_bench --pipeline=50 -n 20000 --ratio 0:1 --qps=0 --key_maximum=1`
Before: the average pipelining latency was 10ms
After: the average pipelining latency is 5ms.
Avg latency: pipelined_latency_usec / total_pipelined_squashed_commands
Also, improved counting of squashed commands - to count actual squashed ones.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
This commit generalizes the machanism of running transaction callbacks during scheduling, removing the need for specialized ScheduleUniqueShard/RunQuickie. Instead, transactions can be run now during ScheduleInShard - called "immediate" runs - if the transaction is concluding and either only a single shard is active or the operation can be safely repeated if scheduling failed (idempotent commands, like MGET).
Updates transaction stats to mirror the new changes more closely.
---------
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
* chore: add malloc-based stats and decommit
Provides more stats and control with glibc-malloc based allocator.
For example,
with v1.15.0 (--proactor_threads=2), empty database, `info memory`returns
```
used_memory:614576
used_memory_human:600.2KiB
used_memory_peak:614576
used_memory_peak_human:600.2KiB
used_memory_rss:19922944
used_memory_rss_human:19.00MiB
```
then during `memtier_benchmark -n 300000 --key-maximum 100000 --ratio 0:1 --threads=30 -c 100` (i.e GET-only with 3k connections):
```
used_memory:614576
used_memory_human:600.2KiB
used_memory_peak:614576
used_memory_peak_human:600.2KiB
used_memory_rss:59985920
used_memory_rss_human:57.21MiB
used_memory_peak_rss:59985920
```
connections overhead grows by ~39MB.
when the traffic stops, `used_memory_rss_human` becomes `30.35MiB`
and we do not know where 11MB gets lost and `MEMORY DECOMMIT` does not reduce the RSS.
With this change, `memory malloc-stats` return during the memtier traffic
```
malloc arena: 394862592
malloc fordblks: 94192
```
i.e. 395MB virtual memory was allocated by malloc and only 94KB is chunks available for reuse.
395MB is arena virtual memory, and not RSS obviously, but at least we have some visibility into malloc reservations.
The RSS usage is the same ~57MB and the difference between virtual and RSS is due to the fact we reserve fiber stacks of size 131KB but we touch less.
After the traffic stops, `arena` is reduced to 134520832 bytes, and fordblks are 133016592, i.e. majority of reserved ranges are also free (available to reuse) in the malloc pools.
RSS goes down similarly to before to ~31MB.
So far, this PR only demonstrated the increased visibility to mmapped ranges reserved by glibc malloc.
The additional functional change is in `MEMORY DECOMMIT` that now trims malloc RSS usage from reserved but unused (fordblks) pages
by calling `malloc_trim`.
After the call, RSS is: `used_memory_rss_human:20.29MiB` which is almost the same as when we started the empty process.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore: fix build for older glibc environments
Disable these extensions for alpine and use legacy version
for older glibc libraries.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore: add oom stats to /metrics
Expose oom/cmd errors when we reject executing a command if we reached OOM state (controlled by oom_deny_ratio flag).
Expose oom/insert errors when we do not insert a new key or do not grow a dashtable (controlled by table_growth_margin).
Move OOM command check to a place that covers all types of transactions - including multi and squashing transactions.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Previously, transactions would run out of order only when all shards determined that the keys locks were free. With this change, each shard might decide to run out of order independently if the locks are free. COORD_OOO is now deprecated and the OUT_OF_ORDER per-shard flag should is used to indicate it
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
* feat: track differrent patterns of multi/exec transactions
This information is exposed via "DEBUG EXEC" command.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore: address comments + add basic squasher stats
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. How many transactions we processed by type
2. How many transactions we processed by width (number of unique shards).
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* feat: DispatchTracker
Use a DispatchTracker to track ongoing dispatches for commands that change global state
---------
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
* fix(server): client pause fix on pipeline squash
allow squashing commands on pause
move await on client pause inside InvokeCommand - this way all flows of command invoke will read pause state
Signed-off-by: adi_holden <adi@dragonflydb.io>
1. add flag maxmemory_ratio
2. When current used memory * maxmemory_ratio > maxmemory_limit denyoom
commands will return oom error.
Signed-off-by: adi_holden <adi@dragonflydb.io>
* feat: run tx-schedule inline if the dest shard is on the same thread (#908)
The optimization is applied within ScheduleSingleHop call.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* fix(server): Don't inline schedule when in LOADING
* Fix the another pre-emption bug with inline scheduling
* Better locking around journal callbacks
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Co-authored-by: Roman Gershman <roman@dragonflydb.io>
1. now pipeline_cache_capacity tracks how many bytes are allocated via free_req_pool_ cache.
it's been shown now via "pipeline_cache_bytes" stat in "info memory" command.
2. a small refactoring of server_state code into server_state.cc
3. Reduce dependency of dfly_transaction on facade lib.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
feat(server): support monitor command - allowing user to debug commands
from all connections by using a connection as monitors for the this
(#344)
Signed-off-by: Boaz Sade <boaz@dragonflydb.io>
