Also, stop deduplicating keys by default, but keep it as a run-time flag.
All in all, this PR improves MGET (100% miss) throughput by at least 7%:
from 1.23M to 1.32M qps.
(100% miss traffic allows measuring more effectively the impact of this code)
Also, finally fix TieredStorageTest.FlushAll test.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* fix(glob): Correctly match longer strings with '*' or '**' pattern
If string is longer than 16 chars we are using reflex library for
matching. When used pattern is '*' or '**' we are going to remove
trailing and leading star and have empty pattern. We should, in this,
edge case set manually star pattern.
Fixes#4948
Signed-off-by: mkaruza <mario@dragonflydb.io>
* Core matching test
---------
Signed-off-by: mkaruza <mario@dragonflydb.io>
1. Fix a crash bug in RESETSTAT when number of shards is less than number of threads.
2. Tune regtests parameters for pipelining tests to pass.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Before: StoredCmd always copied the backing buffer of the commands.
this of course sub-optimal if the bucking buffer exists during the life-time
of StoredCmd. This is exactly the case in `Service::DispatchManyCommands`.
This PR:
1. Adds support for both owned and non-owned arguments.
2. Improves the interfaces around StoredCmd and removes some code duplication.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. Eliminate replies reverse call - will allow to unite replies and cmd vectors into one.
2. Introduce SwitchTxCmd function to avoid code duplication.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Fixes#4998.
1. Reduces agressive yielding when reading multiple requests since it humpers pipeline efficiency.
Now we yield consistently based on cpu time spend since the last resume point (via flag with sane defaults).
2. Increases socket read buffer size effectively allowing processing more requests in bulk.
`./dragonfly --cluster_mode=emulated`
latencies (usec) for pipeline sizes 80-199:
p50: 1887, p75: 2367, p90: 2897, p99: 6266
`./dragonfly --cluster_mode=emulated --experimental_cluster_shard_by_slot`
latencies (usec) for pipeline sizes 80-199:
p50: 813, p75: 976, p90: 1216, p99: 3528
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
This is relevant only for cluster-enabled configurations.
Also, inline the cluster config getter functions, as they are on critical path for 100% of requests.
Finally, skip a test that triggers a check-fail bug filed in #5004Fixes#5005
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
The issue is that Scan has high latency (in seconds) when keys are large and there are no matches. Iterating 10k buckets and string matching each of the keys is potentially an expensive operation that depends on the keyspace and the number of actual matches.
* replace heuristic in scan command to throttle based on time
Signed-off-by: kostas <kostas@dragonflydb.io>
* chore(hset_family): Support resp3 format for hrandfield
Return nested arrays if hrandfield is used with values.
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
* feat(dfly_bench): allow regulated throughput in 3 modes
1. Coordinated omission - with --qps=0, each request is sent and then we wait for the response and so on.
For pipeline mode, k requests are sent and then we wait for them to return to send another k
2. qps > 0: we schedule sending requests at frequency "qps" per connection but if pending requests count crosses a limit
we slow down by throttling request sending. This mode enables gentle uncoordinated omission, where the schedule
converges to the real throughput capacity of the backend (if it's slower than the target throughput).
3. qps < 0, similar as (2) but does not adjust its scheduling and may overload the server
if target QPS is too high.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore: change pipelining and coordinated omission logic
Before that the uncoordinated omission only worked without pipelining.
Now, with pipelining mode with send a burst of P requests and then:
a) For coordinated omission - wait for all of them to complete before proceeding
further
b) For non-coordinated omission - we sleep to pace our single connection throughput as
defined by the qps setting.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* evict in heartbeat if expire table is not empty
* add metrics around heartbeat evictions (total evictions and total evicted bytes)
Signed-off-by: kostas <kostas@dragonflydb.io>
fix(set_family): Transfer TTL flag from link to object in delete
When extracting DensePtr from LinkObject we need to transfer TTL flag
before this DensePtr is assigned.
Fixes#3915
Signed-off-by: mkaruza <mario@dragonflydb.io>
* chore: Make snapshotting more responsive
This should improve situation around #4787 -
maybe not solve it completely but improve it significantly.
On my tests when doing snapshotting under read traffic with master
(memtier_benchmark --ratio 0:1 -d 256 --test-time=400 --distinct-client-seed --key-maximum=2000000 -c 5 -t 2 --pipeline=3)
I got drop from 250K qps to 8K qps during the full sync phase.
With this PR, the throughput went up to 70-80K qps.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Close MONITOR connection if we overflow pipeline limits. It can happen
that MONITOR connection dispatches messages slower than they are received
causing memory to go out of bounds (and can result in process crash).
Signed-off-by: mkaruza <mario@dragonflydb.io>
StringSet object doesn't update time when FIELDEXPIRE is called. It will
use base time when object is created. Update object time when we want to
expire field in SET object.
Fixes#4894
Signed-off-by: mkaruza <mario@dragonflydb.io>
