* chore: Make snapshotting more responsive
This should improve situation around #4787 -
maybe not solve it completely but improve it significantly.
On my tests when doing snapshotting under read traffic with master
(memtier_benchmark --ratio 0:1 -d 256 --test-time=400 --distinct-client-seed --key-maximum=2000000 -c 5 -t 2 --pipeline=3)
I got drop from 250K qps to 8K qps during the full sync phase.
With this PR, the throughput went up to 70-80K qps.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Close MONITOR connection if we overflow pipeline limits. It can happen
that MONITOR connection dispatches messages slower than they are received
causing memory to go out of bounds (and can result in process crash).
Signed-off-by: mkaruza <mario@dragonflydb.io>
StringSet object doesn't update time when FIELDEXPIRE is called. It will
use base time when object is created. Update object time when we want to
expire field in SET object.
Fixes#4894
Signed-off-by: mkaruza <mario@dragonflydb.io>
Remove key argument in DbSlice::CallChangeCallbacks which is used only
for DVLOG(2) logging. DbSlice::OnCbFinishBlocking needs to call this
function but it doesn't have key name so it has to retrieve it from
iterator.
Signed-off-by: mkaruza <mario@dragonflydb.io>
Pytest test_exit_on_s3_snapshot_load_err can raise exception on start in
some test environments. Now wait for exception in instance start and
stop.
Signed-off-by: mkaruza <mario@dragonflydb.io>
When setting TTL, the object is cloned. An extra 4 bytes is requested
during allocation. This can result in the object being allocated from a
larger page, eg moving from 16 byte page to 32 byte page. The page block
size is used to report object allocation size. Currently this change in
size is not reflected in the total memory usage of the dense set, so it
remains 16 bytes while the object allocated size is now 32 bytes.
If such an object is later replaced, we deduct the size of the object
from total memory usage of the set. Here we can run into an overflow,
because the size of the object is deducted from the tracked size of the
set, and the former is greater than the latter.
To avoid this, if during setting expiry time the new size is different
from old size, we update the set size.
Exit process if error is reported when we try to initially load snapshot from
cloud storage or local directory.
Fixes#4840
Signed-off-by: mkaruza <mario@dragonflydb.io>
When a search operation is performed on a hash set, expired fields are
removed as a side effect.
If at the end of such an operation the hash set becomes empty, its key
is removed from the database.
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
Bumpup logic is moved to OnCbFinish. Previously keys which are going to
be delete were also bumped up but with this change if key doesn't exists
on callback we will skip it.
Closes#4775
Signed-off-by: mkaruza <mario@dragonflydb.io>
We allowed running heartbeat during global transactions. For example, during replication and while transitioning from full to stable sync we unregister and register journal callbacks with a preemption inbetween. If under preemption heartbeat runs, then we loose those journal entries triggering an lsn check failure.
Signed-off-by: kostas <kostas@dragonflydb.io>
* rename StaticSeeder to DebugPopulateSeeder to show intent
* add a DCHECK in debug populate that triggers if there is a registered replica that hasn't reached stable sync
* small cleanup
Signed-off-by: kostas <kostas@dragonflydb.io>
* feat: debug compression <type>
Extend `debug compression` to support value types.
if no type is given, falls back to testing keys compressability.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* Apply suggestions from code review
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
now it looks like this:
```
> docker run --rm ghcr.io/dragonflydb/dragonfly-dev:ubuntu-f767d82 --version
dragonfly f767d82-f767d82ce78ccbc90ddfb525f4ad4bd9aafcfbed
```
fixes#4830
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
- Removes the warning the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion
Signed-off-by: Thomas Camlong <thomas@ajnart.fr>
