feat(server): Basic multi transaction modes
This commit adds the notion of multi transaction modes that allow controlling the execution and
locking behaviour of multi transactions.
In general, there are four modes:
- GLOBAL: all commands run within a global transaction. There is no need for recording locks. Lua scripts can theoretically run with undeclared keys.
- LOCK_AHEAD: the transaction locks all keys ahead likewise to a regular transaction and schedules itself.
- LOCK_INCREMENTAL: the transaction determines what shards it has keys in and schedules itself on those shards, but locks only when accessing a new key. This allows other transactions to run ooo alonside with a big multi-transaction that accesses a contended key only at its very end.
- NON_ATOMIC: all commands run separately, no atomicity is provided, likewise to a pipeline
This commit only adds support for the first 3 modes to EXEC commands.
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
This change allows to track which transactions are run as out of order.
OOO txs are more performant and inhibit substantially less latency.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* bug(transaction): local result needs to be reset on InitByArgs Fixes#752
Signed-off-by: adi_holden <adi@dragonflydb.io>
* add unit test
Signed-off-by: adi_holden <adi@dragonflydb.io>
---------
Signed-off-by: adi_holden <adi@dragonflydb.io>
The scenario is described in a unit test that reproduces the issue with high chance.
Also added dragonfly_test in repeat=100 mode to CI.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. Before that we did no support a real syntax with <numkey> argument,
now we do.
2. Fix warnings.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
fix(server): Fix a bug when an expired transaction stays in watched queue.
Now we remove the transaction from the watched queues in a consistent manner based on the
keys it was assigned to watch.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
fix(server): Fix redundant locking in multi transactions
Apparently commands that scheduled themselves using "Schedule()" call
crashed under multi transactions. This has been fixed now inside Transaction code.
It has been covered by DflyEngineTest.MultiRename test.
In addition, this PR fixes#468 that opened the rabbit hole of
nasty bugs under multi transactions.
Signed-off-by: Boaz Sade <boaz@dragonflydb.io>
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Signed-off-by: Boaz Sade <boaz@dragonflydb.io>
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
The bug was that if two push operations where queued together in the tx queue,
and the first push awakes pending blpop, then the PollExecution function would continue with the
second push before switching to blpop, which contradicts the spec.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Partially implements #6.
Before, each shard lazily updated its clock used for the expiry evaluation.
Now, the clock value is set during the transaction scheduling phase and is assigned
to each transaction. From now on DbSlice methods use this value when checking whether
the entry is expired via passed DbContext argument.
Also, implemented transactionally consistent TIME command and
verify that time is the same during the transaction. See
https://ably.com/blog/redis-keys-do-not-expire-atomically for motivation.
Still have not implemented any lamport style updates for background processes
(not sure if it's the right way to proceed).
In more detail, RdbSaver uses AlignedBuffer that writes into io::Sink in chunks of 4KB.
It's great for the direct file I/O, but bad for sockets that receive blocks of 4KB with garbage
at the end. I improved the code around this and actually simplified the logic, so now AlignedBuffer
is just another Sink that is passed into serializer when writing into files. When sending to
sockets a socket sink is passed instead.
Also many other unrelated changes grouped into this pretty big cr.
1. dashtable readability improvements.
2. Move methods from facade::ConnectionContext - into facade::Service,
make ConnectionContext a dumb object.
3. Optionally allow journal to be memory only (not backed up by a disk)
by using a ring buffer to store last k entries in each journal slice. Also renamed
journal_shard into journal_slice because journal has presence in each DF thread and not
only in its shards.
4. Introduce journal::Entry that will consolidate any store change that happens in the thread.
5. Introduce GetRandomHex utility function.
6. Introduce two hooks: ServerFamily::OnClose that is called when a connection is closed,
and ServerFamily::BreakOnShutdown that is called when process exits and any background fibers neet to
break early.
7. Pull some noisy info logs out of rdb_load class.
8. Snapshot class now has the ability to subscribe to journal changes, thus it can include concurrent changes into the snapshot.
Currently only journal::Op::VAL is supported (it's part of RDB format anyway).
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. No entries data is written into a journal yet.
2. Introduced a state machine to start and stop journal using a new auxillary command "dfly".
3. string_family currently calls an universal command Journal::RecordEntry that should
save the current key/value of that entry. Please note that we won't use it for all the operations
because for some it's more efficient to record the action itself than the final value.
4. no locking information is recorded yet so atomicity of multi-key operations is not preserved for now.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. Correctly parse keys for EVAL command.
2. Support PUBLISH within lua.
3. Remove spurious failure in debug-mode with the increased verbosity level.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
In rare cases a scheduled transaction is not scheduled correctly and we need
to remove it from the tx-queue in order to re-schedule. When we pull it from tx-queue
and it has been located at the head, we must poll-execute the next txs in the queue.
1. Fix the bug.
2. Improve verbosity loggings to make it easier to follow up on tx flow in release mode.
3. Introduce /txz handler that shows currently pending transactions in the queue.
4. Fix a typo in xdel() function.
5. Add a py-script that reproduces the bug.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. Found dangling transaction pointers that where left in the watch queue. Fix the state machine there.
2. Improved transaction code a bit, merged duplicated code into RunInShard function, got rid of RunNoop.
3. Improved BPopper::Run flow.
4. Added 'DEBUG WATCH' command. Also 'DEBUG OBJECT' now returns shard id and the lock status of the object.
1. Make EngineShardSet to be a process singleton instead of a variable passed everywhere.
2. Implement a heuristic of updating free memory per shard.
3. Make sure that SET command returns OOM when a database reaches its limits.
