Remove stash template parameter because we only use dashtable with a single configuration
of STASH_CNT=4.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
It's only a code move, without functional changes.
This is in preparation to implementing the same path functionality
for flexbuffers objects.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* fix(server): mget crash on same key get
fix: #2465
the bug: on cache mode mget bumps up items. When executing mget with the same key several times i.e mget key key we will invalidate the iterator when we bump up the item in dash table.
the fix: bump up/down items only once by using bumped_items set
This PR also reverts c225113
and updates the bumped stats and bumped_items set if the item was bumped
Signed-off-by: adi_holden <adi@dragonflydb.io>
chore: cosmetic improvements in dash code
1. Better naming
2. Improve improving the interface of ForEachSlot command
3. Wrap the repeating code of updating the bucket version into the UpdateVersion function
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
fix(bug): dashtable split crashes when moving items from the old segment.
Segment's Insert function uses an opportunistic heuristic that chose a bucket with smaller items among two available.
This creates a problem with split that goes from a smaller bucket to the biggest one and moves items to the new segment.
In rare case, the heurstic fills up a the next bucket with items that could reside in earlier buckets and then that bucket
does not have space for its own items. Eventually, items that had enough space in the old segment do not find space in the new one
The fix is to adopt a conservative approach during split and try to use a home bucket first. Home bucket is usually a smaller one.
This approach should be optimal because Split starts with smaller buckets first (can be proven iteratively).
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
In more detail, RdbSaver uses AlignedBuffer that writes into io::Sink in chunks of 4KB.
It's great for the direct file I/O, but bad for sockets that receive blocks of 4KB with garbage
at the end. I improved the code around this and actually simplified the logic, so now AlignedBuffer
is just another Sink that is passed into serializer when writing into files. When sending to
sockets a socket sink is passed instead.
Also many other unrelated changes grouped into this pretty big cr.
1. dashtable readability improvements.
2. Move methods from facade::ConnectionContext - into facade::Service,
make ConnectionContext a dumb object.
3. Optionally allow journal to be memory only (not backed up by a disk)
by using a ring buffer to store last k entries in each journal slice. Also renamed
journal_shard into journal_slice because journal has presence in each DF thread and not
only in its shards.
4. Introduce journal::Entry that will consolidate any store change that happens in the thread.
5. Introduce GetRandomHex utility function.
6. Introduce two hooks: ServerFamily::OnClose that is called when a connection is closed,
and ServerFamily::BreakOnShutdown that is called when process exits and any background fibers neet to
break early.
7. Pull some noisy info logs out of rdb_load class.
8. Snapshot class now has the ability to subscribe to journal changes, thus it can include concurrent changes into the snapshot.
Currently only journal::Op::VAL is supported (it's part of RDB format anyway).
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Before this change Dragonfly evicted items only when it was low on memory and had to grow its main dictionary.
It is not enough because in many cases Dragonfly can grow in memory even when the main dictionary does not grow.
For example, when its dictionary is less than 90% utilized, but the newly added objects require lots of memory.
In addition, the dashtable adds additional segments, when other segments have enough available slots to fill the rest of the free memory.
This change adds another layer of defense that allows evicting items even when dictionary segments are not full.
The eviction is still local with respect to a segment. On my tests it seems that it's much harder to cross maxmemory limit than before.
In addition, we tightened the heuristic that allowes the dashtable to grow. Now it takes into account the average bytes per item
in order to project how much memory the full table takes before adding to it new segments.
This really improves dashtable utilization.
There are still things to improve:
1. the eviction behavior is rough. Once an insert does the eviction it tries to free enough objects to bring memory back.
This may result in very spiky insertion/eviction patterns.
2. The eviction procedure, even though it's limited to a single segment, is quite heavy because it goes over all buckets
in the segment. This may result in weird artifacts where we evict just enough to be under the limit, then add and evict
again and so on.
3. Therefore, we may need a periodic eviction that will compliment this emergency eviction step.
Fixes#224 and partially addresses #256
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. Snapshot now passed db index and rdb save handles by inserting select opcodes when needed.
2. Fix few more bugs in CVCUponInsert.
3. Save lua scripts.
1. Stabilized naming scheme for snapshot files.
The behavior is determined by dbfilename flag.
By default it's 'dump'. Dragonfly checks if the flag has an extension
if not, it automatically saves timestamped files (dump*.rdb) and loads the latest file that is available.
In case the flag has extension like 'dump.rdb' it fallbacks to redis behavior of loading and saving to the
same file.
2. Updated the logo url.
This id one by shifting right slots in a stash bucket of the full segment.
In addition, I added eviction related stats to memory and stats section.
I also updated README with the changes. Finally, I added a flag that allows
to disable http-admin console in dragonfly.
The heart of this feature is ability to bump up entries that has been searched for.
When we bump up an entry withing a dash-table it's moved out of stash buckets or to lower slot ids
within normal buckets. The entry they replace is moved to the place of bumped-up entries.
The next change will be to implement the actual eviction heuristic that will evict keys from the stash buckets.
This will give stash buckets an addition responsibility for caching mode - they will serve as a probation buffer
that holds low-priority or newly added items.
1. Simplify versioning scheme. This also reduces metadata space and increases table capacity
for same memory reservation by ~%5.
2. Fix snapshotting bugs. Recognize duplicate keys during loading.
3. Introduce DEBUG LOAD command to perform manual loading of files.
