1. Move socket read code into a dedicated function.
Remove std:: prefix in the code.
2. Add an optional iouring bufring registration. Currently not being used and is disabled by default.
1. Run CI/Regression tests with HELIO_STACK_CHECK=4096.
This will crash if a fiber stack usage goes below this limit.
2. Increase shard queue stack size to 64KB
3. Increase fiber stack size to 40KB on Debug builds.
4. Updated helio has some changes around the TLS socket code.
In addition we add a helper script to generate self-signed certificates helpful for local development work.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. Better logging in regtests
2. Release resources in dfly_main in more controlled manner.
3. Switch to ignoring signals when unregister signal handlers during the shutdown.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore: Add `--allocator_tracker` for default tracking
Before, in order to use allocation tracker, one had to issue a `MEMORY
TRACK` command. This flag is identical to that, but allows starting
Dragonfly with certain ranges without issuing a command.
While here, fix a bug. Apparently, `absl::InlinedVector<>` has a bug in
the implementation of `max_size()` and so in practice we did not limit
the number of trackers. I switched to use `capacity()` instead, which I
tested and it works well.
Notes:
1. Currently the flag always add 100% "sampling", we can extend that in
the future if need be
2. I added the flag in `dfly_main.cc` with custom initialization,
because it's low level, and I couldn't get it reasonably working with
changes only to `allocation_tracker.cc`
* fixes
1. Issue ping upon connect, add a comment why.
2. log error if dfly_bench disconnects before all the requests were processed.
3. Refactor memcache parsing code into ParseMC function.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Now unit tests will run the same Hearbeat fiber like in prod.
The whole feature was redundant, with just few explicit settings of maxmemory_limit
I succeeeded to make all unit tests pass.
In addition, this change allows passing a global handler that is called by heartbeat from a single thread.
This is not used yet - preparation for the next PR to break hung up replication connections on a master.
Finally, this change has some non-functional clean-ups and warning fixes to improve code quality.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. Add background offloading stats
2. remove direct_fd override - helio is already updated with default=false, so it's not needed anymore.
3. remove redundant tiered_storage_memory_margin flag
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* fix(memcached): Register memcached listener to handle `--maxclients`
We add other listeners to `listeners`, but we seem to have forgotten to
do that to the memcached listener.
* protocol() == REDIS
1. Use clib malloc for allocating fiber stacks but reduce the fiber stack size.
clib malloc uses default 4K OS pages when reserving memory from the OS.
The reason for not using mi_malloc, because we use 2MB large OS pages with mimalloc.
However, allocating stacks is one of the cases, when using smaller 4KB memory pages is actually more
RSS efficient because memory pages become hot at better granularity.
2. Add "memory_fiberstack_vms_bytes" metric exposing fiber stack vm usage.
3. Fix macos dependencies & update ci versions.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
We parse the container limits for heuristics that deduces memory/cpu capacities automatically.
It's ok if we fail on some less common systems since there are manual overrides that allows specifying these
limits explicitly.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
The new logrotate settings assume that dragonfly closes a log file
once it grows to large. It never rotates file that is currently open for writing.
Specifically logrotate:
1. rotate only log files
2. skip those that are currently open by as process.
3. compresses using zstd which is more cpu efficient than gzip
4. does not truncate/create old files as 0-sized blobs - just renames them
Fixes#1935
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* Update Http auth with username default instead of user
* skip auth for /metrics page
* add/improve tests
* fix a bug with admin port requiring auth on http even if nopass was set
* update helio ref
* update listener class to contain its respective Role
* fix http init to only include admin and main listener
1. Cherry-pick changes from Redis 7 that encode integer scores more efficiently
2. Introduces optimization that first checks if the new element should be the last
for listpack sorted sets.
3. Consolidates listpack related constants and tightens usage for listpack.
4. Introduce MEMORY USAGE command.
5. Introduce a small delay before decommitting memory pages back to OS.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* flags from env variables
* querying environment vars
* remove includes
* refactor
* exit for unknown flag with DFLY_ prefix
* reflecting change in the test
* better tests
* refactor + new test case
* refactor test with inner class
* refactor
* revert back test flags as it might affect ci/cd
* fixing flags
* refactor
* remove includes
* refactor
1. Do not use IOUring on non-linux systems.
2. Improve compatibility of the code for non-linux systems.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* remove locking the registry when Validating users
* deep copies the acl categories in the connection context
* streamline updates to the acl categories via the proactor pool
* feat(server): Support limiting the number of open connections.
* * Update helio after the small fix was merged to master
* Don't limit admin connections (and add a test case)
* Resolve CR comments
feat: introduce "config set" command
Solves #1551. In detail, we introduce ConfigRegistry that holds
a curated set of flags that could be changed at run-time.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* feat(log): On startup print to stdout info
On dragonfly startup print:
1 Which file we are logging to and where
2. How to get more help with the flags (--help, --helpfull)
3. Doc links
Add support for cgroup v1 limit checking
For future reference: testing. In order to test this feature (both for
v1 and v2). To do this, create a cgroup and move Dragonfly into it
before running.
One way to do this is using a script, like: cat; ./dragonfly
--logtostderr
Enabling v1 (by default, you should have v2): Edit `/etc/default/grub`,
and look for `GRUB_CMDLINE_LINUX_DEFAULT`. Append to it
`systemd.unified_cgroup_hierarchy=0`, update GRUB using `sudo
updat-grub`, and reboot.
In v1, create a group `mycgroup` and add PID 1234 to it:
```
sudo mkdir -p /sys/fs/cgroup/memory/mycgroup
sudo bash -c "echo 8589934592 >
/sys/fs/cgroup/memory/mycgroup/memory.limit_in_bytes" # set mem limit to
8G
sudo bash -c "echo 1234 > /sys/fs/cgroup/memory/mycgroup/tasks" # assign
PID 1234 to this cgroup
```
In v2:
```
sudo mkdir -p /sys/fs/cgroup/mycgroup
sudo bash -c "echo 8589934592 > /sys/fs/cgroup/mycgroup/memory.max" #
set mem limit to 8G
sudo bash -c "echo 1234 > /sys/fs/cgroup/mycgroup/cgroup.procs" # assign
PID 1234 to this cgroup
```
then test by (using script example from before):
```
$ ./run.sh # contains: cat; ./dragonfly --logtostderr
[1] 1234
[1] + 1234 suspended (tty input) ./run.sh
fg
^D
<look for Dragonfly memory limit>
```
Signed-off-by: talbii <ido@dragonflydb.io>
