* chore: export pipeline related metrics
Export in /metrics
1. Total pipeline queue length
2. Total pipeline commands
3. Total pipelined duration
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. Restrict build context in our dev/weekly builder to ease development iterations.
2. Switch weekly build to debian 12-slim because it's smaller than 24.04
3. Update our prod releases to use ubuntu 22.04
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
The number of keys in an _incoming_ migration indicates how many keys
were received, while for _outgoing_ it shows the total number. Combining
the two can provide the control plane with percentage.
This slightly modified the format of the response.
Fixes#2756
* feat(server): Use mimalloc in SSL calls
Until now, OpenSSL used `malloc()` directly. This PR overrides it to use
mimalloc.
Fixes#2709
* Add generate-tls-files.sh
* WIP: `cluster_mgr.py` to work with remote targets
* Documentation
* No admin port
* Support different hostname move/migrate
* Fix migrate bug
* Fix typo in --help
* fix test
* self.update_id()
Example usage:
```bash
# Create a 2-node cluster
./cluster_mgr.py --action=create --replicas_per_master=1 --num_master=2
# Move (no migration) all slots to first node
./cluster_mgr.py --action=move --target_port=7001 --slot_start=8192 --slot_end=16383
# Fill data - like run memtier
# Migrate all slots to 2nd node. One could measure how long this step takes.
./cluster_mgr.py --action=migrate --target_port=7002 --slot_start=0 --slot_end=16383
```
We had a place in tools/packaging/generate_debian_package.sh that relied on the existence of build-opt,
moreover, if it did not exist the script deadlocked.
1. Added more loggings
2. Removed the loop
3. Removed unnecessary dependency in the script on the build-dir name.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore: fix our release pipeline
Also remove alpine prod.wip file that has not been used and unlikely will be for prod.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Should allow track caches where Dragonfly is not responsive to I/O
due to big CPU tasks. Also, update the local grafana dashboard.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
The dashboard used `dragonfly_up` metric to boostrap itself
but this metric does not exist anymore. I replaced it with `dragonfly_version`
In addition, the exported format changed slightly because I used the
recent grafana version to export.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* fixes#1936
Eviction Implementation
This patch provides a very simple eviction implementation for the interface mentioned above. In my opinion, the eviction algorithm approximates an LRU policy given that normal buckets always store the most recently accessed data while stash buckets are holding less active data.
The algorithm first selects a small set of segments as eviction targets. Starting from the last slot of the last stash bucket in each of the segments, we walk backward to evict key-value pairs stored in each visited slot. The eviction stopped either when a target memory release goal or the max number of evicted key-value pairs is reached. Therefore, we can upper bound the eviction time through the following two parameters that can be set when DF starts. Note that these two parameters could be retrieved and changed by user through CONFIG GET and CONFIG SET commands.
---------
Signed-off-by: Yue Li <61070669+theyueli@users.noreply.github.com>
The new logrotate settings assume that dragonfly closes a log file
once it grows to large. It never rotates file that is currently open for writing.
Specifically logrotate:
1. rotate only log files
2. skip those that are currently open by as process.
3. compresses using zstd which is more cpu efficient than gzip
4. does not truncate/create old files as 0-sized blobs - just renames them
Fixes#1935
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. Move docker build files to separate dir from docker script files
so that they won't be part of build context. Update dockerignore as well
2. Fix lib dependencies for alpine
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Specifiying an exact boost version is not robust.
Also we do not depend on fibers anymore and boost-context is enough.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Now this management script can:
* Create a cluster (before this PR)
* Print an existing cluster configuration
* Shutdown an existing cluster
* Move slots between cluster nodes
To support connecting to a cluster (for all new functions), I had to
change the way admin ports are defined. Instead of having the user
(optionally) specify the first port, they are hard-coded to be the
regular port + 10,000. This is done because we can't detect the admin
port based for an existing cluster (like via `CLUSTER SHARDS`).
This script allows easily setting up a local cluster.
Example invocation:
```
killall dragonfly; ./cluster_mgr.py --num_masters=3 --with_replicas
Setting up a Dragonfly cluster:
- Master nodes: 3
- Ports: 7001...7003
- Admin ports: 8001...8003
- Replicas? True
Starting nodes...
- Log file for node 7001: /tmp/dfly.cluster.node.7001.log
- Log file for node 7002: /tmp/dfly.cluster.node.7002.log
- Log file for node 7003: /tmp/dfly.cluster.node.7003.log
- Log file for node 7004: /tmp/dfly.cluster.node.7004.log
- Log file for node 7005: /tmp/dfly.cluster.node.7005.log
- Log file for node 7006: /tmp/dfly.cluster.node.7006.log
Configuring replication...
- Response for 7004: OK
- Response for 7005: OK
- Response for 7006: OK
Getting IDs...
- ID for 7001: acefdc2da5d397cfcb99239b3c29cbe6ff10d75a
- ID for 7002: a8cc67dfa42e91a94bd7c0903df35d60a39508bd
- ID for 7003: 1ad91af7bd96c89a8da877164b2ebb4cf458cab8
- ID for 7004: d209c3603343e25a18c78bd68304b6d883973bd3
- ID for 7005: bd2b25e95aaf7fdd2b955e50a00093a8272954bf
- ID for 7006: beb5cb07b75c33e3ff938d07725f2688d9bc91e0
Pushing config...
- Push into 7001: OK
- Push into 7002: OK
- Push into 7003: OK
- Push into 7004: OK
- Push into 7005: OK
- Push into 7006: OK
```
