1. Use clib malloc for allocating fiber stacks but reduce the fiber stack size.
clib malloc uses default 4K OS pages when reserving memory from the OS.
The reason for not using mi_malloc, because we use 2MB large OS pages with mimalloc.
However, allocating stacks is one of the cases, when using smaller 4KB memory pages is actually more
RSS efficient because memory pages become hot at better granularity.
2. Add "memory_fiberstack_vms_bytes" metric exposing fiber stack vm usage.
3. Fix macos dependencies & update ci versions.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Also update actions versions to Node 20.
This change allows dragonfly to be built on MacOs.
However, we still have multiple failing tests on MacOS.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* upload only failed test logs
* remove printing log names for passed tests
* print slow tests with --duration
* separate regression and unit logs for CI workflow
* test(memory): Test memory accounting for all types
* slightly faster
* WIP
* working
* Document
* Update test to use DEBUG POPULATE
* Nothing much
* Working
* fix
* yaml
* explicit capture
* fix ci?
* stub tx
chore: UpdateHelio dependency
Add support for asan/ubsan checkers in our dev environment.
Remove more clang warnings.
Once we fix all the problems we will enable them in our CI as well.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
This is needed if we want to allow asynchronous transactional operations during the callback execution.
Also update actions versions.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
We had a place in tools/packaging/generate_debian_package.sh that relied on the existence of build-opt,
moreover, if it did not exist the script deadlocked.
1. Added more loggings
2. Removed the loop
3. Removed unnecessary dependency in the script on the build-dir name.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore: fix our release pipeline
Also remove alpine prod.wip file that has not been used and unlikely will be for prod.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Also, change its build directory to build-release.
Simplify a bit its configuration steps as well. No change in functionality is expected.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
chore: run dragonfly_test with epoll under gdb
Also, update helio that provide a stacktrace under musl libc (alpine linux).
This version of helio updates absl version as well.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* add a python script to print the most recent log
* if CI timeouts, print the most recent log
* replace global timeout with timeout command
* upload all logs on failure()
* print uid + port + the log files for each df instance
* chore(regression): test bptree on regression pytests
1. stop passing the flag use_zset_tree as it is true on default
2. fix ci test to run replication tests
3. change replication tests seeder to sometimes add more than 128 values
to zset to test the pbtree impl
Signed-off-by: adi_holden <adi@dragonflydb.io>
1. Move docker build files to separate dir from docker script files
so that they won't be part of build context. Update dockerignore as well
2. Fix lib dependencies for alpine
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
fix: use "cd" instead of woarking-directory that does not work
Also, use GITHUB_WORKSPACE due to https://github.com/actions/runner/issues/2058
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* Add a coverage workflow to dragonfly.
* Try and make the testing permissive
* Apply suggestions from code review
Signed-off-by: Roy Jacobson <roi.jacobson1@gmail.com>
* Remove redundant parts
---------
Signed-off-by: Roy Jacobson <roi.jacobson1@gmail.com>
* test(server): test transaction locked keys
1. add test utility class that will add suspension to transaction
execution.
2. add test for locked keys in transaction
Signed-off-by: adi_holden <adi@dragonflydb.io>
* feat(server): Add `--lock_on_hashtags` mode.
This new mode effectively locks hashtags (i.e. strings within {curly
braces}) instead of the full keys being used.
This can allow scripts to access undeclared keys if they all use a
common hashtag, like for the case of BullMQ.
To make sure this mode is tested, I added a way to specify flags via env
variables, and modified `ci.yml` to run all tests using this mode as well.
While at it, I also added `--cluster_mode=emulated` mode to CI.
* The bug - if all commands inside multi trasaction are eval commands
and global scripts mode is on, we did ignored the trasaction
and run each eval separately.
*Fix - run all evals under multi inside the global lock
* Change multi eval run only if scripts are in global mode and multi
mode is not non atomic
* Fix test flags setup
* Skip test ContendedList as it fails
* change default exec mode for txs to lock ahead
Currently, During docker release we don't actually build the
alpine release but the new docker run test ends up trying to
run it and fails. This adds the same toggle that we use for
build to prevent the test step.
This commit updates the weekly docker build to use `dragonfly-weekly`
image tag so that we get better separation. We also now push these
images with the github sha commit tags. We also update `latest` as
these get pushed.
It's a bit more efficient than Boost.Fibers due to better integrations
of Fibers with Proactor loop.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
This PR updates the container release workflow to delete the
unstripped packages so that we prevent the stripped packages from
being overwritten.
This should reduce the size of the container images significantly.
This commit adds a scheduled job that runs 8 AM Israel time every day,
with common build configuration flags so that we can be sure
that building from source for known configurations is possible.
Golden files need to be updated as version is updated in the
`Chart.yaml` file. This commit adds the step to update the golden
files in the release pipeline.
To make that possible, The following additional changes are also
required:
- `Go.Work`: This adds support multi-module projects as we will
have the tests along with the operator in sub-folders. This
is required to run go cmds from the root directory. (like
running tests in this case).
- `.helmignore`: This updates the Helm chart to ignore the `/ci`
folder during helm packaging as those are not required for
users of the chart.
This commit adds a new `golden_test.go` file into the helm chart
which essentially renders the chart with a set of values and
compares that with the expected golden file, and errors if
they don't match.
This builds on the existing CI values files in `ci/` directory. As
this is the first time, The golden files are rendered and added. This
means for all the future changes, The golden files can be updated
by running `go test -update` and the CI will fail if the golden files
are not updated. By doing this, Both the committer and reviewer
can be sure that the changes are intentional, without having to render
the chart manually.
feat(server): Basic multi transaction modes
This commit adds the notion of multi transaction modes that allow controlling the execution and
locking behaviour of multi transactions.
In general, there are four modes:
- GLOBAL: all commands run within a global transaction. There is no need for recording locks. Lua scripts can theoretically run with undeclared keys.
- LOCK_AHEAD: the transaction locks all keys ahead likewise to a regular transaction and schedules itself.
- LOCK_INCREMENTAL: the transaction determines what shards it has keys in and schedules itself on those shards, but locks only when accessing a new key. This allows other transactions to run ooo alonside with a big multi-transaction that accesses a contended key only at its very end.
- NON_ATOMIC: all commands run separately, no atomicity is provided, likewise to a pipeline
This commit only adds support for the first 3 modes to EXEC commands.
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
* regression replication will not run on github schedual
* allow running regression replication manually
Signed-off-by: adi_holden <adi@dragonflydb.io>
