* Add a coverage workflow to dragonfly.
* Try and make the testing permissive
* Apply suggestions from code review
Signed-off-by: Roy Jacobson <roi.jacobson1@gmail.com>
* Remove redundant parts
---------
Signed-off-by: Roy Jacobson <roi.jacobson1@gmail.com>
* test(server): test transaction locked keys
1. add test utility class that will add suspension to transaction
execution.
2. add test for locked keys in transaction
Signed-off-by: adi_holden <adi@dragonflydb.io>
* feat(server): Add `--lock_on_hashtags` mode.
This new mode effectively locks hashtags (i.e. strings within {curly
braces}) instead of the full keys being used.
This can allow scripts to access undeclared keys if they all use a
common hashtag, like for the case of BullMQ.
To make sure this mode is tested, I added a way to specify flags via env
variables, and modified `ci.yml` to run all tests using this mode as well.
While at it, I also added `--cluster_mode=emulated` mode to CI.
* The bug - if all commands inside multi trasaction are eval commands
and global scripts mode is on, we did ignored the trasaction
and run each eval separately.
*Fix - run all evals under multi inside the global lock
* Change multi eval run only if scripts are in global mode and multi
mode is not non atomic
* Fix test flags setup
* Skip test ContendedList as it fails
* change default exec mode for txs to lock ahead
Currently, During docker release we don't actually build the
alpine release but the new docker run test ends up trying to
run it and fails. This adds the same toggle that we use for
build to prevent the test step.
This commit updates the weekly docker build to use `dragonfly-weekly`
image tag so that we get better separation. We also now push these
images with the github sha commit tags. We also update `latest` as
these get pushed.
It's a bit more efficient than Boost.Fibers due to better integrations
of Fibers with Proactor loop.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
This PR updates the container release workflow to delete the
unstripped packages so that we prevent the stripped packages from
being overwritten.
This should reduce the size of the container images significantly.
This commit adds a scheduled job that runs 8 AM Israel time every day,
with common build configuration flags so that we can be sure
that building from source for known configurations is possible.
Golden files need to be updated as version is updated in the
`Chart.yaml` file. This commit adds the step to update the golden
files in the release pipeline.
To make that possible, The following additional changes are also
required:
- `Go.Work`: This adds support multi-module projects as we will
have the tests along with the operator in sub-folders. This
is required to run go cmds from the root directory. (like
running tests in this case).
- `.helmignore`: This updates the Helm chart to ignore the `/ci`
folder during helm packaging as those are not required for
users of the chart.
This commit adds a new `golden_test.go` file into the helm chart
which essentially renders the chart with a set of values and
compares that with the expected golden file, and errors if
they don't match.
This builds on the existing CI values files in `ci/` directory. As
this is the first time, The golden files are rendered and added. This
means for all the future changes, The golden files can be updated
by running `go test -update` and the CI will fail if the golden files
are not updated. By doing this, Both the committer and reviewer
can be sure that the changes are intentional, without having to render
the chart manually.
feat(server): Basic multi transaction modes
This commit adds the notion of multi transaction modes that allow controlling the execution and
locking behaviour of multi transactions.
In general, there are four modes:
- GLOBAL: all commands run within a global transaction. There is no need for recording locks. Lua scripts can theoretically run with undeclared keys.
- LOCK_AHEAD: the transaction locks all keys ahead likewise to a regular transaction and schedules itself.
- LOCK_INCREMENTAL: the transaction determines what shards it has keys in and schedules itself on those shards, but locks only when accessing a new key. This allows other transactions to run ooo alonside with a big multi-transaction that accesses a contended key only at its very end.
- NON_ATOMIC: all commands run separately, no atomicity is provided, likewise to a pipeline
This commit only adds support for the first 3 modes to EXEC commands.
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
* regression replication will not run on github schedual
* allow running regression replication manually
Signed-off-by: adi_holden <adi@dragonflydb.io>
The scenario is described in a unit test that reproduces the issue with high chance.
Also added dragonfly_test in repeat=100 mode to CI.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
This should greatly improve readability of the chart itself
- moved the `Pod` spec for both the `Deployment` and `StatefulSet` into a common template in `_pod.yaml`
- replace a bunch of `if $value; print $value`-type blocks with `with $value; print .`
- replaced `command.set` in `values.yaml` with `command` directly
- this was broken anyways, as the chart wrongly referenced `command.cmd` for both `Deployment` and `StatefulSet`
- populated contrib/charts/dragonfly/ci/ folder for development/CI purposes
Signed-off-by: Philipp Born <git@pborn.eu>
Signed-off-by: Philipp Born <git@pborn.eu>
- make use of docker buildx caching when possible (helpful with local docker builds)
- introduce a reusable container workflow which is triggered by docker-release and docker-weekly workflows
- added an alpine-dev Dockerfile
- split release.sh contents into different Makefile targets
- make use of job matrix to build alpine + ubuntu in parallel
- make alpine build optional by checking for Dockerfile presence
-- as the pre-built binaries don't work with alpine, because of glibc <-> musl incompatibilities
Signed-off-by: Philipp Born <git@pborn.eu>
Adding a regression pipeline
A dedicated github actions pipeline that build dragonfly and runs pytests located under "tests/dragonfly" every 3 days
Fixes#291
Signed-off-by: zNNiz <96877065+zNNiz@users.noreply.github.com>
Docker release pipeline can not push to protected main branch using
the service token. This switches to PAT secret.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. Reads from external storage support now o_direct mode.
2. Simplify write unloading logic. Make pending buffer a ring buffer with
a predefined capacity.
3. Add more tiered stats to info command
1. Docker build now builds for arm64 as well.
2. Add bind option to specify on which interface the server should listen.
3. Automatically deduce maxmemory setting.
1. Basic support of sets and hsets. No intset or dict-based hset yet.
2. Make sure that rdbloader actually loads the data into shards.
3. Support object metadata like expiry time.
4. Support lists loading.
