Commit Graph

2840 Commits

Author SHA1 Message Date
adiholden
db67b35f8e
fix server: fix stats of pipeline squashed commands (#4132)
Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-11-14 20:57:05 +02:00
dependabot[bot]
a887d822a9
chore(deps): bump github/codeql-action from 3.27.0 to 3.27.1 in the actions group (#4115)
chore(deps): bump github/codeql-action in the actions group

Bumps the actions group with 1 update: [github/codeql-action](https://github.com/github/codeql-action).


Updates `github/codeql-action` from 3.27.0 to 3.27.1
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](662472033e...4f3212b617)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-11-14 19:28:25 +02:00
Shahar Mike
1513134247
fix: Use MOVED error type for moved replies (#4125)
**The problem:**

When in cluster mode, `MOVED` replies (which are arguably not even errors) are aggregated per slot-id + remote host, and displayed in `# Errorstats` as such. For example, in a server that does _not_ own 8k slots, we will aggregate 8k different errors, and their counts (in memory).

This slows down all `INFO` replies, takes a lot of memory, and also makes `INFO` replies very long.

**The fix:**

Use `type` `MOVED` for moved replies, making them all the same under `# Errorstats`

Fixes #4118
2024-11-14 13:12:38 +02:00
Shahar Mike
f04e5497bc
fix: Do not use cc_ in connection if it's null (#4131)
* fix: Do not use `cc_` in connection if it's null

This is a rare condition, which we know can happen during shutdown (see
[here](https://github.com/dragonflydb/dragonfly/pull/3873#issue-2568503374))

* add comment
2024-11-14 12:41:51 +02:00
Roman Gershman
ab6088f5d6
chore: simplify BumpUps deduplication (#4098)
* chore: simplify BumpUps deduplication

This pr #2474 introduced iterator protection by
tracking which keys where bumped up during the transaction operation.
This was done by managing keys view set. However, this can be simplified
using fingerprints. Also, fingerprints do not require that the original keys exist.

In addition, this #3241 PR introduces FetchedItemsRestorer that tracks bumped set and
saves it to protect against fiber context switch. My claim is that it's redundant.
Since we only keep the auto-laundering iterators, when a fiber preempts these iterators recognize it
(see IteratorT::LaunderIfNeeded) and refresh themselves anyway.

To summarize: fetched_items_ protect us from iterator invalidation during atomic scenarios,
and auto-laundering protects us from everything else, so fetched_items_ can be cleared in that case.
---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-11-13 17:27:53 +02:00
adiholden
fb84d47b4d
feat server: experimental_new_io flag add as deprecated (#4127)
* feat server: experimental_new_io flag add as deprecated

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-11-13 13:29:40 +00:00
Borys
4bc9ad6f01
test: add test for snapshoting during migration (#4108)
* test: add test for snapshotting during migration

* test: add test to run replication after migration
2024-11-13 13:40:00 +02:00
Kostas Kyrimis
91c236ab2f
fix: slow CI tests (#4117)
* refactor test_big_value_serialization
* remove no needed replication tests for big value

---------

Signed-off-by: kostas <kostas@dragonflydb.io>
2024-11-13 10:32:31 +02:00
romange
7815c8d3d7 chore(helm-chart): update to v1.25.1 2024-11-12 08:04:45 +00:00
Roman Gershman
fa8f3f5564
fix: regression in squashing code when determining eval commands (#4116)
The regression was caused by #3947 and it causes crashes in bullmq.
It has not been found till now because python client sends commands in uppercase.
Fixes #4113

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Co-authored-by: Kostas Kyrimis <kostas@dragonflydb.io>
2024-11-11 19:54:47 +00:00
Kostas Kyrimis
0facd6fd8f
fix: skip Send() in SinkReplyBuilder::Flush() when vec is empty (#4114)
* skip Send() in SinkReplyBuilder::Flush() when vec is empty

Signed-off-by: kostas <kostas@dragonflydb.io>
2024-11-11 17:53:30 +00:00
adiholden
3ab244616a
feat(server) : snapshot traverse physical buckets (#4084)
Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-11-11 14:49:20 +02:00
Roman Gershman
79aa5d490d
chore: support rdb loading and container utils with QList. (#4109)
generic_family_test and rdb_test pass with qlist enabled.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-11-11 12:13:10 +02:00
romange
745db2c82f chore(helm-chart): update to v1.25.0 2024-11-11 07:48:44 +00:00
Roman Gershman
50473b56aa
chore: decouple reply_builder from ConnectionContext (#4069)
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-11-10 21:14:24 +02:00
Roman Gershman
9b7af7d750
chore: implement Erase for a range (#4106)
chore: implement Erase with a range

Also migrate more unit tests from valkey repo.
Finally, fix OpTrim

All tests `list_family_test --list_experimental_v2` pass.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>

chore: implement OpTrim with QList
2024-11-10 16:33:34 +00:00
Stepan Bagritsevich
503bb4ed33
fix(search_family): Process wrong field types in indexes for the FT.SEARCH and FT.AGGREGATE commands (#4070)
* fix(search_family): Process wrong field types in indexes for the FT.SEARCH and FT.AGGREGATE commands

fixes  #3986
---------

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>
2024-11-10 16:56:25 +02:00
Roman Gershman
f745f3133d
chore: Implement LMOVE over QList (#4104)
* chore: Implement LMOVE over QList

All tests in list_family_test besides LTrim pass.

---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-11-10 13:47:01 +00:00
Roman Gershman
1eef773d0a
fix: test_noreply_pipeline flakiness (#4102)
Fixes #3896. Now we retry several times.
In my checks this should significantly reduce the failure probability.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-11-10 13:39:24 +02:00
Borys
ab86d358b0
fix: update instruction how to build from the source for ubuntu (#4103) 2024-11-10 11:21:07 +00:00
Roman Gershman
be96e6cf99
chore: change Namespaces to be a global pointer (#4032)
* chore: change Namespaces to be a global pointer

Before the namespaces object was defined globally.
However it has non-trivial d'tor that is being called after main exits.
It's quite dangerous to have global non-POD objects being defined globally.
For example, if we used LOG(INFO) inside the Clear function , that would crash dragonfly on exit.

Ths PR changes it to be a global pointer.

---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-11-10 10:45:53 +00:00
Roman Gershman
9366c67464
chore: Implement list Pop/Erase functionality with QList (#4099)
Adjusted OpRem/OpBPop/OpPop

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-11-10 12:37:42 +02:00
Roman Gershman
c43ba5f2cc
chore: RdbSerializer::SaveListObject supports QList (#4101)
Now, `./rdb_test --list_experimental_v2` passes.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-11-10 12:35:24 +02:00
Vladislav
eadce55b67
chore: remove old io (#3953)
* chore: Remove old IO

* fix: fix last error accounting
* chore: use unique_ptr<char> in MGetResponse storage

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-11-10 11:56:41 +02:00
adiholden
2d49a28c15
fix(server): handle running script load inside multi (#4074)
Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-11-10 09:34:40 +02:00
Roman Gershman
75c961e7ed
chore: Add initial bindings for QList in list_family (#4093)
The feature is guarded by list_experimental_v2 flag, which is disabled.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-11-10 07:40:50 +02:00
Roman Gershman
1819e51f78
chore: implement QList::Erase functionality (#4092)
* chore: implement QList::Erase functionality

    Also add a parametrized test covering fill/compress options.
    Fix a compression bug introduced when copying the code.
    Introduce move c'tor/operator.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>

* chore: implement QList::Replace functionality

Also add a parametrized test covering fill/compress options.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>

---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-11-09 20:06:32 +02:00
Kostas Kyrimis
d3ef0a3630
fix(sanitizers): disable false positive FtProfileInvalidQuery (#4090)
* disable FtProfileInvalidQuery 

Signed-off-by: kostas <kostas@dragonflydb.io>
2024-11-08 18:46:54 +02:00
Roman Gershman
5838ed99b4
chore: implement QList::Insert function (#4087)
* chore: implement QList::Insert function

Also add a basic test for Insertion.
Prepare the skeleton for Replace functionality.
---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-11-08 11:33:37 +00:00
Vladislav
b860b712c0
chore(acl): Implicit categories (#3411)
Most of our CO:: categories became meaningless with the introduction of acl. For example, CO::FAST literally doesn't mean anything, it's never read or used.

* add implicit categories

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-11-08 12:40:56 +02:00
Roman Gershman
c9537bb52e
chore: implement QList::Iterate (#4086)
Also, copy the main flow for the Insert function.
It's not finished yet, hence no tests yet.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-11-08 11:19:54 +02:00
Blanky
43060ba79e
chore(dependabot): Allows only security patches (#4075)
* feat(dependabot): Add dependabot to check for vulnerabilies and update dependencies

---------

Signed-off-by: BLANKatGITHUB <131886247+BLANKatGITHUB@users.noreply.github.com>
Signed-off-by: Blanky <131886247+BLANKatGITHUB@users.noreply.github.com>
2024-11-08 08:08:33 +02:00
Roman Gershman
41d8df6e11
Add index based iterator to QList interface (#4083)
chore: add index based iterator to the interface

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-11-07 19:26:57 +00:00
Roman Gershman
2794239827
chore: Add QList::Iterator (#4082)
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-11-07 20:51:09 +02:00
Roman Gershman
d5a0ce4004
chore: add push functionality to QList class (#4079)
The code is copied from quicklist.c
No functional changes.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-11-07 12:14:42 +02:00
Roman Gershman
c75683277b
chore: add QList class (#4078)
This class will consolidate list functionality across multiple layers, and eventually replace the original quicklist code.
Right now we added the skeleton files and defined most of the interface.
The implementation is not done yet.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-11-07 11:13:12 +02:00
Roman Gershman
c19af2bd43
chore: parser clean ups (#4077)
1. Eliminate redundant states
2. Eliminate redundant member variables

Added some comments. No functional changes.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-11-07 10:07:48 +02:00
Roman Gershman
f8b3fa0d7b
feat: support GCS storage for saving/loading snapshots (#4064) 2024-11-06 12:11:59 +02:00
adiholden
ae3faf59fb
feat(server): dont use channel for replication / save df (#4041)
* feat server: dont use channel for replication / save df

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-11-05 16:50:01 +02:00
Roman Gershman
7df8c268d8
chore: eliminate redundant ConnectionContext arguments (#4065) 2024-11-05 10:40:04 +02:00
dependabot[bot]
54c67a9198
chore(deps): bump pytest-repeat from 0.9.1 to 0.9.3 in /tests/dragonfly (#4057)
Bumps [pytest-repeat](https://github.com/pytest-dev/pytest-repeat) from 0.9.1 to 0.9.3.
- [Release notes](https://github.com/pytest-dev/pytest-repeat/releases)
- [Changelog](https://github.com/pytest-dev/pytest-repeat/blob/main/CHANGES.rst)
- [Commits](https://github.com/pytest-dev/pytest-repeat/compare/v0.9.1...v0.9.3)

---
updated-dependencies:
- dependency-name: pytest-repeat
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-11-04 23:47:24 +02:00
dependabot[bot]
aa4a61894d
chore(deps): bump docker/build-push-action from 5 to 6 (#4055)
Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 5 to 6.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v5...v6)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-11-04 22:38:06 +02:00
dependabot[bot]
d6ec5910d1
chore(deps): bump github/codeql-action from 3.24.9 to 3.27.0 (#4054)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.24.9 to 3.27.0.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](1b1aada464...662472033e)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-11-04 22:37:51 +02:00
dependabot[bot]
8dc12a39e5
chore(deps): bump ossf/scorecard-action from 2.3.1 to 2.4.0 (#4053)
Bumps [ossf/scorecard-action](https://github.com/ossf/scorecard-action) from 2.3.1 to 2.4.0.
- [Release notes](https://github.com/ossf/scorecard-action/releases)
- [Changelog](https://github.com/ossf/scorecard-action/blob/main/RELEASE.md)
- [Commits](0864cf1902...62b2cac7ed)

---
updated-dependencies:
- dependency-name: ossf/scorecard-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-11-04 22:37:07 +02:00
dependabot[bot]
f67103f9b0
chore(deps): bump mozilla-actions/sccache-action from 0.0.4 to 0.0.6 (#4051)
Bumps [mozilla-actions/sccache-action](https://github.com/mozilla-actions/sccache-action) from 0.0.4 to 0.0.6.
- [Release notes](https://github.com/mozilla-actions/sccache-action/releases)
- [Commits](https://github.com/mozilla-actions/sccache-action/compare/v0.0.4...v0.0.6)

---
updated-dependencies:
- dependency-name: mozilla-actions/sccache-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-11-04 22:36:20 +02:00
dependabot[bot]
18f7d9bbd5
chore(deps): bump mikepenz/action-junit-report from 4 to 5 (#4052)
Bumps [mikepenz/action-junit-report](https://github.com/mikepenz/action-junit-report) from 4 to 5.
- [Release notes](https://github.com/mikepenz/action-junit-report/releases)
- [Commits](https://github.com/mikepenz/action-junit-report/compare/v4...v5)

---
updated-dependencies:
- dependency-name: mikepenz/action-junit-report
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-11-04 22:35:47 +02:00
dependabot[bot]
94ee96afd0
chore(deps): bump redis-om from 0.2.2 to 0.3.3 in /tests/dragonfly (#4060)
Bumps [redis-om](https://github.com/redis/redis-om-python) from 0.2.2 to 0.3.3.
- [Release notes](https://github.com/redis/redis-om-python/releases)
- [Commits](https://github.com/redis/redis-om-python/compare/v0.2.2...v0.3.3)

---
updated-dependencies:
- dependency-name: redis-om
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-11-04 22:35:18 +02:00
dependabot[bot]
eedb04509d
chore(deps): bump github.com/gruntwork-io/terratest from 0.41.11 to 0.47.2 in /contrib/charts/dragonfly (#4058)
chore(deps): bump github.com/gruntwork-io/terratest

Bumps [github.com/gruntwork-io/terratest](https://github.com/gruntwork-io/terratest) from 0.41.11 to 0.47.2.
- [Release notes](https://github.com/gruntwork-io/terratest/releases)
- [Commits](https://github.com/gruntwork-io/terratest/compare/v0.41.11...v0.47.2)

---
updated-dependencies:
- dependency-name: github.com/gruntwork-io/terratest
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-11-04 22:34:48 +02:00
dependabot[bot]
b70f9703f4
chore(deps): bump tomli from 2.0.1 to 2.0.2 in /tests/dragonfly (#4059)
Bumps [tomli](https://github.com/hukkin/tomli) from 2.0.1 to 2.0.2.
- [Changelog](https://github.com/hukkin/tomli/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hukkin/tomli/compare/2.0.1...2.0.2)

---
updated-dependencies:
- dependency-name: tomli
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-11-04 22:31:29 +02:00
dependabot[bot]
86b64d910a
chore(deps): bump github.com/redis/go-redis/v9 from 9.5.1 to 9.7.0 in /tools/replay (#4062)
chore(deps): bump github.com/redis/go-redis/v9 in /tools/replay

Bumps [github.com/redis/go-redis/v9](https://github.com/redis/go-redis) from 9.5.1 to 9.7.0.
- [Release notes](https://github.com/redis/go-redis/releases)
- [Changelog](https://github.com/redis/go-redis/blob/master/CHANGELOG.md)
- [Commits](https://github.com/redis/go-redis/compare/v9.5.1...v9.7.0)

---
updated-dependencies:
- dependency-name: github.com/redis/go-redis/v9
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-11-04 22:31:01 +02:00