Commit Graph

1512 Commits

Author SHA1 Message Date
Andy Dunstall
f008965527
feat(aws): s3 snapshot error handling (#2002)
* feat(s3): support bucket only dir

* feat(s3): improve error handling

* chore: update helio
2023-10-10 13:03:46 +01:00
Shahar Mike
b1bd2103d7
opt(server): If unspecified, use num_shards == proactor_threads (#1998)
* opt(server): If unspecified, use num_shards == proactor_threads

* Use same config for tests, per Roman's request
2023-10-10 14:12:39 +03:00
Kostas Kyrimis
cbba6f4e11
fix(acl/admin_port): skip acl validation on admin port (#1997)
* skip all acl related functionality on admin port
2023-10-10 08:28:07 +03:00
Roman Gershman
6e76f8e6cc
fix: logrotate for dragonfly logs (#1972)
The new logrotate settings assume that dragonfly closes a log file
once it grows to large. It never rotates file that is currently open for writing.

Specifically logrotate:

1. rotate only log files
2. skip those that are currently open by as process.
3. compresses using zstd which is more cpu efficient than gzip
4. does not truncate/create old files as 0-sized blobs - just renames them

Fixes #1935

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2023-10-09 20:08:37 +03:00
Roman Gershman
2876a9ecab
fix: make --help flag work for development builds (#2004)
Before that SOURCE_PATH_FROM_BUILD_ENV define has not been defined
for the development builds.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2023-10-09 19:44:14 +03:00
Yue Li
17e331c16c
feat(streams): support XAUTOCLAIM command (#1969)
fixes: #1899

---------

Signed-off-by: Yue Li <61070669+theyueli@users.noreply.github.com>
2023-10-09 06:59:22 -07:00
Kostas Kyrimis
5471827448
update(requirepass): behave as redis requirepass (#1996)
* requirepass also updates ACL default user password
* update config set requirepass to include the new behaviour
* add tests
* fix non existent default user when loading empty files
2023-10-09 16:34:42 +03:00
Rounak Nandanwar
eb50741855
feat: add support for XINFO STREAM stream_name (#1533)
* feat: add support for XINFO STREAM stream_name

Signed-off-by: rounaknandanwar <rounak.nandanwar@gmail.com>
Co-authored-by: rounak.nandanwar <rounak.nandanwar@gojek.com>
2023-10-09 13:33:45 +03:00
Vladislav
ee68f1bccc
fix: fix queue backpressure access (#2000)
* fix: fix queue backpressure access


---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-10-07 13:07:27 +03:00
Vladislav
60ec6a3a27
fix: fix eval in multi interpreter borrow (#1999)
* fix: fix eval in multi interpreter borrow

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-10-07 13:07:06 +03:00
Vladislav
e84d9a65d8
fix(server): Add additional metrics (#1975)
* fix(server): Clean up metrics collection
* feat(server): Replication memory metrics
* fix(server): Limit dispatch queue size

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-10-06 14:16:22 +03:00
Andy Dunstall
2d28b48481
feat(aws): add s3 awsv2 (#1929)
* feat(aws): add s3 awsv2

* feat(aws): add s3 snapshot test

* feat(aws): disable ec2 metadata by default

* feat(aws): add s3 disable payload signing flag

* chore: update helio

* fix: fix requirements.txt

* feat(s3): update sign payload flag

* chore: update helio
2023-10-06 10:24:56 +01:00
Kostas Kyrimis
0c1402c4ab
fix(regTests): test_cancel_replication_immediately races (#1990) 2023-10-06 09:38:14 +03:00
Vladislav
f3c07174aa
feat(search): Plug in polymorphic allocator (#1819)
* feat(search): Plug in polymorphic allocator

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-10-05 18:09:11 +03:00
Kostas Kyrimis
e98d5e60c8
fix(regTests): adjust small network buff test (#1991)
* Seeder runs infinitely
2023-10-05 16:37:40 +03:00
Ujjwal gupta
f69a7693f5
Docs: fix typos and grammar (#1983) 2023-10-04 14:11:10 +03:00
Yue Li
06dc497bf0
feat(streams): support LASTID option for XCLAIM command. (#1968)
fixes #1898

here's the spec of last_id option from Redis

 *  * 6. LASTID <id>:
 *      Update the consumer group last ID with the specified ID if the
 *      current last ID is smaller than the provided one.
 *      This is used for replication / AOF, so that when we read from a
 *      consumer group, the XCLAIM that gets propagated to give ownership
 *      to the consumer, is also used in order to update the group current
 *      ID.
2023-10-03 12:50:41 -07:00
Kostas Kyrimis
8b7a43d214
fix(Listener): skip auth when requirepass is empty on http (#1985)
* Fix a small bug on http when username was required even if requirepass was empty
2023-10-03 19:56:19 +03:00
Vladislav
e15900fd94
chore: DEBUG SHARDS (#1987)
* chore: DEBUG SHARDS

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>

* fix: fixes

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-10-03 15:57:35 +00:00
Vladislav
1ea0c39d6a
fix(server): Fix async lua (#1982)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-10-03 16:13:38 +03:00
omahs
d16195bfb7
fix: typos (#1986) 2023-10-03 13:05:04 +00:00
Vladislav
20b924f9d5
fix(regtests): Reduce load on redis replication test (#1984)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-10-03 12:28:42 +03:00
Kostas Kyrimis
945d3a39e4
chore(Http): change authed username to default and skip auth on metrics (#1950)
* Update Http auth with username default instead of user
* skip auth for /metrics page
* add/improve tests
* fix a bug with admin port requiring auth on http even if nopass was set
* update helio ref
* update listener class to contain its respective Role
* fix http init to only include admin and main listener
2023-10-03 10:45:37 +03:00
Kostas Kyrimis
38c4e3b588
chore(dfly_main): refactor to use RAII (#1981) 2023-10-02 20:47:38 +03:00
Kostas Kyrimis
59dd48d47a
fix(RegTests): fix replication test_network_disconnect_active_stream (#1980)
* The issue was similar with test `network_disconnect_small_buffers` but this time the debug build could be slow enough for the replication to not finish. As a consequence, by the time the test reached the assertion, the log did not contain the expected output.
2023-10-02 17:35:25 +03:00
adiholden
84d4ba4d69
chore(regression): test bptree on regression pytests (#1963)
* chore(regression): test bptree on regression pytests

1. stop passing the flag use_zset_tree as it is true on default
2. fix ci test to run replication tests
3. change replication tests seeder to sometimes add more than 128 values
   to zset to test the pbtree impl

Signed-off-by: adi_holden <adi@dragonflydb.io>
2023-10-02 14:07:50 +00:00
Roman Gershman
57441f9863
chore: Print useful info when printing DflyInstance objects (#1977) 2023-10-02 12:02:46 +03:00
Vladislav
bc48bed6ad
fix(regtests): Colored per-instance log (#1971)
* fix(regtests): Colored per-instance log

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-10-02 10:36:41 +03:00
Vladislav
ee8a661e24
feat(search): Basic sorting (#1941)
* feat(search): Basic sorting

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-10-01 20:27:26 +03:00
Roman Gershman
4be9551806
chore: adjust verbosity level 1 not be over-verbose in prod (#1974)
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2023-10-01 14:31:39 +03:00
Roman Gershman
43d55fa6d7
chore: rename owner to conn (#1973)
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2023-10-01 10:56:18 +03:00
Andy Dunstall
e6b8cd1d76
feat(cmd): add restricted commands flag (#1967) 2023-09-29 16:16:06 +01:00
Kostas Kyrimis
ba4eba991d
fix(regTests): adjust network disconnect small buffer to pass on (#1970) 2023-09-29 16:50:01 +03:00
Roman Gershman
017b931eba
chore: preallocate lua arrays (#1964)
saves 1-2% due to SetGlobalArrayInternal calls with lots of arguments/keys.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2023-09-28 14:09:05 +03:00
Roy Jacobson
4ecab6e786
Add a feature flag for partial_sync (#1957) 2023-09-28 13:15:05 +03:00
Roy Jacobson
7ad29ab919
fix(regtests): Limit infinite loop in replication_test (#1961) 2023-09-28 11:45:03 +03:00
Yue Li
a2457e3410
feat(streams): support entries_read and lag for XINFO GROUPS (#1952)
entries_read and lag have been added to the output of XINFO GROUPS since Redis 7.0. This patch supports both for Dragonfly. This patch also fixes a bug that incorrectly sets the initial value of entries_read when a consumer group is created.

fixes #1948
2023-09-28 01:35:19 -07:00
Roy Jacobson
5c9c9255d2
chore: Small refactor of DflyInstance (#1951)
* Move to its own file
* Unify self.args and self.params.args earlier so it can be inspected.
2023-09-28 10:11:11 +03:00
Kostas Kyrimis
949cedf66c
fix(AclFamily): do not allow to delete default user (#1954)
* do not allow to delete default user
* upon loading an acl file, if default does not exist create them
* add test
2023-09-28 09:22:45 +03:00
Kostas Kyrimis
8dd34b9ce3
fix: small typos on flags (#1953)
* Polish small typos on flags
2023-09-27 15:02:04 +00:00
Kostas Kyrimis
b82825cd6c
fix: test_cancel_replication_immediately (#1942)
* Give enough time to get replication canceled when the CI server is under heavy load
2023-09-27 17:56:26 +03:00
Yue Li
bcdebc35e9
bug(streams): entries_read is not updated on xreadgroup command (#1946)
fixes #1945
2023-09-27 05:03:19 -07:00
Roy Jacobson
ab903612f1
fix(regtests): Bind to 127.0.0.1 and not localhost (#1937)
* fix(regtests): Bind to 127.0.0.1 and not localhost

* Split start into start/serve
2023-09-27 14:58:52 +03:00
Roy Jacobson
fe5c6968ea
chore: Change the DNS log message to something that makes sense externally (#1947) 2023-09-27 13:17:42 +03:00
Yue Li
c47469ed17
feat(streams): support XACK command (#1923)
fixes #1869
2023-09-27 00:14:57 -07:00
Vladislav
b523f35dc9
feat(search): Early limit cutoff (#1943)
* feat(search): Early limit cutoff

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-09-27 09:25:02 +03:00
Roman Gershman
b67e4d8bb6
chore: improve doubles encoding for listpacks (#1938)
chore: reduce double encoding for listpacks

Following memory improvements by Redis 7, use double convrsion library to represent
double values with less space for listpacks.

The change is to use double conversion library instead of plain sprintf inside
zzlInsertAt. This requires to move zzlInsertAt to Dragonfly codebase.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2023-09-26 22:24:37 +03:00
Roman Gershman
589ee1cf40
chore: flip default sorted set implementation to bptree (#1933)
Also, bring back the default max listpack entries count for zset to 128.
The reason for this - I've added some optimizations that improved listpack
performance and also because I would like to write an article about it
and I need to compare Dragonfly to Redis7 that has this setting set to 128
by default.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2023-09-26 12:41:47 +03:00
iko1
1dbaab564a
fix: set the key index within the json.debug command (#1932)
* fix: set the key index within the json.debug command

Signed-off-by: iko1 <me@remotecpp.dev>
2023-09-26 12:39:42 +03:00
Roy Jacobson
d50b492e1f
feat(replication): First iteration on partial sync. (#1836)
First iteration on partial sync.
2023-09-26 10:35:50 +03:00