Commit Graph

193 Commits

Author SHA1 Message Date
Kostas Kyrimis
50545fc176
feat(AclFamily): add AUTH for acl members (#1732)
* Extends AUTH command to authenticate ACL users
* Add tests
2023-08-25 18:56:15 +03:00
Vladislav
3520697b22
fix: fix search test (#1739)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-08-25 10:01:36 +03:00
Vladislav
84871b8dce
feat: search index persistence (#1721)
* feat: WIP search index persistence

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>

* Update src/server/search/doc_index.cc

Co-authored-by: Kostas Kyrimis  <kostaskyrim@gmail.com>
Signed-off-by: Vladislav <vladislav.oleshko@gmail.com>

* fix: foxes

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
Signed-off-by: Vladislav <vladislav.oleshko@gmail.com>
Co-authored-by: Kostas Kyrimis <kostaskyrim@gmail.com>
2023-08-24 13:41:10 +03:00
Kostas Kyrimis
bd87fb75fa
feat(AclFamily): add acl setuser command (#1725)
* Add `ACL SETUSER` command
* Add tests
2023-08-24 13:24:18 +03:00
Kostas Kyrimis
7f89bf37d8
fix(AclFamily): commands parsing and error handling (#1726)
* Fixed broken ACL command (prev df crashed)
* Fixed broken ACL LIST STR (now prints error)
* Added tests
2023-08-23 14:23:41 +03:00
Roy Jacobson
331e6a4d47
chore: Mark slow pytests as 'slow' so they can be excluded. (#1720) 2023-08-23 13:04:49 +03:00
Kostas Kyrimis
898061d738
feat(AclFamilly): add acl list command (#1722)
* Add acl-family source and header
* Add `ACL LIST` command
* Add a simple test to check the default user
2023-08-22 15:33:14 +00:00
Vladislav
ac79167530
fix: Add small timeout to monitor (#1718)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-08-21 10:20:43 +00:00
Vladislav
e0f3684e5d
feat: json paths in search (#1695)
* feat: json paths in search

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
Signed-off-by: Vladislav <vladislav.oleshko@gmail.com>
Co-authored-by: Roy Jacobson <roi.jacobson1@gmail.com>
2023-08-18 15:40:59 +03:00
Vladislav
c65b9cf63d
fix: Fix squashing, pytest arg formatting (#1712)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-08-18 09:28:19 +03:00
Vladislav
4fbd0e38dd
feat: Pipeline squashing (#1619)
* feat: Pipeline squashing

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
Signed-off-by: Vladislav <vladislav.oleshko@gmail.com>
Co-authored-by: Kostas Kyrimis <kostaskyrim@gmail.com>
2023-08-17 16:06:48 +03:00
Vladislav
71fa2f275e
fix: MONITOR now works for multi transactions (#1675)
* fix: fix monitoring for multi transactions

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-08-17 12:50:16 +03:00
talbii
16c2353faf
introduce --replicaof flag (#1583)
* introduce `--replicaof` flag

Closes #1381.

The behvaiour of `--replicaof` is similar to `REPLICAOF`. On startup, the instance continuously attempts to connect to master. Stop using the normal `REPLICAOF NO ONE` command.

The flag expects format `<IPv4/host>:<port>` or `[<IPv6>]:<port>`.

---------

Signed-off-by: talbii <ido@dragonflydb.io>
Signed-off-by: talbii <41526934+talbii@users.noreply.github.com>
2023-08-09 14:42:08 +03:00
Vladislav
7f1a2d9e38
fix(pytest): Raise exception if instance.stop timed out (#1660)
fix(pytest): Raise if instance.stop timed out

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-08-07 19:10:15 +03:00
Vladislav
279d7cc068
feat: FtList, FtDropIndex and FtInfo (#1649)
* feat: FtList, FtDropIndex and FtInfo

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-08-06 22:01:58 +03:00
Dor Avrahami
0b6c0bbe53
feat: support for cron expressions based snapshot (update). (#1620)
Requested by #1590.
Introducing a new flag --snapshot_cron, enabling users to use cronjob expressions to time snapshot saves.
Cronjob expressions are parsed using a third party library croncpp.
This PR continues #1599, updating cron expressions to crontab style,
up to minutes resolution instead of seconds.

Signed-off-by: Dor Avrahami <da19965@gmail.com>
2023-08-02 12:58:45 +03:00
Shahar Mike
3b0bd212f4
fix(test): Fix failing tests. (#1612)
Solution is to wait until snapshot is ready, instead of hard coding a
sleep schedule. Also don't reuse files by other test cases.
2023-08-01 08:30:17 +00:00
Dor Avrahami
977fc18e25
feat: support for cron expressions based snapshot. (#1599)
Introducing a new flag `--snapshot_cron`, which enables users
to use cron expressions to time snapshot saves.

Signed-off-by: Dor Avrahami <da19965@gmail.com>
2023-07-31 20:26:01 +03:00
Roy Jacobson
4c85d5825d
tests: Add a password to TLS configurations (#1603)
Add a password to TLS configurations
2023-07-31 08:48:36 +00:00
adiholden
3ed0bb480f
Fix(regression test): fix test_flushall_in_full_sync (#1597)
* Fix(regression test): fix test_flushall_in_full_sync

The bug: the test checks the replication using role command on replica
The replica updates the status to full sync when starting the full sync
flow, but actually the master did not start snapshoting yet.
The fix: check the status using role command on master, because master
updates the status only after snapshoting started.

Signed-off-by: adi_holden <adi@dragonflydb.io>
2023-07-31 09:24:37 +03:00
Roy Jacobson
85619e0010
sec: Adjust flag validation for TLS. (#1582)
* sec: Adjust flag checks when using TLS.

* Trust default certificates if no specific roots are given

* Add regression tests for the different scenarios

* Validate that client connections work as well
2023-07-30 19:24:39 +02:00
adiholden
57dc42530e
Fix(regression test): test_tls_replication (#1580)
The test fails sometimes when starting master after killing it.
The reason for this is that OS did not release port untill we started
master again.
The fix - adding sleep after kill
After we will have randomly selected ports on pytest we can remove this
sleep.

Signed-off-by: adi_holden <adi@dragonflydb.io>
2023-07-23 10:27:12 +03:00
Kostas Kyrimis
9698d6fea2
test(replication): add admin port replication tests (#1561) 2023-07-20 14:35:41 +03:00
Kostas Kyrimis
078d152ae0
feat: add replication over tls (#1525)
1. Introduces `tls_replication` flag to allow tls connections for replicas
2. Add pytests
2023-07-19 18:21:46 +00:00
Kostas Kyrimis
7944af3c62
feat: Add black formatter to the project (#1544)
Add black formatter and run it on pytests
2023-07-17 13:13:12 +03:00
adiholden
c27fa8d674
fix(regression test): fix in shutdown and replication pytests (#1530)
* fix(regression_test): fix in shutdown and replication pytests

- skip test_gracefull_shutdown test
- fix test_take_over_seeder test:
    bug: the dbfilename was not unique, therefore between different runs the server reload
    the snapshot of the last test run and this failed the test.
    fix: use random dbfilename
- fix test_take_over_timeout test:
    bug: REPLTAKEOVER timeout was not small enough for opt dfly build
    fix: decrease timeout

Signed-off-by: adi_holden <adi@dragonflydb.io>
2023-07-11 09:56:20 +03:00
Kostas Kyrimis
77a223d36d
fix: add tls-ca-cert-file and tls-ca-cert-dir flags to allow tls certificate validation (#1515)
1. add tls-ca-cert-file flag
2. add tls-ca-cert-dir flag
3. enables redis-cli to connect over tls without --insecure flag by properly validating certificate wtih CA
2023-07-11 08:28:18 +03:00
Shahar Mike
a6745850ab
Fix regression test failures. (#1529)
The issue was that, sometimes, the ID generated for one of the nodes
contained the slot ID that was used in the test (either 5259 or 5260).
This caused the test to replace the "slot" part of the id, which in turn
caused the node to think that it no longer owns any slot.
2023-07-10 10:59:58 +03:00
Roy Jacobson
3904a4f628
Fix regression test failures on old Python versions (#1521)
Signed-off-by: adiholden <adi@dragonflydb.io>
Co-authored-by: adiholden <adi@dragonflydb.io>
2023-07-06 15:27:39 +03:00
Kostas Kyrimis
15481b81ce
feat(replication): allow non-tls connections between replica and master on admin port #1419 (#1490)
1. Add new flag no_tls_on_admin_port
2. Add replication tests for no_tls_on_admin_port
2023-07-06 14:04:45 +03:00
Roy Jacobson
0f69d32b11
takeover: Cancel blocking commands (#1514)
* fix: Cancel blocking commands when performing a takeover

* Add some comments

* Make CancelBlocking a method of ConnectionContext

* add a small todo
2023-07-05 17:09:10 +02:00
Roy Jacobson
177a21b266
Fix a bug and add a timeout test for takeover. (#1512) 2023-07-03 16:46:38 +02:00
Roy Jacobson
4babed54d3
feat: Support atomic replica takeover (#1314)
* fix(server): Initialize ServerFamily with all listeners.

- Add a test for CLIENT LIST which is the visible result of this.

* use std move

* feat: Implement replicas take over

* Basic test

* Address CR comments

* Write a better test. Sadly it fails

* chore: Expose AwaitDispatches for reuse in takeover

* Ensure that no commands can execute during or after a takeover

* CR progress

* Actually disable the expiration

* Improve tests coverage

* Fix the dispatch waiting code

* Improve testing coverage and fix a shutdown snaphot bug

* don't replicate a replica
2023-07-02 16:11:28 +02:00
Roy Jacobson
52192e0596
fix(server): Initialize ServerFamily with all listeners. (#1485)
* fix(server): Initialize ServerFamily with all listeners.

- Add a test for CLIENT LIST which is the visible result of this.

* use std move
2023-07-02 10:01:54 +02:00
Vladislav
1ee0e30255
fix: disable prohibited eval in multi test (#1505)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-07-02 10:50:03 +03:00
Vladislav
cfca751848
feat: global eval in exec (#1443)
Enables execution of global lua scripts inside multi/exec transactions if the defualt script config enables global execution for scripts. This change is only a fix and does not provide any safeguards against other execution scenarios (namely enabling globality with script flags). In the future, the proper execution mode should be determined more carefully by inspecting the scripts to be executed

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
Co-authored-by: Kostas Kyrimis  <kostaskyrim@gmail.com>
2023-07-01 22:12:05 +03:00
Kostas Kyrimis
542b9783b7
fix(regression): remove test case for info persistence (#1492)
The test case for checking is_loading == 1 is inherently racy because
the client can connect at any time before or after the dragonfly
instance loads the snapshot.
2023-07-01 11:57:54 +03:00
Kostas Kyrimis
3eaeef096c
fix: misspelled replication on pytest redis_replication (#1501) 2023-07-01 11:56:43 +03:00
Shahar Mike
5c11beb919
feat(cluster): Send flush slots cmd from masters to replicas. (#1484)
This fixes potential data diffs between master and replica upon slot moving

Fixes #1320
2023-06-28 22:04:51 +03:00
Roy Jacobson
cbe72e353d
fix: Increase the duration of the seeding in test_replication_info. (#1479) 2023-06-26 10:41:25 +03:00
Kostas Kyrimis
0670a81488
fix(regression): failing info persistence on snapshot pytests (#1473) 2023-06-23 16:04:44 +03:00
Kostas Kyrimis
10479a4fba
fix: failing assertion on snapshot regression tests when dbfly instance shuts down (#1463)
This PR is a temporary solution for clients that are not properly
removed from the connection pool triggering an active client assertion
during dragonfly instance shutdown
2023-06-22 13:47:36 +03:00
Roy Jacobson
0a7328f1e1
Expose replication lag metrics (#1400)
* feat(server): Expose lag metrics over prometheus.

---------

Signed-off-by: Roy Jacobson <roy@dragonflydb.io>
2023-06-22 11:12:41 +02:00
Vladislav
fea4e1dd4d
fix: rename memcache to memcacheD (#1461)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-06-22 11:55:20 +03:00
adiholden
98e84f804a
feat(regression github): send failed tests names to chat (#1459)
* feat(regression) : send failed tests names

Signed-off-by: adi_holden <adi@dragonflydb.io>
2023-06-22 09:47:12 +03:00
Kostas Kyrimis
99f3284910
feat(server): Add missing fields to INFO PERSISTENCE command (#1408) (#1438)
* Add loading field
* Add rdb_changes_since_last_save field
2023-06-21 23:35:36 +03:00
Vladislav
f25098bb98
fix: add numpy to regtests (neede for knn) (#1454)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-06-21 09:18:42 +03:00
Vladislav
a9d9b4935c
feat: Implement KNN interface (#1412)
Add external KNN interface

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-06-20 14:10:08 +03:00
Vladislav
6d4d740d6e
fix: Don't remove non-concluding tx from queue on ooo runs (#1427)
* fix: Don't remove non-concluding tx from queue on ooo runs

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-06-18 21:14:28 +03:00
Roman Gershman
69e6ad799a
fix: remove bad check-fail in the transaction code (#1420)
fix: remove bad check-fail in the transaction code.

Fixes #1421.

The failure reproduces for dragongly running with a single thread where all the
arguments grouped within the same ShardData

Also, we improve verbosity levels inside reply_builder.cc.
For that we extend SinkReplyBuilder to support protocol errors reporting
and we remove ad-hoc code for this from dragonfly_connection.
Required to track errors easily with `--vmodule=reply_builder=1`

Finally, a pytest is added to cover the issue.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2023-06-18 07:03:08 +03:00