Commit Graph

10871 Commits

Author SHA1 Message Date
chenyang8094
fa60049648
Fix EVAL timeout test failed on freebsd (#10098)
* Refactor EVAL timeout test
* since the test used r config set appendonly yes which generates a rewrite, it missed it's purpose
* Fix the bug that start_server returns before redis starts ready, which affects when multiple tests share the same dir.
* Elapsed time tracking no loner needed

Co-authored-by: Oran Agra <oran@redislabs.com>
2022-01-24 22:31:35 +02:00
Binbin
495ac8b79a
Fix outdated save key word in redis.conf (#10166)
For some complex data types, server.dirty actually counts
the number of elements that have been changed.
And in FLUSHDB or FLUSHALL, we count the number of keys.

So the word "key" is not strictly correct and is outdated.
Some discussion can be seen at #8140.
2022-01-24 22:02:42 +02:00
yoav-steinberg
7eadc5ee70
Support function flags in script EVAL via shebang header (#10126)
In #10025 we added a mechanism for flagging certain properties for Redis Functions.
This lead us to think we'd like to "port" this mechanism to Redis Scripts (`EVAL`) as well. 

One good reason for this, other than the added functionality is because it addresses the
poor behavior we currently have in `EVAL` in case the script performs a (non DENY_OOM) write operation
during OOM state. See #8478 (And a previous attempt to handle it via #10093) for details.
Note that in Redis Functions **all** write operations (including DEL) will return an error during OOM state
unless the function is flagged as `allow-oom` in which case no OOM checking is performed at all.

This PR:
- Enables setting `EVAL` (and `SCRIPT LOAD`) script flags as defined in #10025.
- Provides a syntactical framework via [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) for
  additional script annotations and even engine selection (instead of just lua) for scripts.
- Provides backwards compatibility so scripts without the new annotations will behave as they did before.
- Appropriate tests.
- Changes `EVAL[SHA]/_RO` to be flagged as `STALE` commands. This makes it possible to flag individual
  scripts as `allow-stale` or not flag them as such. In backwards compatibility mode these commands will
  return the `MASTERDOWN` error as before.
- Changes `SCRIPT LOAD` to be flagged as a `STALE` command. This is mainly to make it logically
  compatible with the change to `EVAL` in the previous point. It enables loading a script on a stale server
  which is technically okay it doesn't relate directly to the server's dataset. Running the script does, but that
  won't work unless the script is explicitly marked as `allow-stale`.

Note that even though the LUA syntax doesn't support hash tag comments `.lua` files do support a shebang
tag on the top so they can be executed on Unix systems like any shell script. LUA's `luaL_loadfile` handles
this as part of the LUA library. In the case of `luaL_loadbuffer`, which is what Redis uses, I needed to fix the
input script in case of a shebang manually. I did this the same way `luaL_loadfile` does, by replacing the
first line with a single line feed character.
2022-01-24 16:50:02 +02:00
Viktor Söderqvist
857dc5bacd
Disable keyspec module API in 7.0 RC1 (#10135)
The keyspec API is not yet released and there is a plan to change it
in #10108, which is going to be included in RC2. Therefore, we hide
it in RC1 to avoid introducing a breaking change in RC2.

Co-authored-by: Oran Agra <oran@redislabs.com>
2022-01-24 15:02:07 +02:00
chenyang8094
6dc3f09cb9
Fix AOFRW limit test occasional failures on slower machines (#10164) 2022-01-24 14:55:24 +02:00
ny0312
b40a9ba5fd
Fix flaky cluster tests in 24-links.tcl (#10157)
* Fix flaky cluster test "Disconnect link when send buffer limit reached"
* Fix flaky cluster test "Each node has two links with each peer"

Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2022-01-23 17:28:32 -08:00
Binbin
7e5ded2ad0
Fix timing issue in sentinel CKQUORUM test (#10036)
A test failure was reported in Daily CI (test-centos7-tls).
`CKQUORUM detects failover authorization cannot be reached`.

```
CKQUORUM detects failover authorization cannot be reached: FAILED:
Expected 'invalid command name "OK 4 usable Sentinels. Quorum and failover authorization can be reached"' to match '*NOQUORUM*'
```

It seems that current sentinel does not confirm that the other
sentinels are actually `down`, and then check the quorum.
It at least take 3 seconds on my machine, and we can see there
will be a timing issue with the hard code `after 5000`.

In this commit, we check the response of `SENTINEL SENTINELS mymaster`
to ensure that other sentinels are actually `down` in the view the
current sentinel. Solve the timing issue due to sentinel monitor mechanism.
2022-01-23 13:54:50 +02:00
Binbin
23325c135f
sub-command support for ACL CAT and COMMAND LIST. redisCommand always stores fullname (#10127)
Summary of changes:
1. Rename `redisCommand->name` to `redisCommand->declared_name`, it is a
  const char * for native commands and SDS for module commands.
2. Store the [sub]command fullname in `redisCommand->fullname` (sds).
3. List subcommands in `ACL CAT`
4. List subcommands in `COMMAND LIST`
5. `moduleUnregisterCommands` now will also free the module subcommands.
6. RM_GetCurrentCommandName returns full command name

Other changes:
1. Add `addReplyErrorArity` and `addReplyErrorExpireTime`
2. Remove `getFullCommandName` function that now is useless.
3. Some cleanups about `fullname` since now it is SDS.
4. Delete `populateSingleCommand` function from server.h that is useless.
5. Added tests to cover this change.
6. Add some module unload tests and fix the leaks
7. Make error messages uniform, make sure they always contain the full command
  name and that it's quoted.
7. Fixes some typos

see the history in #9504, fixes #10124

Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: guybe7 <guy.benoish@redislabs.com>
2022-01-23 10:05:06 +02:00
guybe7
a6fd2a46d1
Improved handling of subcommands (don't allow ACL on first-arg of a sub-command) (#10147)
Recently we added extensive support for sub-commands in for redis 7.0,
this meant that the old ACL mechanism for
sub-commands wasn't needed, or actually was improved (to handle both include
and exclude control, like for commands), but only for real sub-commands.
The old mechanism in ACL was renamed to first-arg, and was able to match the
first argument of any command (including sub-commands).
We now realized that we might wanna completely delete that first-arg feature some
day, so the first step was not to give it new capabilities in 7.0 and it didn't have before.

Changes:
1. ACL: Block the first-arg mechanism on subcommands (we keep if in non-subcommands
  for backward compatibility)
2. COMMAND: When looking up a command, insist the command name doesn't contain
  extra words. Example: When a user issues `GET key` we want `lookupCommand` to return
  `getCommand` but when if COMMAND calls `lookupCommand` with `get|key` we want it to fail.

Other changes:
1. ACLSetUser: prevent a redundant command lookup
2022-01-22 14:09:40 +02:00
Madelyn Olson
55c81f2cd3
ACL V2 - Selectors and key based permissions (#9974)
* Implemented selectors which provide multiple different sets of permissions to users
* Implemented key based permissions 
* Added a new ACL dry-run command to test permissions before execution
* Updated module APIs to support checking key based permissions

Co-authored-by: Oran Agra <oran@redislabs.com>
2022-01-20 13:05:27 -08:00
guybe7
10bbeb6837
Add command tips to COMMAND DOCS (#10104)
Adding command tips (see https://redis.io/topics/command-tips) to commands.

Breaking changes:
1. Removed the "random" and "sort_for_script" flags. They are now command tips.
(this isn't affecting redis behavior since #9812, but could affect some client applications
that's relying on COMMAND command flags)

Summary of changes:
1. add BLOCKING flag (new flag) for all commands that could block. The ACL category with
  the same name is now implicit.
2. move RANDOM flag to a `nondeterministic_output` tip
3. move SORT_FOR_SCRIPT flag to `nondeterministic_output_order` tip
3. add REQUEST_POLICY and RESPONSE_POLICY where appropriate as documented in the tips
4. deprecate (ignore) the `random` flag for RM_CreateCommand

Other notes:
1. Proxies need to send `RANDOMKEY` to all shards and then select one key randomly.
  The other option is to pick a random shard and transfer `RANDOMKEY `to it, but that scheme
  fails if this specific shard is empty
2. Remove CMD_RANDOM from `XACK` (i.e. XACK does not have RANDOM_OUTPUT)
   It was added in 9e4fb96ca1, I guess by mistake.
   Also from `(P)EXPIRETIME` (new command, was flagged "random" by mistake).
3. Add `nondeterministic_output` to `OBJECT ENCODING` (for the same reason `XTRIM` has it:
   the reply may differ depending on the internal representation in memory)
4. RANDOM on `HGETALL` was wrong (there due to a limitation of the old script sorting logic), now
  it's `nondeterministic_output_order`
5. Unrelated: Hide CMD_PROTECTED from COMMAND
2022-01-20 11:32:11 +02:00
Meir Shpilraien (Spielrein)
9c60292250
Added functions support to redis-check-rdb (#10154)
The PR added the missing verification for functions on redis-check-rdb.
The verification only verifies the rdb structure and does not try to load the functions code and
verify more advance checks (like compilation of the function code).
2022-01-20 11:10:33 +02:00
perryitay
c4b788230c
Adding module api for processing commands during busy jobs and allow flagging the commands that should be handled at this status (#9963)
Some modules might perform a long-running logic in different stages of Redis lifetime, for example:
* command execution
* RDB loading
* thread safe context

During this long-running logic Redis is not responsive.

This PR offers 
1. An API to process events while a busy command is running (`RM_Yield`)
2. A new flag (`ALLOW_BUSY`) to mark the commands that should be handled during busy
  jobs which can also be used by modules (`allow-busy`)
3. In slow commands and thread safe contexts, this flag will start rejecting commands with -BUSY only
  after `busy-reply-threshold`
4. During loading (`rdb_load` callback), it'll process events right away (not wait for `busy-reply-threshold`),
  but either way, the processing is throttled to the server hz rate.
5. Allow modules to Yield to redis background tasks, but not to client commands

* rename `script-time-limit` to `busy-reply-threshold` (an alias to the pre-7.0 `lua-time-limit`)

Co-authored-by: Oran Agra <oran@redislabs.com>
2022-01-20 09:05:53 +02:00
Madelyn Olson
22172a4aa6
Fix double key declaration for renamenx and change flag to INSERT (#10137)
* Fix double key declaration for renamenx
* Change the flag from UPDATE to INSERT, since it can not affect the data in the key
2022-01-19 19:30:32 -08:00
Meir Shpilraien (Spielrein)
c556c57e5e
Added AOF rewrite support for functions. (#10141)
Function PR was merged without AOF rw support because we thought this feature was going
to be removed on Redis 7.

Tests was added on aofrw.tcl
Other existing aofrw tests where slow due to unwanted rdb-key-save-delay

Co-authored-by: Oran Agra <oran@redislabs.com>
2022-01-19 21:21:42 +02:00
David CARLIER
834fa5870c
zmalloc_get_rss openbsd implementation (#10149)
Add support for getting the RSS in OpenBSD
2022-01-19 20:56:12 +02:00
Binbin
3720de78c9
Fix fcall_ro key spec flags in commands.c (#10145)
In #10122, we modify the key spec flags to `RO` and `ACCESS`.
But forgot to call generate-command-code.py. Also formatted
it to follow the Python PEP8.
2022-01-19 16:07:28 +02:00
Wen Hui
68a8d0b46d
Add sentinel config set test case (#10114) 2022-01-19 11:57:51 +02:00
Ozan Tezcan
72e1b5de4d
Fix replica count check in migration tests. (#10140)
Tests were not using loop index as node id, checking replica count of the same node over and over.
2022-01-19 11:36:24 +02:00
Ozan Tezcan
1af0a2c5ae
Fix eventloop module test for valgrind (#10139)
was eating too much memory, and taking too long with valgrind
2022-01-19 09:13:51 +02:00
Oran Agra
eef9c6b0ee
New detailed key-spec flags (RO, RW, OW, RM, ACCESS, UPDATE, INSERT, DELETE) (#10122)
The new ACL key based permissions in #9974 require the key-specs (#8324) to have more
explicit flags rather than just READ and WRITE. See discussion in #10040

This PR defines two groups of flags:
One about how redis internally handles the key (mutually-exclusive).
The other is about the logical operation done from the user's point of view (3 mutually exclusive
write flags, and one read flag, all optional).
In both groups, if we can't explicitly flag something as explicit read-only, delete-only, or
insert-only, we flag it as `RW` or `UPDATE`.
here's the definition from the code:
```
/* Key-spec flags *
 * -------------- */
/* The following refer what the command actually does with the value or metadata
 * of the key, and not necessarily the user data or how it affects it.
 * Each key-spec may must have exaclty one of these. Any operation that's not
 * distinctly deletion, overwrite or read-only would be marked as RW. */
#define CMD_KEY_RO (1ULL<<0)     /* Read-Only - Reads the value of the key, but
                                  * doesn't necessarily returns it. */
#define CMD_KEY_RW (1ULL<<1)     /* Read-Write - Modifies the data stored in the
                                  * value of the key or its metadata. */
#define CMD_KEY_OW (1ULL<<2)     /* Overwrite - Overwrites the data stored in
                                  * the value of the key. */
#define CMD_KEY_RM (1ULL<<3)     /* Deletes the key. */
/* The follwing refer to user data inside the value of the key, not the metadata
 * like LRU, type, cardinality. It refers to the logical operation on the user's
 * data (actual input strings / TTL), being used / returned / copied / changed,
 * It doesn't refer to modification or returning of metadata (like type, count,
 * presence of data). Any write that's not INSERT or DELETE, would be an UPADTE.
 * Each key-spec may have one of the writes with or without access, or none: */
#define CMD_KEY_ACCESS (1ULL<<4) /* Returns, copies or uses the user data from
                                  * the value of the key. */
#define CMD_KEY_UPDATE (1ULL<<5) /* Updates data to the value, new value may
                                  * depend on the old value. */
#define CMD_KEY_INSERT (1ULL<<6) /* Adds data to the value with no chance of,
                                  * modification or deletion of existing data. */
#define CMD_KEY_DELETE (1ULL<<7) /* Explicitly deletes some content
                                  * from the value of the key. */
```

Unrelated changes:
- generate-command-code.py is only compatible with python3 (modified the shabang)
- generate-command-code.py print file on json parsing error
- rename `shard_channel` key-spec flag to just `channel`.
- add INCOMPLETE flag in input spec of SORT and SORT_RO
2022-01-18 16:00:00 +02:00
Wang Yuan
d697daa7a5
Use const char pointer in redismodule.h as far as possible (#10064)
When I used C++ to develop a redis module. i  used `string.data()` as the second parameter `ele`
of  `RedisModule_DigestAddStringBuffer`, but there is a warning, since we never change the `ele`,
i think we should use `const char` for it.

This PR adds const to just a handful of module APIs that required it, all not very widely used.
The implication is a breaking change in terms of compilation error that's easy to resolve, and no ABI impact.
The affected APIs are around Digest, Info injection, and Cluster bus messages.
2022-01-18 15:55:20 +02:00
Ozan Tezcan
99ab4236af
Add event loop support to the module API (#10001)
Modules can now register sockets/pipe to the Redis main thread event loop and do network operations asynchronously. Previously, modules had to maintain an event loop and another thread for asynchronous network operations.

Also, if a module is calling API functions after doing some network operations, it had to synchronize its event loop thread's access with Redis main thread by locking the GIL, causing contention on the lock. After this commit, no synchronization is needed as module can operate in Redis main thread context. So, this commit may improve the performance for some use cases.

Added three functions to the module API:

* RedisModule_EventLoopAdd(int fd, int mask, RedisModuleEventLoopFunc func, void *user_data)
* RedisModule_EventLoopDel(int fd, int mask)
* RedisModule_EventLoopAddOneShot(RedisModuleEventLoopOneShotFunc func, void *user_data) - This function can be called from other threads to trigger callback on Redis main thread. Callback will be triggered only once. If Redis main thread is sleeping, this call will wake up the Redis main thread.
Event loop callbacks are called by Redis main thread after locking the GIL. Inside callbacks, modules can operate as if they are holding the GIL.

Added REDISMODULE_EVENT_EVENTLOOP event with two subevents:

* REDISMODULE_SUBEVENT_EVENTLOOP_BEFORE_SLEEP
* REDISMODULE_SUBEVENT_EVENTLOOP_AFTER_SLEEP

These events are for modules that want to participate in the before and after sleep action. e.g It might be useful to implement batching : Read data from the network, write all to a file in one go on BEFORE_SLEEP event.
2022-01-18 13:10:07 +02:00
Yossi Gottlieb
25e6d4d459
Fix additional AOF filename issues. (#10110)
This extends the previous fix (#10049) to address any form of
non-printable or whitespace character (including newlines, quotes,
non-printables, etc.)

Also, removes the limitation on appenddirname, to align with the way
filenames are handled elsewhere in Redis.
2022-01-18 12:52:27 +02:00
Meir Shpilraien (Spielrein)
51f9bed3dd
Fix FUNCTION LOAD ignores unknown parameter. (#10131)
Following discussion on: https://github.com/redis/redis/issues/9899#issuecomment-1014689385
Raise error if unknows parameter is given to `FUNCTION LOAD`.

Before the fix:
```
127.0.0.1:6379> function load LUA lib2 foo bar "local function test1() return 5 end redis.register_function('test1', test1)"
OK
```

After the fix:
```
127.0.0.1:6379> function load LUA lib2 foo bar "local function test1() return 5 end redis.register_function('test1', test1)"
(error) ERR Unkowns option given: foo
```
2022-01-18 10:29:52 +02:00
Oran Agra
ae89958972
Set repl-diskless-sync to yes by default, add repl-diskless-sync-max-replicas (#10092)
1. enable diskless replication by default
2. add a new config named repl-diskless-sync-max-replicas that enables
   replication to start before the full repl-diskless-sync-delay was
   reached.
3. put replica online sooner on the master (see below)
4. test suite uses repl-diskless-sync-delay of 0 to be faster
5. a few tests that use multiple replica on a pre-populated master, are
   now using the new repl-diskless-sync-max-replicas
6. fix possible timing issues in a few cluster tests (see below)

put replica online sooner on the master 
----------------------------------------------------
there were two tests that failed because they needed for the master to
realize that the replica is online, but the test code was actually only
waiting for the replica to realize it's online, and in diskless it could
have been before the master realized it.

changes include two things:
1. the tests wait on the right thing
2. issues in the master, putting the replica online in two steps.

the master used to put the replica as online in 2 steps. the first
step was to mark it as online, and the second step was to enable the
write event (only after getting ACK), but in fact the first step didn't
contains some of the tasks to put it online (like updating good slave
count, and sending the module event). this meant that if a test was
waiting to see that the replica is online form the point of view of the
master, and then confirm that the module got an event, or that the
master has enough good replicas, it could fail due to timing issues.

so now the full effect of putting the replica online, happens at once,
and only the part about enabling the writes is delayed till the ACK.

fix cluster tests 
--------------------
I added some code to wait for the replica to sync and avoid race
conditions.
later realized the sentinel and cluster tests where using the original 5
seconds delay, so changed it to 0.

this means the other changes are probably not needed, but i suppose
they're still better (avoid race conditions)
2022-01-17 14:11:11 +02:00
zhaozhao.zz
90916f16a5
show subcommands latencystats (#10103)
since `info commandstats` already shows sub-commands, we should do the same in `info latencystats`.
similarly, the LATENCY HISTOGRAM command now shows sub-commands (with their full name) when:
* asking for all commands
* asking for a specific container command
* asking for a specific sub-command)

Co-authored-by: Oran Agra <oran@redislabs.com>
2022-01-17 12:32:32 +02:00
Binbin
26ef5132a6
Fix timing issue in PSETEX/PEXPIRE sub-second expire tests (#10121)
These two tests have a high probability of failure
on MacOS. Or it takes many retries to succeed.
Keys often expire before we can access them.

So this time we try to avoid this by reducing the time
of the first `after`, or removeing the first `after`.

The results of doing `20/81` and `0/101` are:
- PEXPIRE (20/81): 1069/1949
- PEXPIREAT (20/81): 1093/1949

- PEXPIRE (0/101): 31936 / 31936
- PEXPIREAT (0/101): 31936 / 31936

The first number is the number of times that the
test succeeded without any retries.
The second number is the total number of executions.

And we can see that `0/101` doesn't even need an extra
retries. Also reduces the time required for testing.
So in the end we chose `0/100`, i.e. remove the first `after`.

As for `PEXPIREAT`, there is no failure, but we still changed
it together, using `0/201`, after 2W tests, none of them failed.
2022-01-17 10:42:13 +02:00
chenyang8094
d7aeb2ce5f
Use am instead of using server.aof_manifest directly to call getBaseAndIncrAppendOnlyFilesSize (#10123) 2022-01-17 09:02:09 +02:00
David CARLIER
50fa627b90
zmalloc_get_rss netbsd impl fix proposal. (#10116)
Seems like the previous implementation was broken (always returning 0)

since kinfo_proc2 is used the KERN_PROC2 sysctl oid is more appropriate
and also the query's length was not necessarily accurate (6 here).
2022-01-16 10:03:09 +02:00
sundb
32e7b46a17
Fix quicklist node not being recompressed correctly after inserting a new node before or after it (#10120)
### Describe
Fix crash found by CI, Introduced by #9849.
When we do any operation on the quicklist, we should make sure that all nodes
of the quicklist should not be in the recompressed state.

### Issues
This PR fixes two issues with incorrect recompression.
1. The current quicklist node is full and the previous node isn't full,
    the current node is not recompressed correctly after inserting elements into the previous node.
2. The current quicklist node is full and the next node isn't full,
    the current node is not recompressed correctly after inserting elements into the next node.

### Test
Add two tests to cover incorrect compression issues.

### Other
Fix unittest test failure caused by assertion introduced by #9849.
2022-01-16 08:54:40 +02:00
David CARLIER
7da7d2aa8e
checkTcpBacklogSettings check for solaris based systems. (#10109) 2022-01-15 20:39:05 +02:00
Ali-Akber Saifee
98414aca7f
Fix typo in doc for PUBSUB command (#10117)
Replace "Pub/Sun" -> "Pub/Sub"
2022-01-15 19:57:24 +02:00
Binbin
440d28091b
Fix function no-cluster flag test (#10115)
Fixes cluster test introduced in #10066.
```
Function no-cluster flag: ERR Error registering functions: @user_function: 1: wrong number of arguments to redis.register_function
```
2022-01-15 09:13:53 +02:00
Meir Shpilraien (Spielrein)
4db4b43417
Function Flags support (no-writes, no-cluster, allow-state, allow-oom) (#10066)
# Redis Functions Flags

Following the discussion on #10025 Added Functions Flags support.
The PR is divided to 2 sections:
* Add named argument support to `redis.register_function` API.
* Add support for function flags

## `redis.register_function` named argument support

The first part of the PR adds support for named argument on `redis.register_function`, example:
```
redis.register_function{
    function_name='f1',
    callback=function()
        return 'hello'
    end,
    description='some desc'
}
```

The positional arguments is also kept, which means that it still possible to write:
```
redis.register_function('f1', function() return 'hello' end)
```

But notice that it is no longer possible to pass the optional description argument on the positional
argument version. Positional argument was change to allow passing only the mandatory arguments
(function name and callback). To pass more arguments the user must use the named argument version.

As with positional arguments, the `function_name` and `callback` is mandatory and an error will be
raise if those are missing. Also, an error will be raise if an unknown argument name is given or the
arguments type is wrong.

Tests was added to verify the new syntax.

## Functions Flags

The second part of the PR is adding functions flags support.
Flags are given to Redis when the engine calls `functionLibCreateFunction`, supported flags are:

* `no-writes` - indicating the function perform no writes which means that it is OK to run it on:
   * read-only replica
   * Using FCALL_RO
   * If disk error detected
   
   It will not be possible to run a function in those situations unless the function turns on the `no-writes` flag

* `allow-oom` - indicate that its OK to run the function even if Redis is in OOM state, if the function will
  not turn on this flag it will not be possible to run it if OOM reached (even if the function declares `no-writes`
  and even if `fcall_ro` is used). If this flag is set, any command will be allow on OOM (even those that is
  marked with CMD_DENYOOM). The assumption is that this flag is for advance users that knows its
  meaning and understand what they are doing, and Redis trust them to not increase the memory usage.
  (e.g. it could be an INCR or a modification on an existing key, or a DEL command)

* `allow-state` - indicate that its OK to run the function on stale replica, in this case we will also make
  sure the function is only perform `stale` commands and raise an error if not.

* `no-cluster` - indicate to disallow running the function if cluster is enabled.

Default behaviure of functions (if no flags is given):
1. Allow functions to read and write
2. Do not run functions on OOM
3. Do not run functions on stale replica
4. Allow functions on cluster

### Lua API for functions flags

On Lua engine, it is possible to give functions flags as `flags` named argument:

```
redis.register_function{function_name='f1', callback=function() return 1 end, flags={'no-writes', 'allow-oom'}, description='description'}
```

The function flags argument must be a Lua table that contains all the requested flags, The following
will result in an error:
* Unknown flag
* Wrong flag type

Default behaviour is the same as if no flags are used.

Tests were added to verify all flags functionality

## Additional changes
* mark FCALL and FCALL_RO with CMD_STALE flag (unlike EVAL), so that they can run if the function was
  registered with the `allow-stale` flag.
* Verify `CMD_STALE` on `scriptCall` (`redis.call`), so it will not be possible to call commands from script while
  stale unless the command is marked with the `CMD_STALE` flags. so that even if the function is allowed while
  stale we do not allow it to bypass the `CMD_STALE` flag of commands.
* Flags section was added to `FUNCTION LIST` command to provide the set of flags for each function:
```
> FUNCTION list withcode
1)  1) "library_name"
    2) "test"
    3) "engine"
    4) "LUA"
    5) "description"
    6) (nil)
    7) "functions"
    8) 1) 1) "name"
          2) "f1"
          3) "description"
          4) (nil)
          5) "flags"
          6) (empty array)
    9) "library_code"
   10) "redis.register_function{function_name='f1', callback=function() return 1 end}"
```
* Added API to get Redis version from within a script, The redis version can be provided using:
   1. `redis.REDIS_VERSION` - string representation of the redis version in the format of MAJOR.MINOR.PATH
   2. `redis.REDIS_VERSION_NUM` - number representation of the redis version in the format of `0x00MMmmpp`
      (`MM` - major, `mm` - minor,  `pp` - patch). The number version can be used to check if version is greater or less 
      another version. The string version can be used to return to the user or print as logs.

   This new API is provided to eval scripts and functions, it also possible to use this API during functions loading phase.
2022-01-14 14:02:02 +02:00
Peter Hickey
38a5116728
Fix typo in deps/README.md (#10113) 2022-01-14 13:54:22 +02:00
Binbin
56a802057e
Fix kill aof rewrite child test (#10107)
The dbs doesn't have any keys, `rdb-key-save-delay`
config has no effect that cause the rewrite to complete.

It was introduced in #10015.
2022-01-13 12:38:41 +02:00
Ozan Tezcan
f41cc87088
Added RM_MonotonicMicroseconds() API to provide monotonic time function (#10101)
Added RM_MonotonicMicroseconds(). Modules can use monotonic timestamp counter for measurements.
2022-01-13 11:36:03 +02:00
chenyang8094
e9bff7978a
Always create base AOF file when redis start from empty. (#10102)
Force create a BASE file (use a foreground `rewriteAppendOnlyFile`) when redis starts from an
empty data set and  `appendonly` is  yes.

The reasoning is that normally, after redis is running for some time, and the AOF has gone though
a few rewrites, there's always a base rdb file. and the scenario where the base file is missing, is
kinda rare (happens only at empty startup), so this change normalizes it.
But more importantly, there are or could be some complex modules that are started with some
configuration, when they create persistence they write that configuration to RDB AUX fields, so
that can can always know with which configuration the persistence file they're loading was
created (could be critical). there is (was) one scenario in which they could load their persisted data,
and that configuration was missing, and this change fixes it.

Add a new module event: REDISMODULE_SUBEVENT_PERSISTENCE_SYNC_AOF_START, similar to
REDISMODULE_SUBEVENT_PERSISTENCE_AOF_START which is async.

Co-authored-by: Oran Agra <oran@redislabs.com>
2022-01-13 08:49:26 +02:00
Binbin
20c33fe6a8
Show subcommand full name in error log / ACL LOG (#10105)
Use `getFullCommandName` to get the full name of the command.
It can also get the full name of the subcommand, like "script|help".

Before:
```
> SCRIPT HELP
(error) NOPERM this user has no permissions to run the 'help' command or its subcommand

> ACL LOG
    7) "object"
    8) "help"
```

After:
```
> SCRIPT HELP
(error) NOPERM this user has no permissions to run the 'script|help' command

> ACL LOG
    7) "object"
    8) "script|help"
```

Fix #10094
2022-01-12 20:05:14 +02:00
Binbin
e22146b07a
Add script tests to cover keys with expiration time set (#10096)
This commit adds some tests that the test cases will
access the keys with expiration time set in the script call.
There was no test case for this part before. See #10080

Also there is a test will cover #1525. we block the time so
that the key can not expire in the middle of the script execution.

Other changes:
1. Delete `evalTimeSnapshot` and just use `scriptTimeSnapshot` in it's place.
2. Some cleanups to scripting.tcl.
3. better names for tests that run in a loop to make them distinctable
2022-01-11 22:43:18 +02:00
Ozan Tezcan
6790d848c5
Reuse temporary client objects for blocked clients by module (#9940)
Added a pool for temporary client objects to reuse in module operations.
By reusing temporary clients, we are avoiding expensive createClient()/freeClient()
calls and improving performance of RM_BlockClient() and  RM_GetThreadSafeContext() calls. 

This commit contains two optimizations: 

1 - RM_BlockClient() and RM_GetThreadSafeContext() calls create temporary clients and they are freed in
RM_UnblockClient() and RM_FreeThreadSafeContext() calls respectively. Creating/destroying client object
takes quite time. To avoid that, added a pool of temporary clients. Pool expands when more clients are needed.
Also, added a cron function to shrink the pool and free unused clients after some time. Pool starts with zero
clients in it. It does not have max size and can grow unbounded as we need it. We will keep minimum of 8
temporary clients in the pool once created. Keeping small amount of clients to avoid client allocation costs
if temporary clients are required after some idle period.

2 - After unblocking a client (RM_UnblockClient()), one byte is written to pipe to wake up Redis main thread.
If there are many clients that will be unblocked, each operation requires one write() call which is quite expensive.
Changed code to avoid subsequent calls if possible. 

There are a few more places that need temporary client objects (e.g RM_Call()). These are now using the same
temporary client pool to make things more centralized.
2022-01-11 19:00:56 +02:00
Oran Agra
3204a03574
Move doc metadata from COMMAND to COMMAND DOCS (#10056)
Syntax:
`COMMAND DOCS [<command name> ...]`

Background:
Apparently old version of hiredis (and thus also redis-cli) can't
support more than 7 levels of multi-bulk nesting.

The solution is to move all the doc related metadata from COMMAND to a
new COMMAND DOCS sub-command.

The new DOCS sub-command returns a map of commands (not an array like in COMMAND),
And the same goes for the `subcommands` field inside it (also contains a map)

Besides that, the remaining new fields of COMMAND (hints, key-specs, and
sub-commands), are placed in the outer array rather than a nested map.
this was done mainly for consistency with the old format.

Other changes:
---
* Allow COMMAND INFO with no arguments, which returns all commands, so that we can some day deprecated
  the plain COMMAND (no args)

* Reduce the amount of deferred replies from both COMMAND and COMMAND
  DOCS, especially in the inner loops, since these create many small
  reply objects, which lead to many small write syscalls and many small
  TCP packets.
  To make this easier, when populating the command table, we count the
  history, args, and hints so we later know their size in advance.
  Additionally, the movablekeys flag was moved into the flags register.
* Update generate-commands-json.py to take the data from both command, it
  now executes redis-cli directly, instead of taking input from stdin.
* Sub-commands in both COMMAND (and COMMAND INFO), and also COMMAND DOCS,
  show their full name. i.e. CONFIG 
*   GET will be shown as `config|get` rather than just `get`.
  This will be visible both when asking for `COMMAND INFO config` and COMMAND INFO config|get`, but is
  especially important for the later.
  i.e. imagine someone doing `COMMAND INFO slowlog|get config|get` not being able to distinguish between the two
  items in the array response.
2022-01-11 17:16:16 +02:00
guybe7
5009b43dc6
Module events: Fail RM_SubscribeToServerEvent if event is too new (#9987)
We must fail RM_SubscribeToServerEvent in case a module, that
was compiled with a new redismodule.h, tries to subscribe to an
event that doesn't exist on an old redis-server
2022-01-11 14:33:39 +02:00
Binbin
39feee8e3a
LPOP/RPOP with count against non existing list return null array (#10095)
It used to return `$-1` in RESP2, now we will return `*-1`.
This is a bug in redis 6.2 when COUNT was added, the `COUNT`
option was introduced in #8179. Fix #10089.

the documentation of [LPOP](https://redis.io/commands/lpop) says
```
When called without the count argument:
Bulk string reply: the value of the first element, or nil when key does not exist.

When called with the count argument:
Array reply: list of popped elements, or nil when key does not exist.
```
2022-01-11 14:26:13 +02:00
小令童鞋
1e25bdf780
fix redis crached by using eval with access to volatile keys (#10080)
This is a recent regression from the Redis Functions commits
2022-01-11 07:56:31 +02:00
Madelyn Olson
d0949b7c5c
Fix timing issue with cluster hostname test (#10086) 2022-01-10 16:21:05 -08:00
chenyang8094
bd46a2abf4
Support whitespace characters in appendfilename, and ban them in appenddirname (#10049)
1. Ban whitespace characters in `appenddirname`   
2. Handle the case where `appendfilename` contains spaces (for backwards compatibility)
2022-01-10 09:09:39 +02:00
Binbin
a1ae260e8a
Make sure replicas don't write their own replies to the replication link (#10081)
The following steps will crash redis-server:
```
[root]# cat crash
PSYNC replicationid -1
SLOWLOG GET
GET key
[root]# nc 127.0.0.1 6379 < crash
```

This one following #10020 and the crash was reported in #10076.

Other changes about the output info:
1. Cmd with a full name by using `getFullCommandName`, now it will print the right
   subcommand name like `slowlog|get`.
2. Print the full client info by using `catClientInfoString`, the info is also valuable.:
2022-01-10 08:21:16 +02:00
Wen Hui
d7479107c7
update codes according to new json file (#7425)
readonly/readwrite only set client flags for slave in cluster mode,
so it should be ok for setting ok-stale and ok-loading command flag
2022-01-10 08:00:04 +02:00