Commit Graph

10824 Commits

Author SHA1 Message Date
chenyang8094
bd46a2abf4
Support whitespace characters in appendfilename, and ban them in appenddirname (#10049)
1. Ban whitespace characters in `appenddirname`   
2. Handle the case where `appendfilename` contains spaces (for backwards compatibility)
2022-01-10 09:09:39 +02:00
Binbin
a1ae260e8a
Make sure replicas don't write their own replies to the replication link (#10081)
The following steps will crash redis-server:
```
[root]# cat crash
PSYNC replicationid -1
SLOWLOG GET
GET key
[root]# nc 127.0.0.1 6379 < crash
```

This one following #10020 and the crash was reported in #10076.

Other changes about the output info:
1. Cmd with a full name by using `getFullCommandName`, now it will print the right
   subcommand name like `slowlog|get`.
2. Print the full client info by using `catClientInfoString`, the info is also valuable.:
2022-01-10 08:21:16 +02:00
Wen Hui
d7479107c7
update codes according to new json file (#7425)
readonly/readwrite only set client flags for slave in cluster mode,
so it should be ok for setting ok-stale and ok-loading command flag
2022-01-10 08:00:04 +02:00
Madelyn Olson
e8e02f900c
Changed latency histogram output to omit trailing 0s and periods (#10075)
Changed latency percentile output to omit trailing 0s and periods
2022-01-09 17:04:18 -08:00
Binbin
a84c964d37
Fix crash when error [sub]command name contains | (#10082)
The following error commands will crash redis-server:
```
> get|
Error: Server closed the connection
> get|set
Error: Server closed the connection
> get|other
```

The reason is in #9504, we use `lookupCommandBySds` for find the
container command. And it split the command (argv[0]) with `|`.
If we input something like `get|other`, after the split, `get`
will become a valid command name, pass the `ERR unknown command`
check, and finally crash in `addReplySubcommandSyntaxError`

In this case we do not need to split the command name with `|`
and just look in the commands dict to find if `argv[0]` is a
container command.

So this commit introduce a new function call `isContainerCommandBySds`
that it will return true if a command name is a container command.

Also with the old code, there is a incorrect error message:
```
> config|get set
(error) ERR Unknown subcommand or wrong number of arguments for 'set'. Try CONFIG|GET HELP.
```

The crash was reported in #10070.
2022-01-09 13:06:51 +02:00
Itamar Haber
75c50a1563
Fixes 'since' for GEORADIUS[BYMEMBER]_RO (#10034) 2022-01-09 11:13:32 +02:00
YEONCHEOL JANG
b9669829c8
Fixes typo for redis.conf (#10072)
Fixes typo for redis.conf
2022-01-09 08:08:55 +02:00
guybe7
7cd6a64d2f
lpGetInteger returns int64_t, avoid overflow (#10068)
Fix #9410

Crucial for the ms and sequence deltas, but I changed all
calls, just in case (e.g. "flags")

Before this commit:
`ms_delta` and `seq_delta` could have overflown, causing `currid` to be wrong,
which in turn would cause `streamTrim` to trim the entire rax node (see new test)
2022-01-07 15:31:05 +02:00
Viktor Söderqvist
e88f6acb94
Build commands.c in Makefile (#10039)
With this rule, the script to generate commands.c from JSON runs whenever commands.o is built if any of commands/*.json are modified. Without such rule, it's easy to forget to run the script when updating the JSON files.

It's a follow-up on #9656 and #9951.
2022-01-06 17:59:37 +02:00
Meir Shpilraien (Spielrein)
885f6b5ceb
Redis Function Libraries (#10004)
# Redis Function Libraries

This PR implements Redis Functions Libraries as describe on: https://github.com/redis/redis/issues/9906.

Libraries purpose is to provide a better code sharing between functions by allowing to create multiple
functions in a single command. Functions that were created together can safely share code between
each other without worrying about compatibility issues and versioning.

Creating a new library is done using 'FUNCTION LOAD' command (full API is described below)

This PR introduces a new struct called libraryInfo, libraryInfo holds information about a library:
* name - name of the library
* engine - engine used to create the library
* code - library code
* description - library description
* functions - the functions exposed by the library

When Redis gets the `FUNCTION LOAD` command it creates a new empty libraryInfo.
Redis passes the `CODE` to the relevant engine alongside the empty libraryInfo.
As a result, the engine will create one or more functions by calling 'libraryCreateFunction'.
The new funcion will be added to the newly created libraryInfo. So far Everything is happening
locally on the libraryInfo so it is easy to abort the operation (in case of an error) by simply
freeing the libraryInfo. After the library info is fully constructed we start the joining phase by
which we will join the new library to the other libraries currently exist on Redis.
The joining phase make sure there is no function collision and add the library to the
librariesCtx (renamed from functionCtx). LibrariesCtx is used all around the code in the exact
same way as functionCtx was used (with respect to RDB loading, replicatio, ...).
The only difference is that apart from function dictionary (maps function name to functionInfo
object), the librariesCtx contains also a libraries dictionary that maps library name to libraryInfo object.

## New API
### FUNCTION LOAD
`FUNCTION LOAD <ENGINE> <LIBRARY NAME> [REPLACE] [DESCRIPTION <DESCRIPTION>] <CODE>`
Create a new library with the given parameters:
* ENGINE - REPLACE Engine name to use to create the library.
* LIBRARY NAME - The new library name.
* REPLACE - If the library already exists, replace it.
* DESCRIPTION - Library description.
* CODE - Library code.

Return "OK" on success, or error on the following cases:
* Library name already taken and REPLACE was not used
* Name collision with another existing library (even if replace was uses)
* Library registration failed by the engine (usually compilation error)

## Changed API
### FUNCTION LIST
`FUNCTION LIST [LIBRARYNAME <LIBRARY NAME PATTERN>] [WITHCODE]`
Command was modified to also allow getting libraries code (so `FUNCTION INFO` command is no longer
needed and removed). In addition the command gets an option argument, `LIBRARYNAME` allows you to
only get libraries that match the given `LIBRARYNAME` pattern. By default, it returns all libraries.

### INFO MEMORY
Added number of libraries to `INFO MEMORY`

### Commands flags
`DENYOOM` flag was set on `FUNCTION LOAD` and `FUNCTION RESTORE`. We consider those commands
as commands that add new data to the dateset (functions are data) and so we want to disallows
to run those commands on OOM.

## Removed API
* FUNCTION CREATE - Decided on https://github.com/redis/redis/issues/9906
* FUNCTION INFO - Decided on https://github.com/redis/redis/issues/9899

## Lua engine changes
When the Lua engine gets the code given on `FUNCTION LOAD` command, it immediately runs it, we call
this run the loading run. Loading run is not a usual script run, it is not possible to invoke any
Redis command from within the load run.
Instead there is a new API provided by `library` object. The new API's: 
* `redis.log` - behave the same as `redis.log`
* `redis.register_function` - register a new function to the library

The loading run purpose is to register functions using the new `redis.register_function` API.
Any attempt to use any other API will result in an error. In addition, the load run is has a time
limit of 500ms, error is raise on timeout and the entire operation is aborted.

### `redis.register_function`
`redis.register_function(<function_name>, <callback>, [<description>])`
This new API allows users to register a new function that will be linked to the newly created library.
This API can only be called during the load run (see definition above). Any attempt to use it outside
of the load run will result in an error.
The parameters pass to the API are:
* function_name - Function name (must be a Lua string)
* callback - Lua function object that will be called when the function is invokes using fcall/fcall_ro
* description - Function description, optional (must be a Lua string).

### Example
The following example creates a library called `lib` with 2 functions, `f1` and `f1`, returns 1 and 2 respectively:
```
local function f1(keys, args)
    return 1
end

local function f2(keys, args)
    return 2
end

redis.register_function('f1', f1)
redis.register_function('f2', f2)
```

Notice: Unlike `eval`, functions inside a library get the KEYS and ARGV as arguments to the
functions and not as global.

### Technical Details

On the load run we only want the user to be able to call a white list on API's. This way, in
the future, if new API's will be added, the new API's will not be available to the load run
unless specifically added to this white list. We put the while list on the `library` object and
make sure the `library` object is only available to the load run by using [lua_setfenv](https://www.lua.org/manual/5.1/manual.html#lua_setfenv) API. This API allows us to set
the `globals` of a function (and all the function it creates). Before starting the load run we
create a new fresh Lua table (call it `g`) that only contains the `library` API (we make sure
to set global protection on this table just like the general global protection already exists
today), then we use [lua_setfenv](https://www.lua.org/manual/5.1/manual.html#lua_setfenv)
to set `g` as the global table of the load run. After the load run finished we update `g`
metatable and set `__index` and `__newindex` functions to be `_G` (Lua default globals),
we also pop out the `library` object as we do not need it anymore.
This way, any function that was created on the load run (and will be invoke using `fcall`) will
see the default globals as it expected to see them and will not have the `library` API anymore.

An important outcome of this new approach is that now we can achieve a distinct global table
for each library (it is not yet like that but it is very easy to achieve it now). In the future we can
decide to remove global protection because global on different libraries will not collide or we
can chose to give different API to different libraries base on some configuration or input.

Notice that this technique was meant to prevent errors and was not meant to prevent malicious
user from exploit it. For example, the load run can still save the `library` object on some local
variable and then using in `fcall` context. To prevent such a malicious use, the C code also make
sure it is running in the right context and if not raise an error.
2022-01-06 13:39:38 +02:00
Ozan Tezcan
568c2e039b
Set errno to EEXIST in redisFork() if child process exists (#10059)
Callers of redisFork() are logging `strerror(errno)` on failure.
`errno` is not set when there is already a child process, causing printing
current value of errno which was set before `redisFork()` call. 

Setting errno to EEXIST on this failure to provide more meaningful error message.
2022-01-06 09:54:21 +02:00
filipe oliveira
5dd15443ac
Added INFO LATENCYSTATS section: latency by percentile distribution/latency by cumulative distribution of latencies (#9462)
# Short description

The Redis extended latency stats track per command latencies and enables:
- exporting the per-command percentile distribution via the `INFO LATENCYSTATS` command.
  **( percentile distribution is not mergeable between cluster nodes ).**
- exporting the per-command cumulative latency distributions via the `LATENCY HISTOGRAM` command.
  Using the cumulative distribution of latencies we can merge several stats from different cluster nodes
  to calculate aggregate metrics .

By default, the extended latency monitoring is enabled since the overhead of keeping track of the
command latency is very small.
 
If you don't want to track extended latency metrics, you can easily disable it at runtime using the command:
 - `CONFIG SET latency-tracking no`

By default, the exported latency percentiles are the p50, p99, and p999.
You can alter them at runtime using the command:
- `CONFIG SET latency-tracking-info-percentiles "0.0 50.0 100.0"`


## Some details:
- The total size per histogram should sit around 40 KiB. We only allocate those 40KiB when a command
  was called for the first time.
- With regards to the WRITE overhead As seen below, there is no measurable overhead on the achievable
  ops/sec or full latency spectrum on the client. Including also the measured redis-benchmark for unstable
  vs this branch. 
- We track from 1 nanosecond to 1 second ( everything above 1 second is considered +Inf )

## `INFO LATENCYSTATS` exposition format

   - Format: `latency_percentiles_usec_<CMDNAME>:p0=XX,p50....` 

## `LATENCY HISTOGRAM [command ...]` exposition format

Return a cumulative distribution of latencies in the format of a histogram for the specified command names.

The histogram is composed of a map of time buckets:
- Each representing a latency range, between 1 nanosecond and roughly 1 second.
- Each bucket covers twice the previous bucket's range.
- Empty buckets are not printed.
- Everything above 1 sec is considered +Inf.
- At max there will be log2(1000000000)=30 buckets

We reply a map for each command in the format:
`<command name> : { `calls`: <total command calls> , `histogram` : { <bucket 1> : latency , < bucket 2> : latency, ...  } }`

Co-authored-by: Oran Agra <oran@redislabs.com>
2022-01-05 14:01:05 +02:00
sundb
4d3c4cfac7
Show the elapsed time of single test and speed up some tests (#10058)
Following #10038.

This PR introduces two changes.
1. Show the elapsed time of a single test in the test output, in order to have a more
detailed understanding of the changes in test run time.

2. Speedup two tests related to `key-load-delay` configuration.
other tests do not seem to be affected by #10003.
2022-01-05 13:49:01 +02:00
Binbin
9538088751
Fix typos in aof.c / redis.conf (#10057) 2022-01-05 12:06:02 +02:00
Ozan Tezcan
d1b5b63872
Fix typo in multi test (#10054) 2022-01-05 10:16:04 +02:00
Binbin
b7f9e9ae39
Add tests for blocking XREAD[GROUP] when the stream ran dry (#10035)
The purpose of this commit is to add some tests to
cover #5299, which was fixed in #5300 but without tests.

This commit should close #5306 and #5299.
2022-01-04 21:48:49 +02:00
Yuta Hongo
8deb9a4f1e
Stringify JSON key of --json option result (#10046)
About RESP3 an ordered collection of key-value pairs, keys and value can
be any other RESP3 type, but a key should be string in JSON spec.
2022-01-04 17:24:29 +02:00
yoav-steinberg
65a7635793
redis-cli --replica reads dummy empty rdb instead of full snapshot (#10044)
This makes redis-cli --replica much faster and reduces COW/fork risks on server side.
This commit also improves the RDB filtering via REPLCONF rdb-filter-only to support no "include" specifiers at all.
2022-01-04 17:09:22 +02:00
Matthieu MOREL
d5a3b3f5ec
Setup dependabot for github-actions and codespell (#9857)
This sets up  dependabot to check weekly updates for pip and github-actions dependencies.
If it finds an update it will create a PR to update the dependency. More information can be found here

It includes the update of:

* vmactions/freebsd-vm from 0.1.4 to 0.1.5
* codespell from 2.0.0 to 2.1.0

Also includes spelling fixes found by the latest version of codespell.
Includes a dedicated .codespell folder so dependabot can read a requirements.txt file and every files dedicated to codespell can be grouped in the same place

Co-Authored-By: Matthieu MOREL <mmorel-35@users.noreply.github.com>
Co-Authored-By: MOREL Matthieu <matthieu.morel@cnp.fr>
2022-01-04 16:19:28 +02:00
Binbin
c57e41c029
Print error messages in monitor/pubsub when errors occurs (#10050)
In monitor/pubsub mode, if the server closes the connection,
for example, use `CLIENT KILL`, redis-cli will exit directly
without printing any error messages.

This commit ensures that redis-cli will try to print the
error messages before exiting. Also there is a minor cleanup
for restart, see the example below.

before:
```
127.0.0.1:6379> monitor
OK
[root@ redis]#

127.0.0.1:6379> subscribe channel
Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "channel"
3) (integer) 1
[root@ redis]#

127.0.0.1:6379> restart
127.0.0.1:6379> get keyUse 'restart' only in Lua debugging mode.
(nil)
```

after:
```
127.0.0.1:6379> monitor
OK
Error: Server closed the connection
[root@ redis]#

127.0.0.1:6379> subscribe channel
Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "channel"
3) (integer) 1
Error: Server closed the connection
[root@ redis]#

127.0.0.1:6379> restart
Use 'restart' only in Lua debugging mode.
```
2022-01-04 14:45:10 +02:00
王辉
747b08bee0
Fix C11_ATOMIC detection on GNU Make 4.3 (#10033)
Older version of GNU Make (<4.3) required quoting of number signs (#) to
avoid them being treated as a comment. Newer versions will treat this
quote as a literal.

This issue and a proposed solution is discussed here:
https://lists.gnu.org/archive/html/info-gnu/2020-01/msg00004.html

Co-authored-by: Yossi Gottlieb <yossigo@gmail.com>
2022-01-04 14:05:00 +02:00
guybe7
ac84b1cd82
Ban snapshot-creating commands and other admin commands from transactions (#10015)
Creating fork (or even a foreground SAVE) during a transaction breaks the atomicity of the transaction.
In addition to that, it could mess up the propagated transaction to the AOF file.

This change blocks SAVE, PSYNC, SYNC and SHUTDOWN from being executed inside MULTI-EXEC.
It does that by adding a command flag, so that modules can flag their commands with that flag too.

Besides it changes BGSAVE, BGREWRITEAOF, and CONFIG SET appendonly, to turn the
scheduled flag instead of forking righ taway.

Other changes:
* expose `protected`, `no-async-loading`, and `no_multi` flags in COMMAND command
* add a test to validate propagation of FLUSHALL inside a transaction.
* add a test to validate how CONFIG SET that errors reacts in a transaction

Co-authored-by: Oran Agra <oran@redislabs.com>
2022-01-04 13:37:47 +02:00
zhaozhao.zz
2e1979a21e
use startEvictionTimeProc() in config set maxmemory (#10019)
This would mean that the effects of `CONFIG SET maxmemory` may not be visible once the command returns.
That could anyway happen since incremental eviction was added in redis 6.2 (see #7653)

We do this to fix one of the propagation bugs about eviction see #9890 and #10014.
2022-01-04 13:08:10 +02:00
chenyang8094
87789fae0b
Implement Multi Part AOF mechanism to avoid AOFRW overheads. (#9788)
Implement Multi-Part AOF mechanism to avoid overheads during AOFRW.
Introducing a folder with multiple AOF files tracked by a manifest file.

The main issues with the the original AOFRW mechanism are:
* buffering of commands that are processed during rewrite (consuming a lot of RAM)
* freezes of the main process when the AOFRW completes to drain the remaining part of the buffer and fsync it.
* double disk IO for the data that arrives during AOFRW (had to be written to both the old and new AOF files)

The main modifications of this PR:
1. Remove the AOF rewrite buffer and related code.
2. Divide the AOF into multiple files, they are classified as two types, one is the the `BASE` type,
  it represents the full amount of data (Maybe AOF or RDB format) after each AOFRW, there is only
  one `BASE` file at most. The second is `INCR` type, may have more than one. They represent the
  incremental commands since the last AOFRW.
3. Use a AOF manifest file to record and manage these AOF files mentioned above.
4. The original configuration of `appendfilename` will be the base part of the new file name, for example:
  `appendonly.aof.1.base.rdb` and `appendonly.aof.2.incr.aof`
5. Add manifest-related TCL tests, and modified some existing tests that depend on the `appendfilename`
6. Remove the `aof_rewrite_buffer_length` field in info.
7. Add `aof-disable-auto-gc` configuration. By default we're automatically deleting HISTORY type AOFs.
  It also gives users the opportunity to preserve the history AOFs. just for testing use now.
8. Add AOFRW limiting measure. When the AOFRW failures reaches the threshold (3 times now),
  we will delay the execution of the next AOFRW by 1 minute. If the next AOFRW also fails, it will be
  delayed by 2 minutes. The next is 4, 8, 16, the maximum delay is 60 minutes (1 hour). During the limit
  period, we can still use the 'bgrewriteaof' command to execute AOFRW immediately.
9. Support upgrade (load) data from old version redis.
10. Add `appenddirname` configuration, as the directory name of the append only files. All AOF files and
  manifest file will be placed in this directory.
11. Only the last AOF file (BASE or INCR) can be truncated. Otherwise redis will exit even if
  `aof-load-truncated` is enabled.

Co-authored-by: Oran Agra <oran@redislabs.com>
2022-01-03 19:14:13 +02:00
Meir Shpilraien (Spielrein)
78a62c0124
Fix OOM error not raised of functions (#10048)
OOM Error did not raise on functions due to a bug.
Added test to verify the fix.
2022-01-03 19:04:29 +02:00
Madelyn Olson
5460c10047
Implement clusterbus message extensions and cluster hostname support (#9530)
Implement the ability for cluster nodes to advertise their location with extension messages.
2022-01-02 19:48:29 -08:00
Harkrishn Patro
9f8885760b
Sharded pubsub implementation (#8621)
This commit implements a sharded pubsub implementation based off of shard channels.

Co-authored-by: Harkrishn Patro <harkrisp@amazon.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2022-01-02 16:54:47 -08:00
Binbin
b8ba942ac2
Add DUMP RESTORE tests for redis-cli -x and -X options (#10041)
This commit adds DUMP RESTORES tests for the -x and -X options.
I wanted to add it in #9980 which introduce the -X option, but
back then i failed due to some errors (related to redis-cli call).
2022-01-02 13:58:22 +02:00
yoav-steinberg
2ff3fc1790
Make sure replicas don't write their own replies to the replication link (#10020)
Since #9166 we have an assertion here to make sure replica clients don't write anything to their buffer.
But in reality a replica may attempt write data to it's buffer simply by sending a command on the replication link.
This command in most cases will be rejected since #8868 but it'll still generate an error.
Actually the only valid command to send on a replication link is 'REPCONF ACK` which generates no response.

We want to keep the design so that replicas can send commands but we need to avoid any situation where we start
putting data in their response buffers, especially since they aren't used anymore. This PR makes sure to disconnect
a rogue client which generated a write on the replication link that cause something to be written to the response buffer.

To recreate the bug this fixes simply connect via telnet to a redis server and write sync\r\n wait for the the payload to
be written and then write any command (valid or invalid), such as ping\r\n on the telnet connection. It'll crash the server.
2022-01-02 11:18:43 +02:00
Joey from AWS
09c668f220
Report slot to keys map size in MEMORY STATS in cluster mode (#10017)
Report slot to keys map size in MEMORY STATS in cluster mode
Report dictMetadataSize in MEMORY USAGE command as well
2022-01-02 10:39:59 +02:00
Viktor Söderqvist
45a155bd0f
Wait for replicas when shutting down (#9872)
To avoid data loss, this commit adds a grace period for lagging replicas to
catch up the replication offset.

Done:

* Wait for replicas when shutdown is triggered by SIGTERM and SIGINT.

* Wait for replicas when shutdown is triggered by the SHUTDOWN command. A new
  blocked client type BLOCKED_SHUTDOWN is introduced, allowing multiple clients
  to call SHUTDOWN in parallel.
  Note that they don't expect a response unless an error happens and shutdown is aborted.

* Log warning for each replica lagging behind when finishing shutdown.

* CLIENT_PAUSE_WRITE while waiting for replicas.

* Configurable grace period 'shutdown-timeout' in seconds (default 10).

* New flags for the SHUTDOWN command:

    - NOW disables the grace period for lagging replicas.

    - FORCE ignores errors writing the RDB or AOF files which would normally
      prevent a shutdown.

    - ABORT cancels ongoing shutdown. Can't be combined with other flags.

* New field in the output of the INFO command: 'shutdown_in_milliseconds'. The
  value is the remaining maximum time to wait for lagging replicas before
  finishing the shutdown. This field is present in the Server section **only**
  during shutdown.

Not directly related:

* When shutting down, if there is an AOF saving child, it is killed **even** if AOF
  is disabled. This can happen if BGREWRITEAOF is used when AOF is off.

* Client pause now has end time and type (WRITE or ALL) per purpose. The
  different pause purposes are *CLIENT PAUSE command*, *failover* and
  *shutdown*. If clients are unpaused for one purpose, it doesn't affect client
  pause for other purposes. For example, the CLIENT UNPAUSE command doesn't
  affect client pause initiated by the failover or shutdown procedures. A completed
  failover or a failed shutdown doesn't unpause clients paused by the CLIENT
  PAUSE command.

Notes:

* DEBUG RESTART doesn't wait for replicas.

* We already have a warning logged when a replica disconnects. This means that
  if any replica connection is lost during the shutdown, it is either logged as
  disconnected or as lagging at the time of exit.

Co-authored-by: Oran Agra <oran@redislabs.com>
2022-01-02 09:50:15 +02:00
yoav-steinberg
1bf6d6f11e
Generate RDB with Functions only via redis-cli --functions-rdb (#9968)
This is needed in order to ease the deployment of functions for ephemeral cases, where user
needs to spin up a server with functions pre-loaded.

#### Details:

* Added `--functions-rdb` option to _redis-cli_.
* Functions only rdb via `REPLCONF rdb-filter-only functions`. This is a placeholder for a space
  separated inclusion filter for the RDB. In the future can be `REPLCONF rdb-filter-only
  "functions db:3 key-patten:user*"` and a complementing `rdb-filter-exclude` `REPLCONF`
  can also be added.
* Handle "slave requirements" specification to RDB saving code so we can use the same RDB
  when different slaves express the same requirements (like functions-only) and not share the
  RDB when their requirements differ. This is currently just a flags `int`, but can be extended to
  a more complex structure with various filter fields.
* make sure to support filters only in diskless replication mode (not to override the persistence file),
  we do that by forcing diskless (even if disabled by config)

other changes:
* some refactoring in rdb.c (extract portion of a big function to a sub-function)
* rdb_key_save_delay used in AOFRW too
* sendChildInfo takes the number of updated keys (incremental, rather than absolute)

Co-authored-by: Oran Agra <oran@redislabs.com>
2022-01-02 09:39:01 +02:00
sundb
888e92eb57
Fix a valgrind test failure due to slowly shutdown (#10038)
This pr is mainly to solve the problem that redis process cannot be exited normally, due to changes in #10003.
When a test uses the `key-load-delay` config to delay loading, but does not reset it at the end of the test, will lead to server wait for the loading to reach the event
loop (once in 2mb) before actually shutting down.
2022-01-01 17:45:13 +02:00
Madelyn Olson
39ced1afa1
Constrain error message to just node-id (#10022) 2021-12-31 09:32:33 -08:00
sundb
73951abe7b
Fix when the master connection is disconnected, replication retry read indefinitely (#10032)
Now if redis is still loading when we receive sigterm, we will wait for the loading to reach the event
loop (once in 2mb) before actually shutting down. See #10003.

This change caused valgrind CI to fail.
See https://github.com/redis/redis/runs/4662901673?check_suite_focus=true

This pr is mainly to solve the problem that redis process cannot be exited normally.
When the master is disconnected, if repl is processing diskless loading and using `connRead` to read data from master,
it may enter an infinite retry state, which does not handle `connRead` returning 0(master connection disconnected).
2021-12-31 14:08:04 +02:00
Viktor Söderqvist
e4b3a257ee
Modules: Mark all APIs non-experimental (#9983)
These exist for quite some time, and are no longer experimental
2021-12-30 12:17:22 +02:00
Binbin
4836ae32c7
redis-cli: Add -X option and extend --cluster call take arg from stdin (#9980)
There are two changes in this commit:

1. Add -X option to redis-cli.
Currently `-x` can only be used to provide the last argument,
so you can do `redis-cli dump keyname > key.dump`,
and then do `redis-cli -x restore keyname 0 < key.dump`.

But what if you want to add the replace argument (which comes last?).
oran suggested adding such usage:
`redis-cli -X <tag> restore keyname <tag> replace < key.dump`

i.e. you're able to provide a string in the arguments that's gonna be
substituted with the content from stdin.

Note that the tag name should not conflict with others non-replaced args.
And the -x and -X options are conflicting.

Some usages:
```
[root]# echo mypasswd | src/redis-cli -X passwd_tag mset username myname password passwd_tag                                                   OK
[root]# echo username > username.txt
[root]# head -c -1 username.txt | src/redis-cli -X name_tag mget name_tag password
1) "myname"
2) "mypasswd\n"
```

2. Handle the combination of both `-x` and `--cluster` or `-X` and `--cluster`
Extend the broadcast option to receive the last arg or <tag> arg from the stdin.

Now we can use `redis-cli -x --cluster call <host>:<port> cmd`,
or `redis-cli -X <tag> --cluster call <host>:<port> cmd <tag>`.
(support part of #9899)
2021-12-30 12:10:04 +02:00
Viktor Söderqvist
5006eab552
Module API doc formatting from #9656 (#10026)
Add blank before lists, so that they will be rendered as lists.

Commands RM_GetCommand and RM_CreateSubcommand were added in #9656.
2021-12-30 11:34:41 +02:00
Itamar Haber
aec8c57701
Add missing metadata to the commands SSOT files. (#10016)
Add missing information about commands, mainly from reviewing redis-doc and removing
the metadata from it (https://github.com/redis/redis-doc/pull/1722)

* Reintroduces CLUSTER S****S (supported by Redis) but missing from the JSON / docs (related? #9675).
  Note that without that json file, the command won't work (breaking change)
* Adds the `replicas` argument (exists in Redis) to `CLIENT KILL`.
* Adds `history` entries to several commands based on redis-doc's man pages.
* Adds `since` to applicable command arguments based on `history` (this basically makes
  some of `history` redundant - perhaps at a later stage).
* Uses proper semantic versioning in all version references.
* Also removes `geoencodeCommand` and `geodecodeCommand` header
  declarations per b96af595a5.
2021-12-29 21:57:40 +02:00
Ozan Tezcan
b0c06e904a
Fixed typo in test tag (for needs:debug) (#10021) 2021-12-28 16:23:02 +02:00
perryitay
43229e4f10
Safe and organized exit when receiving sigterm while loading (#10003)
on the signal handler we are enabling server.shutdown_asap flag instead of just doing `exit()`,
and then catch it on the whileBlockedCron() where we prepare for shutdown correctly.

this is a more ore organized and safe termination, the old approach was missing these for example:
1. removal of the pidfile
2. shutdown event to modules
2021-12-28 13:25:56 +02:00
guybe7
266d95066d
Remove incomplete fix of a broader problem (#10013)
Preventing COFIG SET maxmemory from propagating is just the tip of the iceberg.
Module that performs a write operation in a notification can cause any
command to be propagated, based on server.dirty

We need to come up with a better solution.
2021-12-28 10:19:58 +02:00
David CARLIER
2c5573e894
Check somaxconn system settings on macOS, FreeBSD and OpenBSD. (#9972)
Co-authored-by: Yossi Gottlieb <yossigo@gmail.com>
2021-12-28 09:20:10 +02:00
chenyang8094
af0b50f83a
Tests: don't rely on the response of MEMORY USAGE when mem_allocator is not jemalloc (#10010)
It turns out that libc malloc can return an allocation of a different size on requests of the same size.
this means that matching MEMORY USAGE of one key to another copy of the same data can fail.

Solution:
Keep running the test that calls MEMORY USAGE, but ignore the response.
We do that by introducing a new utility function to get the memory usage, which always returns 1
when the allocator is not jemalloc.

Other changes:
Some formatting for datatype2.tcl

Co-authored-by: Oran Agra <oran@redislabs.com>
2021-12-27 21:37:21 +02:00
Itamar Haber
f810510bb2
Adds utils/gen-commands-json.py (#9958)
Following #9656, this script generates a "commands.json" file from the output
of the new COMMAND. The output of this script is used in redis/redis-doc#1714
and by redis/redis-io#259. This also converts a couple of rogue dashes (in 
'key-specs' and 'multiple-token' flags) to underscores (continues #9959).
2021-12-27 19:31:13 +02:00
chenyang8094
317464a386
Fix failing test due to recent change in transaction propagation (#10006)
PR #9890 may have introduced a problem.
There are tests that use MULTI-EXEC to make sure two BGSAVE / BGREWRITEAOF are executed together.
But now it's not valid to run run commands that create a snapshot inside a transaction (gonna be blocked soon)
This PR modifies the test not to rely on MULTI-EXEC.

Co-authored-by: Oran Agra <oran@redislabs.com>
2021-12-27 15:18:17 +02:00
guybe7
0f15e025e6
Fix race in propagation test (#10012)
There's a race between testing DBSIZE and the thread starting.
If the thread hadn't started by the time we checked DBISZE, no
keys will have been evicted.
The correct way is to check the evicted_keys stat.
2021-12-27 12:31:24 +02:00
Binbin
e84ccc3f56
santize dump payload: fix carsh when zset with NAN score (#10002)
`zslInsert` with a NAN score will crash the server.
This one found by the `corrupt-dump-fuzzer`.
2021-12-26 11:40:11 +02:00
Meir Shpilraien (Spielrein)
365cbf46a7
Add FUNCTION DUMP and RESTORE. (#9938)
Follow the conclusions to support Functions in redis cluster (#9899)

Added 2 new FUNCTION sub-commands:
1. `FUNCTION DUMP` - dump a binary payload representation of all the functions.
2. `FUNCTION RESTORE <PAYLOAD> [FLUSH|APPEND|REPLACE]` - give the binary payload extracted
   using `FUNCTION DUMP`, restore all the functions on the given payload. Restore policy can be given to
   control how to handle existing functions (default is APPEND):
   * FLUSH: delete all existing functions.
   * APPEND: appends the restored functions to the existing functions. On collision, abort.
   * REPLACE: appends the restored functions to the existing functions. On collision,
     replace the old function with the new function.

Modify `redis-cli --cluster add-node` to use `FUNCTION DUMP` to get existing functions from
one of the nodes in the cluster, and `FUNCTION RESTORE` to load the same set of functions
to the new node. `redis-cli` will execute this step before sending the `CLUSTER MEET` command
to the new node. If `FUNCTION DUMP` returns an error, assume the current Redis version do not
support functions and skip `FUNCTION RESTORE`. If `FUNCTION RESTORE` fails, abort and do not send
the `CLUSTER MEET` command. If the new node already contains functions (before the `FUNCTION RESTORE`
is sent), abort and do not add the node to the cluster. Test was added to verify
`redis-cli --cluster add-node` works as expected.
2021-12-26 09:03:37 +02:00
Meir Shpilraien (Spielrein)
08ff606b0b
Changed fuction name to be case insensitive. (#9984)
Use case insensitive string comparison for function names (like we do for commands and configs)
In addition, add verification that the functions only use the following characters: [a-zA-Z0-9_]
2021-12-26 08:37:24 +02:00