+ + + +

+ +CommandReference: Contents
  Redis Command Reference
    Connection handling
    Commands operating on string values
    Commands operating on the key space
    Commands operating on lists
    Commands operating on sets
    Multiple databases handling commands
    Sorting
    Persistence control commands
    Remote server control commands +

+ +

CommandReference

+ +

Redis Command Reference

Every command name links to a specific wiki page describing the behavior of the command.

Commands operating on string values

SET key value set a key to a string value
GET key return the string value of the key
SETNX key value set a key to a string value if the key does not exist
INCR key increment the integer value of key
INCRBY key integer increment the integer value of key by integer
INCR key decrement the integer value of key
DECRBY key integer decrement the integer value of key by integer
EXISTS key test if a key exists
DEL key delete a key
TYPE key return the type of the value stored at key

Commands operating on the key space

KEYS pattern return all the keys matching a given pattern
RANDOMKEY return a random key from the key space
RENAME oldname newname rename the old key in the new one, destroing the newname key if it already exists
RENAMENX oldname newname rename the old key in the new one, if the newname key does not already exist
DBSIZE return the number of keys in the current db

Commands operating on lists

RPUSH key value Append an element to the tail of the List value at key
LPUSH key value Append an element to the head of the List value at key
LLEN key Return the length of the List value at key
LRANGE key start end Return a range of elements from the List at key
LTRIM key start end Trim the list at key to the specified range of elements
LINDEX key index Return the element at index position from the List at key
LSET key index value Set a new value as the element at index position of the List at key
LREM key count value Remove the first-N, last-N, or all the elements matching value from the List at key
LPOP key Return and remove (atomically) the first element of the List at key
RPOP key Return and remove (atomically) the last element of the List at key

Commands operating on sets

SADD key member Add the specified member to the Set value at key
SREM key member Remove the specified member from the Set value at key
SCARD key Return the number of elements (the cardinality) of the Set at key
SISMEMBER key member Test if the specified value is a member of the Set at key
SINTER key1 key2 ... keyN Return the intersection between the Sets stored at key1, key2, ..., keyN
SINTERSTORE dstkey key1 key2 ... keyN Compute the intersection between the Sets stored at key1, key2, ..., keyN, and store the resulting Set at dstkey
SMEMBERS key Return all the members of the Set value at key

Multiple databases handling commands

SELECT index Select the DB having the specified index
MOVE key dbindex Move the key from the currently selected DB to the DB having as index dbindex
FLUSHDB Remove all the keys of the currently selected DB
FLUSHALL Remove all the keys from all the databases

Sorting

SORT key BY pattern LIMIT start end GET pattern ASC|DESC ALPHA Sort a Set or a List accordingly to the specified parameters

Persistence control commands

SAVE Synchronously save the DB on disk
BGSAVE Asynchronously save the DB on disk
LASTSAVE Return the UNIX time stamp of the last successfully saving of the dataset on disk
SHUTDOWN Synchronously save the DB on disk, then shutdown the server

Remote server control commands

INFO provide information and statistics about the server

+ +

+ + + +

+ +FAQ: Contents
  Why I need Redis if there is already memcachedb, Tokyo Cabinet, ...?
  Isn't this key-value thing just hype?
  Can I backup a Redis DB while the server is working?
  What's the Redis memory footprint?
  I like Redis high level operations and features, but I don't like it takes everything in memory and I can't have a dataset larger the memory. Plans to change this?
  Ok but I absolutely need to have a DB larger than memory, still I need the Redis features
  What happens if Redis runs out of memory?
  What Redis means actually?
  Why did you started the Redis project? +

+ +

FAQ

+ +

Why I need Redis if there is already memcachedb, Tokyo Cabinet, ...?

Memcachedb is basically memcached done persistent. Redis is a different evolution +path in the key-value DBs, the idea is that the main advantages of key-value DBs +are retained even without a so severe loss of comfort of plain key-value DBs. +So Redis offers more features:

Keys can store different data types, not just strings. Notably Lists and Sets. For example if you want to use Redis as a log storage system for different computers every computer can just RPUSH data to the computer_ID key. Don't want to save more than 1000 log lines per computer? Just issue a LTRIM computer_ID 0 999 command to trim the list after every push.

Another example is about Sets. Imagine to build a social news site like Reddit. Every time a user upvote a given news you can just add to the news_ID_upmods key holding a value of type SET the id of the user that did the upmodding. Sets can also be used to index things. Every key can be a tag holding a SET with the IDs of all the objects associated to this tag. Using Redis set intersection you obtain the list of IDs having all this tags at the same time.

We wrote a simple Twitter Clone using just Redis as database. Download the source code from the download section and imagine to write it with a plain key-value DB without support for lists and sets... it's much harder.

Multiple DBs. Using the SELECT command the client can select different datasets. This is useful because Redis provides a MOVE atomic primitive that moves a key form a DB to another one, if the target DB already contains such a key it returns an error: this basically means a way to perform locking in distributed processing.

So what is Redis really about? The User interface with the programmer. Redis aims to export to the programmer the right tools to model a wide range of problems. Sets, Lists with O(1) push operation, lrange and ltrim, server-side fast intersection between sets, are primitives that allow to model complex problems with a key value database.

Isn't this key-value thing just hype?

I imagine key-value DBs, in the short term future, to be used like you use memory in a program, with lists, hashes, and so on. With Redis it's like this, but this special kind of memory containing your data structures is shared, atomic, persistent.

When we write code it is obvious, when we take data in memory, to use the most sensible data structure for the work, right? Incredibly when data is put inside a relational DB this is no longer true, and we create an absurd data model even if our need is to put data and get this data back in the same order we put it inside (an ORDER BY is required when the data should be already sorted. Strange, dont' you think?).

Key-value DBs bring this back at home, to create sensible data models and use the right data structures for the problem we are trying to solve.

Can I backup a Redis DB while the server is working?

Yes you can. When Redis saves the DB it actually creates a temp file, then rename(2) that temp file name to the destination file name. So even while the server is working it is safe to save the database file just with the cp unix command. Note that you can use master-slave replication in order to have redundancy of data, but if all you need is backups, cp or scp will do the work pretty well.

What's the Redis memory footprint?

Worst case scenario: 1 Million keys with the key being the natural numbers from 0 to 999999 and the string "Hello World" as value use 100MB on my Intel macbook (32bit). Note that the same data stored linearly in an unique string takes something like 16MB, this is the norm because with small keys and values there is a lot of overhead. Memcached will perform similarly.

With large keys/values the ratio is much better of course.

64 bit systems will use much more memory than 32 bit systems to store the same keys, especially if the keys and values are small, this is because pointers takes 8 bytes in 64 bit systems. But of course the advantage is that you can have a lot of memory in 64 bit systems, so to run large Redis servers a 64 bit system is more or less required.

I like Redis high level operations and features, but I don't like it takes everything in memory and I can't have a dataset larger the memory. Plans to change this?

The whole key-value hype started for a reason: performances. Redis takes the whole dataset in memory and writes asynchronously on disk in order to be very fast, you have the best of both worlds: hyper-speed and persistence of data, but the price to pay is exactly this, that the dataset must fit on your computers RAM.

If the data is larger then memory, and this data is stored on disk, what happens is that the bottleneck of the disk I/O speed will start to ruin the performances. Maybe not in benchmarks, but once you have real load with distributed key accesses the data must come from disk, and the disk is damn slow. Not only, but Redis supports higher level data structures than the plain values. To implement this things on disk is even slower.

Redis will always continue to hold the whole dataset in memory because this days scalability requires to use RAM as storage media, and RAM is getting cheaper and cheaper. Today it is common for an entry level server to have 16 GB of RAM! And in the 64-bit era there are no longer limits to the amount of RAM you can have in theory.

Ok but I absolutely need to have a DB larger than memory, still I need the Redis features

One possible solution is to use both MySQL and Redis at the same time, basically take the state on Redis, and all the things that get accessed very frequently: user auth tokens, Redis Lists with chronologically ordered IDs of the last N-comments, N-posts, and so on. Then use MySQL as a simple storage engine for larger data, that is just create a table with an auto-incrementing ID as primary key and a large BLOB field as data field. Access MySQL data only by primary key (the ID). The application will run the high traffic queries against Redis but when there is to take the big data will ask MySQL for specific resources IDs.

What happens if Redis runs out of memory?

With modern operating systems malloc() returning NULL is not common, usually the server will start swapping and Redis performances will be disastrous so you'll know it's time to use more Redis servers or get more RAM.

However it is planned to add a configuration directive to tell Redis to stop accepting queries but instead to SAVE the latest data and quit if it is using more than a given amount of memory. Also the new INFO command (work in progress in this days) will report the amount of memory Redis is using so you can write scripts that monitor your Redis servers checking for critical conditions.

Update: redis SVN is able to know how much memory it is using and report it via the INFO command.

What Redis means actually?

Redis means two things: +

it's a joke on the word Redistribute (instead to use just a Relational DB redistribute your workload among Redis servers)
it means REmote DIctionary Server

Why did you started the Redis project?

In order to scale LLOOGG. But after I got the basic server working I liked the idea to share the work with other guys, and Redis was turned into an open source project. + +

+ +

+ + + +

+ +ProtocolSpecification: Contents
  Protocol Specification
    Networking layer
    Simple INLINE commands
    Bulk commands
    Bulk replies
    Bulk reply error reporting
    Multi-Bulk replies
    Nil elements in Multi-Bulk replies
    Multi-Bulk replies errors
    Status code reply
    Integer reply
    Single line reply
    Multiple commands and pipelining +

+ +

ProtocolSpecification

+ +

Protocol Specification

The Redis protocol is a compromise between being easy to parse by a computer +and being easy to parse by an human. Before reading this section you are +strongly encouraged to read the "REDIS TUTORIAL" section of this README in order +to get a first feeling of the protocol playing with it by TELNET.

Networking layer

A client connects to a Redis server creating a TCP connection to the port 6973. +Every redis command or data transmitted by the client and the server is +terminated by "\r\n" (CRLF).

Simple INLINE commands

The simplest commands are the inline commands. This is an example of a +server/client chat (the server chat starts with S:, the client chat with C:)

+C: PING
+S: +PONG
+

An inline command is a CRLF-terminated string sent to the client. The server +usually replies to inline commands with a single line that can be a number +or a return code.

When the server replies with a status code (that is a one line reply just indicating if the operation succeeded or not), if the first character of the +reply is a "+" then the command succeeded, if it is a "-" then the following +part of the string is an error.

The following is another example of an INLINE command returning an integer:

+C: EXISTS somekey
+S: 0
+

Since 'somekey' does not exist the server returned '0'.

Note that the EXISTS command takes one argument. Arguments are separated +simply by spaces.

Bulk commands

A bulk command is exactly like an inline command, but the last argument +of the command must be a stream of bytes in order to send data to the server. +the "SET" command is a bulk command, see the following example:

+C: SET mykey 6
+C: foobar
+S: +OK
+

The last argument of the commnad is '6'. This specify the number of DATA +bytes that will follow (note that even this bytes are terminated by two +additional bytes of CRLF).

All the bulk commands are in this exact form: instead of the last argument +the number of bytes that will follow is specified, followed by the bytes, +and CRLF. In order to be more clear for the programmer this is the string +sent by the client in the above sample:

"SET mykey 6\r\nfoobar\r\n"

Bulk replies

The server may reply to an inline or bulk command with a bulk reply. See +the following example:

+C: GET mykey
+S: 6
+S: foobar
+

A bulk reply is very similar to the last argument of a bulk command. The +server sends as the first line the number of bytes of the actual reply +followed by CRLF, then the bytes are sent followed by additional two bytes +for the final CRLF. The exact sequence sent by the server is:

"6\r\nfoobar\r\n"

+If the requested value does not exist the bulk reply will use the special +value 'nil' instead to send the line containing the number of bytes to read. +This is an example:

+C: GET nonexistingkey
+S: nil
+

The client library API should not return an empty string, but a nil object. +For example a Ruby library should return 'nil' while a C library should return +NULL.

Bulk reply error reporting

Bulk replies can signal errors, for example trying to use GET against a list +value is not permitted. Bulk replies use a negative bytes count in order to +signal an error. An error string of ABS(bytes_count) bytes will follow. See +the following example:

+S: GET alistkey
+S: -38
+S: -ERR Requested element is not a string
+

-38 means: sorry your operation resulted in an error, but a 38 bytes string +that explains this error will follow. Client APIs should abort on this kind +of errors, for example a PHP client should call the die() function.

The following commands reply with a bulk reply: GET, KEYS, LINDEX, LPOP, RPOP

Multi-Bulk replies

Commands similar to LRANGE needs to return multiple values (every element +of the list is a value, and LRANGE needs to return more than a single element). This is accomplished using multiple bulk writes, +prefixed by an initial line indicating how many bulk writes will follow. +Example:

+C: LRANGE mylist 0 3
+S: 4
+S: 3
+S: foo
+S: 3
+S: bar
+S: 5
+S: Hello
+S: 5
+S: World
+

The first line the server sent is "4\r\n" in order to specify that four bulk +write will follow. Then every bulk write is transmitted.

If the specified key does not exist instead of the number of elements in the +list, the special value 'nil' is sent. Example:

+C: LRANGE nokey 0 1
+S: nil
+

A client library API SHOULD return a nil object and not an empty list when this +happens. This makes possible to distinguish between empty list and non existing ones.

Nil elements in Multi-Bulk replies

Single elements of a multi bulk reply may have -1 length, in order to signal that this elements are missing and not empty strings. This can happen with the SORT command when used with the GET pattern option when the specified key is missing. Example of a multi bulk reply containing an empty element:

+S: 3
+S: 3
+S: foo
+S: -1
+S: 3
+S: bar
+

The second element is nul. The client library should return something like this:

+["foo",nil,"bar"]
+

Multi-Bulk replies errors

Like bulk reply errors Multi-bulk reply errors are reported using a negative +count. Example:

+C: LRANGE stringkey 0 1
+S: -38
+S: -ERR Requested element is not a string
+

The following commands reply with a multi-bulk reply: LRANGE, LINTER

Check the Bulk replies errors section for more information.

Status code reply

As already seen a status code reply is in the form of a single line string +terminated by "\r\n". For example:

++OK
+

and

+-ERR no suck key
+

are two examples of status code replies. The first character of a status code reply is always "+" or "-".

The following commands reply with a status code reply: +PING, SET, SELECT, SAVE, BGSAVE, SHUTDOWN, RENAME, LPUSH, RPUSH, LSET, LTRIM

Integer reply

This type of reply is just a CRLF terminated string representing an integer. For example "0\r\n", or "1000\r\n" are integer replies.

With commands like INCR or LASTSAVE using the integer reply to actually return a value there is no special meaning for the returned integer. It is just an incremental number for INCR, a UNIX time for LASTSAVE and so on.

Some commands like EXISTS will return 1 for true and 0 for false.

Other commands like SADD, SREM and SETNX will return 1 if the operation was actually done, 0 otherwise, and a negative value if the operation is invalid (for example SADD against a non-set value), accordingly to this table: +

+-1 no such key
+-2 operation against the a key holding a value of the wrong type
+-3 source and destiantion objects/dbs are the same
+-4 argument out of range
+

+In all this cases it is mandatory that the client raises an error instead to pass the negative value to the caller. Please check the commands documentation for the exact behaviour.

The following commands will reply with an integer reply: SETNX, DEL, EXISTS, INCR, INCRBY, DECR, DECRBY, DBSIZE, LASTSAVE, RENAMENX, MOVE, LLEN, SADD, SREM, SISMEMBER, SCARD

The commands that will never return a negative integer (commands that can't fail) are: INCR, DECR, INCRBY, DECRBY, LASTSAVE, EXISTS, SETNX, DEL, DBSIZE.

Single line reply

This replies are just single line strings terminated by CRLF. Only two commands reply in this way currently, RANDOMKEY and TYPE.

Multiple commands and pipelining

A client can use the same connection in order to issue multiple commands. +Pipelining is supported so multiple commands can be sent with a single +write operation by the client, it is not needed to read the server reply +in order to issue the next command. All the replies can be read at the end.

Usually Redis server and client will have a very fast link so this is not +very important to support this feature in a client implementation, still +if an application needs to issue a very large number of commands in short +time to use pipelining can be much faster.

+

+ +

+ + + +

+ +README: Contents
  Introduction
  Beyond key-value databases
  What are the differences between Redis and Memcached?
  What are the differences between Redis and Tokyo Cabinet / Tyrant?
  Does Redis support locking?
  Multiple databases support
  Redis Data Types
    Implementation Details
  Redis Tutorial
  License
  Credits +

+ +

README

+ +

Introduction

Redis is a database. To be more specific redis is a very simple database +implementing a dictionary where keys are associated with values. For example +I can set the key "surname_1992" to the string "Smith".

Redis takes the whole dataset in memory, but the dataset is persistent +since from time to time Redis writes a dump of the dataset on disk asynchronously. The dump is loaded every time the server is restarted. This means that if a system crash occurs the last few queries can get lost (that is acceptable in many applications), so we supported master-slave replication from the early days.

Beyond key-value databases

In most key-value databases keys and values are simple strings. In Redis keys are just strings too, but the associated values can be Strings, Lists and Sets, and there are commands to perform complex atomic operations against this data types, so you can think at Redis as a data structures server.

For example you can append elements to a list stored at the key "mylist" using the LPUSH or RPUSH operation in O(1). Later you'll be able to get a range of elements with LRANGE or trim the list with LTRIM. Sets are very flexible too, it is possible to add and remove elements from Sets (unsorted collections of strings), and then ask for server-side intersection of Sets.

All this features, the support for sorting Lists and Sets, allow to use Redis as the sole DB for your scalable application without the need of any relational database. We wrote a simple Twitter clone in PHP + Redis to show a real world example, the link points to an article explaining the design and internals in very simple words.

What are the differences between Redis and Memcached?

In the following ways:

Memcached is not persistent, it just holds everything in memory without saving since its main goal is to be used as a cache. Redis instead can be used as the main DB for the application. We wrote a simple Twitter clone using only Redis as database.

Like memcached Redis uses a key-value model, but while keys can just be strings, values in Redis can be lists and sets, and complex operations like intersections, set/get n-th element of lists, pop/push of elements, can be performed against sets and lists. It is possible to use lists as message queues.

What are the differences between Redis and Tokyo Cabinet / Tyrant?

Redis and Tokyo can be used for the same applications, but actually they are ery different beasts:

Tokyo is purely key-value, everything beyond key-value storing of strings is delegated to an embedded Lua interpreter. AFAIK there is no way to guarantee atomicity of operations like pushing into a list, and every time you want to have data structures inside a Tokyo key you have to perform some kind of object serialization/de-serialization.

Tokyo stores data on disk, synchronously, this means you can have datasets bigger than memory, but that under load, like every kind of process that relay on the disk I/O for speed, the performances may start to degrade. With Redis you don't have this problems but you have another problem: the dataset in every single server must fit in your memory.

Redis is generally an higher level beast in the operations supported. Things like SORTing, Server-side set-intersections, can't be done with Tokyo. But Redis is not an on-disk DB engine like Tokyo: the latter can be used as a fast DB engine in your C project without the networking overhead just linking to the library. Still remember that in many scalable applications you need multiple servers talking with multiple servers, so the server-client model is almost always needed.

Does Redis support locking?

No, the idea is to provide atomic primitives in order to make the programmer +able to use redis with locking free algorithms. For example imagine you have +10 computers and 1 redis server. You want to count words in a very large text. +This large text is split among the 10 computers, every computer will process +its part and use Redis's INCR command to atomically increment a counter +for every occurrence of the word found.

INCR/DECR are not the only atomic primitives, there are others like PUSH/POP +on lists, POP RANDOM KEY operations, UPDATE and so on. For example you can +use Redis like a Tuple Space (http://en.wikipedia.org/wiki/Tuple_space) in +order to implement distributed algorithms.

(News: locking with key-granularity is now planned)

Multiple databases support

Another synchronization primitive is the support for multiple DBs. By default DB 0 is selected for every new connection, but using the SELECT command it is possible to select a different database. The MOVE operation can move an item from one DB to another atomically. This can be used as a base for locking free algorithms together with the 'RANDOMKEY' or 'POPRANDOMKEY' commands.

Redis Data Types

Redis supports the following three data types as values:

Strings: just any sequence of bytes. Redis strings are binary safe so they can not just hold text, but images, compressed data and everything else.
Lists: lists of strings, with support for operations like append a new string on head, on tail, list length, obtain a range of elements, truncate the list to a given length, sort the list, and so on.
Sets: an unsorted set of strings. It is possible to add or delete elements from a set, to perform set intersection, union, subtraction, and so on.

+Values can be Strings, Lists or Sets. Keys can be a subset of strings not containing newlines ("\n") and spaces (" ").

Note that sometimes strings may hold numeric vaules that must be parsed by +Redis. An example is the INCR command that atomically increments the number +stored at the specified key. In this case Redis is able to handle integers +that can be stored inside a 'long long' type, that is a 64-bit signed integer.

Implementation Details

Strings are implemented as dynamically allocated strings of characters. +Lists are implemented as doubly linked lists with cached length. +Sets are implemented using hash tables that use chaining to resolve collisions.

Redis Tutorial

(note, you can skip this section if you are only interested in "formal" doc.)

Later in this document you can find detailed information about Redis commands, +the protocol specification, and so on. This kind of documentation is useful +but... if you are new to Redis it is also BORING! The Redis protocol is designed +so that is both pretty efficient to be parsed by computers, but simple enough +to be used by humans just poking around with the 'telnet' command, so this +section will show to the reader how to play a bit with Redis to get an initial +feeling about it, and how it works.

To start just compile redis with 'make' and start it with './redis-server'. +The server will start and log stuff on the standard output, if you want +it to log more edit redis.conf, set the loglevel to debug, and restart it.

You can specify a configuration file as unique parameter:

./redis-server /etc/redis.conf

+This is NOT required. The server will start even without a configuration file +using a default built-in configuration.

Now let's try to set a key to a given value:

+$ telnet localhost 6379
+Trying 127.0.0.1...
+Connected to localhost.
+Escape character is '^]'.
+SET foo 3  
+bar
++OK
+

The first line we sent to the server is "set foo 3". This means "set the key +foo with the following three bytes I'll send you". The following line is +the "bar" string, that is, the three bytes. So the effect is to set the +key "foo" to the value "bar". Very simple!

(note that you can send commands in lowercase and it will work anyway, +commands are not case sensitive)

Note that after the first and the second line we sent to the server there +is a newline at the end. The server expects commands terminated by "\r\n" +and sequence of bytes terminated by "\r\n". This is a minimal overhead from +the point of view of both the server and client but allows us to play with +Redis with the telnet command easily.

The last line of the chat between server and client is "+OK". This means +our key was added without problems. Actually SET can never fail but +the "+OK" sent lets us know that the server received everything and +the command was actually executed.

Let's try to get the key content now:

+GET foo
+3
+bar
+

Ok that's very similar to 'set', just the other way around. We sent "get foo", +the server replied with a first line that is just a number of bytes the value +stored at key contained, followed by the actual bytes. Again "\r\n" are appended +both to the bytes count and the actual data.

What about requesting a non existing key?

+GET blabla
+nil
+

When the key does not exist instead of the length just the "nil" string is sent. +Another way to check if a given key exists or not is indeed the EXISTS command:

+EXISTS nokey
+0
+EXISTS foo
+1
+

As you can see the server replied '0' the first time since 'nokey' does not +exist, and '1' for 'foo', a key that actually exists.

Ok... now you know the basics, read the REDIS COMMAND REFERENCE section to +learn all the commands supported by Redis and the PROTOCOL SPECIFICATION +section for more details about the protocol used if you plan to implement one +for a language missing a decent client implementation.

License

Redis is released under the BSD license. See the COPYING file for more information.

Credits

Redis is written and maintained by Salvatore Sanfilippo, Aka 'antirez'.

Enjoy, +antirez + +

+ +

+ + + +

+ +TwitterAlikeExample: Contents
  A case study: Design and implementation of a simple Twitter clone using only the Redis key-value store as database and PHP
  Key-value stores basics
    Atomic operations
    Beyond key-value stores
      The set data type
    Prerequisites
    Data layout
    Following, followers and updates
    Authentication
    Updates
    Paginating updates
    Following users
  Making it horizontally scalable
      Hashing the key
      Special keys +

+ +

TwitterAlikeExample

+ +

A case study: Design and implementation of a simple Twitter clone using only the Redis key-value store as database and PHP

In this article I'll explain the design and the implementation of a simple clone of Twitter written using PHP and Redis as only database. The programming community uses to look at key-value stores like special databases that can't be used as drop in replacement for a relational database for the development of web applications. This article will try to prove the contrary.

Our Twitter clone, called Retwis, is structurally simple, has very good performances, and can be distributed among N web servers and M Redis servers with very little efforts. You can find the source code here.

We use PHP for the example since it can be read by everybody. The same (or... much better) results can be obtained using Ruby, Python, Erlang, and so on. +

Key-value stores basics

+The essence of a key-value store is the ability to store some data, called value, inside a key. This data can later be retrieved only if we know the exact key used to store it. There is no way to search something by value. So for example I can use the command SET to store the value bar at key foo:

+SET foo bar
+

Redis will store our data permanently, so we can later ask for "What is the value stored at key foo?" and Redis will reply with bar:

+GET foo => bar
+

Other common operations provided by key-value stores are DEL used to delete a given key, and the associated value, SET-if-not-exists (called SETNX on Redis) that sets a key only if it does not already exist, and INCR that is able to atomically increment a number stored at a given key:

+SET foo 10
+INCR foo => 11
+INCR foo => 12
+INCR foo => 13
+

Atomic operations

+So far it should be pretty simple, but there is something special about INCR. Think about this, why to provide such an operation if we can do it ourself with a bit of code? After all it is as simple as:

+x = GET foo
+x = x + 1
+SET foo x
+

The problem is that doing the increment this way will work as long as there is only a client working with the value x at a time. See what happens if two computers are accessing this data at the same time:

+x = GET foo (yields 10)
+y = GET foo (yields 10)
+x = x + 1 (x is now 11)
+y = y + 1 (y is now 11)
+SET foo x (foo is now 11)
+SET foo y (foo is now 11)
+

Something is wrong with that! We incremented the value two times, but instead to go from 10 to 12 our key holds 11. This is because the INCR operation done with GET / increment / SET is not an atomic operation. Instead the INCR provided by Redis, Memcached, ..., are atomic implementations, the server will take care to protect the get-increment-set for all the time needed to complete in order to prevent simultaneous accesses.

What makes Redis different from other key-value stores is that it provides more operations similar to INCR that can be used together to model complex problems. This is why you can use Redis to write whole web applications without using an SQL database and without to get mad. +

Beyond key-value stores

+In this section we will see what Redis features we need to build our Twitter clone. The first thing to know is that Redis values can be more than strings. Redis supports Lists and Sets as values, and there are atomic operations to operate against this more advanced values so we are safe even with multiple accesses against the same key. Let's start from Lists:

+LPUSH mylist a (now mylist holds one element list 'a')
+LPUSH mylist b (now mylist holds 'b,a')
+LPUSH mylist c (now mylist holds 'c,b,a')
+

LPUSH means Left Push, that is, add an element to the left (or to the head) of the list stored at mylist. If the key mylist does not exist it is automatically created by Redis as an empty list before the PUSH operation. As you can imagine, there is also the RPUSH operation that adds the element on the right of the list (on the tail).

This is very useful for our Twitter clone. Updates of users can be stored into a list stored at username:updates for instance. There are operations to get data or information from Lists of course. For instance LRANGE returns a range of the list, or the whole list.

+LRANGE mylist 0 1 => c,b
+

LRANGE uses zero-based indexes, that is the first element is 0, the second 1, and so on. The command aguments are LRANGE key first-index last-index. The last index argument can be negative, with a special meaning: -1 is the last element of the list, -2 the penultimate, and so on. So in order to get the whole list we can use:

+LRANGE mylist 0 -1 => c,b,a
+

Other important operations are LLEN that returns the length of the list, and LTRIM that is like LRANGE but instead of returning the specified range trims the list, so it is like Get range from mylist, Set this range as new value but atomic. We will use only this List operations, but make sure to check the Redis documentation to discover all the List operations supported by Redis. +

The set data type

+There is more than Lists, Redis also supports Sets, that are unsorted collection of elements. It is possible to add, remove, and test for existence of members, and perform intersection between different Sets. Of course it is possible to ask for the list or the number of elements of a Set. Some example will make it more clear. Keep in mind that SADD is the add to set operation, SREM is the remove from set operation, sismember is the test if it is a member operation, and SINTER is perform intersection operation. Other operations are SCARD that is used to get the cardinality (the number of elements) of a Set, and SMEMBERS that will return all the members of a Set.

+SADD myset a
+SADD myset b
+SADD myset foo
+SADD myset bar
+SCARD myset => 4
+SMEMBERS myset => bar,a,foo,b
+

Note that SMEMBERS does not return the elements in the same order we added them, since Sets are unsorted collections of elements. When you want to store the order it is better to use Lists instead. Some more operations against Sets:

+SADD mynewset b
+SADD mynewset foo
+SADD mynewset hello
+SINTER myset mynewset => foo,b
+

SINTER can return the intersection between Sets but it is not limited to two sets, you may ask for intersection of 4,5 or 10000 Sets. Finally let's check how SISMEMBER works:

+SISMEMBER myset foo => 1
+SISMEMBER myset notamember => 0
+

Ok I think we are ready to start coding! +

Prerequisites

+If you didn't download it already please grab the source code of Retwis. It's a simple tar.gz file with a few of .php files inside. The implementation is very simple. You will find the PHP library client inside (redis.php) that is used to talk with the Redis server from PHP. This library was written by Ludovico Magnocavallo and you are free to reuse this in your own projects, but for updated version of the library please download the Redis distribution.

Another thing you probably want is a working Redis server. Just get the source, compile with make, and run with ./redis-server and you are done. No configuration is required at all in order to play with it or to run Retwis in your computer. +

Data layout

+Working with a relational database this is the stage were the database layout should be produced in form of tables, indexes, and so on. We don't have tables, so what should be designed? We need to identify what keys are needed to represent our objects and what kind of values this keys need to hold.

Let's start from Users. We need to represent this users of course, with the username, userid, password, followers and following users, and so on. The first question is, what should identify an user inside our system? The username can be a good idea since it is unique, but it is also too big, and we want to stay low on memory. So like if our DB was a relational one we can associate an unique ID to every user. Every other reference to this user will be done by id. That's very simple to do, because we have our atomic INCR operation! When we create a new user we can do something like this, assuming the user is callled "antirez":

+INCR global:nextUserId => 1000
+SET uid:1000:username antirez
+SET uid:1000:password p1pp0
+

We use the global:nextUserId key in order to always get an unique ID for every new user. Then we use this unique ID to populate all the other keys holding our user data. This is a Design Pattern with key-values stores! Keep it in mind. +Besides the fields already defined, we need some more stuff in order to fully define an User. For example sometimes it can be useful to be able to get the user ID from the username, so we set this key too:

+SET username:antirez:uid 1000
+

This may appear strange at first, but remember that we are only able to access data by key! It's not possible to tell Redis to return the key that holds a specific value. This is also our strength, this new paradigm is forcing us to organize the data so that everything is accessible by primary key, speaking with relational DBs language. +

Following, followers and updates

+There is another central need in our system. Every user has followers users and following users. We have a perfect data structure for this work! That is... Sets. So let's add this two new fields to our schema:

+uid:1000:followers => Set of uids of all the followers users
+uid:1000:following => Set of uids of all the following users
+

Another important thing we need is a place were we can add the updates to display in the user home page. We'll need to access this data in chronological order later, from the most recent update to the older ones, so the perfect kind of Value for this work is a List. Basically every new update will be LPUSHed in the user updates key, and thanks to LRANGE we can implement pagination and so on. Note that we use the words updates and posts interchangeably, since updates are actually "little posts" in some way.

+uid:1000:posts => a List of post ids, every new post is LPUSHed here.
+

Authentication

+Ok we have more or less everything about the user, but authentication. We'll handle authentication in a simple but robust way: we don't want to use PHP sessions or other things like this, our system must be ready in order to be distributed among different servers, so we'll take the whole state in our Redis database. So all we need is a random string to set as the cookie of an authenticated user, and a key that will tell us what is the user ID of the client holding such a random string. We need two keys in order to make this thing working in a robust way:

+SET uid:1000:auth fea5e81ac8ca77622bed1c2132a021f9
+SET auth:fea5e81ac8ca77622bed1c2132a021f9 1000
+

In order to authenticate an user we'll do this simple work (login.php): +

Get the username and password via the login form
Check if the username:<username>:uid key actually exists
If it exists we have the user id, (i.e. 1000)
Check if uid:1000:password matches, if not, error message
Ok authenticated! Set "fea5e81ac8ca77622bed1c2132a021f9" (the value of uid:1000:auth) as "auth" cookie

+This is the actual code:

+include("retwis.php");
+
+# Form sanity checks
+if (!gt("username") || !gt("password"))
+    goback("You need to enter both username and password to login.");
+
+# The form is ok, check if the username is available
+$username = gt("username");
+$password = gt("password");
+$r = redisLink();
+$userid = $r->get("username:$username:id");
+if (!$userid)
+    goback("Wrong username or password");
+$realpassword = $r->get("uid:$userid:password");
+if ($realpassword != $password)
+    goback("Wrong useranme or password");
+
+# Username / password OK, set the cookie and redirect to index.php
+$authsecret = $r->get("uid:$userid:auth");
+setcookie("auth",$authsecret,time()+3600*24*365);
+header("Location: index.php");
+

This happens every time the users log in, but we also need a function isLoggedIn in order to check if a given user is already authenticated or not. These are the logical steps preformed by the isLoggedIn function: +

Get the "auth" cookie from the user. If there is no cookie, the user is not logged in, of course. Let's call the value of this cookie <authcookie>
Check if auth:<authcookie> exists, and what the value (the user id) is (1000 in the exmple).
In order to be sure check that uid:1000:auth matches.
Ok the user is authenticated, and we loaded a bit of information in the $User global variable.

+The code is simpler than the description, possibly:

+function isLoggedIn() {
+    global $User, $_COOKIE;
+
+    if (isset($User)) return true;
+
+    if (isset($_COOKIE['auth'])) {
+        $r = redisLink();
+        $authcookie = $_COOKIE['auth'];
+        if ($userid = $r->get("auth:$authcookie")) {
+            if ($r->get("uid:$userid:auth") != $authcookie) return false;
+            loadUserInfo($userid);
+            return true;
+        }
+    }
+    return false;
+}
+
+function loadUserInfo($userid) {
+    global $User;
+
+    $r = redisLink();
+    $User['id'] = $userid;
+    $User['username'] = $r->get("uid:$userid:username");
+    return true;
+}
+

loadUserInfo as separated function is an overkill for our application, but it's a good template for a complex application. The only thing it's missing from all the authentication is the logout. What we do on logout? That's simple, we'll just change the random string in uid:1000:auth, remove the old auth:<oldauthstring> and add a new auth:<newauthstring>.

Important: the logout procedure explains why we don't just authenticate the user after the lookup of auth:<randomstring>, but double check it against uid:1000:auth. The true authentication string is the latter, the auth:<randomstring> is just an authentication key that may even be volatile, or if there are bugs in the program or a script gets interrupted we may even end with multiple auth:<something> keys pointing to the same user id. The logout code is the following (logout.php):

+include("retwis.php");
+
+if (!isLoggedIn()) {
+    header("Location: index.php");
+    exit;
+}
+
+$r = redisLink();
+$newauthsecret = getrand();
+$userid = $User['id'];
+$oldauthsecret = $r->get("uid:$userid:auth");
+
+$r->set("uid:$userid:auth",$newauthsecret);
+$r->set("auth:$newauthsecret",$userid);
+$r->delete("auth:$oldauthsecret");
+
+header("Location: index.php");
+

That is just what we described and should be simple to undestand. +

Updates

+Updates, also known as posts, are even simpler. In order to create a new post on the database we do something like this:

+INCR global:nextPostId => 10343
+SET post:10343 "$owner_id|$time|I'm having fun with Retwis"
+

As you can se the user id and time of the post are stored directly inside the string, we don't need to lookup by time or user id in the example application so it is better to compact everything inside the post string.

After we create a post we obtain the post id. We need to LPUSH this post id in every user that's following the author of the post, and of course in the list of posts of the author. This is the file update.php that shows how this is performed:

+include("retwis.php");
+
+if (!isLoggedIn() || !gt("status")) {
+    header("Location:index.php");
+    exit;
+}
+
+$r = redisLink();
+$postid = $r->incr("global:nextPostId");
+$status = str_replace("\n"," ",gt("status"));
+$post = $User['id']."|".time()."|".$status;
+$r->set("post:$postid",$post);
+$followers = $r->smembers("uid:".$User['id'].":followers");
+if ($followers === false) $followers = Array();
+$followers[] = $User['id']; /* Add the post to our own posts too */
+
+foreach($followers as $fid) {
+    $r->push("uid:$fid:posts",$postid,false);
+}
+# Push the post on the timeline, and trim the timeline to the
+# newest 1000 elements.
+$r->push("global:timeline",$postid,false);
+$r->ltrim("global:timeline",0,1000);
+
+header("Location: index.php");
+

The core of the function is the foreach. We get using SMEMBERS all the followers of the current user, then the loop will LPUSH the post against the uid:<userid>:posts of every follower.

Note that we also maintain a timeline with all the posts. In order to do so what is needed is just to LPUSH the post against global:timeline. Let's face it, do you start thinking it was a bit strange to have to sort things added in chronological order using ORDER BY with SQL? I think so indeed. +

Paginating updates

+Now it should be pretty clear how we can user LRANGE in order to get ranges of posts, and render this posts on the screen. The code is simple:

+function showPost($id) {
+    $r = redisLink();
+    $postdata = $r->get("post:$id");
+    if (!$postdata) return false;
+
+    $aux = explode("|",$postdata);
+    $id = $aux[0];
+    $time = $aux[1];
+    $username = $r->get("uid:$id:username");
+    $post = join(array_splice($aux,2,count($aux)-2),"|");
+    $elapsed = strElapsed($time);
+    $userlink = "<a class=\"username\" href=\"profile.php?u=".urlencode($username)."\">".utf8entities($username)."</a>";
+
+    echo('<div class="post">'.$userlink.' '.utf8entities($post)."<br>");
+    echo('<i>posted '.$elapsed.' ago via web</i></div>');
+    return true;
+}
+
+function showUserPosts($userid,$start,$count) {
+    $r = redisLink();
+    $key = ($userid == -1) ? "global:timeline" : "uid:$userid:posts";
+    $posts = $r->lrange($key,$start,$start+$count);
+    $c = 0;
+    foreach($posts as $p) {
+        if (showPost($p)) $c++;
+        if ($c == $count) break;
+    }
+    return count($posts) == $count+1;
+}
+

showPost will simply convert and print a Post in HTML while showUserPosts get range of posts passing them to showPosts.

Following users

If user id 1000 (antirez) wants to follow user id 1001 (pippo), we can do this with just two SADD:

+SADD uid:1000:following 1001
+SADD uid:1001:followers 1000
+

Note the same pattern again and again, in theory with a relational database the list of following and followers is a single table with fields like following_id and follower_id. With queries you can extract the followers or following of every user. With a key-value DB that's a bit different as we need to set both the 1000 is following 1001 and 1001 is followed by 1000 relations. This is the price to pay, but on the other side accessing the data is simpler and ultra-fast. And having this things as separated sets allows us to do interesting stuff, for example using SINTER we can have the intersection of 'following' of two different users, so we may add a feature to our Twitter clone so that it is able to say you at warp speed, when you visit somebody' else profile, "you and foobar have 34 followers in common" and things like that.

You can find the code that sets or removes a following/follower relation at follow.php. It is trivial as you can see. +

Making it horizontally scalable

+Gentle reader, if you reached this point you are already an hero, thank you. Before to talk about scaling horizontally it is worth to check the performances on a single server. Retwis is amazingly fast, without any kind of cache. On a very slow and loaded server, apache benchmark with 100 parallel clients issuing 100000 requests measured the average pageview to take 5 milliseconds. This means you can serve millions of users every day with just a single Linux box, and this one was monkey asses slow! Go figure with more recent hardware.

So, first of all, probably you will not need more than one server for a lot of applications, even when you have a lot of users. But let's assume we are Twitter and need to handle a huge amount of traffic. What to do? +

Hashing the key

+The first thing to do is to hash the key and issue the request on different servers based on the key hash. There are a lot of well known algorithms to do so, for example check the Redis Ruby library client that implements consistent hashing, but the general idea is that you can turn your key into a number, and than take the reminder of the division of this number by the number of servers you have:

+server_id = crc32(key) % number_of_servers
+

This has a lot of problems since if you add one server you need to move too much keys and so on, but this is the general idea even if you use a better hashing scheme like consistent hashing.

Ok, are key accesses distributed among the key space? Well, all the user data will be partitioned among different servers. There are no inter-keys operations used (like SINTER, otherwise you need to care that things you want to intersect will end in the same server. This is why Redis unlike memcached does not force a specific hashing scheme, it's application specific). Btw there are keys that are accessed more frequently.

Special keys

For example every time we post a new message, we need to increment the global:nextPostId key. How to fix this problem? A Single server will get a lot if increments. The simplest way to handle this is to have a dedicated server just for increments. This is probably an overkill btw unless you have really a lot of traffic. There is another trick. The ID does not really need to be an incremental number, but just it needs to be unique. So you can get a random string long enough to be unlikely (almost impossible, if it's md5-size) to collide, and you are done. We successfully eliminated our main problem to make it really horizontally scalable!

There is another one: global:timeline. There is no fix for this, if you need to take something in order you can split among different servers and then merge when you need to get the data back, or take it ordered and use a single key. Again if you really have so much posts per second, you can use a single server just for this. Remember that with commodity hardware Redis is able to handle 100000 writes for second, that's enough even for Twitter, I guess.

Please feel free to use the comments below for questions and feedbacks. + +

+ +

Benchmarks

BgsaveCommand

CommandReference

Credits

DbsizeCommand

DelCommand

DesignPatterns

ExistsCommand

FAQ

FlushallCommand

FlushdbCommand

GetCommand

IncrCommand

InfoCommand

KeysCommand

LastsaveCommand

LindexCommand

LlenCommand

LpopCommand

LrangeCommand

LremCommand

LsetCommand

LtrimCommand

MoveCommand

ProtocolSpecification

QuitCommand

README

RandomkeyCommand

RenameCommand

RenamenxCommand

ReplyTypes