cae1403191
1. Fix memory corruption bug in quicklist and listpack due to wrong assert usage. 2. Add HSETNX command and fix skip_exists semantics in OpSet function. 3. Add tests for hsetnx. 4. Add MEMORY USAGE decorator. 5. redis_parser accepts arrays upto 8192 elements. 6. Fix listpack replace call when passing an empty string. 7. Implement "debug object" command. |
||
|---|---|---|
| .github/workflows | ||
| .vscode | ||
| doc | ||
| helio@1a5a762691 | ||
| patches | ||
| src | ||
| tests | ||
| tools/docker | ||
| .clang-format | ||
| .dockerignore | ||
| .gitignore | ||
| .gitmodules | ||
| .gitorderfile | ||
| CMakeLists.txt | ||
| LICENSE | ||
| README.md | ||
| TODO.md | ||
Dragonfly
A novel memory store that supports Redis and Memcached commands. For more detailed status of what's implemented - see below.
Features include:
- High throughput reaching millions of QPS on a single node.
- TLS support.
- Pipelining mode.
- A novel cache design, which does not require specifying eviction policies.
- Memory efficiency that can save 20-40% for regular workloads and even more for cache like workloads
Building from source
I've tested the build on Ubuntu 21.04+. Requires: CMake, Ninja, boost, libunwind8-dev
sudo apt install ninja-build
sudo apt install libunwind-dev
sudo apt-get install libboost-all-dev
git clone --recursive https://github.com/romange/dragonfly
cd dragonfly && ./helio/blaze.sh -release
cd build-opt && ninja dragonfly
Running
./dragonfly --logtostderr
for more options, run ./dragonfly --help
Milestone - Source Available
API 1.0
- String family
- SET
- SETNX
- GET
- DECR
- INCR
- DECRBY
- GETSET
- INCRBY
- MGET
- MSET
- MSETNX
- SUBSTR
- Generic family
- DEL
- ECHO
- EXISTS
- EXPIRE
- EXPIREAT
- PING
- RENAME
- RENAMENX
- SELECT
- TTL
- TYPE
- SORT
- Server Family
- QUIT
- DBSIZE
- BGSAVE
- SAVE
- DBSIZE
- DEBUG
- EXEC
- FLUSHALL
- FLUSHDB
- INFO
- MULTI
- SHUTDOWN
- LASTSAVE
- SLAVEOF/REPLICAOF
- SYNC
- Set Family
- SADD
- SCARD
- SDIFF
- SDIFFSTORE
- SINTER
- SINTERSTORE
- SISMEMBER
- SMOVE
- SPOP
- SRANDMEMBER
- SREM
- SMEMBERS
- SUNION
- SUNIONSTORE
- List Family
- LINDEX
- LLEN
- LPOP
- LPUSH
- LRANGE
- LREM
- LSET
- LTRIM
- RPOP
- RPOPLPUSH
- RPUSH
- SortedSet Family
- ZADD
- ZCARD
- ZINCRBY
- ZRANGE
- ZRANGEBYSCORE
- ZREM
- ZREMRANGEBYSCORE
- ZREVRANGE
- ZSCORE
- Not sure whether these are required for the initial release.
- AUTH
- BGREWRITEAOF
- KEYS
- MONITOR
- RANDOMKEY
- MOVE
API 2.0
- List Family
- BLPOP
- BRPOP
- BRPOPLPUSH
- BLMOVE
- LINSERT
- LPUSHX
- RPUSHX
- String Family
- SETEX
- APPEND
- PREPEND (dragonfly specific)
- BITCOUNT
- BITFIELD
- BITOP
- BITPOS
- GETBIT
- GETRANGE
- INCRBYFLOAT
- PSETEX
- SETBIT
- SETRANGE
- STRLEN
- HashSet Family
- HSET
- HMSET
- HDEL
- HEXISTS
- HGET
- HMGET
- HLEN
- HINCRBY
- HINCRBYFLOAT
- HGETALL
- HKEYS
- HSETNX
- HVALS
- HSCAN
- PubSub family
- PUBLISH
- PUBSUB
- PUBSUB CHANNELS
- SUBSCRIBE
- UNSUBSCRIBE
- PSUBSCRIBE
- PUNSUBSCRIBE
- Server Family
- WATCH
- UNWATCH
- DISCARD
- CLIENT KILL/LIST/UNPAUSE/PAUSE/GETNAME/SETNAME/REPLY/TRACKINGINFO
- COMMAND
- COMMAND COUNT/GETKEYS/INFO
- CONFIG GET/REWRITE/SET/RESETSTAT
- MIGRATE
- ROLE
- SLOWLOG
- PSYNC
- TIME
- LATENCY...
- Generic Family
- SCAN
- PEXPIREAT
- PEXPIRE
- DUMP
- EVAL
- EVALSHA
- OBJECT
- PERSIST
- PTTL
- RESTORE
- SCRIPT LOAD
- SCRIPT DEBUG/KILL/FLUSH/EXISTS
- Set Family
- SSCAN
- Sorted Set Family
- ZCOUNT
- ZINTERSTORE
- ZLEXCOUNT
- ZRANGEBYLEX
- ZRANK
- ZREMRANGEBYLEX
- ZREMRANGEBYRANK
- ZREVRANGEBYSCORE
- ZREVRANK
- ZUNIONSTORE
- ZSCAN
- HYPERLOGLOG Family
- PFADD
- PFCOUNT
- PFMERGE
In addition, we want to support efficient expiry (TTL) and cache eviction algorithms. We should implement basic memory management support. For Master/Slave replication we should design a distributed log format.
Memchache API
- set
- get
- replace
- add
- stats (partial)
- append
- prepend
- delete
- flush_all
- incr
- decr
- version
- quit
Commands that I prefer avoid implementing before launch:
- PUNSUBSCRIBE
- PSUBSCRIBE
- HYPERLOGLOG
- SCRIPT DEBUG
- OBJECT
- DUMP/RESTORE
- CLIENT
Also, I would omit keyspace notifications. For that I would like to deep dive and learn exact use-cases for this API.
Random commands we implemented as decorators along the way
- ROLE (2.8) decorator for for master withour replicas
- UNLINK (4.0) decorator for DEL command
- BGSAVE
- FUNCTION FLUSH
Milestone Stability
APIs 3,4,5 without cluster support, without modules, without memory introspection commands. Without geo commands and without support for keyspace notifications, without streams. Design config support. ~10-20 commands overall... Probably implement cluster-API decorators to allow cluster-configured clients to connect to a single instance.
Design decisions along the way
Expiration deadlines with relative accuracy
I decided to limit the expiration range to 365 days. Moreover, expiration deadlines
with millisecond precision (PEXPIRE/PSETEX etc) will be rounded to closest second
**for deadlines greater than 33554431ms (approximately 560 minutes). In other words,
expiries of PEXPIRE key 10010 will expire exactly after 10 seconds and 10ms. However,
PEXPIRE key 34000300 will expire after 34000 seconds (i.e. 300ms earlier). Similarly,
PEXPIRE key 34000800 will expire after 34001 seconds, i.e. 200ms later.
Such rounding has at most 0.002% error which I hope is acceptable for large ranges. If it breaks your use-cases - talk to me or open an issue and explain your case.
For more detailed differences between this and Redis implementations see here.