Netty emulates the Redis server

original
2016/08/28 10:23
Reading 8.1K

The client and server of Redis use RESP(Redis Serialization Protocol) The client and server exchange data through TCP connection. The server's default port number is 6379. The commands or data sent by the client and server are always \r\n (CRLF) ending.

RESP supports five data types:

Status reply: "+" indicates correct status information, and "+" indicates specific information, such as:

 redis 127.0.0.1:6379> set ss sdf OK

In fact, the data it actually replies to is:+OK r n
Error reply: Start with "-" to indicate the error status information, followed by specific information, such as:

 redis 127.0.0.1:6379> incr ss (error) ERR value is not an integer or out of range

Integer reply: Start with ":" to indicate the reply to some operations, such as DEL, EXISTS, INCR, etc

 redis 127.0.0.1:6379> incr aa (integer) 1

Bulk reply: Start with "$" to indicate the string length of the next line. The specific string is in the next line

Multiple bulk replies: Starting with "*", it indicates the total number of lines in the message body (excluding the current line) "*" is the specific number of lines

 redis 127.0.0.1:6379> get ss "sdf" Client ->Server *2\r\n $3\r\n get\r\n $2\r\n ss\r\n Server ->Client $3\r\n sdf\r\n

Note: All the above are XX replies, which does not mean that the protocol format is only applicable to the server ->client, and the client ->server also uses the above protocol format. In fact, the unification of the two terminal protocol format is more convenient for expansion

To get back to the main point, we use netty to simulate the Redis server, which can be divided into the following steps:

 1. An underlying communication framework is required. Netty4.0.25 is selected here 2. The data passed through by the client needs to be decoded. In fact, the above five data types need to be processed separately 3. After decoding, we encapsulate the commands into more understandable commands, such as set<name>foo hello<params> 4. Once you have a command, you can execute it. In fact, we can connect to the existing Redis server, but this is just a simple simulation 5. After processing, the reply is encapsulated and then encoded. The next five data types need to be returned according to different commands 6. Test and verify whether the correct results can be returned by connecting to the Redis server simulated by Netty through Redis cli

The above ideas refer to a project on github: https://github.com/spullara/redis-protocol The test code is simplified on this basis

Step 1: Communication framework netty

 <dependency> <groupId>io.netty</groupId> <artifactId>netty-all</artifactId> <version>4.0.25.Final</version> </dependency>

Step 2: Data type decoding

 public class RedisCommandDecoder extends ReplayingDecoder<Void> { public static final char CR = '\r'; public static final char LF = '\n'; public static final byte DOLLAR_BYTE = '$'; public static final byte ASTERISK_BYTE = '*'; private byte[][] bytes; private int arguments = 0; @Override protected void decode(ChannelHandlerContext ctx, ByteBuf in, List<Object> out) throws Exception { if (bytes !=  null) { int numArgs = bytes.length; for (int i = arguments;  i < numArgs; i++) { if (in.readByte() == DOLLAR_BYTE) { int l = RedisReplyDecoder.readInt(in); if (l > Integer. MAX_VALUE) { throw new IllegalArgumentException( "Java only supports arrays up to " + Integer. MAX_VALUE + " in size"); } int size = (int) l; bytes[i] = new byte[size]; in.readBytes(bytes[i]); if (in.bytesBefore((byte) CR) !=  0) { throw new RedisException("Argument doesn't end in CRLF"); } // Skip CRLF(\r\n) in.skipBytes(2); arguments++; checkpoint(); } else { throw new IOException("Unexpected character"); } } try { out.add(new Command(bytes)); } finally { bytes = null; arguments = 0; } } else if (in.readByte() == ASTERISK_BYTE) { int l = RedisReplyDecoder.readInt(in); if (l > Integer. MAX_VALUE) { throw new IllegalArgumentException( "Java only supports arrays up to " + Integer. MAX_VALUE + " in size"); } int numArgs = (int) l; if (numArgs < 0) { throw new RedisException("Invalid size: " + numArgs); } bytes = new byte[numArgs][]; checkpoint(); decode(ctx, in, out); } else { in.readerIndex(in.readerIndex() - 1); byte[][] b = new byte[1][]; b[0] = in.readBytes(in.bytesBefore((byte) CR)).array(); in.skipBytes(2); out.add(new Command(b, true)); } } }

First, initialize the two-dimensional array byte [] [] bytes by receiving multiple batch types starting with "*", so as to read the first data ending r nas the length of the array, and then process the batch types starting with "$".
In addition to dealing with the familiar batch and multiple batch types, the above also deals with the data without any identification. In fact, there is a special name called Inline command:
Sometimes only telnet connects to the Redis service, or just sends a command to the Redis service for detection. Although the Redis protocol can be easily implemented, the use of Interactive sessions is not ideal, and the Redis cli is not always available. For these reasons, Redis supports special commands to implement the situations described above. The design of these commands is very user-friendly, and they are called Inline commands.

Step 3: Encapsulate the command object

From the second step, we can see that both commandName and params are uniformly placed in the byte two-dimensional array, and finally encapsulated in the command object

 public class Command { public static final byte[] EMPTY_BYTES = new byte[0]; private final Object name; private final Object[] objects; private final boolean inline; public Command(Object[] objects) { this(null, objects, false); } public Command(Object[] objects, boolean inline) { this(null, objects, inline); } private Command(Object name, Object[] objects, boolean inline) { this.name = name; this.objects = objects; this.inline = inline; } public byte[] getName() { if (name !=  null) return getBytes(name); return getBytes(objects[0]); } public boolean isInline() { return inline; } private byte[] getBytes(Object object) { byte[] argument; if (object == null) { argument = EMPTY_BYTES; } else if (object instanceof byte[]) { argument = (byte[]) object; } else if (object instanceof ByteBuf) { argument = ((ByteBuf) object).array(); } else if (object instanceof String) { argument = ((String) object).getBytes(Charsets.UTF_8); } else { argument = object.toString().getBytes(Charsets.UTF_8); } return argument; } public void toArguments(Object[] arguments, Class<?>[] types) { for (int position = 0;  position < types.length; position++) { if (position >= arguments.length) { throw new IllegalArgumentException( "wrong number of arguments for '" + new String(getName()) + "' command"); } if (objects.length - 1 > position) { arguments[position] = objects[1 + position]; } } } }

All data is placed in the Object array, and you can know that Object [0] is the commandName through the getName method

Step 4: Execute the command

After decoding and encapsulation, we need to implement the handler class to process messages

 public class RedisCommandHandler extends SimpleChannelInboundHandler<Command> { private Map<String, Wrapper> methods = new HashMap<String, Wrapper>(); interface Wrapper { Reply<?> execute(Command command) throws RedisException; } public RedisCommandHandler(final RedisServer rs) { Class<? extends RedisServer> aClass = rs.getClass(); for (final Method method : aClass.getMethods()) { final Class<?>[] types = method.getParameterTypes(); methods.put(method.getName(), new Wrapper() { @Override public Reply<?> execute(Command command) throws RedisException { Object[] objects = new Object[types.length]; try { command.toArguments(objects, types); return (Reply<?>)  method.invoke(rs, objects); } catch (Exception e) { return new ErrorReply("ERR " + e.getMessage()); } } }); } } @Override public void channelReadComplete(ChannelHandlerContext ctx) throws Exception { ctx.flush(); } @Override protected void channelRead0(ChannelHandlerContext ctx, Command msg) throws Exception { String name = new String(msg.getName()); Wrapper wrapper = methods.get(name); Reply<?> reply; if (wrapper == null) { reply = new ErrorReply("unknown command '" + name + "'"); } else { reply = wrapper.execute(msg); } if (reply == StatusReply. QUIT) { ctx.close(); } else { if (msg.isInline()) { if (reply == null) { reply = new InlineReply(null); } else { reply = new InlineReply(reply.data()); } } if (reply == null) { reply = ErrorReply.NYI_REPLY; } ctx.write(reply); } } }

When instantiating the handler, a RedisServer object is passed in. This method is really used to process Redis commands. Theoretically, this object should support all Redis commands, but only two methods are provided for testing:

 public interface RedisServer { public BulkReply get(byte[] key0) throws RedisException; public StatusReply set(byte[] key0, byte[] value1) throws RedisException; }

In the channelRead0 method, we can get the encapsulated command method, and then execute the operation through the command name. The RedisServer here is also very simple, just using a simple hashmap to temporarily save data.

Step 5: Encapsulate reply

Step 4: After processing the command, we can see that a Reply object is returned

 public interface Reply<T> { byte[] CRLF = new byte[] { RedisReplyDecoder.CR,  RedisReplyDecoder.LF }; T data(); void write(ByteBuf os) throws IOException; }

According to the five types mentioned above plus an inline command, splice according to different data formats, such as StatusReply:

 public void write(ByteBuf os) throws IOException { os.writeByte('+'); os.writeBytes(statusBytes); os.writeBytes(CRLF); }

So the Encoder corresponding to the Decoder is very simple:

 public class RedisReplyEncoder extends MessageToByteEncoder<Reply<?>> { @Override public void encode(ChannelHandlerContext ctx,  Reply<?> msg, ByteBuf out) throws Exception { msg.write(out); } }

Just return the encapsulated Reply to the client

The last step: test

Startup class:

 public class Main { private static Integer port = 6379; public static void main(String[] args) throws InterruptedException { final RedisCommandHandler commandHandler = new RedisCommandHandler( new SimpleRedisServer()); ServerBootstrap b = new ServerBootstrap(); final DefaultEventExecutorGroup group = new DefaultEventExecutorGroup(1); try { b.group(new NioEventLoopGroup(), new NioEventLoopGroup()) .channel(NioServerSocketChannel.class) .option(ChannelOption.SO_BACKLOG, 100).localAddress(port) .childOption(ChannelOption.TCP_NODELAY, true) .childHandler(new ChannelInitializer<SocketChannel>() { @Override public void initChannel(SocketChannel ch) throws Exception { ChannelPipeline p = ch.pipeline(); p.addLast(new RedisCommandDecoder()); p.addLast(new RedisReplyEncoder()); p.addLast(group, commandHandler); } }); ChannelFuture f = b.bind().sync(); f.channel().closeFuture().sync(); } finally { group.shutdownGracefully(); } } }

The ChannelPipeline adds RedisCommandDecoder, RedisReplyEncoder and RedisCommandHandler respectively. At the same time, the port we start is the same as the port of the Redis server six thousand three hundred and seventy-nine

Open the redis cli program:

 redis 127.0.0.1:6379> get dsf (nil) redis 127.0.0.1:6379> set dsf dsfds OK redis 127.0.0.1:6379> get dsf "dsfds" redis 127.0.0.1:6379>

It can be seen from the results that there is no difference between the Redis server and the Redis server

summary

The significance of doing this is actually to treat it as a Redis proxy The proxy server is used for sharding. The client does not directly access the Redis server. For the client, the background Redis cluster is completely transparent.

Personal blog: codingo.xyz

Expand to read the full text
Loading
Click to join the discussion 🔥 (10) Post and join the discussion 🔥
Reward
ten comment
one hundred and eleven Collection
ten fabulous
 Back to top
Top