redis该如何分区-译文（原创）

写在最前，最近一直在研究redis的使用，包括redis应用场景、性能优化、可行性。这是看到redis官网中一个链接，主要是讲解redis数据分区的，既然是官方推荐的，那我就翻译一下，与大家共享。

Partitioning: how to split data among multiple Redis instances.

 

分区：如何把数据存储在多个实例中。

 

Partitioning is the process of splitting your data into multiple Redis instances, so that every instance will only contain a subset of your keys. The first part of this document will introduce you to the concept of partitioning, the second part will show you the alternatives for Redis partitioning.

分区是把你的数据分割存储在多个redis实例中的一个过程，每个实例中只保存一部分key。本文件的第一部分将介绍你到分区的概念，第二部分说明如何使用redis分区。

Why partitioning is useful

为什么分区是有效的

Partitioning in Redis serves two main goals:

在redis服务器中使用分区有两个主要作用：

• It allows for much larger databases, using the sum of the memory of many computers. Without partitioning you are limited to the amount of memory a single computer can support.
  他可以利用多台计算机的内存共同构建一个大型数据库。不使用分区的情况下你会单个计算机的内存限制。
• It allows to scale the computational power to multiple cores and multiple computers, and the network bandwidth to multiple computers and network adapters.
  他可以在多核和多台计算机之间扩展，并且适应不同的计算机带宽。

Partitioning basics

 

分区的基本概念

 

There are different partitioning criteria. Imagine we have four Redis instances R0, R1, R2, R3, and many keys representing users like user:1, user:2, ... and so forth, we can find different ways to select in which instance we store a given key. In other words there are different systems to map a given key to a given Redis server.

有多种分区方式。比如：我们有四个redis实例：R0, R1, R2, R3和许多代表用户的键（像 user:1, user:2）等等，我可以用不同的方式来从中选择一个实例来存储一个键。换句话说，有不同的系统来映射给定的键存储到给定的redis服务器中。

One of the simplest way to perform partitioning is called range partitioning, and is accomplished by mapping ranges of objects into specific Redis instances. For example I could say, users from ID 0 to ID 10000 will go into instanceR0, while users form ID 10001 to ID 20000 will go into instance R1 and so forth.

一个最简单的分区方法就是范围分区，并通过具体的实例对象来映射该范围。比如，id 1到10000的用户存储到R0中，10001到20000的用户存储到R1中，依此类推。

This systems works and is actually used in practice, however it has the disadvantage that there is to take a table mapping ranges to instances. This table needs to be managed and we need a table for every kind of object we have. Usually with Redis it is not a good idea.
这个方案是可以被应用到实践中的，但是他有一个缺点就是他需要一个表来存储每个实例存储范围的映射关系。这个表是需要维护的，并且我们需要为我们每一种对象创建这么一张表。所以在使用redis时，这不是一个很好的方案。

An alternative to to range partitioning is hash partitioning. This scheme works with any key, no need for a key in the form object_name:<id> as is as simple as this:

散列分区：一种可以替代范围分区的分区方式。该方案适用于任何键，他简单到不需要使用这样的键（object_name:<id>）：

• Take the key name and use an hash function to turn it into a number. For instance I could use the crc32 hash function. So if the key is foobar I do crc32(foobar) that will output something like 93024922.
  使用一个哈希函数把key转换成一个数字。例如：我可以使用CRC32算法。所以如果key是foobar，那么执行CRC32(foobar)的结果就是像93024922一样的东西。
• I use a modulo operation with this number in order to turn it into a number between 0 and 3, so that I can map this number to one of the four Redis instances I've. So 93024922 modulo 4 equals 2, so I know my key foobar should be stored into the R2 instance. Note: the modulo operation is just the rest of the division, usually it is implemented by the% operator in many programming languages.
  我是用一种取模的函数把一个号码转换到0到3中的一个数字，这样我就可以把这个数字映射到4个redis实例中的一个实例上。93024922模4等于2，这样我就知道foobar这个key应该存放到R2实例中。提示：取模运算是他工程里的说法，通常我们在程序语言设计中只需要使用%（取余）就可以了。

There are many other ways to perform partitioning, but with this two examples you should get the idea. One advanced form of hash partitioning is called consistent hashing and is implemented by a few Redis clients and proxies.

通过这两个例子，你应该能想到还有很多其他的划分方式。哈希分区是一种先进的分区形式，它也被叫做一致性分区，他由几个redis客户端和代理实现。

Different implementations of partitioning

 

不同的划分方式的实现

 

Partitioning can be responsibility of different parts of a software stack.

分区可以由一个软件栈的不同职责区域完成。

• Client side partitioning means that the clients directly select the right node where to write or read a given key. Many Redis clients implement client side partitioning.
  客户端实现分区：是指有客户端直接选在合适的借点进行读写键。许多redis客户端都实现了这种分区方式。
• Proxy assisted partitioning means that our clients send requests to a proxy that is able to speak the Redis protocol, instead of sending requests directly to the right Redis instance. The proxy will make sure to forward our request to the right Redis instance accordingly to the configured partitioning schema, and will send the replies back to the client. The Redis and Memcached proxy Twemproxy implements proxy assisted partitioning.
  代理辅助分区： 是指客户端把请求通过redis协议发送给代理，而不是直接发送给真正的redis实例服务器。这个代理会确保我们的请求根据配置分区架构发送到正确的redis实例上，并返回给客户端。redis和memcached的代理都是用 Twemproxy（twitter的一个代理框架）来实现代理服务分区的。
• Query routing means that you can send your query to a random instance, and the instance will make sure to forward your query to the right node. Redis Cluster implements an hybrid form of query routing, with the help of the client (the request is not directly forwarded from a Redis instance to another, but the client gets redirected to the right node).
  查询路由：是指你可以把一个请求发送给一个随机的实例，这时实例会把该查询转发给正确的节点。Redis集群实现了一种混合查询路由，客户端的请求不用直接从一个实例转发到另一个实例，而是被重定向到正确的节点。

Disadvantages of partitioning
分区的一些缺点

Some features of Redis don't play very well with partitioning:

redis分区在有些方面做的并不好：

• Operations involving multiple keys are usually not supported. For instance you can't perform the intersection between two sets if they are stored in keys that are mapped to different Redis instances (actually there are ways to do this, but not directly).
  不支持涉及多个键的操作。比如你不能操作映射在两个redis实例上的两个集合的交叉集。（其实可以做到这一点，但是需要间接的解决）
• Redis transactions involving multiple keys can not be used.
  redis之间多个键的事务不能使用。
• The partitioning granuliary is the key, so it is not possible to shard a dataset with a single huge key like a very big sorted set.
  使用类似于一个大的排序集合将单一的数据集进行分片是不太可能的。因为分区关键是键。
• When partitioning is used, data handling is more complex, for instance you have to handle multiple RDB / AOF files, and to make a backup of your data you need to aggregate the persistence files from multiple instances and hosts.
  如果使用分区，数据的处理会变得复杂，你不得不对付多个redis数据库和AOF文件，不得在多个实例和主机之间持久化你的数据。
• Adding and removing capacity can be complex. For instance Redis Cluster plans to support mostly transparent rebalancing of data with the ability to add and remove nodes at runtime, but other systems like client side partitioning and proxies don't support this feature. However a technique called Presharding helps in this regard.
  添加和删除节点也会变得复杂。比如redis集群计划支持透明的运行时添加和删除节点，但是像客户端分区或者代理分区的特性就不会再被支持。不过Presharding(预分片)可以在这方面提供帮助。

Data store or cache?
作为数据存储还是作为缓存使用？

Partitioning when using Redis ad a data store or cache is conceptually the same, however there is a huge difference. While when Redis is used as a data store you need to be sure that a given key always maps to the same instance, when Redis is used as a cache if a given node is unavailable it is not a big problem if we start using a different node, altering the key-instance map as we wish to improve the availability of the system (that is, the ability of the system to reply to our queries).
使用redis存储数据或者缓存数据在概念上是相同的，但是使用过程中这两者有巨大的差距。当redis被当作持久化数据存储服务器使用的时候意味着对于相同的键值必须被映射到相同的实例上面，但是如果把redis当作数据缓存器，当我们使用不同的节点的时候，找不到对应键值的对象不是什么大问题(缓存就是随时准备好牺牲自己)，改变键值和实例映射逻辑可以提供系统的可用性(也就是系统处理查询请求的能力)。

Consistent hashing implementations are often able to switch to other nodes if the preferred node for a given key is not available. Similarly if you add a new node, part of the new keys will start to be stored on the new node.
一致性哈希可以为给定的键值不可用的情况下能够切换到其他的节点上。同样的，你添加一个新的节点，部分新的键值开始存储到新添加的节点上面。

The main concept here is the following:
主要的概念如下：

• If Redis is used as a cache scaling up and down using consistent hashing is easy.
  如果redis只作为缓存服务器来使用，那么用哈希是相当容易的。
• If Redis is used as a store, we need to take the map between keys and nodes fixed, and a fixed number of nodes. Otherwise we need a system that is able to rebalance keys between nodes when we add or remove nodes, and currently only Redis Cluster is able to do this, but Redis Cluster is not production ready.
  若果redis被作为数据持久化服务器，我们需要提供节点和键值的固定映射，还有一组固定的redis实例节点。否则我们需要一个系统来为我们增加或者删除键值和节点，目前，redis集群可以做到这点，但是redis集群还没有发布正式版本。

Presharding

 

预分片

 

We learned that a problem with partitioning is that, unless we are using Redis as a cache, to add and remove nodes can be tricky, and it is much simpler to use a fixed keys-instances map.

从分区的概念中，我们可以了解，除非只把redis当作缓存服务器来使用，否则添加和删除redis节点都会非常复杂。相反使用固定的键值和实例映射确实很简单的。

However the data storage needs may vary over the time. Today I can live with 10 Redis nodes (instances), but tomorrow I may need 50 nodes.

然而数据存储会经常需要变化。今天我只需要10个redis节点（实例），但是明天我可能会需要50个节点。

Since Redis is extremely small footprint and lightweight (a spare instance uses 1 MB of memory), a simple approach to this problem is to start with a lot of instances since the start. Even if you start with just one server, you can decide to live in a distributed world since your first day, and run multiple Redis instances in your single server, using partitioning.

因为redis足够轻量和小巧(一个备用实例使用1M的内存)，解决这个问题的简单方法就是一开始就使用大量的实例节点。即使你开始是有一个服务器，你可以换成分布式的结构，因为可以在单个服务器上通过分区分方式来运行多个redis节点。

And you can select this number of instances to be quite big since the start. For example, 32 or 64 instances could do the trick for most users, and will provide enough room for growth.

你可以选择的实例可数可以非常大。例如，32或者64个实例能够满足绝大多数的用户，并且可以为其提供足够的增长空间。

In this way as your data storage needs increase and you need more Redis servers, what to do is to simply move instances from one server to another. Once you add the first additional server, you will need to move half of the Redis instances from the first server to the second, and so forth.

通过这样的方法来满足数据存储需求的增加时你只需要更多的redis服务器，然后把一个节点移动到另外的服务器上面。一旦你添加了额外的服务器，你可以将一半的redis的实例移动到第二个等等。

Using Redis replication you will likely be able to do the move with minimal or no downtime for your users:
你可以使用redis 的主从复制来减少服务的停止时间：

• Start empty instances in your new server.
  在新服务器上开启新的redis空实例。
• Move data configuring these new instances as slaves for your source instances.
  将节点的数据配置移动到新的从服务器上
• Stop your clients.
  停止你的redis客户端。
• Update the configuration of the moved instances with the new server IP address.
  在新的服务器上更新移动过来的节点配置。
• Send the SLAVEOF NO ONE command to the slaves in the new server.
  发送slave no one 命令到新服务器的从节点。
• Restart your clients with the new updated configuration.
  使用新的配置重启客户端。
• Finally shut down the no longer used instances in the old server.
  最后永久关闭老服务器上不再使用的节点。

Implementations of Redis partitioning
redis分区实践。

So far we covered Redis partitioning in theory, but what about practice? What system should you use?

到目前为止，我们讲了分区的原理。但是该如何实战？你应该使用什么样的系统？

Redis Cluster

redis集群

Unfortunately Redis Cluster is currently not production ready, however you can get more information about it reading the specification or checking the partial implementation in the unstable branch of the Redis GitHub repositoriy.

不幸的是redis集群的正式版还没有发布，但是你可以在github上得到不稳定版，看一看他的规范和实现方式。

Once Redis Cluster will be available, and if a Redis Cluster complaint client is available for your language, Redis Cluster will be the de facto standard for Redis partitioning.

一旦redis集群正式版发布，并且提供的客户端语言接口可用，那么这种方式将成为标准的redis分区方式。

Redis Cluster is a mix between query routing and client side partitioning.

redis集群是一个查询路由和客户端分区的混合体。

Twemproxy

 

Twemproxy 框架

 

Twemproxy is a proxy developed at Twitter for the Memcached ASCII and the Redis protocol. It is single threaded, it is written in C, and is extremely fast. It is open source software released under the terms of the Apache 2.0 license.
Twemproxy是一个由Twitter开发的适合memached和redis协议的代理。它是单线程工作，使用C语言实现的，非常的快速。并且是Apache 2.0版权申明下的开源软件。

Twemproxy supports automatic partitioning among multiple Redis instances, with optional node ejection if a node is not available (this will change the keys-instances map, so you should use this feature only if you are using Redis as a cache).

Twemproxy支持自动在多个redis节点分区，如果某个节点不可用，将会被自动屏蔽(这将改变键值和节点映射表，所以如果你把redis当作缓存服务器使用你应该使用这个功能)。

It is not a single point of failure since you can start multiple proxies and instruct your clients to connect to the first that accepts the connection.

你可以启用多个代理，让你的客户端得到可用的连接，这样不会发生单点故障。

Basically Twemproxy is an intermediate layer between clients and Redis instances, that will reliably handle partitioning for us with minimal additional complexities. Currently it is the suggested way to handle partitioning with Redis.

Twemproxy基本上是redis和客户端的一个过渡层，通过简化使用让我们使用可靠的分区。目前这也是使用redis分区的推荐方案。

You can read more about Twemproxy in this antirez blog post.

你可以在antirez的博客发现有关Twemproxy的更多知识。

Clients supporting consistent hashing

客户端一致性哈希实现。

An alternative to Twemproxy is to use a client that implements client side partitioning via consistent hashing or other similar algorithms. There are multiple Redis clients with support for consistent hashing, notably Redis-rb and Predis.
替代Twemproxy的一种方案是使用客户端一致性哈西或者其他类似的算法。有需要redis客户端支持一致性哈西，比如Redis-rb和Predis。

Please check the full list of Redis clients to check if there is a mature client with consistent hashing implementation for your language.

请检查列表已确定是否有成熟的一直性哈希实现的，并且适合于你的编程语言的客户端。

转载请注明出处：http://www.cnblogs.com/eric-z/p/3995502.html