What is libconhash

libconhash is a consistent hashing library which can be compiled both on Windows and Linux platforms, with the following features:

  1. High performance and easy to use, libconhash uses a red-black tree to manage all nodes to achieve high performance.
  2. By default, it uses the MD5 algorithm, but it also supports user-defined hash functions.
  3. Easy to scale according to the node's processing capacity.

Consistent hashing

Why you need consistent hashing

Now we will consider the common way to do load balance. The machine number chosen to cache object o will be:

Hide   Copy Code
hash(o) mod n

Here, n is the total number of cache machines. While this works well until you add or remove cache machines:

    1. When you add a cache machine, then object o will be cached into the machine:
Hide   Copy Code
hash(o) mod (n+1)
  1. When you remove a cache machine, then object o will be cached into the machine:

    Hide   Copy Code
    hash(o) mod (n-1)

So you can see that almost all objects will hashed into a new location. This will be a disaster since the originating content servers are swamped with requests from the cache machines. And this is why you need consistent hashing.

Consistent hashing can guarantee that when a cache machine is removed, only the objects cached in it will be rehashed; when a new cache machine is added, only a fairly few objects will be rehashed.

Now we will go into consistent hashing step by step.

Hash space

Commonly, a hash function will map a value into a 32-bit key, 0~2^32-1. Now imagine mapping the range into a circle, then the key will be wrapped, and 0 will be followed by 2^32-1, as illustrated in figure 1.

Figure 1

Map object into hash space

Now consider four objects: object1~object4. We use a hash function to get their key values and map them into the circle, as illustrated in figure 2.

Figure 2
Hide   Copy Code
hash(object1) = key1;
.....
hash(object4) = key4;

Map the cache into hash space

The basic idea of consistent hashing is to map the cache and objects into the same hash space using the same hash function.

Now consider we have three caches, A, B and C, and then the mapping result will look like in figure 3.

Hide   Copy Code
hash(cache A) = key A;
....
hash(cache C) = key C;

Figure 3

Map objects into cache

Now all the caches and objects are hashed into the same space, so we can determine how to map objects into caches. Take object obj for example, just start from where obj is and head clockwise on the ring until you find a server. If that server is down, you go to the next one, and so forth. See figure 3 above.

According to the method, object1 will be cached into cache A; object2 and object3 will be cached into cache C, and object4 will be cached into cache B.

Add or remove cache

Now consider the two scenarios, a cache is down and removed; and a new cache is added.

If cache B is removed, then only the objects that cached in B will be rehashed and moved to C; in the example, see object4 illustrated in figure 4.

Figure 4

If a new cache D is added, and D is hashed between object2 and object3 in the ring, then only the objects that are between D and B will be rehashed; in the example, see object2, illustrated in figure 5.

Figure 5

Virtual nodes

It is possible to have a very non-uniform distribution of objects between caches if you don't deploy enough caches. The solution is to introduce the idea of "virtual nodes".

Virtual nodes are replicas of cache points in the circle, each real cache corresponds to several virtual nodes in the circle; whenever we add a cache, actually, we create a number of virtual nodes in the circle for it; and when a cache is removed, we remove all its virtual nodes from the circle.

Consider the above example. There are two caches A and C in the system, and now we introduce virtual nodes, and the replica is 2, then three will be 4 virtual nodes. Cache A1 and cache A2 represent cache A; cache C1 and cache C2 represent cache C, illustrated as in figure 6.

Figure 6

Then, the map from object to the virtual node will be:

Hide   Copy Code
objec1->cache A2; objec2->cache A1; objec3->cache C1; objec4->cache C2

When you get the virtual node, you get the cache, as in the above figure.

So object1 and object2 are cached into cache A, and object3 and object4 are cached into cache. The result is more balanced now.

So now you know what consistent hashing is.

Using the code

Interfaces of libconhash

Hide   Shrink   Copy Code
/* initialize conhash library
* @pfhash : hash function, NULL to use default MD5 method
* return a conhash_s instance
*/
CONHASH_API struct conhash_s* conhash_init(conhash_cb_hashfunc pfhash); /* finalize lib */
CONHASH_API void conhash_fini(struct conhash_s *conhash); /* set node */
CONHASH_API void conhash_set_node(struct node_s *node,
const char *iden, u_int replica); /*
* add a new node
* @node: the node to add
*/
CONHASH_API int conhash_add_node(struct conhash_s *conhash,
struct node_s *node); /* remove a node */
CONHASH_API int conhash_del_node(struct conhash_s *conhash,
struct node_s *node);
... /*
* lookup a server which object belongs to
* @object: the input string which indicates an object
* return the server_s structure, do not modify the value,
* or it will cause a disaster
*/
CONHASH_API const struct node_s*
conhash_lookup(const struct conhash_s *conhash,
const char *object);

Libconhash is very easy to use. There is a sample in the project that shows how to use the library.

First, create a conhash instance. And then you can add or remove nodes of the instance, and look up objects.

The update node's replica function is not implemented yet.

Hide   Copy Code
/* init conhash instance */
struct conhash_s *conhash = conhash_init(NULL);
if(conhash)
{
/* set nodes */
conhash_set_node(&g_nodes[0], "titanic", 32);
/* ... */ /* add nodes */
conhash_add_node(conhash, &g_nodes[0]);
/* ... */
printf("virtual nodes number %d\n", conhash_get_vnodes_num(conhash));
printf("the hashing results--------------------------------------:\n"); /* lookup object */
node = conhash_lookup(conhash, "James.km");
if(node) printf("[%16s] is in node: [%16s]\n", str, node->iden);
}

Reference

 

License

This article, along with any associated source code and files, is licensed under The BSD License

Consistent hashing的更多相关文章

  1. Consistent hashing —— 一致性哈希

    原文地址:http://www.codeproject.com/Articles/56138/Consistent-hashing 基于BSD License What is libconhash l ...

  2. 一致性 hash 算法( consistent hashing )a

    一致性 hash 算法( consistent hashing ) 张亮 consistent hashing 算法早在 1997 年就在论文 Consistent hashing and rando ...

  3. 一致性哈希算法 - consistent hashing

    1 基本场景比如你有 N 个 cache 服务器(后面简称 cache ),那么如何将一个对象 object 映射到 N 个 cache 上呢,你很可能会采用类似下面的通用方法计算 object 的 ...

  4. 一致性 hash 算法( consistent hashing )

    consistent hashing 算法早在 1997 年就在论文 Consistent hashing and random trees 中被提出,目前在cache 系统中应用越来越广泛: 1 基 ...

  5. 【转】一致性hash算法(consistent hashing)

    consistent hashing 算法早在 1997 年就在论文 Consistent hashing and random trees 中被提出,目前在 cache 系统中应用越来越广泛: 1  ...

  6. Consistent Hashing算法-搜索/负载均衡

    在做服务器负载均衡时候可供选择的负载均衡的算法有很多,包括:  轮循算法(Round Robin).哈希算法(HASH).最少连接算法(Least Connection).响应速度算法(Respons ...

  7. 一致性hash算法 - consistent hashing

    consistent hashing 算法早在 1997 年就在论文 Consistent hashing and random trees 中被提出,目前在 cache 系统中应用越来越广泛: 1 ...

  8. Consistent Hashing原理与实现

    原理介绍: consistent hashing原理介绍来自博客:http://blog.csdn.net/sparkliang/article/details/5279393, 多谢博主的分享 co ...

  9. 一致性哈希算法(consistent hashing)样例+測试。

    一个简单的consistent hashing的样例,非常easy理解. 首先有一个设备类,定义了机器名和ip: public class Cache { public String name; pu ...

  10. _00013 一致性哈希算法 Consistent Hashing 新的讨论,并出现相应的解决

    笔者博文:妳那伊抹微笑 博客地址:http://blog.csdn.net/u012185296 个性签名:世界上最遥远的距离不是天涯,也不是海角,而是我站在妳的面前.妳却感觉不到我的存在 技术方向: ...

随机推荐

  1. [Java Web]Hibernate基础总结(四)

    性能优化 在大数据量遍历时(比如查找消息敏感词),须要手动使用clear方法释放缓存中的数据,防止缓存中数据过多浪费内存. 1+N问题:将Fetch设为LAZY能够在须要时才发出sql语句,或者设置B ...

  2. js模块化规范AMD、CMD、CommonJS...

    1. AMD 1.1 什么是AMD? AMD 英文名 Asynchronous Module Definition ,中文名 异步模块定义 .这是一个浏览器模块化开发的规范. 由于浏览器环境执行环境的 ...

  3. JSP、servlet、SQL三者之间的数据传递

    JSP.servlet.SQL三者之间的数据传递 博客分类: web开发 JSPservletSQL数据库连接池web开发  前言: 最近一直在做WEB开发,现总结一下这一段时间的体会和感触. 切记, ...

  4. 项目红色感叹号eclipse因Web App Libraries中的jar包missing导致项目红色感叹号

    症状: 如题 分析: 修改.更换或者删除了WEB-INF/lib中的jar包 解决方案: 右击项目>build path>Libraries 直接remove Web App Librar ...

  5. usb 安装系统

    写在前面 本文是先安装windows再安装linux,并通过windows引导linux的启动项.这样方便linux的反复重装.折腾等. 光盘安装和U盘安装基本差不多,只是U盘安装多了把镜像文件写到U ...

  6. php中的 file_get_contents(‘php://input’)

    今天用CI框架整合微信发现了一个没见过的 file_get_contents('php://input'); 这个家伙是干嘛用的,然后自己直接写了验证的代码.好吧 废话不多说直接看下面的代码. < ...

  7. oracle+SQL优化实例

    1.     减少I/O操作: SELECT COUNT(CASE WHEN empno>20 THEN 1 END) c1,COUNT(CASE WHEN empno<20 THEN 1 ...

  8. 一个共通的viewModel搞定所有的分页查询一览及数据导出(easyui + knockoutjs + mvc4.0)

    前言 大家看标题就明白了我想写什么了,在做企业信息化系统中可能大家写的最多的一种页面就是查询页面了.其实每个查询页面,除了条件不太一样,数据不太一样,其它的其实都差不多.所以我就想提取一些共通的东西出 ...

  9. 第二百三十六节,Bootstrap辅组类和响应式工具

    Bootstrap辅组类和响应式工具 学习要点: 1.辅组类 2.响应式工具 本节课我们主要学习一下 Bootstrap 的辅组类和响应式工具,辅助类提供了一组类来辅 组页面设计,而响应式工具则利用媒 ...

  10. TCP/IP协议族-----15、传输控制协议(TCP)