memcached使用libevent 和 多线程模式
一、libevent的使用
首先我们知道,memcached是使用了iblievet作为网络框架的,而iblievet又是单线程模型的基于linux下epoll事件的异步模型。因此,其基本的思想就是 对可读,可写,超时,出错等事件进行绑定函数,等有其事件发生,对其绑定函数回调。
可以减掉了解一下 libevent基本api调用
- struct event_base *base;
- base = event_base_new();//初始化libevent
event_base_new对比epoll,可以理解为epoll里的epoll_create。
event_base内部有一个循环,循环阻塞在epoll调用上,当有一个事件发生的时候,才会去处理这个事件。其中,这个事件是被绑定在event_base上面的,每一个事件就会对应一个struct event,可以是监听的fd。
其中struct event 使用event_new 来创建和绑定,使用event_add来启用,例如:
- struct event *listener_event;
- listener_event = event_new(base, listener, EV_READ|EV_PERSIST, do_accept, (void*)base);
参数说明:
base:event_base类型,event_base_new的返回值
listener:监听的fd,listen的fd
EV_READ|EV_PERSIST:事件的类型及属性
do_accept:绑定的回调函数
(void*)base:给回调函数的参数
event_add(listener_event, NULL);
对比epoll:
event_new相当于epoll中的epoll_wait,其中的epoll里的while循环,在libevent里使用event_base_dispatch。
event_add相当于epoll中的epoll_ctl,参数是EPOLL_CTL_ADD,添加事件。
注:libevent支持的事件及属性包括(使用bitfield实现,所以要用 | 来让它们合体)
EV_TIMEOUT: 超时
EV_READ: 只要网络缓冲中还有数据,回调函数就会被触发
EV_WRITE: 只要塞给网络缓冲的数据被写完,回调函数就会被触发
EV_SIGNAL: POSIX信号量
EV_PERSIST: 不指定这个属性的话,回调函数被触发后事件会被删除
EV_ET: Edge-Trigger边缘触发,相当于EPOLL的ET模式
事件创建添加之后,就可以处理发生的事件了,相当于epoll里的epoll_wait,在libevent里使用event_base_dispatch启动event_base循环,直到不再有需要关注的事件。
有了上面的分析,结合之前做的epoll服务端程序,对于一个服务器程序,流程基本是这样的:
1. 创建socket,bind,listen,设置为非阻塞模式
2. 创建一个event_base,即
- struct event_base * event_base_new(void)
3. 创建一个event,将该socket托管给event_base,指定要监听的事件类型,并绑定上相应的回调函数(及需要给它的参数)。即
- struct event * event_new(struct event_base *base, evutil_socket_t fd, short events, void (*cb)(evutil_socket_t, short, void *), void *arg)
4. 启用该事件,即
- int event_add(struct event *ev, const struct timeval *tv)
5. 进入事件循环,即
- int event_base_dispatch(struct event_base *event_base)
- /* initialize main thread libevent instance */
- main_base = event_init();
最后会调用
- /* enter the event loop */
- if (event_base_loop(main_base, 0) != 0) {
- retval = EXIT_FAILURE;
- }
在该对象内部循环。不退出。
- static void conn_init(void) {
- freetotal = 200;
- freecurr = 0;
- if ((freeconns = calloc(freetotal, sizeof(conn *))) == NULL) {
- fprintf(stderr, "Failed to allocate connection structures\n");
- }
- return;
- }
- /*
- * Returns a connection from the freelist, if any.
- */
- conn *conn_from_freelist() {
- conn *c;
- pthread_mutex_lock(&conn_lock);
- if (freecurr > 0) {
- c = freeconns[--freecurr];
- } else {
- c = NULL;
- }
- pthread_mutex_unlock(&conn_lock);
- return c;
- }
3.那么conn的结构体内部长什么样子呢?
- typedef struct conn conn;
- struct conn {
- int sfd;
- sasl_conn_t *sasl_conn;
- enum conn_states state;
- enum bin_substates substate;
- struct event event;
- short ev_flags;
- short which; /** which events were just triggered */
- char *rbuf; /** buffer to read commands into */
- char *rcurr; /** but if we parsed some already, this is where we stopped */
- int rsize; /** total allocated size of rbuf */
- int rbytes; /** how much data, starting from rcur, do we have unparsed */
- char *wbuf;
- char *wcurr;
- int wsize;
- int wbytes;
- /** which state to go into after finishing current write */
- enum conn_states write_and_go;
- void *write_and_free; /** free this memory after finishing writing */
- char *ritem; /** when we read in an item's value, it goes here */
- int rlbytes;
- /* data for the nread state */
- /**
- * item is used to hold an item structure created after reading the command
- * line of set/add/replace commands, but before we finished reading the actual
- * data. The data is read into ITEM_data(item) to avoid extra copying.
- */
- void *item; /* for commands set/add/replace */
- /* data for the swallow state */
- int sbytes; /* how many bytes to swallow */
- /* data for the mwrite state */
- struct iovec *iov;
- int iovsize; /* number of elements allocated in iov[] */
- int iovused; /* number of elements used in iov[] */
- struct msghdr *msglist;
- int msgsize; /* number of elements allocated in msglist[] */
- int msgused; /* number of elements used in msglist[] */
- int msgcurr; /* element in msglist[] being transmitted now */
- int msgbytes; /* number of bytes in current msg */
- item **ilist; /* list of items to write out */
- int isize;
- item **icurr;
- int ileft;
- char **suffixlist;
- int suffixsize;
- char **suffixcurr;
- int suffixleft;
- enum protocol protocol; /* which protocol this con<pre name="code" class="cpp"> if (sigignore(SIGPIPE) == -1) {
- perror("failed to ignore SIGPIPE; sigaction");
- exit(EX_OSERR);
- }
nection speaks */ enum network_transport transport; /* what transport is used by this connection */ /* data for UDP clients */ int request_id; /* Incoming UDP request ID, if this is a UDP "connection" */ struct sockaddr request_addr; /* Who sent the most recent request */ socklen_t request_addr_size; unsigned char *hdrbuf; /* udp packet headers */ int hdrsize; /* number of headers' worth of space is allocated */ bool noreply; /* True if the reply should not be sent. */ /* current stats command */ struct { char *buffer; size_t size; size_t offset; } stats; /* Binary protocol stuff */ /* This is where the binary header goes */ protocol_binary_request_header binary_header; uint64_t cas; /* the cas to return */ short cmd; /* current command being processed */ int opaque; int keylen; conn *next; /* Used for generating a list of conn structures */ LIBEVENT_THREAD *thread; /* Pointer to the thread object serving this connection */};
这里的所有字段就是在处理数据需要用到的。这里不详细描述。以后会慢慢分解。
- if (sigignore(SIGPIPE) == -1) {
- perror("failed to ignore SIGPIPE; sigaction");
- exit(EX_OSERR);
- }
初始化多线程模型,并且每个线程一个iblievent的事件模型就是调用event_init函数。
- /* start up worker threads if MT mode */
- thread_init(settings.num_threads, main_base);
内部实现不详细。主要是调用pthread_create函数。
- if (settings.port && server_sockets(settings.port, tcp_transport,
- portnumber_file)) {
- vperror("failed to listen on TCP port %d", settings.port);
- exit(EX_OSERR);
- }
然后调用下面的函数:
- static int server_socket(const char *interface,
- int port,
- enum network_transport transport,
- FILE *portnumber_file)
因为,一个主机可能会有多个网卡,比如双线机房,联通或者电信,因此内部实现会出现以下代码:
- for (next= ai; next; next= next->ai_next) {
- conn *listen_conn_add;
- if ((sfd = new_socket(next)) == -1) {
- /* getaddrinfo can return "junk" addresses,
- * we make sure at least one works before erroring.
- */
- if (errno == EMFILE) {
- /* ...unless we're out of fds */
- perror("server_socket");
- exit(EX_OSERR);
- }
- continue;
- }
而
- static int new_socket(struct addrinfo *ai)
- if (!(listen_conn_add = conn_new(sfd, conn_listening,
- EV_READ | EV_PERSIST, 1,
- transport, main_base))) {
- fprintf(stderr, "failed to create listening connection\n");
- exit(EXIT_FAILURE);
- }
- listen_conn_add->next = listen_conn;
- listen_conn = listen_conn_add;
- static conn *listen_conn = NULL;
作为全局的静态的变量。无头结点的单链表
- conn *conn_new(const int sfd, enum conn_states init_state,
- const int event_flags,
- const int read_buffer_size, enum network_transport transport,
- struct event_base *base) {
- conn *c = conn_from_freelist();
该函数主要是做了哪些动作呢?
- event_set(&c->event, sfd, event_flags, event_handler, (void *)c);
- event_base_set(base, &c->event);
- c->ev_flags = event_flags;
- if (event_add(&c->event, 0) == -1) {
- if (conn_add_to_freelist(c)) {
- conn_free(c);
- }
- perror("event_add");
- return NULL;
- }
这一步就是,讲sfd上的事件绑定event_handler 函数,就是当有该连接上来的时候有数据进行可读的时候绑定,回调。
- <span style="font-family: Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255);">最终event_handler函数会调用</span>
- static void drive_machine(conn *c)
函数。那么这个函数做了哪些工作呢?
- while (!stop) {
- switch(c->state) {
- case conn_listening:
- addrlen = sizeof(addr);
- if ((sfd = accept(c->sfd, (struct sockaddr *)&addr, &addrlen)) == -1)
- <pre name="code" class="cpp">/*
- * Dispatches a new connection to another thread. This is only ever called
- * from the main thread, either during initialization (for UDP) or because
- * of an incoming connection.
- */
- void dispatch_conn_new(int sfd, enum conn_states init_state, int event_flags,
- int read_buffer_size, enum network_transport transport) {
- CQ_ITEM *item = cqi_new();
- char buf[1];
- int tid = (last_thread + 1) % settings.num_threads;
- LIBEVENT_THREAD *thread = threads + tid;
- last_thread = tid;
- item->sfd = sfd;
- item->init_state = init_state;
- item->event_flags = event_flags;
- item->read_buffer_size = read_buffer_size;
- item->transport = transport;
- cq_push(thread->new_conn_queue, item);
- MEMCACHED_CONN_DISPATCH(sfd, thread->thread_id);
- buf[0] = 'c';
- if (write(thread->notify_send_fd, buf, 1) != 1) {
- perror("Writing to thread notify pipe");
- }
- }
- /* Listen for notifications from other threads */
- event_set(&me->notify_event, me->notify_receive_fd,
- EV_READ | EV_PERSIST, thread_libevent_process, me);
- event_base_set(me->base, &me->notify_event);
- if (event_add(&me->notify_event, 0) == -1) {
- fprintf(stderr, "Can't monitor libevent notify pipe\n");
- exit(1);
- }
- static void thread_libevent_process(int fd, short which, void *arg)
- conn *conn_new(const int sfd, enum conn_states init_state,
- const int event_flags,
- const int read_buffer_size, enum network_transport transport,
- struct event_base *base)
同样,调用
- conn *c = conn_from_freelist();
取出一个conn* ,然后进行初始化,这个时候和上文讲到的一样了,知识状态不同了,
- enum conn_states {
- conn_listening, /**< the socket which listens for connections */
- conn_new_cmd, /**< Prepare connection for next command */
- conn_waiting, /**< waiting for a readable socket */
- conn_read, /**< reading in a command line */
- conn_parse_cmd, /**< try to parse a command from the input buffer */
- conn_write, /**< writing out a simple response */
- conn_nread, /**< reading in a fixed number of bytes */
- conn_swallow, /**< swallowing unnecessary bytes w/o storing */
- conn_closing, /**< closing this connection */
- conn_mwrite, /**< writing out many items sequentially */
- conn_max_state /**< Max state value (used for assertion) */
- };
也就是
- static void drive_machine(conn *c)
的核心逻辑了。通过设置状态,然后调用不同的代码,
- /*
- * Sets a connection's current state in the state machine. Any special
- * processing that needs to happen on certain state transitions can
- * happen here.
- */
- static void conn_set_state(conn *c, enum conn_states state) {
- assert(c != NULL);
- assert(state >= conn_listening && state < conn_max_state);
- if (state != c->state) {
- if (settings.verbose > 2) {
- fprintf(stderr, "%d: going from %s to %s\n",
- c->sfd, state_text(c->state),
- state_text(state));
- }
- if (state == conn_write || state == conn_mwrite) {
- MEMCACHED_PROCESS_COMMAND_END(c->sfd, c->wbuf, c->wbytes);
- }
- c->state = state;
- }
- }
到此,网络框架部分已经基本处理完成。起始这个框架是非常简单而且实用的。
memcached使用libevent 和 多线程模式的更多相关文章
- 分布式缓存系统 Memcached 半同步/半异步模式
在前面工作线程初始化的分析中讲到Memcached采用典型的Master_Worker模式,也即半同步/半异步的高效网络并发模式.其中主线程(异步线程)负责接收客户端连接,然后分发给工作线程,具体由工 ...
- 使用libevent进行多线程socket编程demo
最近要对一个用libevent写的C/C++项目进行修改,要改成多线程的,故做了一些学习和研究. libevent是一个用C语言写的开源的一个库.它对socket编程里的epoll/select等功能 ...
- libevent实现多线程
libevent并不是线程安全的,但这不代表libevent不支持多线程模式.前几天在微博上看到ruanyf发了条微博说到apache和nginx的并发模型,看到评论很多人都说不对于是自己又查了下,总 ...
- Libevent 的多线程操作
起因是event_base 跨线程add/remove event 导致崩溃或者死循环. 据查:libvent 1.4.x是非线程安全的,要跨线程执行event_add,会有问题.因此传统做法是通过p ...
- winform 承载 WCF 注意,可能不是工作在多线程模式下
[ServiceBehavior(ConcurrencyMode = ConcurrencyMode.Multiple, InstanceContextMode = InstanceContextMo ...
- 多线程模式之MasterWorker模式
多线程模式之MasterWorker模式 Master-Worker模式的核心思想是,系统由两类进程协作工作:Master进程和Worker进程.Master进程负责接收和分配任务,Worker进程负 ...
- Servlet单实例多线程模式
http://kakajw.iteye.com/blog/920839 前言:Servlet/JSP技术和ASP.PHP等相比,由于其多线程运行而具有很高的执行效率.由于Servlet/JSP默认是以 ...
- Netty-主从Reactor多线程模式的源码实现
Netty--主从Reactor多线程模式的源码实现 总览 EventLoopGroup到底是什么? EventLoopGroup是一个存储EventLoop的容器,同时他应该具备线程池的功能. gr ...
- 多线程模式下高并发的环境中唯一确保单例模式---DLC双端锁
DLC双端锁,CAS,ABA问题 一.什么是DLC双端锁?有什么用处? 为了解决在多线程模式下,高并发的环境中,唯一确保单例模式只能生成一个实例 多线程环境中,单例模式会因为指令重排和线程竞争的原因会 ...
随机推荐
- 解释一下python中的继承
当一个类继承另一个类,它就被称为一个子类/派生类,继承父类/基类/超类.它会继承/获取所有类成员(属性和方法) 继承能让我们重新使用代码,也能更容易的创建和维护应用 单继承:一个类继承单个基类 多继承 ...
- Python基础(6)_函数
一 为何要有函数? 不加区分地将所有功能的代码垒到一起,问题是: 代码可读性差 代码冗余 代码可扩展差 如何解决? 函数即工具,事先准备工具的过程是定义函数,拿来就用指的就是函数调用 结论:函数使用必 ...
- [笔记]Go语言在Linux环境下输出彩色字符
Go语言要打印彩色字符与Linux终端输出彩色字符类似,以黑色背景高亮绿色字体为例: fmt.Printf("\n %c[1;40;32m%s%c[0m\n\n", 0x1B, & ...
- loadrunner脚本篇——Run-time Settings之ContentCheck
运用场景(很少用到): ContentCheck的设置可用来让VuGen检测存在错误的站点页面.如果被测的Web应用没有使用自定义的错误页面,那么这里不用添加规则,因为LR在回放时候,可以默认的捕捉到 ...
- .ssh中的文件的分别意义
当我们在用户的主目录使用如下命令: cd (进入个人主目录,默认为/home/hadoop) ssh-keygen -t rsa -P '' (注:最后是二个单引号) 表示在用户的主目录创建ssh登陆 ...
- css背景透明文字不透明
测试背景透明度为0.3.文字不透明: background-color: #000; /* 一.CSS3的opacity */ opacity: 0.3; /* 兼容浏览器为:firefox,chro ...
- android 加固防止反编译-重新打包
http://blog.csdn.net/u010921385/article/details/52505094 1.需要加密的Apk(源Apk) 2.壳程序Apk(负责解密Apk工作) 3.加密工具 ...
- SpringBoot 通用Validator
第一步,pom.xml引入hibernate-validator <dependency> <groupId>org.hibernate</groupId> < ...
- 【转载】如何简单地理解Python中的if __name__ == '__main__'
原帖:https://blog.csdn.net/yjk13703623757/article/details/77918633 通俗的理解__name__ == '__main__':假如你叫小明. ...
- 如何去掉Intellij IDEA过多的警告 设置警告级别
Intellij IDEA的代码提示系统很强大,根据严格的代码规范,包括简洁程度,运行效率,潜在bug提前发现等等给你做出了除编译器之外的大量额外提示.但这些提示有时会给我们带来困扰,比如弄的界面很乱 ...