Building a Non-blocking TCP server using OTP principles
转自:https://erlangcentral.org/wiki/index.php/Building_a_Non-blocking_TCP_server_using_OTP_principles
Building a Non-blocking TCP server using OTP principles
From ErlangCentral Wiki
Contents |
Author
Serge Aleynikov <saleyn at gmail.com>
Overview
A reader of this tutorial is assumed to be familiar with gen_server and gen_fsm behaviours, TCP socket communications using gen_tcp module, active and passive socket modes, and OTP supervision principles.
OTP provides a convenient framework for building reliable applications. This is in part accomplished by abstracting common functionality into a set of reusable behaviours such as gen_server and gen_fsm that are linked to OTP's supervision hierarchy.
There are several known TCP server designs. The one we are going to cover involves one process listening for client connections and spawning an FSM process per connecting client. While there is support for TCP communications available in OTP through the gen_tcp module, there is no standard behavior for building non-blocking TCP servers using OTP standard guidelines. By non-blocking we imply that the listening process and the client-handling FSMs should not make any blocking calls and be readily responsive to incoming control messages (such as changes in system configuration, restart requests, etc.) without causing timeouts. Note that blocking in the context of Erlang means blocking an Erlang process rather than the emulator's OS process(es).
In this tutorial we will show how to build a non-blocking TCP server using gen_server and gen_fsm behaviours that offers flow control and is fully compliant with OTP application design principles.
A reader who is new to the OTP framework is encouraged to read Joe Armstrong's tutorial on how to build A Fault-tolerant Server using blocking gen_tcp:connect/3 and gen_tcp:accept/1 calls without involving OTP.
This tutorial was inspired by several threads (e.g. one, two) on the Erlang Questions mailing list mentioning an approach to building non-blocking asynchronous TCP servers.
Server Design
The design of our server will include the main application's supervisor tcp_server_app process with one_for_one restart strategy and two child specifications. The first one being a listening process implemented as a gen_server behaviour that will wait for asynchronous notifications of client socket connections. The second one is another supervisor tcp_client_sup responsible for starting client handling FSMs and logging abnormal disconnects via standard SASL error reports.
For the sake of simplicity of this tutorial, the client handling FSM (tcp_echo_fsm) will implement an echo server that will echo client's requests back to the client.
+----------------+
| tcp_server_app |
+--------+-------+
| (one_for_one)
+----------------+---------+
| |
+-------+------+ +-------+--------+
| tcp_listener | + tcp_client_sup |
+--------------+ +-------+--------+
| (simple_one_for_one)
+-----|---------+
+-------|--------+|
+--------+-------+|+
| tcp_echo_fsm |+
+----------------+
Application and Supervisor behaviours
In order to build an OTP application we need to construct modules implementing an application and supervisor behaviour callback functions. While traditionally these functionalities are implemented in separate modules, given their succinctness we'll combine them in one module.
As an added bonus we implement a get_app_env function that illustrates how to process configuration options as well as command-line options given to the emulator at start-up.
The two instances of init/1 function are for two tiers of supervision hierarchy. Since two different restart strategies for each supervisor are needed, we implement them at different tiers.
Upon application's startup the tcp_server_app:start/2 callback function calls supervisor:start_link/2 that creates main application's supervisor calling tcp_server_app:init([Port, Module]) callback. This supervisor creates a tcp_listener process and a child supervisor tcp_client_sup responsible for spawning client connections. The Module argument in the init function is the name of client-connection handling FSM (in this case tcp_echo_fsm).
TCP Server Application (tcp_server_app.erl) |
||
|
Listener Process
One of the shortcomings of the gen_tcp module is that it only exports interface to a blocking accept call. This leads most of developers working on an implementation of a TCP server build a custom process linked to a supervisor using proc_lib or come up with some other proprietary design.
Examining prim_inet module reveals an interesting fact that the actual call to inet driver to accept a client socket is asynchronous. While this is a non-documented property, which means that the OTP team is free to change this implementation, we will exploit this functionality in the construction of our server.
The listener process is implemented as a gen_server behaviour:
TCP Listener Process (tcp_listener.erl) |
||
|
In this module init/1 call takes two parameters - the port number that the TCP listener should be started on and the name of a protocol handling module for client connections. The initialization function opens a listening socket in passive {active, false} mode. This is done so that we have flow control of the data received on the connected client sockets that will inherit this option from the listening socket.
The most interesting part of this code is the prim_inet:async_accept/2 call as well as the handling of asynchronous inet_async messages. In order to get this working we also needed to copy some of the internal OTP code encapsulated in the set_sockopt/2 function that handles socket registration with inet database and copying some options to the client socket.
As soon as a client socket is connected inet driver will notify the listening process using {inet_async, ListSock, Ref, {ok, CliSocket}} message. At this point we'll instantiate a new client socket handling process and set its ownership of the CliSocket.
Client Socket Handling Process
While tcp_listener is a generic implementation, tcp_echo_fsm is a mere stub FSM for illustrating how to write TCP servers. This modules needs to export two functions - one start_link/0 for a tcp_client_sup supervisor and another set_socket/2 for the listener process to notify the client connection handling FSM process that it is now the owner of the socket, and can begin receiving messages by setting the {active, once} or {active, true} option.
We would like to highlight the synchronization pattern used between the listening process and client connection-handling FSM to avoid possible message loss due to dispatching some messages from the socket to the wrong (listening) process. The process owning the listening socket has it open with {active, false}. After accepting the client's socket that socket inherits its socket options (including {active, false}) from the listener, transfers ownership of the socket to the newly spawned client connection-handling FSM by calling gen_tcp:controlling_process/2 and calls Module:set_socket/2 to notify the FSM that it can start receiving messages from the socket. Until the FSM process enables message delivery by setting the active mode on the socket by calling inet:setopts(Socket, [{active, once}]), the data sent by the TCP sender stays in the socket buffer.
When socket ownership is transfered to FSM in the 'WAIT_FOR_SOCKET' state the FSM sets {active, once} option to let inet driver send it one TCP message at a time. This is the OTP way of preserving flow control and avoiding process message queue flooding with TCP data and crashing the system in case of a fast-producer-slow-consumer case.
The FSM states are implemented by special functions in the tcp_echo_fsm module that use a naming convention with capital case state names enclosed in single quotes. The FSM consists of two states. 'WAIT_FOR_SOCKET' is the initial state in which the FSM is waiting for assignment of socket ownership, and 'WAIT_FOR_DATA' is the state that represents awaiting for TCP message from a client. In this state FSM also handles a special 'timeout' message that signifies no activity from a client and causes the process to stop and close client connection.
TCP Client Socket Handling FSM (tcp_echo_fsm.erl) |
||
|
Application File
Another required part of building an OTP application is creation of an application file that includes application name, version, startup module and environment.
Application File (tcp_server.app) |
||
|
Compiling
Create the following directory structure for this application:
./tcp_server
./tcp_server/ebin/
./tcp_server/ebin/tcp_server.app
./tcp_server/src/tcp_server_app.erl
./tcp_server/src/tcp_listener.erl
./tcp_server/src/tcp_echo_fsm.erl
$ cd tcp_server/src
$ for f in tcp*.erl ; do erlc +debug_info -o ../ebin $f
Running
We are going to start an Erlang shell with SASL support so that we can view all progress and error reports for our TCP application. Also we are going to start appmon application in order to examine visually the supervision hierarchy.
$ cd ../ebin
$ erl -boot start_sasl
...
1> appmon:start().
{ok,<0.44.0>}
2> application:start(tcp_server).
ok
Now click on the tcp_server button in the appmon's window in order to display supervision hierarchy of the tcp_server application.
3> {ok,S} = gen_tcp:connect({127,0,0,1},2222,[{packet,2}]).
{ok,#Port<0.150>}
The step above initiated a new client connection to the echo server.
4> gen_tcp:send(S,<<"hello">>).
ok
5> f(M), receive M -> M end.
{tcp,#Port<0.150>,"hello"}
We verified that the echo server works as expected. Now let's try to crash the client connection on the server and watch for the supervisor generating an error report entry on screen.
6> [{_,Pid,_,_}] = supervisor:which_children(tcp_client_sup).
[{undefined,<0.64.0>,worker,[]}]
7> exit(Pid,kill).
true
=SUPERVISOR REPORT==== 31-Jul-2007::14:33:49 ===
Supervisor: {local,tcp_client_sup}
Context: child_terminated
Reason: killed
Offender: [{pid,<0.77.0>},
{name,undefined},
{mfa,{tcp_echo_fsm,start_link,[]}},
{restart_type,temporary},
{shutdown,2000},
{child_type,worker}]
Note that if you are putting this server under a stress test with many incoming connections, the listener process may fail to accept new connections after the number of open file descriptors reaches the limit set by the operating system. In that case you will see the error:
"too many open files"
If you are running Linux/UNIX, google for a solution (which ultimately boils down to increasing the per-process limit by setting "ulimit -n ..." option).
Conclusion
OTP provides building blocks for constructing non-blocking TCP servers. This tutorial showed how to create a simple TCP server with flow control using standard OTP behaviours. As an exercise the reader is encouraged to try abstracting generic non-blocking TCP server functionality into a stand-along behaviour.
Sample Implementations
View source |
Discuss this page |
Page history |
What links here |
Related changes
Building a Non-blocking TCP server using OTP principles的更多相关文章
- 【转载】C# Tutorial - Simple Threaded TCP Server
http://tech.pro/tutorial/704/csharp-tutorial-simple-threaded-tcp-server In this tutorial I'm going t ...
- socket - socketserver - start TCP server
前面提到如何使用socket模块启动tcpserver: 创建socket:sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 绑定ip: ...
- [转] 3个学习Socket编程的简单例子:TCP Server/Client, Select
以前都是采用ACE的编写网络应用,最近由于工作需要,需要直接只用socket接口编写CS的代码,重新学习这方面的知识,给出自己所用到的3个简单例子,都是拷贝别人的程序.如果你能完全理解这3个例子,估计 ...
- [转]一个基于完成端口的TCP Server Framework,浅析IOCP
[转]一个基于完成端口的TCP Server Framework,浅析IOCP http://www.cppblog.com/adapterofcoms/archive/2010/06/26/1187 ...
- Socket TCP Server一个端口可以有多少个长连接?受到什么影响?linux最大文件句柄数量总结
Socket TCP Server一个端口可以有多少个长连接? 网上答案很多,不知道那个才是正确的 理论上是无限的 16.Linux中,一个端口能够接受tcp链接数量的理论上限是? A.1024 B. ...
- 基于 LWIP 建立 TCP Server 与主机通信实验
LWIP 版本:2.0.3 上一篇文章是写如何将 LWIP 移植到板子上,今天晚上记录基于 LWIP 实现与主机的网络通信. 先是打开了原子的实验例程,大概浏览了一遍,觉得 TCP 网络网络通信也就是 ...
- swoole深入学习 2. tcp Server和tcp Client
这节来学习Swoole最基础的Server和Client.会通过创建一个tcp Server来讲解. server <?php class Server { private $serv; pub ...
- Modbus库开发笔记之九:利用协议栈开发Modbus TCP Server应用
前面我们已经完成了Modbus协议栈的开发,但这不是我们的目的.我们开发它的目的当然是要使用它来解决我们的实际问题.接下来我们就使用刚开发的Modbus协议栈开发一个Modbus TCP Server ...
- Modbus库开发笔记之三:Modbus TCP Server开发
在完成了前面的工作后,我们就可以实现有针对性的应用了,首先我们来实现Modbus TCP的服务器端应用.当然我们不是做具体的应用,而是对Modbus TCP的服务器端应用进行封装以供有需要时调用. 这 ...
随机推荐
- ThreadLocal 类说明
1 ThreadLocal 不是一个线程,而是保存线程本地化对象的容器.当运行于多线程环境的某个对象使用 ThreadLocal 维护变量时,ThreadLocal 为每一个使用该变量的线程分配一个独 ...
- linux中init.d文件夹的说明
一.简单说明 /etc/init.d 是 /etc/rc.d/init.d 的软链接(soft link).可以通过 ll 命令查看. ls -ld /etc/init.d lrwxrwxrwx. r ...
- mysql-5.7中innodb_buffer_pool页面淘汰算法
一. 什么是innodb_buffer_pool: innodb_buffer_pool是一块内存区域,innodb用它来缓存数据,索引,undo,change buffer ... : 这块区域又被 ...
- unity5, Configurable Joint: Anchor, Connected Anchor, Auto Configure Connected Anchor
configurable joint加在轮子上,connected body是车身. 这种情况下,Anchor=(0,0,0)表示轮子一端joint锚点取carWheelCenter Connecte ...
- Solr4:查询参数fq的用法(对结果进行过滤;两组关键词组合查询)
Solr查询参数文档可以参考: http://wiki.apache.org/solr/CommonQueryParameters#head-6522ef80f22d0e50d2f12ec487758 ...
- 初学 Haskell 练习:算24点
其中用到了 Monad 做不确定性计算.运行速度很快. -- woodfox, Oct 10, 2014 import Control.Applicative import Control.Monad ...
- linux安装rzsz(lrzsz)
lrzsz是一个unix通信套件提供的,X,Y和ZModem文件传输协议,可以用在Windows与linux系统之间的文件传输,体积小速度快,可以与xshell工具配合使用. (1)在线安装 yum ...
- java.net.DatagramPacket/java.net.DatagramSocket-UDP Socket编程
UDP 的 Java 支持 UDP 协议提供的服务不同于 TCP 协议的端到端服务,它是面向非连接的,属不可靠协议,UDP 套接字在使用前不需要进行连接.实际上,UDP 协议只实现了两个功能: 在 I ...
- Maven工程pom.xml文件秒变gradle工程的命令
下面是一个maven工程,我想把它转成gradle项目,怎么办? 打开cmd命令行窗口,切换到你的maven工程的pom.xml文件所在目录,然后执行如下命令: gradle init --type ...
- UBoot启动代码第一阶段流程
http://blog.csdn.net/xautfengzi/article/details/7470134 前段时间了看了UBoot的源码,放了一段时间之后忘得差不多了.现做一些注释,方便以后温习 ...