Building a Non-blocking TCP server using OTP principles
转自:https://erlangcentral.org/wiki/index.php/Building_a_Non-blocking_TCP_server_using_OTP_principles
Building a Non-blocking TCP server using OTP principles
From ErlangCentral Wiki
Contents |
Author
Serge Aleynikov <saleyn at gmail.com>
Overview
A reader of this tutorial is assumed to be familiar with gen_server and gen_fsm behaviours, TCP socket communications using gen_tcp module, active and passive socket modes, and OTP supervision principles.
OTP provides a convenient framework for building reliable applications. This is in part accomplished by abstracting common functionality into a set of reusable behaviours such as gen_server and gen_fsm that are linked to OTP's supervision hierarchy.
There are several known TCP server designs. The one we are going to cover involves one process listening for client connections and spawning an FSM process per connecting client. While there is support for TCP communications available in OTP through the gen_tcp module, there is no standard behavior for building non-blocking TCP servers using OTP standard guidelines. By non-blocking we imply that the listening process and the client-handling FSMs should not make any blocking calls and be readily responsive to incoming control messages (such as changes in system configuration, restart requests, etc.) without causing timeouts. Note that blocking in the context of Erlang means blocking an Erlang process rather than the emulator's OS process(es).
In this tutorial we will show how to build a non-blocking TCP server using gen_server and gen_fsm behaviours that offers flow control and is fully compliant with OTP application design principles.
A reader who is new to the OTP framework is encouraged to read Joe Armstrong's tutorial on how to build A Fault-tolerant Server using blocking gen_tcp:connect/3 and gen_tcp:accept/1 calls without involving OTP.
This tutorial was inspired by several threads (e.g. one, two) on the Erlang Questions mailing list mentioning an approach to building non-blocking asynchronous TCP servers.
Server Design
The design of our server will include the main application's supervisor tcp_server_app process with one_for_one restart strategy and two child specifications. The first one being a listening process implemented as a gen_server behaviour that will wait for asynchronous notifications of client socket connections. The second one is another supervisor tcp_client_sup responsible for starting client handling FSMs and logging abnormal disconnects via standard SASL error reports.
For the sake of simplicity of this tutorial, the client handling FSM (tcp_echo_fsm) will implement an echo server that will echo client's requests back to the client.
+----------------+
| tcp_server_app |
+--------+-------+
| (one_for_one)
+----------------+---------+
| |
+-------+------+ +-------+--------+
| tcp_listener | + tcp_client_sup |
+--------------+ +-------+--------+
| (simple_one_for_one)
+-----|---------+
+-------|--------+|
+--------+-------+|+
| tcp_echo_fsm |+
+----------------+
Application and Supervisor behaviours
In order to build an OTP application we need to construct modules implementing an application and supervisor behaviour callback functions. While traditionally these functionalities are implemented in separate modules, given their succinctness we'll combine them in one module.
As an added bonus we implement a get_app_env function that illustrates how to process configuration options as well as command-line options given to the emulator at start-up.
The two instances of init/1 function are for two tiers of supervision hierarchy. Since two different restart strategies for each supervisor are needed, we implement them at different tiers.
Upon application's startup the tcp_server_app:start/2 callback function calls supervisor:start_link/2 that creates main application's supervisor calling tcp_server_app:init([Port, Module]) callback. This supervisor creates a tcp_listener process and a child supervisor tcp_client_sup responsible for spawning client connections. The Module argument in the init function is the name of client-connection handling FSM (in this case tcp_echo_fsm).
TCP Server Application (tcp_server_app.erl) |
||
|
Listener Process
One of the shortcomings of the gen_tcp module is that it only exports interface to a blocking accept call. This leads most of developers working on an implementation of a TCP server build a custom process linked to a supervisor using proc_lib or come up with some other proprietary design.
Examining prim_inet module reveals an interesting fact that the actual call to inet driver to accept a client socket is asynchronous. While this is a non-documented property, which means that the OTP team is free to change this implementation, we will exploit this functionality in the construction of our server.
The listener process is implemented as a gen_server behaviour:
TCP Listener Process (tcp_listener.erl) |
||
|
In this module init/1 call takes two parameters - the port number that the TCP listener should be started on and the name of a protocol handling module for client connections. The initialization function opens a listening socket in passive {active, false} mode. This is done so that we have flow control of the data received on the connected client sockets that will inherit this option from the listening socket.
The most interesting part of this code is the prim_inet:async_accept/2 call as well as the handling of asynchronous inet_async messages. In order to get this working we also needed to copy some of the internal OTP code encapsulated in the set_sockopt/2 function that handles socket registration with inet database and copying some options to the client socket.
As soon as a client socket is connected inet driver will notify the listening process using {inet_async, ListSock, Ref, {ok, CliSocket}} message. At this point we'll instantiate a new client socket handling process and set its ownership of the CliSocket.
Client Socket Handling Process
While tcp_listener is a generic implementation, tcp_echo_fsm is a mere stub FSM for illustrating how to write TCP servers. This modules needs to export two functions - one start_link/0 for a tcp_client_sup supervisor and another set_socket/2 for the listener process to notify the client connection handling FSM process that it is now the owner of the socket, and can begin receiving messages by setting the {active, once} or {active, true} option.
We would like to highlight the synchronization pattern used between the listening process and client connection-handling FSM to avoid possible message loss due to dispatching some messages from the socket to the wrong (listening) process. The process owning the listening socket has it open with {active, false}. After accepting the client's socket that socket inherits its socket options (including {active, false}) from the listener, transfers ownership of the socket to the newly spawned client connection-handling FSM by calling gen_tcp:controlling_process/2 and calls Module:set_socket/2 to notify the FSM that it can start receiving messages from the socket. Until the FSM process enables message delivery by setting the active mode on the socket by calling inet:setopts(Socket, [{active, once}]), the data sent by the TCP sender stays in the socket buffer.
When socket ownership is transfered to FSM in the 'WAIT_FOR_SOCKET' state the FSM sets {active, once} option to let inet driver send it one TCP message at a time. This is the OTP way of preserving flow control and avoiding process message queue flooding with TCP data and crashing the system in case of a fast-producer-slow-consumer case.
The FSM states are implemented by special functions in the tcp_echo_fsm module that use a naming convention with capital case state names enclosed in single quotes. The FSM consists of two states. 'WAIT_FOR_SOCKET' is the initial state in which the FSM is waiting for assignment of socket ownership, and 'WAIT_FOR_DATA' is the state that represents awaiting for TCP message from a client. In this state FSM also handles a special 'timeout' message that signifies no activity from a client and causes the process to stop and close client connection.
TCP Client Socket Handling FSM (tcp_echo_fsm.erl) |
||
|
Application File
Another required part of building an OTP application is creation of an application file that includes application name, version, startup module and environment.
Application File (tcp_server.app) |
||
|
Compiling
Create the following directory structure for this application:
./tcp_server
./tcp_server/ebin/
./tcp_server/ebin/tcp_server.app
./tcp_server/src/tcp_server_app.erl
./tcp_server/src/tcp_listener.erl
./tcp_server/src/tcp_echo_fsm.erl
$ cd tcp_server/src
$ for f in tcp*.erl ; do erlc +debug_info -o ../ebin $f
Running
We are going to start an Erlang shell with SASL support so that we can view all progress and error reports for our TCP application. Also we are going to start appmon application in order to examine visually the supervision hierarchy.
$ cd ../ebin
$ erl -boot start_sasl
...
1> appmon:start().
{ok,<0.44.0>}
2> application:start(tcp_server).
ok
Now click on the tcp_server button in the appmon's window in order to display supervision hierarchy of the tcp_server application.
3> {ok,S} = gen_tcp:connect({127,0,0,1},2222,[{packet,2}]).
{ok,#Port<0.150>}
The step above initiated a new client connection to the echo server.
4> gen_tcp:send(S,<<"hello">>).
ok
5> f(M), receive M -> M end.
{tcp,#Port<0.150>,"hello"}
We verified that the echo server works as expected. Now let's try to crash the client connection on the server and watch for the supervisor generating an error report entry on screen.
6> [{_,Pid,_,_}] = supervisor:which_children(tcp_client_sup).
[{undefined,<0.64.0>,worker,[]}]
7> exit(Pid,kill).
true
=SUPERVISOR REPORT==== 31-Jul-2007::14:33:49 ===
Supervisor: {local,tcp_client_sup}
Context: child_terminated
Reason: killed
Offender: [{pid,<0.77.0>},
{name,undefined},
{mfa,{tcp_echo_fsm,start_link,[]}},
{restart_type,temporary},
{shutdown,2000},
{child_type,worker}]
Note that if you are putting this server under a stress test with many incoming connections, the listener process may fail to accept new connections after the number of open file descriptors reaches the limit set by the operating system. In that case you will see the error:
"too many open files"
If you are running Linux/UNIX, google for a solution (which ultimately boils down to increasing the per-process limit by setting "ulimit -n ..." option).
Conclusion
OTP provides building blocks for constructing non-blocking TCP servers. This tutorial showed how to create a simple TCP server with flow control using standard OTP behaviours. As an exercise the reader is encouraged to try abstracting generic non-blocking TCP server functionality into a stand-along behaviour.
Sample Implementations
View source |
Discuss this page |
Page history |
What links here |
Related changes
Building a Non-blocking TCP server using OTP principles的更多相关文章
- 【转载】C# Tutorial - Simple Threaded TCP Server
http://tech.pro/tutorial/704/csharp-tutorial-simple-threaded-tcp-server In this tutorial I'm going t ...
- socket - socketserver - start TCP server
前面提到如何使用socket模块启动tcpserver: 创建socket:sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 绑定ip: ...
- [转] 3个学习Socket编程的简单例子:TCP Server/Client, Select
以前都是采用ACE的编写网络应用,最近由于工作需要,需要直接只用socket接口编写CS的代码,重新学习这方面的知识,给出自己所用到的3个简单例子,都是拷贝别人的程序.如果你能完全理解这3个例子,估计 ...
- [转]一个基于完成端口的TCP Server Framework,浅析IOCP
[转]一个基于完成端口的TCP Server Framework,浅析IOCP http://www.cppblog.com/adapterofcoms/archive/2010/06/26/1187 ...
- Socket TCP Server一个端口可以有多少个长连接?受到什么影响?linux最大文件句柄数量总结
Socket TCP Server一个端口可以有多少个长连接? 网上答案很多,不知道那个才是正确的 理论上是无限的 16.Linux中,一个端口能够接受tcp链接数量的理论上限是? A.1024 B. ...
- 基于 LWIP 建立 TCP Server 与主机通信实验
LWIP 版本:2.0.3 上一篇文章是写如何将 LWIP 移植到板子上,今天晚上记录基于 LWIP 实现与主机的网络通信. 先是打开了原子的实验例程,大概浏览了一遍,觉得 TCP 网络网络通信也就是 ...
- swoole深入学习 2. tcp Server和tcp Client
这节来学习Swoole最基础的Server和Client.会通过创建一个tcp Server来讲解. server <?php class Server { private $serv; pub ...
- Modbus库开发笔记之九:利用协议栈开发Modbus TCP Server应用
前面我们已经完成了Modbus协议栈的开发,但这不是我们的目的.我们开发它的目的当然是要使用它来解决我们的实际问题.接下来我们就使用刚开发的Modbus协议栈开发一个Modbus TCP Server ...
- Modbus库开发笔记之三:Modbus TCP Server开发
在完成了前面的工作后,我们就可以实现有针对性的应用了,首先我们来实现Modbus TCP的服务器端应用.当然我们不是做具体的应用,而是对Modbus TCP的服务器端应用进行封装以供有需要时调用. 这 ...
随机推荐
- MATLAB(1)——基本调试方法(Debug)
作者:桂. 时间:2017-02-28 07:06:30 链接:http://www.cnblogs.com/xingshansi/articles/6477185.html 声明:转载请注明出处, ...
- NodeJS写日志_Log4js使用详解
今天和大家分享一下NodeJS中写日志的一个常用第三方包:Log4js. 跟随主流Blog特色,先简单介绍下Log4js的基本信息.介绍Log4js之前,需要先说一下Log4***,Log4***是由 ...
- cocos2dx当节点存在缩放时要注意的问题
cocos2dx(所有引擎也均如此),如果一个节点存在缩放,一定不要忘了其局部空间里的单位长度也会发生变化.其子节点位移,局部空间转世界空间结果等都会受影响. 有时候我们想将父节点的缩放转移到子节点中 ...
- How to set JAVA environment variables in Linux or CentOS
How to set JAVA environment variables JAVA_HOME and PATH in Linux After installing new java (jdk or ...
- C#操作txt文件并清空添加操作
C#操作txt文件,进行清空添加操作的例子.代码: //把txt清空 FileStream stream = File.Open(Adr,FileMode.OpenOrCreate,FileAcces ...
- git 从远程git服务上拉代码 git服务器非默认端口
从服务器上拉代码有如下报错: fatal: Not a git repository (or any of the parent directories): .git 初始代本地版本库: [root@ ...
- C# 获取指定目录下所有文件信息
/// <summary> /// 返回指定目录下所有文件信息 /// </summary> /// <param name="strDirectory&quo ...
- python文件和目录操作方法大全
一.python中对文件.文件夹操作时经常用到的os模块和shutil模块常用方法. 1.得到当前工作目录,即当前Python脚本工作的目录路径: os.getcwd()2.返回指定目录下的所有文件和 ...
- Spark的基本说明
1.关于Application 用户程序,一个Application由一个在Driver运行的功能代码和多个Executor上运行的代码组成(工作在不同的节点上). 又分成多个Job,每个Job由多个 ...
- sqlserver中sp_executesql使用实例(获取动态sql输出结果)
语法 sp_executesql [ @stmt = ] stmt [ {, [@params=] N'@parameter_name data_type [ [ OUT [ PUT ][,. ...