class  TransportServer

 bootstrap.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel ch) throws Exception {
RpcHandler rpcHandler = appRpcHandler;
for (TransportServerBootstrap bootstrap : bootstraps) {
rpcHandler = bootstrap.doBootstrap(ch, rpcHandler);
}
context.initializePipeline(ch, rpcHandler);
}
});

每次有新client连接完成,server都会调用匿名类的initChannel方法,在调用context.initializePipeline(ch, rpcHandler)时,通过createChannelHandler方法,创建TransportResponseHandler用于处理响应消息(Channel直接被隐藏在TransportResponseHandler中),创建TransportRequestHandler 用于处理请求消息(Channel被封装在TransportClient中,TransportClient又被隐藏在TransportRequestHandler),最后返回的TransportChannelHandler包含三者。

public TransportChannelHandler initializePipeline(
SocketChannel channel,
RpcHandler channelRpcHandler) {
try {
TransportChannelHandler channelHandler = createChannelHandler(channel, channelRpcHandler);
channel.pipeline()
.addLast("encoder", encoder)
.addLast(TransportFrameDecoder.HANDLER_NAME, NettyUtils.createFrameDecoder())
.addLast("decoder", decoder)
.addLast("idleStateHandler", new IdleStateHandler(0, 0, conf.connectionTimeoutMs() / 1000))
// NOTE: Chunks are currently guaranteed to be returned in the order of request, but this
// would require more logic to guarantee if this were not part of the same event loop.
.addLast("handler", channelHandler);
return channelHandler;
} catch (RuntimeException e) {
logger.error("Error while initializing Netty pipeline", e);
throw e;
}
}
private TransportChannelHandler createChannelHandler(Channel channel, RpcHandler rpcHandler) {
TransportResponseHandler responseHandler = new TransportResponseHandler(channel);
TransportClient client = new TransportClient(channel, responseHandler);
TransportRequestHandler requestHandler = new TransportRequestHandler(channel, client,
rpcHandler);
return new TransportChannelHandler(client, responseHandler, requestHandler,
conf.connectionTimeoutMs(), closeIdleConnections);
}
class TransportChannelHandler
//原始消息到达,根据消息类型调用不同处理方式。此处以RequestMessage消息为例
public void channelRead0(ChannelHandlerContext ctx, Message request) throws Exception {
if (request instanceof RequestMessage) {
requestHandler.handle((RequestMessage) request); //上文实例的TransportRequestHandler
} else {
responseHandler.handle((ResponseMessage) request);
}
}
class  TransportRequestHandler 

  @Override
public void handle(RequestMessage request) {
if (request instanceof ChunkFetchRequest) {
processFetchRequest((ChunkFetchRequest) request);
} else if (request instanceof RpcRequest) {
processRpcRequest((RpcRequest) request); //以rpc请求为例
} else if (request instanceof OneWayMessage) {
processOneWayMessage((OneWayMessage) request);
} else if (request instanceof StreamRequest) {
processStreamRequest((StreamRequest) request);
} else {
throw new IllegalArgumentException("Unknown request type: " + request);
}
} private void processRpcRequest(final RpcRequest req) {
try {
rpcHandler.receive(reverseClient, req.body().nioByteBuffer(), new RpcResponseCallback() { //reverseClient上文传递的值
@Override
public void onSuccess(ByteBuffer response) {
respond(new RpcResponse(req.requestId, new NioManagedBuffer(response)));
} @Override
public void onFailure(Throwable e) {
respond(new RpcFailure(req.requestId, Throwables.getStackTraceAsString(e)));
}
});
} catch (Exception e) {
logger.error("Error while invoking RpcHandler#receive() on RPC id " + req.requestId, e);
respond(new RpcFailure(req.requestId, Throwables.getStackTraceAsString(e)));
} finally {
req.body().release();
}
} class NettyRpcHandler override def receive(
client: TransportClient,
message: ByteBuffer,
callback: RpcResponseCallback): Unit = {
val messageToDispatch = internalReceive(client, message) //将client封装在messageToDispatch 中
dispatcher.postRemoteMessage(messageToDispatch, callback) //此方法是将消息封装成inbox形式,放入inbox消息队列
} internalReceive调用
val requestMessage = nettyEnv.deserialize[RequestMessage](client, message) class NettyRpcEnv
private[netty] def deserialize[T: ClassTag](client: TransportClient, bytes: ByteBuffer): T = {
NettyRpcEnv.currentClient.withValue(client) {
deserialize { () =>
javaSerializerInstance.deserialize[T](bytes) //反序列化时调用NettyRpcEndpointRef实例的readObject方法,而下面两个DynamicVariable在此处被替换为新值
}
}
} object NettyRpcEnv
//DynamicVariable可以用来在指定作用域替换数据
private[netty] val currentEnv = new DynamicVariable[NettyRpcEnv](null)
private[netty] val currentClient = new DynamicVariable[TransportClient](null) class NettyRpcEndpointRef private def readObject(in: ObjectInputStream): Unit = { //反序列化,为client、nettyEnv填充新值,而非通过网络传送过来的空值
in.defaultReadObject()
nettyEnv = NettyRpcEnv.currentEnv.value
client = NettyRpcEnv.currentClient.value
}

spark-rpc是如何实现将netty的Channel隐藏在inbox中的的更多相关文章

  1. spark2.1源码分析3:spark-rpc如何实现将netty的Channel隐藏在inbox中

    class TransportServer bootstrap.childHandler(new ChannelInitializer<SocketChannel>() { @Overri ...

  2. Spark RPC框架源码分析(一)简述

    Spark RPC系列: Spark RPC框架源码分析(一)运行时序 Spark RPC框架源码分析(二)运行时序 Spark RPC框架源码分析(三)运行时序 一. Spark rpc框架概述 S ...

  3. Spark RPC框架源码分析(二)RPC运行时序

    前情提要: Spark RPC框架源码分析(一)简述 一. Spark RPC概述 上一篇我们已经说明了Spark RPC框架的一个简单例子,Spark RPC相关的两个编程模型,Actor模型和Re ...

  4. org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse

    跑sparkPis示例程序 [root@node01 bin]# ./spark-submit --master spark://node01:7077 --class org.apache.spar ...

  5. spark RPC详解

    前段时间看spark,看着迷迷糊糊的.最近终于有点头绪,先梳理了一下spark rpc相关的东西,先记录下来. 1,概述 个人认为,如果把分布式系统(HDFS, HBASE,SPARK等)比作一个人, ...

  6. Spark RPC

    在Spark中,对于网络调用的底层封装(粘包拆包,编解码,链路管理等)都是在common/network-common包中实现的(详见[common/network-common]).在common/ ...

  7. spark 源码分析之六--Spark RPC剖析之Dispatcher和Inbox、Outbox剖析

    在上篇 spark 源码分析之五 -- Spark内置RPC机制剖析之一创建NettyRPCEnv 中,涉及到了Diapatcher 内容,未做过多的剖析.本篇来剖析一下它的工作原理. Dispatc ...

  8. spark 源码分析之八--Spark RPC剖析之TransportContext和TransportClientFactory剖析

    spark 源码分析之八--Spark RPC剖析之TransportContext和TransportClientFactory剖析 TransportContext 首先官方文档对Transpor ...

  9. spark 源码分析之十--Spark RPC剖析之TransportResponseHandler、TransportRequestHandler和TransportChannelHandler剖析

    spark 源码分析之十--Spark RPC剖析之TransportResponseHandler.TransportRequestHandler和TransportChannelHandler剖析 ...

随机推荐

  1. Ubuntu系统下Anaconda使用方法总结

    前言:个人认为Anaconda比pip使用起来要方便很多,因为它创建虚拟环境和下载各种库都很方便.但是有时候conda的安装方法中没有某些库,必须使用pip来进行安装.所以,主要用Anaconda,偶 ...

  2. mayavi与X11的一些坑总结

    1. Mayavi:https://docs.enthought.com/mayavi/mayavi/ 适合渲染各种和图形有关的数据.在大批量处理数据时,一般不希望对每一个渲染好的对象都将其显示出来, ...

  3. Ubuntu如何使用Vscode写C++代码

    (一). 下载Vscode (1). 打开Ubuntu软件 (2). 搜索Vscode,下载并运行. (二). 安装C++ (1). 看到右边的列表,点击最下面的 (2). 然后我们搜索和下载 C++ ...

  4. 每天一本电子书 - JavaScript for Kids: A Playful Introduction to Programming

    JavaScript for Kids: A Playful Introduction to Programming 作者: Nick Morgan  出版社: No Starch Press 副标题 ...

  5. 部署在sae上的servlet程序出现is not a javax.servlet.Servlet 错误

    sae本身提供了servlet jar包, 部署时删掉你lib目录下的servlet jar包.

  6. Spring-AOP 基于注解的实现

    一.AOP: 是对OOP编程方式的一种补充.翻译过来为“面向切面编程”. 可以理解为一个拦截器框架,但是这个拦截器会非常武断,如果它拦截一个类,那么它就会拦截这个类中的所有方法.如对一个目标列的代理, ...

  7. Matlab 提取R,G,B颜色分量

    >> im = imread('ny.png'); >> r = im(:,:,1); >> g = im(:,:,2); >> b = im(:,:, ...

  8. Java第五周总结

    Java抽象类与接口的区别 很多常见的面试题都会出诸如抽象类和接口有什么区别,什么情况下会使用抽象类和什么情况你会使用接口这样的问题.本文我们将仔细讨论这些话题. 在讨论它们之间的不同点之前,我们先看 ...

  9. HTML相关内容

    一,HTML标签的分类 1,单标签和双标签 2,内联标签:span,b,u,i,a,img   块级标签:h1~h6,p,div,hr,ul,ol,li 二,常用标签 1,head内常用标签 标签 含 ...

  10. [spoj Favorite Dice ][期望dp]

    (1)https://vjudge.net/problem/SPOJ-FAVDICE 题意:有一个n面的骰子,每一面朝上的概率相同,求所有面都朝上过至少一次的总次数期望. 题解:令dp[i]表示 i ...