基于Kurento的WebRTC移动视频群聊技术方案
说在前面的话:视频实时群聊天有三种架构:
Mesh架构:终端之间互相连接,没有中心服务器,产生的问题,每个终端都要连接n-1个终端,每个终端的编码和网络压力都很大。群聊人数N不可能太大。
Router架构:终端之间引入中心服务器,学名MCU(Multi Point Control Unit),每个终端的视频流都发布到MCU服务器上,然后服务器负责编码发布多视频流的工作,减轻客户端的压力。
Mix架构:在Router架构基础上,多个视频流在服务器端被合为一个视频流,减轻网络压力。
下面讲我们的选择,在MCU方面有licode、kurento等解决方案。kurento在视频群聊领域有专门的kurento Room解决方案,官方还提供一个kurento room server的样例实现。
首先可以考虑不是一个Kurento Room Demo作为搭建方案原型的MCU组件。
Room Demo的部署可见:http://doc-kurento-room.readthedocs.io/en/stable/demo_deployment.html
其中碰到一些Maven编译问题:
- Unable to initialise extensions Component descriptor role: 'com.jcraft.jsch.UIKeyboardInteractive', implementation: 'org.apache.maven.wagon.providers.ssh.jsch.interactive.PrompterUIKeyboardInteractive', role hint: 'default' has a hint, but there are other implementations that don't
Maven的安装版本需要时3.0以上
还有碰到找不到bower命令行问题。bower是Node.js下面的一个包管理工具,安装node.js以后用npm安装即可
最后按照部署指南网页中的命令启动服务器即可。
Demo服务器有两部分,一部分是Demo Web服务器,二是把官方的kurento room server也集成到了这个demo中。不用再架设独立的kurento room server
说说Android段的实施:再说一个公司:http://www.nubomedia.eu/,这家公司提供实时媒体通信开源云服务,核心组件可能是kurento media server,它的官网和kurento官网用一个模板,about里面显示两家组织有联系,kurento官方提供的JavaClient因为底层API原因在android上不肯用,这个nubomedia组织提供了一个kurento android client的实现,同时还提供了一个kurento room client的实现以及room使用案例:https://github.com/nubomedia-vtt/nubo-test,这家公司对其开发的开源方案管理非常及时,早晨提个接口的issue,下午已经commit了代码修改。
这个案例虽然支持room沟通,但视频沟通是基于room发布订阅机制做的双人聊天。略改一下代码应该就可以实现多人聊天不过这家组织提供的两个client实现和官方的接口高度相似。主要改的是PeerVideoActivity这个类,下面我share一个基本走通多端通信的这个类的代码,供大家参考:
- package fi.vtt.nubotest;
- import android.app.ListActivity;
- import android.content.SharedPreferences;
- import android.graphics.PixelFormat;
- import android.opengl.GLSurfaceView;
- import android.os.Bundle;
- import android.os.Handler;
- import android.util.Log;
- import android.view.Menu;
- import android.view.MenuItem;
- import android.view.View;
- import android.view.WindowManager;
- import android.widget.TextView;
- import android.widget.Toast;
- import org.webrtc.IceCandidate;
- import org.webrtc.MediaStream;
- import org.webrtc.PeerConnection;
- import org.webrtc.RendererCommon;
- import org.webrtc.SessionDescription;
- import org.webrtc.VideoRenderer;
- import org.webrtc.VideoRendererGui;
- import java.util.Map;
- import fi.vtt.nubomedia.kurentoroomclientandroid.RoomError;
- import fi.vtt.nubomedia.kurentoroomclientandroid.RoomListener;
- import fi.vtt.nubomedia.kurentoroomclientandroid.RoomNotification;
- import fi.vtt.nubomedia.kurentoroomclientandroid.RoomResponse;
- import fi.vtt.nubomedia.webrtcpeerandroid.NBMMediaConfiguration;
- import fi.vtt.nubomedia.webrtcpeerandroid.NBMPeerConnection;
- import fi.vtt.nubomedia.webrtcpeerandroid.NBMWebRTCPeer;
- import fi.vtt.nubotest.util.Constants;
- /**
- * Activity for receiving the video stream of a peer
- * (based on PeerVideoActivity of Pubnub's video chat tutorial example.
- */
- public class PeerVideoActivity extends ListActivity implements NBMWebRTCPeer.Observer, RoomListener {
- private static final String TAG = "PeerVideoActivity";
- private NBMMediaConfiguration peerConnectionParameters;
- private NBMWebRTCPeer nbmWebRTCPeer;
- private SessionDescription localSdp;
- private SessionDescription remoteSdp;
- private String PaticipantID;
- private VideoRenderer.Callbacks localRender;
- private VideoRenderer.Callbacks remoteRender;
- private GLSurfaceView videoView;
- private SharedPreferences mSharedPreferences;
- private int publishVideoRequestId;
- private int sendIceCandidateRequestId;
- private TextView mCallStatus;
- private String username, calluser;
- private boolean backPressed = false;
- private Thread backPressedThread = null;
- private static final int LOCAL_X_CONNECTED = 72;
- private static final int LOCAL_Y_CONNECTED = 72;
- private static final int LOCAL_WIDTH_CONNECTED = 25;
- private static final int LOCAL_HEIGHT_CONNECTED = 25;
- // Remote video screen position
- private static final int REMOTE_X = 0;
- private static int REMOTE_Y = 0;
- private static final int REMOTE_WIDTH = 25;
- private static final int REMOTE_HEIGHT = 25;
- private Handler mHandler;
- private CallState callState;
- private enum CallState{
- IDLE, PUBLISHING, PUBLISHED, WAITING_REMOTE_USER, RECEIVING_REMOTE_USER,PATICIPANT_JOINED,RECEIVING_PATICIPANT,
- }
- @Override
- public void onCreate(Bundle savedInstanceState) {
- super.onCreate(savedInstanceState);
- callState = CallState.IDLE;
- setContentView(R.layout.activity_video_chat);
- getWindow().addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);
- mHandler = new Handler();
- Bundle extras = getIntent().getExtras();
- if (extras == null || !extras.containsKey(Constants.USER_NAME)) {
- ;
- Toast.makeText(this, "Need to pass username to PeerVideoActivity in intent extras (Constants.USER_NAME).",
- Toast.LENGTH_SHORT).show();
- finish();
- return;
- }
- this.username = extras.getString(Constants.USER_NAME, "");
- Log.i(TAG, "username: " + username);
- if (extras.containsKey(Constants.CALL_USER)) {
- this.calluser = extras.getString(Constants.CALL_USER, "");
- Log.i(TAG, "callUser: " + calluser);
- }
- this.mCallStatus = (TextView) findViewById(R.id.call_status);
- TextView prompt = (TextView) findViewById(R.id.receive_prompt);
- prompt.setText("Receive from " + calluser);
- this.videoView = (GLSurfaceView) findViewById(R.id.gl_surface);
- // Set up the List View for chatting
- RendererCommon.ScalingType scalingType = RendererCommon.ScalingType.SCALE_ASPECT_FILL;
- VideoRendererGui.setView(videoView, null);
- localRender = VideoRendererGui.create( LOCAL_X_CONNECTED, LOCAL_Y_CONNECTED,
- LOCAL_WIDTH_CONNECTED, LOCAL_HEIGHT_CONNECTED,
- scalingType, true);
- NBMMediaConfiguration.NBMVideoFormat receiverVideoFormat = new NBMMediaConfiguration.NBMVideoFormat(352, 288, PixelFormat.RGB_888, 20);
- peerConnectionParameters = new NBMMediaConfiguration( NBMMediaConfiguration.NBMRendererType.OPENGLES,
- NBMMediaConfiguration.NBMAudioCodec.OPUS, 0,
- NBMMediaConfiguration.NBMVideoCodec.VP8, 0,
- receiverVideoFormat,
- NBMMediaConfiguration.NBMCameraPosition.FRONT);
- nbmWebRTCPeer = new NBMWebRTCPeer(peerConnectionParameters, this, localRender, this);
- nbmWebRTCPeer.initialize();
- Log.i(TAG, "PeerVideoActivity initialized");
- mHandler.postDelayed(publishDelayed, 4000);
- MainActivity.getKurentoRoomAPIInstance().addObserver(this);
- callState = CallState.PUBLISHING;
- mCallStatus.setText("Publishing...");
- }
- private Runnable publishDelayed = new Runnable() {
- @Override
- public void run() {
- nbmWebRTCPeer.generateOffer("derp", true);
- }
- };
- @Override
- public boolean onCreateOptionsMenu(Menu menu) {
- // Inflate the menu; this adds items to the action bar if it is present.
- getMenuInflater().inflate(R.menu.menu_video_chat, menu);
- return true;
- }
- @Override
- public boolean onOptionsItemSelected(MenuItem item) {
- // Handle action bar item clicks here. The action bar will
- // automatically handle clicks on the Home/Up button, so long
- // as you specify a parent activity in AndroidManifest.xml.
- int id = item.getItemId();
- //noinspection SimplifiableIfStatement
- if (id == R.id.action_settings) {
- return true;
- }
- return super.onOptionsItemSelected(item);
- }
- @Override
- protected void onStart() {
- super.onStart();
- }
- @Override
- protected void onPause() {
- nbmWebRTCPeer.stopLocalMedia();
- super.onPause();
- }
- @Override
- protected void onResume() {
- super.onResume();
- nbmWebRTCPeer.startLocalMedia();
- }
- @Override
- protected void onStop() {
- endCall();
- super.onStop();
- }
- @Override
- protected void onDestroy() {
- super.onDestroy();
- }
- @Override
- public void onBackPressed() {
- // If back button has not been pressed in a while then trigger thread and toast notification
- if (!this.backPressed){
- this.backPressed = true;
- Toast.makeText(this,"Press back again to end.",Toast.LENGTH_SHORT).show();
- this.backPressedThread = new Thread(new Runnable() {
- @Override
- public void run() {
- try {
- Thread.sleep(5000);
- backPressed = false;
- } catch (InterruptedException e){ Log.d("VCA-oBP","Successfully interrupted"); }
- }
- });
- this.backPressedThread.start();
- }
- // If button pressed the second time then call super back pressed
- // (eventually calls onDestroy)
- else {
- if (this.backPressedThread != null)
- this.backPressedThread.interrupt();
- super.onBackPressed();
- }
- }
- public void hangup(View view) {
- finish();
- }
- public void receiveFromRemote(View view){
- Log.e(TAG,"--->receiveFromRemote");
- if (callState == CallState.PUBLISHED){
- callState = CallState.WAITING_REMOTE_USER;
- nbmWebRTCPeer.generateOffer("remote", false);
- runOnUiThread(new Runnable() {
- @Override
- public void run() {
- mCallStatus.setText("Waiting remote stream...");
- }
- });
- }
- }
- /**
- * Terminates the current call and ends activity
- */
- private void endCall() {
- callState = CallState.IDLE;
- try
- {
- if (nbmWebRTCPeer != null) {
- nbmWebRTCPeer.close();
- nbmWebRTCPeer = null;
- }
- }
- catch (Exception e){e.printStackTrace();}
- }
- @Override
- public void onLocalSdpOfferGenerated(final SessionDescription sessionDescription, NBMPeerConnection nbmPeerConnection) {
- Log.e(TAG,"--->onLocalSdpOfferGenerated");
- if (callState == CallState.PUBLISHING || callState == CallState.PUBLISHED) {
- localSdp = sessionDescription;
- Log.e(TAG,"--->onLocalSdpOfferGenerated:publish");
- runOnUiThread(new Runnable() {
- @Override
- public void run() {
- if (MainActivity.getKurentoRoomAPIInstance() != null) {
- Log.d(TAG, "Sending " + sessionDescription.type);
- publishVideoRequestId = ++Constants.id;
- // String sender = calluser + "_webcam";
- // MainActivity.getKurentoRoomAPIInstance().sendReceiveVideoFrom(sender, localSdp.description, publishVideoRequestId);
- MainActivity.getKurentoRoomAPIInstance().sendPublishVideo(localSdp.description, false, publishVideoRequestId);
- }
- }
- });
- } else { // Asking for remote user video
- Log.e(TAG,"--->onLocalSdpOfferGenerated:remote");
- remoteSdp = sessionDescription;
- // nbmWebRTCPeer.selectCameraPosition(NBMMediaConfiguration.NBMCameraPosition.BACK);
- runOnUiThread(new Runnable() {
- @Override
- public void run() {
- if (MainActivity.getKurentoRoomAPIInstance() != null) {
- Log.e(TAG, "Sending--> " +calluser+ sessionDescription.type);
- publishVideoRequestId = ++Constants.id;
- String sender = calluser + "_webcam";
- MainActivity.getKurentoRoomAPIInstance().sendReceiveVideoFrom(sender, remoteSdp.description, publishVideoRequestId);
- }
- }
- });
- }
- }
- @Override
- public void onLocalSdpAnswerGenerated(SessionDescription sessionDescription, NBMPeerConnection nbmPeerConnection) {
- }
- @Override
- public void onIceCandidate(IceCandidate iceCandidate, NBMPeerConnection nbmPeerConnection) {
- Log.e(TAG,"--->onIceCandidate");
- sendIceCandidateRequestId = ++Constants.id;
- if (callState == CallState.PUBLISHING || callState == CallState.PUBLISHED){
- Log.e(TAG,"--->onIceCandidate:publish");
- MainActivity.getKurentoRoomAPIInstance().sendOnIceCandidate(this.username, iceCandidate.sdp,
- iceCandidate.sdpMid, Integer.toString(iceCandidate.sdpMLineIndex), sendIceCandidateRequestId);
- } else{
- Log.e(TAG,"--->onIceCandidate:"+this.calluser);
- MainActivity.getKurentoRoomAPIInstance().sendOnIceCandidate(this.calluser, iceCandidate.sdp,
- iceCandidate.sdpMid, Integer.toString(iceCandidate.sdpMLineIndex), sendIceCandidateRequestId);
- }
- }
- @Override
- public void onIceStatusChanged(PeerConnection.IceConnectionState iceConnectionState, NBMPeerConnection nbmPeerConnection) {
- Log.i(TAG, "onIceStatusChanged");
- }
- @Override
- public void onRemoteStreamAdded(MediaStream mediaStream, NBMPeerConnection nbmPeerConnection) {
- if (callState == CallState.PUBLISHING || callState == CallState.PUBLISHED) {
- Log.e(TAG, "-->onRemoteStreamAdded-->no");
- return;
- }
- Log.e(TAG, "-->onRemoteStreamAdded");
- RendererCommon.ScalingType scalingType = RendererCommon.ScalingType.SCALE_ASPECT_FILL;
- remoteRender = VideoRendererGui.create( REMOTE_X, REMOTE_Y,
- REMOTE_WIDTH, REMOTE_HEIGHT,
- scalingType, false);
- REMOTE_Y = REMOTE_Y+25;
- nbmWebRTCPeer.attachRendererToRemoteStream(remoteRender, mediaStream);
- runOnUiThread(new Runnable() {
- @Override
- public void run() {
- mCallStatus.setText("");
- }
- });
- }
- @Override
- public void onRemoteStreamRemoved(MediaStream mediaStream, NBMPeerConnection nbmPeerConnection) {
- Log.i(TAG, "onRemoteStreamRemoved");
- }
- @Override
- public void onPeerConnectionError(String s) {
- Log.e(TAG, "onPeerConnectionError:" + s);
- }
- @Override
- public void onRoomResponse(RoomResponse response) {
- Log.e(TAG, "-->OnRoomResponse:" + response);
- if (Integer.valueOf(response.getId()) == publishVideoRequestId){
- SessionDescription sd = new SessionDescription(SessionDescription.Type.ANSWER,
- response.getValue("sdpAnswer").get(0));
- if (callState == CallState.PUBLISHING){
- callState = CallState.PUBLISHED;
- nbmWebRTCPeer.processAnswer(sd, "derp");
- } else if (callState == CallState.WAITING_REMOTE_USER){
- callState = CallState.RECEIVING_REMOTE_USER;
- nbmWebRTCPeer.processAnswer(sd, "remote");
- } else if (callState == CallState.PATICIPANT_JOINED){
- callState = CallState.RECEIVING_PATICIPANT;
- nbmWebRTCPeer.processAnswer(sd, this.PaticipantID);
- //NOP
- }
- }
- }
- @Override
- public void onRoomError(RoomError error) {
- Log.e(TAG, "OnRoomError:" + error);
- }
- @Override
- public void onRoomNotification(RoomNotification notification) {
- Log.e(TAG, "OnRoomNotification--> (state=" + callState.toString() + "):" + notification);
- if(notification.getMethod().equals("iceCandidate")) {
- Map<String, Object> map = notification.getParams();
- String sdpMid = map.get("sdpMid").toString();
- int sdpMLineIndex = Integer.valueOf(map.get("sdpMLineIndex").toString());
- String sdp = map.get("candidate").toString();
- IceCandidate ic = new IceCandidate(sdpMid, sdpMLineIndex, sdp);
- Log.e(TAG, "callState-->" + callState);
- if (callState == CallState.PUBLISHING || callState == CallState.PUBLISHED) {
- nbmWebRTCPeer.addRemoteIceCandidate(ic, "derp");
- }else if(callState==CallState.PATICIPANT_JOINED || callState== CallState.RECEIVING_PATICIPANT){
- nbmWebRTCPeer.addRemoteIceCandidate(ic,this.PaticipantID);
- }else {
- nbmWebRTCPeer.addRemoteIceCandidate(ic, "remote");
- }
- }
- if(notification.getMethod().equals("participantPublished"))
- {
- Map<String, Object> map = notification.getParams();
- final String user = map.get("id").toString();
- this.calluser = user;
- this.PaticipantID = "pt_"+this.calluser;
- PeerVideoActivity.this.runOnUiThread(new Runnable() {
- @Override
- public void run() {
- callState = CallState.PATICIPANT_JOINED;
- nbmWebRTCPeer.generateOffer(PaticipantID, false);
- }
- });
- }
- }
- @Override
- public void onRoomConnected() {
- }
- @Override
- public void onRoomDisconnected() {
- }
- }
再就是android room demo中的MainActivity的添加cert的代码要去掉注释,让这段代码生效,就可以连通服务器了。
在iOS的实施方面,上面这家公司也提供了一个工具包:https://github.com/nubomediaTI/Kurento-ios ,工具包里面也有demo
Web方面,最上面官方的哪个demo就足够参考了
后记:很荣幸这篇博客获得了很多CSDN程序员的关注和询问,这只能证明我很荣幸有机会在去年的那个时间点(16年7月)在大家之前处理了一个后续大家都很关注的技术问题,而处理这个问题主要用到的服务器端room server项目和android端nubo test项目,官方在后续好像都做了一定的升级,反而是我自己搞完这个之后,因为产品设计的原因,后来再没有深入地去生产实施这个东西,甚至开发笔记本关于这个项目的源码项目好像都已经删除了,对于大家提出的问题,早期的我还能答一答,后面的我估计你们用到的源码和我用到的源码估计都不是一个版本了,再就是里面的代码细节也基本忘得差不多,在这儿我建议后续开发这个功能可以去深入阅读分析Kurento官方(https://github.com/Kurento)和欧洲媒体服务云服务商nubomedia官方(https://github.com/nubomedia-vtt)的代码示例和文档。我面给出的代码样例是基于nubomedia一对一视聊样例改的,官方原始代码样例在这段时间内都有了变更。在掌握大的基本WebRTC通信的原理的前提下,我觉得改新的代码估计也不会太难。
基于Kurento的WebRTC移动视频群聊技术方案的更多相关文章
- 网易云信技术分享:IM中的万人群聊技术方案实践总结
本文来自网易云信团队的技术分享,原创发表于网易云信公众号,原文链接:mp.weixin.qq.com/s/LT2dASI7QVpcOVxDAsMeVg,收录时有改动. 1.引言 在不了解IM技术的人眼 ...
- android 开发,视频群聊引发短信异常
说到 NDK 开发,其实是为了有些时候为了项目需求需要调用底层的一些 C/C++ 的一些东西:另外就是为了效率更加高些. 但是很多时候能不用就不用:这个是啥原因?个人感觉有些时候是觉得麻烦,首先要配置 ...
- 一套高可用、易伸缩、高并发的IM群聊架构方案设计实践
本文原题为“一套高可用群聊消息系统实现”,由作者“于雨氏”授权整理和发布,内容有些许改动,作者博客地址:alexstocks.github.io.应作者要求,如需转载,请联系作者获得授权. 一.引言 ...
- ASP.NET SignalR 与LayIM配合,轻松实现网站客服聊天室(四) 添加表情、群聊功能
休息了两天,还是决定把这个尾巴给收了.本篇是最后一篇,也算是草草收尾吧.今天要加上表情功能和群聊.基本上就差不多了,其他功能,读者可以自行扩展或者优化.至于我写的代码方面,自己也没去重构.好的,我们开 ...
- 使用java做一个能赚钱的微信群聊机器人(2020年基于PC端协议最新可用版)
前言 微信群机器人,主要用来管理群聊,提供类似天气查询.点歌.机器人聊天等用途. 由于微信将web端的协议封杀后,很多基于http协议的群聊机器人都失效了,所以这里使用基于PC端协议的插件来实现. 声 ...
- 基于itchat的微信群聊小助手基础开发(一)
前段时间由于要管理微信群,基于itchat开发了一个简单的微信机器人 主要功能有: 图灵机器人功能 群聊昵称格式修改提示 消息防撤回功能 斗图功能 要开发一个基于itchat的最基本的聊天机器人,在g ...
- 基于ejabberd简单实现xmpp群聊离线消息
首先,xmpp服务器是基于ejabberd.离线消息模块是mod_interact,原地址地址:https://github.com/adamvduke/mod_interact: 修改后实现群聊离线 ...
- Flask(4)- flask请求上下文源码解读、http聊天室单聊/群聊(基于gevent-websocket)
一.flask请求上下文源码解读 通过上篇源码分析,我们知道了有请求发来的时候就执行了app(Flask的实例化对象)的__call__方法,而__call__方法返回了app的wsgi_app(en ...
- 基于websocket的单聊.群聊
关于ai.baidu.com的 代码: #########################################核心代码################################### ...
随机推荐
- iOS中类单例方法的一种实现
在Cocos2D编程中,很多情况我们需要类只生成一个实例,这称之为该类的单例类. 一般我们在类中这样实现单例方法: +(instancetype)sharedInstance{ static Foo ...
- Gradle 1.12翻译——第十九章. Gradle 守护进程
有关其他已翻译的章节请关注Github上的项目:https://github.com/msdx/gradledoc/tree/1.12,或访问:http://gradledoc.qiniudn.com ...
- Gitflow工作流程
在工作场合实施Git的时候,有很多种工作流程可供选择,此时反而会让你手足无措.本文罗列了企业团队最常用的一些Git工作流程,包括Centralized Workflow.Feature Branch ...
- Linux多线程实践(2) --线程基本API
POSIX线程库 与线程有关的函数构成了一个完整的系列,绝大多数函数的名字都是以"pthread_"开头,要使用这些函数库,要通过引入头文<pthread.h>,而且链 ...
- 【嵌入式开发】 Linux Kernel 下载 配置 编译 安装 及 驱动简介
作者 : 韩曙亮 转载请出名出处 : http://blog.csdn.net/shulianghan/article/details/38636827 一. Linux 内核简介 1. 内核功能简介 ...
- java组播MulticastSocket
在单播模式中有服务器端和客户端之分,而组播模式与单播模式不同,每个端都是以路由器或交换机做为中转广播站,任意一端向路由器或交换机发送消息,路由或交换机负责发送其他节点,每个节点都是同等的.所以在编程模 ...
- 网站开发进阶(三十)HTML5--本地存储Web Storage
HTML5--本地存储Web Storage Web Storage功能,顾名思义,就是在Web上针对客户端本地储存数据的功能,具体来说Web Storage分为两种: sessionStorage: ...
- SpriteBuilder中不能编辑自定义类或不能给节点添加属性的解决
不能编辑自定义类 你选中一个Sub File(CCBFile)节点,在这个例子中,该节点的Custom class区域灰化禁用且不能修改.这是因为你需要在该Sub File引用的CCB文件中修改Cus ...
- 2014新年福利,居然有人将Ext JS 4.1的文档翻译了
原文:http://damoqiongqiu.iteye.com/blog/1998022
- android Titlebar一行代码实现沉浸式效果
github地址 一个简单易用的导航栏TitleBar,可以轻松实现IOS导航栏的各种效果 整个代码全部集中在TitleBar.java中,所有控件都动态生成,动态布局.不需要引用任何资源文件,拷贝 ...