基于Kurento的WebRTC移动视频群聊技术方案
说在前面的话:视频实时群聊天有三种架构:
Mesh架构:终端之间互相连接,没有中心服务器,产生的问题,每个终端都要连接n-1个终端,每个终端的编码和网络压力都很大。群聊人数N不可能太大。
Router架构:终端之间引入中心服务器,学名MCU(Multi Point Control Unit),每个终端的视频流都发布到MCU服务器上,然后服务器负责编码发布多视频流的工作,减轻客户端的压力。
Mix架构:在Router架构基础上,多个视频流在服务器端被合为一个视频流,减轻网络压力。
下面讲我们的选择,在MCU方面有licode、kurento等解决方案。kurento在视频群聊领域有专门的kurento Room解决方案,官方还提供一个kurento room server的样例实现。
首先可以考虑不是一个Kurento Room Demo作为搭建方案原型的MCU组件。
Room Demo的部署可见:http://doc-kurento-room.readthedocs.io/en/stable/demo_deployment.html
其中碰到一些Maven编译问题:
- Unable to initialise extensions Component descriptor role: 'com.jcraft.jsch.UIKeyboardInteractive', implementation: 'org.apache.maven.wagon.providers.ssh.jsch.interactive.PrompterUIKeyboardInteractive', role hint: 'default' has a hint, but there are other implementations that don't
Maven的安装版本需要时3.0以上
还有碰到找不到bower命令行问题。bower是Node.js下面的一个包管理工具,安装node.js以后用npm安装即可
最后按照部署指南网页中的命令启动服务器即可。
Demo服务器有两部分,一部分是Demo Web服务器,二是把官方的kurento room server也集成到了这个demo中。不用再架设独立的kurento room server
说说Android段的实施:再说一个公司:http://www.nubomedia.eu/,这家公司提供实时媒体通信开源云服务,核心组件可能是kurento media server,它的官网和kurento官网用一个模板,about里面显示两家组织有联系,kurento官方提供的JavaClient因为底层API原因在android上不肯用,这个nubomedia组织提供了一个kurento android client的实现,同时还提供了一个kurento room client的实现以及room使用案例:https://github.com/nubomedia-vtt/nubo-test,这家公司对其开发的开源方案管理非常及时,早晨提个接口的issue,下午已经commit了代码修改。
这个案例虽然支持room沟通,但视频沟通是基于room发布订阅机制做的双人聊天。略改一下代码应该就可以实现多人聊天不过这家组织提供的两个client实现和官方的接口高度相似。主要改的是PeerVideoActivity这个类,下面我share一个基本走通多端通信的这个类的代码,供大家参考:
- package fi.vtt.nubotest;
- import android.app.ListActivity;
- import android.content.SharedPreferences;
- import android.graphics.PixelFormat;
- import android.opengl.GLSurfaceView;
- import android.os.Bundle;
- import android.os.Handler;
- import android.util.Log;
- import android.view.Menu;
- import android.view.MenuItem;
- import android.view.View;
- import android.view.WindowManager;
- import android.widget.TextView;
- import android.widget.Toast;
- import org.webrtc.IceCandidate;
- import org.webrtc.MediaStream;
- import org.webrtc.PeerConnection;
- import org.webrtc.RendererCommon;
- import org.webrtc.SessionDescription;
- import org.webrtc.VideoRenderer;
- import org.webrtc.VideoRendererGui;
- import java.util.Map;
- import fi.vtt.nubomedia.kurentoroomclientandroid.RoomError;
- import fi.vtt.nubomedia.kurentoroomclientandroid.RoomListener;
- import fi.vtt.nubomedia.kurentoroomclientandroid.RoomNotification;
- import fi.vtt.nubomedia.kurentoroomclientandroid.RoomResponse;
- import fi.vtt.nubomedia.webrtcpeerandroid.NBMMediaConfiguration;
- import fi.vtt.nubomedia.webrtcpeerandroid.NBMPeerConnection;
- import fi.vtt.nubomedia.webrtcpeerandroid.NBMWebRTCPeer;
- import fi.vtt.nubotest.util.Constants;
- /**
- * Activity for receiving the video stream of a peer
- * (based on PeerVideoActivity of Pubnub's video chat tutorial example.
- */
- public class PeerVideoActivity extends ListActivity implements NBMWebRTCPeer.Observer, RoomListener {
- private static final String TAG = "PeerVideoActivity";
- private NBMMediaConfiguration peerConnectionParameters;
- private NBMWebRTCPeer nbmWebRTCPeer;
- private SessionDescription localSdp;
- private SessionDescription remoteSdp;
- private String PaticipantID;
- private VideoRenderer.Callbacks localRender;
- private VideoRenderer.Callbacks remoteRender;
- private GLSurfaceView videoView;
- private SharedPreferences mSharedPreferences;
- private int publishVideoRequestId;
- private int sendIceCandidateRequestId;
- private TextView mCallStatus;
- private String username, calluser;
- private boolean backPressed = false;
- private Thread backPressedThread = null;
- private static final int LOCAL_X_CONNECTED = 72;
- private static final int LOCAL_Y_CONNECTED = 72;
- private static final int LOCAL_WIDTH_CONNECTED = 25;
- private static final int LOCAL_HEIGHT_CONNECTED = 25;
- // Remote video screen position
- private static final int REMOTE_X = 0;
- private static int REMOTE_Y = 0;
- private static final int REMOTE_WIDTH = 25;
- private static final int REMOTE_HEIGHT = 25;
- private Handler mHandler;
- private CallState callState;
- private enum CallState{
- IDLE, PUBLISHING, PUBLISHED, WAITING_REMOTE_USER, RECEIVING_REMOTE_USER,PATICIPANT_JOINED,RECEIVING_PATICIPANT,
- }
- @Override
- public void onCreate(Bundle savedInstanceState) {
- super.onCreate(savedInstanceState);
- callState = CallState.IDLE;
- setContentView(R.layout.activity_video_chat);
- getWindow().addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);
- mHandler = new Handler();
- Bundle extras = getIntent().getExtras();
- if (extras == null || !extras.containsKey(Constants.USER_NAME)) {
- ;
- Toast.makeText(this, "Need to pass username to PeerVideoActivity in intent extras (Constants.USER_NAME).",
- Toast.LENGTH_SHORT).show();
- finish();
- return;
- }
- this.username = extras.getString(Constants.USER_NAME, "");
- Log.i(TAG, "username: " + username);
- if (extras.containsKey(Constants.CALL_USER)) {
- this.calluser = extras.getString(Constants.CALL_USER, "");
- Log.i(TAG, "callUser: " + calluser);
- }
- this.mCallStatus = (TextView) findViewById(R.id.call_status);
- TextView prompt = (TextView) findViewById(R.id.receive_prompt);
- prompt.setText("Receive from " + calluser);
- this.videoView = (GLSurfaceView) findViewById(R.id.gl_surface);
- // Set up the List View for chatting
- RendererCommon.ScalingType scalingType = RendererCommon.ScalingType.SCALE_ASPECT_FILL;
- VideoRendererGui.setView(videoView, null);
- localRender = VideoRendererGui.create( LOCAL_X_CONNECTED, LOCAL_Y_CONNECTED,
- LOCAL_WIDTH_CONNECTED, LOCAL_HEIGHT_CONNECTED,
- scalingType, true);
- NBMMediaConfiguration.NBMVideoFormat receiverVideoFormat = new NBMMediaConfiguration.NBMVideoFormat(352, 288, PixelFormat.RGB_888, 20);
- peerConnectionParameters = new NBMMediaConfiguration( NBMMediaConfiguration.NBMRendererType.OPENGLES,
- NBMMediaConfiguration.NBMAudioCodec.OPUS, 0,
- NBMMediaConfiguration.NBMVideoCodec.VP8, 0,
- receiverVideoFormat,
- NBMMediaConfiguration.NBMCameraPosition.FRONT);
- nbmWebRTCPeer = new NBMWebRTCPeer(peerConnectionParameters, this, localRender, this);
- nbmWebRTCPeer.initialize();
- Log.i(TAG, "PeerVideoActivity initialized");
- mHandler.postDelayed(publishDelayed, 4000);
- MainActivity.getKurentoRoomAPIInstance().addObserver(this);
- callState = CallState.PUBLISHING;
- mCallStatus.setText("Publishing...");
- }
- private Runnable publishDelayed = new Runnable() {
- @Override
- public void run() {
- nbmWebRTCPeer.generateOffer("derp", true);
- }
- };
- @Override
- public boolean onCreateOptionsMenu(Menu menu) {
- // Inflate the menu; this adds items to the action bar if it is present.
- getMenuInflater().inflate(R.menu.menu_video_chat, menu);
- return true;
- }
- @Override
- public boolean onOptionsItemSelected(MenuItem item) {
- // Handle action bar item clicks here. The action bar will
- // automatically handle clicks on the Home/Up button, so long
- // as you specify a parent activity in AndroidManifest.xml.
- int id = item.getItemId();
- //noinspection SimplifiableIfStatement
- if (id == R.id.action_settings) {
- return true;
- }
- return super.onOptionsItemSelected(item);
- }
- @Override
- protected void onStart() {
- super.onStart();
- }
- @Override
- protected void onPause() {
- nbmWebRTCPeer.stopLocalMedia();
- super.onPause();
- }
- @Override
- protected void onResume() {
- super.onResume();
- nbmWebRTCPeer.startLocalMedia();
- }
- @Override
- protected void onStop() {
- endCall();
- super.onStop();
- }
- @Override
- protected void onDestroy() {
- super.onDestroy();
- }
- @Override
- public void onBackPressed() {
- // If back button has not been pressed in a while then trigger thread and toast notification
- if (!this.backPressed){
- this.backPressed = true;
- Toast.makeText(this,"Press back again to end.",Toast.LENGTH_SHORT).show();
- this.backPressedThread = new Thread(new Runnable() {
- @Override
- public void run() {
- try {
- Thread.sleep(5000);
- backPressed = false;
- } catch (InterruptedException e){ Log.d("VCA-oBP","Successfully interrupted"); }
- }
- });
- this.backPressedThread.start();
- }
- // If button pressed the second time then call super back pressed
- // (eventually calls onDestroy)
- else {
- if (this.backPressedThread != null)
- this.backPressedThread.interrupt();
- super.onBackPressed();
- }
- }
- public void hangup(View view) {
- finish();
- }
- public void receiveFromRemote(View view){
- Log.e(TAG,"--->receiveFromRemote");
- if (callState == CallState.PUBLISHED){
- callState = CallState.WAITING_REMOTE_USER;
- nbmWebRTCPeer.generateOffer("remote", false);
- runOnUiThread(new Runnable() {
- @Override
- public void run() {
- mCallStatus.setText("Waiting remote stream...");
- }
- });
- }
- }
- /**
- * Terminates the current call and ends activity
- */
- private void endCall() {
- callState = CallState.IDLE;
- try
- {
- if (nbmWebRTCPeer != null) {
- nbmWebRTCPeer.close();
- nbmWebRTCPeer = null;
- }
- }
- catch (Exception e){e.printStackTrace();}
- }
- @Override
- public void onLocalSdpOfferGenerated(final SessionDescription sessionDescription, NBMPeerConnection nbmPeerConnection) {
- Log.e(TAG,"--->onLocalSdpOfferGenerated");
- if (callState == CallState.PUBLISHING || callState == CallState.PUBLISHED) {
- localSdp = sessionDescription;
- Log.e(TAG,"--->onLocalSdpOfferGenerated:publish");
- runOnUiThread(new Runnable() {
- @Override
- public void run() {
- if (MainActivity.getKurentoRoomAPIInstance() != null) {
- Log.d(TAG, "Sending " + sessionDescription.type);
- publishVideoRequestId = ++Constants.id;
- // String sender = calluser + "_webcam";
- // MainActivity.getKurentoRoomAPIInstance().sendReceiveVideoFrom(sender, localSdp.description, publishVideoRequestId);
- MainActivity.getKurentoRoomAPIInstance().sendPublishVideo(localSdp.description, false, publishVideoRequestId);
- }
- }
- });
- } else { // Asking for remote user video
- Log.e(TAG,"--->onLocalSdpOfferGenerated:remote");
- remoteSdp = sessionDescription;
- // nbmWebRTCPeer.selectCameraPosition(NBMMediaConfiguration.NBMCameraPosition.BACK);
- runOnUiThread(new Runnable() {
- @Override
- public void run() {
- if (MainActivity.getKurentoRoomAPIInstance() != null) {
- Log.e(TAG, "Sending--> " +calluser+ sessionDescription.type);
- publishVideoRequestId = ++Constants.id;
- String sender = calluser + "_webcam";
- MainActivity.getKurentoRoomAPIInstance().sendReceiveVideoFrom(sender, remoteSdp.description, publishVideoRequestId);
- }
- }
- });
- }
- }
- @Override
- public void onLocalSdpAnswerGenerated(SessionDescription sessionDescription, NBMPeerConnection nbmPeerConnection) {
- }
- @Override
- public void onIceCandidate(IceCandidate iceCandidate, NBMPeerConnection nbmPeerConnection) {
- Log.e(TAG,"--->onIceCandidate");
- sendIceCandidateRequestId = ++Constants.id;
- if (callState == CallState.PUBLISHING || callState == CallState.PUBLISHED){
- Log.e(TAG,"--->onIceCandidate:publish");
- MainActivity.getKurentoRoomAPIInstance().sendOnIceCandidate(this.username, iceCandidate.sdp,
- iceCandidate.sdpMid, Integer.toString(iceCandidate.sdpMLineIndex), sendIceCandidateRequestId);
- } else{
- Log.e(TAG,"--->onIceCandidate:"+this.calluser);
- MainActivity.getKurentoRoomAPIInstance().sendOnIceCandidate(this.calluser, iceCandidate.sdp,
- iceCandidate.sdpMid, Integer.toString(iceCandidate.sdpMLineIndex), sendIceCandidateRequestId);
- }
- }
- @Override
- public void onIceStatusChanged(PeerConnection.IceConnectionState iceConnectionState, NBMPeerConnection nbmPeerConnection) {
- Log.i(TAG, "onIceStatusChanged");
- }
- @Override
- public void onRemoteStreamAdded(MediaStream mediaStream, NBMPeerConnection nbmPeerConnection) {
- if (callState == CallState.PUBLISHING || callState == CallState.PUBLISHED) {
- Log.e(TAG, "-->onRemoteStreamAdded-->no");
- return;
- }
- Log.e(TAG, "-->onRemoteStreamAdded");
- RendererCommon.ScalingType scalingType = RendererCommon.ScalingType.SCALE_ASPECT_FILL;
- remoteRender = VideoRendererGui.create( REMOTE_X, REMOTE_Y,
- REMOTE_WIDTH, REMOTE_HEIGHT,
- scalingType, false);
- REMOTE_Y = REMOTE_Y+25;
- nbmWebRTCPeer.attachRendererToRemoteStream(remoteRender, mediaStream);
- runOnUiThread(new Runnable() {
- @Override
- public void run() {
- mCallStatus.setText("");
- }
- });
- }
- @Override
- public void onRemoteStreamRemoved(MediaStream mediaStream, NBMPeerConnection nbmPeerConnection) {
- Log.i(TAG, "onRemoteStreamRemoved");
- }
- @Override
- public void onPeerConnectionError(String s) {
- Log.e(TAG, "onPeerConnectionError:" + s);
- }
- @Override
- public void onRoomResponse(RoomResponse response) {
- Log.e(TAG, "-->OnRoomResponse:" + response);
- if (Integer.valueOf(response.getId()) == publishVideoRequestId){
- SessionDescription sd = new SessionDescription(SessionDescription.Type.ANSWER,
- response.getValue("sdpAnswer").get(0));
- if (callState == CallState.PUBLISHING){
- callState = CallState.PUBLISHED;
- nbmWebRTCPeer.processAnswer(sd, "derp");
- } else if (callState == CallState.WAITING_REMOTE_USER){
- callState = CallState.RECEIVING_REMOTE_USER;
- nbmWebRTCPeer.processAnswer(sd, "remote");
- } else if (callState == CallState.PATICIPANT_JOINED){
- callState = CallState.RECEIVING_PATICIPANT;
- nbmWebRTCPeer.processAnswer(sd, this.PaticipantID);
- //NOP
- }
- }
- }
- @Override
- public void onRoomError(RoomError error) {
- Log.e(TAG, "OnRoomError:" + error);
- }
- @Override
- public void onRoomNotification(RoomNotification notification) {
- Log.e(TAG, "OnRoomNotification--> (state=" + callState.toString() + "):" + notification);
- if(notification.getMethod().equals("iceCandidate")) {
- Map<String, Object> map = notification.getParams();
- String sdpMid = map.get("sdpMid").toString();
- int sdpMLineIndex = Integer.valueOf(map.get("sdpMLineIndex").toString());
- String sdp = map.get("candidate").toString();
- IceCandidate ic = new IceCandidate(sdpMid, sdpMLineIndex, sdp);
- Log.e(TAG, "callState-->" + callState);
- if (callState == CallState.PUBLISHING || callState == CallState.PUBLISHED) {
- nbmWebRTCPeer.addRemoteIceCandidate(ic, "derp");
- }else if(callState==CallState.PATICIPANT_JOINED || callState== CallState.RECEIVING_PATICIPANT){
- nbmWebRTCPeer.addRemoteIceCandidate(ic,this.PaticipantID);
- }else {
- nbmWebRTCPeer.addRemoteIceCandidate(ic, "remote");
- }
- }
- if(notification.getMethod().equals("participantPublished"))
- {
- Map<String, Object> map = notification.getParams();
- final String user = map.get("id").toString();
- this.calluser = user;
- this.PaticipantID = "pt_"+this.calluser;
- PeerVideoActivity.this.runOnUiThread(new Runnable() {
- @Override
- public void run() {
- callState = CallState.PATICIPANT_JOINED;
- nbmWebRTCPeer.generateOffer(PaticipantID, false);
- }
- });
- }
- }
- @Override
- public void onRoomConnected() {
- }
- @Override
- public void onRoomDisconnected() {
- }
- }
再就是android room demo中的MainActivity的添加cert的代码要去掉注释,让这段代码生效,就可以连通服务器了。
在iOS的实施方面,上面这家公司也提供了一个工具包:https://github.com/nubomediaTI/Kurento-ios ,工具包里面也有demo
Web方面,最上面官方的哪个demo就足够参考了
后记:很荣幸这篇博客获得了很多CSDN程序员的关注和询问,这只能证明我很荣幸有机会在去年的那个时间点(16年7月)在大家之前处理了一个后续大家都很关注的技术问题,而处理这个问题主要用到的服务器端room server项目和android端nubo test项目,官方在后续好像都做了一定的升级,反而是我自己搞完这个之后,因为产品设计的原因,后来再没有深入地去生产实施这个东西,甚至开发笔记本关于这个项目的源码项目好像都已经删除了,对于大家提出的问题,早期的我还能答一答,后面的我估计你们用到的源码和我用到的源码估计都不是一个版本了,再就是里面的代码细节也基本忘得差不多,在这儿我建议后续开发这个功能可以去深入阅读分析Kurento官方(https://github.com/Kurento)和欧洲媒体服务云服务商nubomedia官方(https://github.com/nubomedia-vtt)的代码示例和文档。我面给出的代码样例是基于nubomedia一对一视聊样例改的,官方原始代码样例在这段时间内都有了变更。在掌握大的基本WebRTC通信的原理的前提下,我觉得改新的代码估计也不会太难。
基于Kurento的WebRTC移动视频群聊技术方案的更多相关文章
- 网易云信技术分享:IM中的万人群聊技术方案实践总结
本文来自网易云信团队的技术分享,原创发表于网易云信公众号,原文链接:mp.weixin.qq.com/s/LT2dASI7QVpcOVxDAsMeVg,收录时有改动. 1.引言 在不了解IM技术的人眼 ...
- android 开发,视频群聊引发短信异常
说到 NDK 开发,其实是为了有些时候为了项目需求需要调用底层的一些 C/C++ 的一些东西:另外就是为了效率更加高些. 但是很多时候能不用就不用:这个是啥原因?个人感觉有些时候是觉得麻烦,首先要配置 ...
- 一套高可用、易伸缩、高并发的IM群聊架构方案设计实践
本文原题为“一套高可用群聊消息系统实现”,由作者“于雨氏”授权整理和发布,内容有些许改动,作者博客地址:alexstocks.github.io.应作者要求,如需转载,请联系作者获得授权. 一.引言 ...
- ASP.NET SignalR 与LayIM配合,轻松实现网站客服聊天室(四) 添加表情、群聊功能
休息了两天,还是决定把这个尾巴给收了.本篇是最后一篇,也算是草草收尾吧.今天要加上表情功能和群聊.基本上就差不多了,其他功能,读者可以自行扩展或者优化.至于我写的代码方面,自己也没去重构.好的,我们开 ...
- 使用java做一个能赚钱的微信群聊机器人(2020年基于PC端协议最新可用版)
前言 微信群机器人,主要用来管理群聊,提供类似天气查询.点歌.机器人聊天等用途. 由于微信将web端的协议封杀后,很多基于http协议的群聊机器人都失效了,所以这里使用基于PC端协议的插件来实现. 声 ...
- 基于itchat的微信群聊小助手基础开发(一)
前段时间由于要管理微信群,基于itchat开发了一个简单的微信机器人 主要功能有: 图灵机器人功能 群聊昵称格式修改提示 消息防撤回功能 斗图功能 要开发一个基于itchat的最基本的聊天机器人,在g ...
- 基于ejabberd简单实现xmpp群聊离线消息
首先,xmpp服务器是基于ejabberd.离线消息模块是mod_interact,原地址地址:https://github.com/adamvduke/mod_interact: 修改后实现群聊离线 ...
- Flask(4)- flask请求上下文源码解读、http聊天室单聊/群聊(基于gevent-websocket)
一.flask请求上下文源码解读 通过上篇源码分析,我们知道了有请求发来的时候就执行了app(Flask的实例化对象)的__call__方法,而__call__方法返回了app的wsgi_app(en ...
- 基于websocket的单聊.群聊
关于ai.baidu.com的 代码: #########################################核心代码################################### ...
随机推荐
- -Dmaven.multiModuleProjectDirectory system propery is not set. Check $M2_HOME environment variable a
eclipse中使用maven插件的时候,运行run as maven build的时候报错 -Dmaven.multiModuleProjectDirectory system propery is ...
- RxVolley使用文档 —— RxVolley = Volley + RxJava + OkHttp
RxVolley使用文档 -- RxVolley = Volley + RxJava + OkHttp 偶然有幸,看到这个框架,便深深的爱上了这个框架,赶紧转载一发到自己的博客上温故而知新,而且作者一 ...
- XML引入以及与html的区别
1.1 引入 HTML: 负责网页的结构 CSS: 负责网页的样式(美观) Javascript: 负责在浏览器端与用户进行交互. 负责静态的网页制作的语言 HTML语言特点: 1)由标签组成. &l ...
- PA 项目关联项目经理
---- 项目关联项目经理 DECLARE p_project_id NUMBER := 155233; l_project_role_id NUMBER := ''; p_employee_id N ...
- 【一天一道LeetCode】#62. Unique Paths
一天一道LeetCode系列 (一)题目 A robot is located at the top-left corner of a m x n grid (marked 'Start' in th ...
- (五十五)iOS多线程之GCD
GCD的全称为Grand Central Dispatch,翻译为大中央调度,是Apple开发的一个多线程编程解决方法. 进程和线程的概念: 正在进行中的程序被称为进程,负责程序运行的内存分配,每一个 ...
- linux的string操作(字符串截取,长度计算)
按指定的字符串截取 1.第一种方法: ${varible##*string} 从左向右截取最后一个string后的字符串 ${varible#*string}从左向右截取第一个string后的字符串 ...
- Java进阶(十八)Java实现定时器(Timer)
Java实现定时器(Timer) 绪 在应用开发中,经常需要一些周期性的操作,比如每5分钟执行某一操作等.对于这样的操作最方便.高效的实现方式就是使用java.util.Timer工具类.java.u ...
- vector的简易实现
vector的简易实现整理自<数据结构与算法分析–C++描述(第3版)>3.4节“向量的实现”.详细可参考<STL源码分析>4.2节. 具体实现代码如下: #ifndef VE ...
- ubuntu 的挂起与休眠
待机 计算机将目前的运行状态等数据存放在内存,关闭硬盘.外设等设备,进入等待状态.此时内存仍然需要电力维持其数据,但整机耗电很少.恢复时计算机从内存读 出数据,回到挂起前的状态,恢复速度较快.一般笔记 ...