RDMA Programming - Base on linux-rdma

首页分类标签留言关于订阅2017-11-08 | 分类 Network  | 标签 RDMA  RoCE  Linux-RDMA

linux-rdma为Linux内核Infiniband子系统drivers/infiniband对应的用户态库,提供了Infiniband Verbs APIRDMA Verbs API.

基本概念

  • Queue Pair(QP)

为了进行RDMA操作,需要在两端建立连接,这通过Queue Pair (QP)来完成,QP相当于socket。通信的两端都需要进行QP的初始化,Communication Manager (CM) 在双方真正建立连接前交换QP信息。

Once a QP is established, the verbs API can be used to perform RDMA reads, RDMA writes, and atomic operations. Serialized send/receive operations, which are similar to socket reads/writes, can be performed as well.

QP对应数据结构struct ibv_qpibv_create_qp用于创建QP.

/**
* ibv_create_qp - Create a queue pair.
*/
struct ibv_qp *ibv_create_qp(struct ibv_pd *pd,
struct ibv_qp_init_attr *qp_init_attr);
  • Completion Queue(CQ)

Completion Queue is an object which contains the completed work requests which were posted to the Work Queues (WQ). Every completion says that a specific WR was completed (both successfully completed WRs and unsuccessfully completed WRs). A Completion Queue is a mechanism to notify the application about information of ended Work Requests (status, opcode, size, source).

对应数据结构struct ibv_cqibv_create_cq用于创建CQ:

/**
* ibv_create_cq - Create a completion queue
* @context - Context CQ will be attached to
* @cqe - Minimum number of entries required for CQ
* @cq_context - Consumer-supplied context returned for completion events
* @channel - Completion channel where completion events will be queued.
* May be NULL if completion events will not be used.
* @comp_vector - Completion vector used to signal completion events.
* Must be >= 0 and < context->num_comp_vectors.
*/
struct ibv_cq *ibv_create_cq(struct ibv_context *context, int cqe,
void *cq_context,
struct ibv_comp_channel *channel,
int comp_vector);
  • Memory Registration (MR)

Memory Registration is a mechanism that allows an application to describe a set of virtually con- tiguous memory locations or a set of physically contiguous memory locations to the network adapter as a virtually contiguous buffer using Virtual Addresses.

对应数据结构struct ibv_mr:

struct ibv_mr {
struct ibv_context *context;
struct ibv_pd *pd;
void *addr;
size_t length;
uint32_t handle;
uint32_t lkey;
uint32_t rkey;
};

Every MR has a remote and a local key (rkey, lkey).

Local keys are used by the local HCA to access local memory, such as during a receive operation.

Remote keys are given to the remote HCA to allow a remote process access to system memory during RDMA operations.

ibv_reg_mr registers a memory region (MR), associates it with a protection domain (PD), and assigns it local and remote keys (lkey, rkey).

/**
* ibv_reg_mr - Register a memory region
*/
struct ibv_mr *ibv_reg_mr(struct ibv_pd *pd, void *addr,
size_t length, int access);
  • Protection Domain (PD)

Object whose components can interact with only each other. These components can be AH, QP, MR, and SRQ.

A protection domain is used to associate Queue Pairs with Memory Regions and Memory Windows , as a means for enabling and controlling network adapter access to Host System memory.

struct ibv_pd is used to implement protection domains:

struct ibv_pd {
struct ibv_context *context;
uint32_t handle;
};

ibv_alloc_pd creates a protection domain (PD). PDs limit which memory regions can be accessed by which queue pairs (QP) providing a degree of protection from unauthorized access.

/**
* ibv_alloc_pd - Allocate a protection domain
*/
struct ibv_pd *ibv_alloc_pd(struct ibv_context *context);
  • Send Request (SR)

An SR defines how much data will be sent, from where, how and, with RDMA, to where. struct ibv_send_wr is used to implement SRs.参考struct ibv_send_wr

示例(IB Verbs API example)

RDMA应用可以使用librdmacm或者libibverbs API编程。前者是对后者的进一步封装。

rc_pingpong是直接使用libibverbs API编程的示例。

一般来说,使用IB Verbs API的基本流程如下:

  • (1) Get the device list

First you must retrieve the list of available IB devices on the local host. Every device in this list contains both a name and a GUID. For example the device names can be: mthca0, mlx4_1.参考这里.

IB devices对应数据结构struct ibv_device:

struct ibv_device {
struct _ibv_device_ops _ops;
enum ibv_node_type node_type;
enum ibv_transport_type transport_type;
/* Name of underlying kernel IB device, eg "mthca0" */
char name[IBV_SYSFS_NAME_MAX];
/* Name of uverbs device, eg "uverbs0" */
char dev_name[IBV_SYSFS_NAME_MAX];
/* Path to infiniband_verbs class device in sysfs */
char dev_path[IBV_SYSFS_PATH_MAX];
/* Path to infiniband class device in sysfs */
char ibdev_path[IBV_SYSFS_PATH_MAX];
};

应用程序通过API ibv_get_device_list获取IB设备列表:

/**
* ibv_get_device_list - Get list of IB devices currently available
* @num_devices: optional. if non-NULL, set to the number of devices
* returned in the array.
*
* Return a NULL-terminated array of IB devices. The array can be
* released with ibv_free_device_list().
*/
struct ibv_device **ibv_get_device_list(int *num_devices);
  • (2) Open the requested device

Iterate over the device list, choose a device according to its GUID or name and open it.参考这里.

应用调用ibv_open_device打开IB设备:

/**
* ibv_open_device - Initialize device for use
*/
struct ibv_context *ibv_open_device(struct ibv_device *device);

返回一个ibv_context对象:

struct ibv_context {
struct ibv_device *device;
struct ibv_context_ops ops;
int cmd_fd;
int async_fd;
int num_comp_vectors;
pthread_mutex_t mutex;
void *abi_compat;
};
  • (3) Allocate a Protection Domain

分配一个PD,参考这里

A Protection Domain (PD) allows the user to restrict which components can interact with only each other.

These components can be AH, QP, MR, MW, and SRQ.

  • (4) Register a memory region

注册一个MR,参考这里.

Any memory buffer which is valid in the process’s virtual space can be registered.

During the registration process the user sets memory permissions and receives local and remote keys (lkey/rkey) which will later be used to refer to this memory buffer.

  • (5) Create a Completion Queue(CQ)

创建一个CQ,参考这里.

A CQ contains completed work requests (WR). Each WR will generate a completion queue entry (CQE) that is placed on the CQ.

The CQE will specify if the WR was completed successfully or not.

  • (6) Create a Queue Pair(QP)

创建QP,参考这里.

Creating a QP will also create an associated send queue and receive queue.

  • (7) Bring up a QP

启动QP,参考这里.

A created QP still cannot be used until it is transitioned through several states, eventually getting to Ready To Send (RTS).

This provides needed information used by the QP to be able send / receive data.

ibv_modify_qp修改QP的状态:

/**
* ibv_modify_qp - Modify a queue pair.
*/
int ibv_modify_qp(struct ibv_qp *qp, struct ibv_qp_attr *attr,
int attr_mask);

例如,对于client/server,需要将QP设置为RTS状态,参考rc_pingpong@pp_connect_ctx.

QP有如下一些状态:

RESET               Newly created, queues empty.
INIT Basic information set. Ready for posting to receive queue.
RTR Ready to Receive. Remote address info set for connected QPs, QP may now receive packets.
RTS Ready to Send. Timeout and retry parameters set, QP may now send packets.
  • (8) Post work requests and poll for completion

Use the created QP for communication operations.

参考pp_post_sendibv_poll_cq.

  • (9) Cleanup
Destroy objects in the reverse order you created them:
Delete QP
Delete CQ
Deregister MR
Deallocate PD
Close device

测试

  • server
# ibv_rc_pingpong -d rxe0 -g 0 -s 128 -r 1 -n 1
local address: LID 0x0000, QPN 0x000011, PSN 0x626753, GID fe80::5054:61ff:fe57:1211
remote address: LID 0x0000, QPN 0x000011, PSN 0x849753, GID fe80::5054:61ff:fe56:1211
256 bytes in 0.00 seconds = 11.38 Mbit/sec
1 iters in 0.00 seconds = 180.00 usec/iter
  • client
# ibv_rc_pingpong -d rxe0 -g 0 172.18.42.162 -s 128 -r 1 -n 1
local address: LID 0x0000, QPN 0x000011, PSN 0x849753, GID fe80::5054:61ff:fe56:1211
remote address: LID 0x0000, QPN 0x000011, PSN 0x626753, GID fe80::5054:61ff:fe57:1211
256 bytes in 0.00 seconds = 16.13 Mbit/sec
1 iters in 0.00 seconds = 127.00 usec/iter

抓包可以查看client与server端的通信流程:

其中,第一个RC Send only为client发送给server的包,参考这里. 然后server回了一个RC Ack,并给client发送了一个RC Send only,参考这里.

前面的一些TCP包为client与server交互的控制信息,参考这里.

Refs

RDMA Programming - Base on linux-rdma的更多相关文章

  1. Git Base For Linux

    GitHub实战系列汇总:http://www.cnblogs.com/dunitian/p/5038719.html Linux安装git,做个记录吧(使用github提供的隐私邮箱) # git官 ...

  2. NVMe over Fabrics又让RDMA技术火了一把

    RDMA是个什么鬼?相信大部分不关心高性能网络的童鞋都不太了解.但是NVMe over Fabrics的出现让搞存储的不得不抽出时间来看看这个东西,这篇文章就来介绍下我所了解的RDMA. RDMA(R ...

  3. [中英对照]Introduction to Remote Direct Memory Access (RDMA) | RDMA概述

    前言: 什么是RDMA? 简单来说,RDMA就是指不通过操作系统(OS)内核以及TCP/IP协议栈在网络上传输数据,因此延迟(latency)非常低,CPU消耗非常少. 下面给出一篇简单介绍RDMA的 ...

  4. [SPDK/NVMe存储技术分析]009 - Introduction to RDMA Send | RDMA Send操作概论

    来源: https://zcopy.wordpress.com/ 说明: 本文不是对原文的逐字逐句翻译,而是摘取核心部分以介绍RDMA Send操作(后面凡是提到RDMA send, 都对应于IBA里 ...

  5. RDMA调研报告&一点随笔

    计算所科研实践随笔 被淹没在论文海里的两个星期. 早上7:10分起床,草草洗漱,7:30出发,开始漫长的1小时通勤.从地铁站的安检口起,队便排的极长,让人看得头皮发麻.下到了轨道旁稍好,但每趟呼啸而来 ...

  6. 基于SoftRoCE 了解RDMA

    RDMA是基于IB技术的内存直接传送,无需内核参与,硬件网卡搞定.IB需要HPC领域的专用硬件,ROCE则是RDMA协议在普通以太网卡的实现,RoCEv1是在MAC上的二层封装,局域网内可以,要通过路 ...

  7. 容器网络启用RDMA高速通讯-Freeflow

    容器网络启用RDMA高速通讯-Freeflow 容器网络启用RDMA高速通讯-Freeflow 本文编译自: Freeflow,https://github.com/openthings/Freefl ...

  8. Revisiting Network Support for RDMA

    重新审视RDMA的网络支持 本文为SIGCOMM 2018会议论文. 笔者翻译了该论文.由于时间仓促,且笔者英文能力有限,错误之处在所难免:欢迎读者批评指正. 本文及翻译版本仅用于学习使用.如果有任何 ...

  9. RDMA的基础概念

    一张图可以简单明确的说明,目前RDMA的几种技术的差别: RDMA是remote Direct memory access的简称,有几个最基本的特点: CPU offload kernel bypas ...

随机推荐

  1. THUPC2019/CTS2019/APIO2019/PKUSC2019游记

    THUPC2019/CTS2019/APIO2019/PKUSC2019游记 5.10 中铺,火车好晃啊 5.11 打了THUPC2019的练习赛,华容道好评(四个小兵,杠鸭!) 5.12 打了THU ...

  2. 腾讯域名使用百度CDN加速配置

    1.百度CDN资源包购买 购买地址 https://console.bce.baidu.com/cdn/#/cdn/package/create 我比较穷所以买的是18块100G的资源包. 2.添加域 ...

  3. django.http.response中HttpResponse 子类

    HttpResponse的子类 Django包含许多处理不同类型的HTTP请求的 HttpResponse 子类.像 HttpResponse 一样,这些类在 django.http 中. HttpR ...

  4. iOS - 数据存储方式(本地化)

    iOS中数据存储方式 一般使用以下4种:(已更新) .NSKeyedAchiever//序列化 存放对象 .NSUserDefaults//本质是plist存储 NSData.NSString.NSN ...

  5. 什么是Familywise Error Rate

    1.什么是Familywise Error Rate(FWE or FWER) 定义:在一系列假设检验中,至少得出一次错误结论的概率. 换句话说,是造成至少一次Type I Error的概率.术语FW ...

  6. npm查看包版本

    点击跳转 ~ 会匹配最近的小版本依赖包,比如~1.2.3会匹配所有1.2.x版本,但是不包括1.3.0 ^ 会匹配最新的大版本依赖包,比如^1.2.3会匹配所有1.x.x的包,包括1.3.0,但是不包 ...

  7. shell 三剑客之 sed pattern 详解

    sed 基础介绍 语法格式 sed 处理过程 sed 选项 cat sed.txt '-p' 打印输出 ,默认输出两次,流输出一次,源文件输出一次 sed 'p' sed.txt -n  只显示处理的 ...

  8. c# 虚属性

  9. 《Python编程:从入门到实践》第二章 变量和简单数据类型 习题答案

    #2.1 print("Hello world!"); #2.2 message="Hello,Python!"; print(message); #2.3 n ...

  10. 爬虫:selenium请求库

    一.介绍 二.安装 三.基本使用 四.选择器 五.等待元素被加载 六.元素交互操作 七.其他 八.项目练习 一.介绍 # selenium最初是一个自动化测试工具,而爬虫中使用它主要是为了解决requ ...