首页/目录

全部文章

八个专题的源码、算法与协议笔记都在这里。

笔记列表

Fast-CDR 源码详细分析

Fast-CDR 源码详细分析

工作区路径:/home/cp/work2/ros2Learn/ros2_humble/src/eProsima/Fast-CDR
版本:1.0.24configure.ac),构建类型 CMake,语言 C++11,许可证 Apache 2.0

eProsima Fast-CDR 是一个轻量级 C++ 序列化库,提供两套 CDR(Common Data Representation)机制:标准 CDRCdr)与 Fast CDRFastCdr)。在 ROS 2 Humble 工作区中,它是 Fast-DDS 的底层依赖,也是 rosidl_typesupport_fastrtps 生成代码所依赖的序列化引擎——所有通过 Fast DDS 传输的 ROS 消息,最终都经由 Fast-CDR 写入/读出字节流。


1. 总体认识

1.1 核心职责

能力 说明
标准 CDR 序列化 遵循 CORBA/DDS 规范,含字节对齐、大小端转换、Encapsulation 头
Fast CDR 序列化 eProsima 自研的简化协议,不做对齐,连续写入,性能更高
缓冲区管理 FastBuffer 提供内部/外部两种内存模式,支持自动扩容
类型覆盖 基本类型、字符串、数组、序列、自定义类型(通过 serialize()/deserialize() 回调)
异常安全 序列化失败时可通过 state 回滚到出错前位置

1.2 在 ROS 2 栈中的位置

ROS 2 应用层类型支持层DDS 中间件网络/共享内存rclcpp Nodestd_msgs / 自定义 msgrosidl_typesupport_fastrtps_cppfastrtps 生成的 serialize/deserializeFast-DDSFast-CDRRTPS 报文 payload
消费者 使用方式
Fast-DDS 直接链接 libfastcdr,序列化/反序列化 RTPS payload
rosidl_typesupport_fastrtps 生成代码调用 eprosima::fastcdr::Cdr<</>>Cdr::alignment()
Fast-DDS-Gen / rosidl 工具链 为 IDL/msg 生成 serialize(Cdr&) / deserialize(Cdr&) 成员函数

注意:ROS 2 默认 RMW 实现(Fast DDS)走 Cdr(标准 CDR) 路径;FastCdr 主要用于 eProsima 内部或对性能更敏感、且双方约定使用 Fast CDR 协议的场景。


2. 目录结构

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
Fast-CDR/
├── include/fastcdr/ # 公开头文件
│ ├── Cdr.h # 标准 CDR(3557 行,核心 API)
│ ├── FastCdr.h # Fast CDR(2104 行)
│ ├── FastBuffer.h # 字节缓冲区 + iterator
│ ├── config.h.in # 构建时生成的配置宏
│ ├── fastcdr_dll.h # DLL 导出 / 自动链接
│ ├── eProsima_auto_link.h # MSVC 自动链接
│ └── exceptions/ # 异常类
│ ├── Exception.h
│ ├── NotEnoughMemoryException.h
│ └── BadParamException.h
├── src/cpp/ # 实现(6 个 .cpp,约 3600 行)
│ ├── Cdr.cpp # 2746 行
│ ├── FastCdr.cpp # 813 行
│ ├── FastBuffer.cpp # 104 行
│ └── exceptions/
├── test/ # gtest 单元测试
│ ├── SimpleTest.cpp # 6931 行,覆盖全部基本类型
│ └── ResizeTest.cpp
├── cmake/ # 构建辅助、打包
├── configure.ac # 版本号 autotools 源
├── CMakeLists.txt # 主构建入口
└── colcon.pkg # colcon 元数据

源码体量很小:约 65 个文件,核心逻辑集中在 3 个头文件 + 3 个实现文件中。


3. FastBuffer — 字节流容器

路径:include/fastcdr/FastBuffer.hsrc/cpp/FastBuffer.cpp

3.1 两种构造模式

构造方式 行为
FastBuffer() 内部分配内存(m_internalBuffer = true),析构时 free()
FastBuffer(char* buf, size_t size) 借用外部缓冲区,不释放

3.2 内存扩容策略

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
bool FastBuffer::resize(
size_t minSizeInc)
{
size_t incBufferSize = BUFFER_START_LENGTH;

if (m_internalBuffer)
{
if (minSizeInc > BUFFER_START_LENGTH)
{
incBufferSize = minSizeInc;
}

if (m_buffer == NULL)
{
m_bufferSize = incBufferSize;

m_buffer = reinterpret_cast<char*>(malloc(m_bufferSize));
// ...
}
else
{
m_bufferSize += incBufferSize;

m_buffer = reinterpret_cast<char*>(realloc(m_buffer, m_bufferSize));
// ...
}
}

return false;
}

要点:

  • 初始分配 200 字节BUFFER_START_LENGTH
  • 每次扩容至少增加 200 字节,或按 minSizeInc 增量
  • 外部缓冲区模式resize() 返回 false,序列化超出容量会抛 NotEnoughMemoryException

3.3 _FastBuffer_iterator — 高效读写迭代器

迭代器封装了 memcpy 读写,避免逐字节循环:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
template<typename _T>
inline
void operator <<(
const _T& data)
{
memcpy(m_currentPosition, &data, sizeof(_T));
}

template<typename _T>
inline
void operator >>(
_T& data)
{
memcpy(&data, m_currentPosition, sizeof(_T));
}

特殊操作符:

  • << iterator:切换底层 buffer 指针,保持相对偏移
  • >> iterator:从另一个 iterator 同步位置索引
  • memcopy / rmemcopy:批量拷贝

4. Cdr — 标准 CDR 序列化

路径:include/fastcdr/Cdr.hsrc/cpp/Cdr.cpp

4.1 关键枚举与配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
typedef enum
{
//! @brief Common CORBA CDR serialization.
CORBA_CDR,
//! @brief DDS CDR serialization.
DDS_CDR
} CdrType;

typedef enum : uint8_t
{
DDS_CDR_WITHOUT_PL = 0x0,
DDS_CDR_WITH_PL = 0x2
} DDSCdrPlFlag;

typedef enum : uint8_t
{
BIG_ENDIANNESS = 0x0,
LITTLE_ENDIANNESS = 0x1
} Endianness;

static const Endianness DEFAULT_ENDIAN;
  • CORBA_CDR:经典 CORBA CDR,无 Encapsulation 头
  • DDS_CDR:DDS 扩展,含 dummy byte、encapsulation kind、options
  • 默认字节序由编译目标决定(FASTCDR_IS_BIG_ENDIAN_TARGET

4.2 Encapsulation(封装头)

DDS 消息在 payload 开头写入/读取 encapsulation,用于声明字节序和 Parameter List 标志:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Cdr& Cdr::read_encapsulation()
{
// DDS_CDR: 先读 dummy byte(必须为 0)
// 再读 encapsulationKind(低 1 位 = 字节序,bit1 = PL 标志)
// 若字节序与当前不同 → 切换 m_swapBytes
// DDS_CDR: 再读 m_options (uint16)
resetAlignment();
return *this;
}

Cdr& Cdr::serialize_encapsulation()
{
// DDS_CDR: 写 dummy=0
// 写 encapsulationKind = m_plFlag | m_endianness
// DDS_CDR: 写 m_options
resetAlignment();
return *this;
}

Encapsulation 字节布局(DDS_CDR):

1
2
3
[ dummy:1B=0 ] [ encapsulationKind:1B ] [ options:2B ]
├ bit0: endianness
└ bit1: parameter list flag

4.3 字节对齐机制

标准 CDR 的核心特征:多字节类型必须按自身大小对齐

静态对齐计算(供 rosidl 生成代码预估序列化大小):

1
2
3
4
5
6
inline static size_t alignment(
size_t current_alignment,
size_t dataSize)
{
return (dataSize - (current_alignment % dataSize)) & (dataSize - 1);
}

实例对齐(序列化过程中):

1
2
3
4
5
6
7
8
9
10
11
12
13
inline size_t alignment(
size_t dataSize) const
{
return dataSize >
m_lastDataSize ? (dataSize - ((m_currentPosition - m_alignPosition) % dataSize)) &
(dataSize - 1) : 0;
}

inline void makeAlign(
size_t align)
{
m_currentPosition += align;
}

优化:若当前要序列化的类型大小 ≤ 上一个类型大小(m_lastDataSize),则无需额外对齐字节——这是 CDR 规范中的 packed 优化。

int16_t 序列化示例:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
Cdr& Cdr::serialize(
const int16_t short_t)
{
size_t align = alignment(sizeof(short_t));
size_t sizeAligned = sizeof(short_t) + align;

if (((m_lastPosition - m_currentPosition) >= sizeAligned) || resize(sizeAligned))
{
m_lastDataSize = sizeof(short_t);
makeAlign(align);

if (m_swapBytes)
{
const char* dst = reinterpret_cast<const char*>(&short_t);
m_currentPosition++ << dst[1];
m_currentPosition++ << dst[0];
}
else
{
m_currentPosition << short_t;
m_currentPosition += sizeof(short_t);
}
return *this;
}
throw NotEnoughMemoryException(...);
}

4.4 大小端转换

  • 构造时:m_swapBytes = (endianness != DEFAULT_ENDIAN)
  • 读 encapsulation 时可能动态切换
  • 多字节类型在 m_swapBytes == true 时逐字节反转写入/读出
  • 提供带 Endianness 参数的 serialize(T, Endianness) 重载,临时切换字节序

4.5 字符串编码

类型 序列化格式
const char* / std::string uint32 length(含 \0)+ 字符数据
const wchar_t* / std::wstring uint32 char_count + 每字符 4 字节(Windows 逐字符,Linux 批量 memcopy)
bool 1 字节:01
long double 对齐到 8 字节边界,占 16 字节(8 字节平台前 8 字节填 0)

4.6 序列 / 数组

  • std::vector<T>:先写 int32 长度,再写元素数组
  • 序列化失败时通过 state 回滚(保证原子性)
  • std::vector<bool> 有特殊模板特化(MSVC / 非 MSVC 分支不同)

4.7 自定义类型

非基本类型通过模板调用对象的成员函数:

1
2
3
4
5
6
template<class _T>
inline Cdr& operator <<(const _T& type_t)
{
type_t.serialize(*this);
return *this;
}

rosidl / Fast-DDS-Gen 为每个 struct 生成 void serialize(eprosima::fastcdr::Cdr&) constvoid deserialize(eprosima::fastcdr::Cdr&)

4.8 state — 序列化快照

Cdr::state 保存四个字段,用于出错回滚或嵌套序列化:

字段 含义
m_currentPosition 当前读写位置
m_alignPosition 对齐基准位置
m_swapBytes 是否字节交换
m_lastDataSize 上次序列化类型大小

5. FastCdr — 无对齐快速序列化

路径:include/fastcdr/FastCdr.hsrc/cpp/FastCdr.cpp

5.1 与 Cdr 的核心差异

特性 Cdr FastCdr
字节对齐 ✅ 按 CDR 规范 ❌ 无对齐,紧凑排列
Encapsulation ✅ CORBA/DDS ❌ 无
大小端 ✅ 支持切换 ❌ 不做字节交换
数组批量写入 对齐后逐元素或 memcopy 直接 memcopy 整块
state 字段 4 个 1 个(仅 position)
典型用途 DDS/ROS 2 互操作 eProsima 内部高性能场景

5.2 基本类型序列化

int16_t 为例——无对齐,直接 memcpy

1
2
3
4
5
6
7
8
9
10
11
FastCdr& serialize(
const int16_t short_t)
{
if (((m_lastPosition - m_currentPosition) >= sizeof(short_t)) || resize(sizeof(short_t)))
{
m_currentPosition << short_t;
m_currentPosition += sizeof(short_t);
return *this;
}
throw exception::NotEnoughMemoryException(...);
}

数组类型直接批量拷贝:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
FastCdr& FastCdr::serializeArray(
const int16_t* short_t,
size_t numElements)
{
size_t totalSize = sizeof(*short_t) * numElements;

if (((m_lastPosition - m_currentPosition) >= totalSize) || resize(totalSize))
{
m_currentPosition.memcopy(short_t, totalSize);
m_currentPosition += totalSize;
return *this;
}
throw NotEnoughMemoryException(...);
}

5.3 long double 平台差异

FastCdr 对 long double 做了大量平台分支:

  • 16 字节平台:直接 memcopy
  • 8 字节平台:写 16 字节(前 8 字节填 0,后 8 字节为值)
  • 支持 __float128:转换为 128 位浮点再写入

这与 Cdr 的处理逻辑一致,保证与 DDS XTypes 128-bit float 布局兼容。


6. 异常体系

路径:include/fastcdr/exceptions/src/cpp/exceptions/

classDiagram
    class exception {
        <<std::exception>>
    }
    class Exception {
        +raise()*
        +what() const
        -m_message
    }
    class NotEnoughMemoryException {
        缓冲区空间不足
    }
    class BadParamException {
        非法参数/格式
    }
    exception <|-- Exception
    Exception <|-- NotEnoughMemoryException
    Exception <|-- BadParamException
异常 触发场景
NotEnoughMemoryException 缓冲区剩余空间不足且无法 resize
BadParamException encapsulation 格式错误、bool 非法值、空指针等

设计特点:

  • 继承 std::exception,同时提供 raise() 用于 re-throw(配合 state 回滚)
  • 序列化复合类型(vector、sequence)在 catch 块中 setState(state_before_error)ex.raise()

7. 构建与配置

7.1 CMake 要点

  • 产物:libfastcdr.so(默认 shared)
  • 版本:从 configure.ac 读取 → 1.0.24
  • 编译特性检测:check_endianness()check_type_sizes()check_stdcxx()
  • 生成 config.h:C++11 支持、大小端、FASTCDR_SIZEOF_LONG_DOUBLE

7.2 config.h 关键宏

1
2
3
4
5
6
#define FASTCDR_VERSION_MAJOR @PROJECT_VERSION_MAJOR@
#define FASTCDR_VERSION_MINOR @PROJECT_VERSION_MINOR@
#define FASTCDR_VERSION_MICRO @PROJECT_VERSION_PATCH@
#define FASTCDR_IS_BIG_ENDIAN_TARGET @FASTCDR_IS_BIG_ENDIAN_TARGET@
#define FASTCDR_HAVE_FLOAT128 @FASTCDR_HAVE_FLOAT128@
#define FASTCDR_SIZEOF_LONG_DOUBLE @FASTCDR_SIZEOF_LONG_DOUBLE@

7.3 colcon 集成

1
2
3
4
5
{
"name": "fastcdr",
"type": "cmake",
"dependencies": ["googletest-distribution"]
}

在 ROS 2 工作区中,fastcdr 作为 Fast-DDS 的前置依赖 被 colcon 自动构建。

7.4 DLL 导出

fastcdr_dll.h 定义 Cdr_DllAPI 宏:

  • Windows 动态库:__declspec(dllexport/dllimport)
  • Linux:空宏(符号默认可见)
  • 支持 EPROSIMA_ALL_DYN_LINK / FASTCDR_DYN_LINK 全局开关

8. 测试

8.1 SimpleTest.cpp

6931 行,使用 gtest,覆盖:

  • 全部基本类型的 serialize/deserialize 往返
  • CORBA_CDR 与 DDS_CDR 两种模式
  • 大端 / 小端
  • 数组、vector、嵌套 array
  • encapsulation 读写
  • state 回滚
  • 空字符串、边界值

8.2 ResizeTest.cpp

专门测试 FastBuffer 内部缓冲区的自动扩容行为。


9. ROS 2 集成详解

9.1 rosidl_typesupport_fastrtps 生成代码

rosidl_typesupport_fastrtps_cpp 为每个 msg 生成:

  1. get_serialized_size() — 调用 Cdr::alignment() 静态方法预估大小
  2. serialize() — 创建 FastBuffer + Cdr,写 encapsulation,逐字段 <<
  3. deserialize() — 读 encapsulation,逐字段 >>

生成模板片段(msg__type_support.cpp.em):

1
2
3
4
eprosima::fastcdr::Cdr::alignment(current_alignment, sizeof(uint32_t));
// ... 每个字段累加对齐与大小
void serialize(eprosima::fastcdr::Cdr & cdr) { /* scdr << field */ }
void deserialize(eprosima::fastcdr::Cdr & cdr) { /* dcdr >> field */ }

9.2 典型序列化流程(Fast DDS RMW)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
ROS msg 对象


rosidl_typesupport_fastrtps_cpp::serialize()

├─ FastBuffer(buffer, size) 或 FastBuffer() + reserve
├─ Cdr scdr(buffer, DEFAULT_ENDIAN, DDS_CDR)
├─ scdr.serialize_encapsulation()
├─ msg.serialize(scdr) // 生成的代码


Fast-DDS 将 buffer 作为 Sample payload 发送


对端 Cdr dcdr(buffer, ..., DDS_CDR)
├─ dcdr.read_encapsulation()
└─ msg.deserialize(dcdr)

9.3 与 CycloneDDS 的对比

中间件 序列化库 CDR 实现
Fast DDS Fast-CDR eprosima::fastcdr::Cdr
CycloneDDS 内置 CDR ddsc 内部实现,不依赖 Fast-CDR

两者 payload 格式均遵循 OMG CDR 规范,但实现独立;同一 msg 在两种 RMW 间二进制 payload 通常兼容(相同 encapsulation + 对齐规则)。


10. Cdr 与 FastCdr 选型

需要序列化?需要与标准 DDS/CORBA 互操作?Cdr + DDS_CDR/CORBA_CDR双方都是 eProsima 且约定 Fast CDR?FastCdrDDS 消息?serialize_encapsulation / read_encapsulation直接写字段

11. 设计特点小结

特点 说明
双引擎 标准兼容(Cdr)+ 性能优化(FastCdr)
Header-heavy 大量 inline 模板在头文件,.cpp 只实现复杂逻辑
零拷贝倾向 iterator + memcopy 批量操作;外部 buffer 模式避免内存复制
异常 + state 回滚 复合类型序列化失败时不留半成品
跨平台 long double 8/16 字节平台分支 + __float128 支持
ROS 2 QL1 声明 Quality Level 1(见 QUALITY.md
轻量 核心库仅 ~3600 行实现代码

12. 推荐阅读顺序

  1. 缓冲区基础FastBuffer.hFastBuffer.cpp — 理解内存模式与 iterator
  2. 标准 CDR 核心Cdr.cppserialize(int16_t)read_encapsulation() — 对齐与 encapsulation
  3. Fast CDR 对比FastCdr.h 中同名 serialize(int16_t) — 体会无对齐差异
  4. ROS 2 生成代码rosidl_typesupport_fastrtps_cpp/resource/msg__type_support.cpp.em
  5. Fast-DDS 使用点Fast-DDS/src/cpp/fastdds/dds/ 中搜索 fastcdr::Cdr
  6. 测试验证test/SimpleTest.cpp 前 200 行 — 了解 API 用法模式
  7. 配置宏:构建后查看 build/fastcdr/include/fastcdr/config.h

13. API 速查

13.1 基本用法(Cdr)

1
2
3
4
5
6
7
8
9
10
11
12
#include <fastcdr/Cdr.h>
#include <fastcdr/FastBuffer.h>

char buffer[512];
eprosima::fastcdr::FastBuffer fastbuffer(buffer, sizeof(buffer));
eprosima::fastcdr::Cdr ser(fastbuffer, eprosima::fastcdr::Cdr::DEFAULT_ENDIAN,
eprosima::fastcdr::Cdr::DDS_CDR);

ser.serialize_encapsulation();
ser << my_msg; // 或 my_msg.serialize(ser)

size_t len = ser.getSerializedDataLength();

13.2 基本用法(FastCdr)

1
2
3
4
5
eprosima::fastcdr::FastBuffer fastbuffer;
eprosima::fastcdr::FastCdr ser(fastbuffer);

ser << int32_t{42} << std::string{"hello"};
// 无 encapsulation,无对齐

13.3 关键公开方法

方法 用途
FastBuffer getBuffer(), reserve(), resize() 缓冲区访问与扩容
Cdr serialize_encapsulation(), read_encapsulation() DDS 封装头
Cdr alignment(), resetAlignment() 对齐计算与控制
Cdr getState(), setState() 快照/回滚
Cdr changeEndianness() 动态切换字节序
Cdr/FastCdr getSerializedDataLength() 已序列化字节数
Cdr/FastCdr operator<< / operator>> 流式序列化/反序列化

文档基于 ROS 2 Humble 工作区中的 Fast-CDR 1.0.24 源码分析生成。

Fast-DDS 源码详细分析

Fast-DDS 源码详细分析

工作区路径:/home/cp/work2/ros2Learn/ros2_humble/src/eProsima/Fast-DDS
版本:2.6.11CMakeLists.txt / package.xml),CMake 工程名 fastrtps(历史命名),语言 C++11+,许可证 Apache 2.0

eProsima Fast-DDS(原 Fast RTPS)是 OMG DDS 规范的 C++ 实现,底层协议为 RTPS(Real-Time Publish-Subscribe)。在 ROS 2 Humble 中,它是默认 RMW 实现 rmw_fastrtps_cpp 所依赖的中间件;所有 ros2 topic pub/echo、节点间通信最终都经由 Fast-DDS 的 RTPS 栈在网络上收发数据。


1. 总体认识

1.1 核心职责

能力 说明
DDS API Participant / Publisher / Subscriber / DataWriter / DataReader / Topic / QoS
RTPS 协议栈 SPDP/SEDP 发现、Reader/Writer 状态机、Heartbeat/AckNack/Gap/Data 子消息
传输层 UDP、TCP(可选 TLS)、共享内存(SHM)、链式传输
序列化 依赖 Fast-CDR 做 CDR payload 编码
XTypes 动态类型、TypeLookup、TypeObject
安全(可选) DDS Security:PKI-DH 认证、Permissions、AES-GCM-GMAC 加密
零拷贝 DataSharing(进程内共享内存 payload pool)

1.2 在 ROS 2 栈中的位置

ROS 2 应用RMW 层类型支持Fast-DDSrclcpp Noderclrmw_fastrtps_cpprmw_fastrtps_shared_cpprosidl_typesupport_fastrtps_cppFast-CDRfastdds::dds APIfastrtps::rtps 协议栈Transport UDP/SHM/TCP
组件 关系
Fast-CDR 序列化引擎(见 Fast-CDR 源码详细分析
foonathan_memory 自定义内存分配器,减少 RTPS 热路径堆分配
rmw_fastrtps_shared_cpp 封装 DomainParticipantDataWriterDataReader
rosidl_typesupport_fastrtps 为每个 msg 生成 TopicDataType + CDR serialize/deserialize

ROS 2 中 Publisher/Subscription 对应 DDS 的 DataWriter/DataReader,而非 DDS Publisher/Subscriber。RMW 为每个 Participant 创建一个 DDS Publisher 和一个 DDS Subscriber 作为容器。


2. 三层 API 架构

Fast-DDS 在同一个 libfastrtps中并存三套 API:

层级 命名空间 头文件路径 实现目录 状态
现代 DDS API eprosima::fastdds::dds include/fastdds/dds/ src/cpp/fastdds/ 主路径 / ROS 2 使用
RTPS 直连 API eprosima::fastrtps::rtps include/fastdds/rtps/ src/cpp/rtps/ 底层协议访问
Legacy Fast RTPS eprosima::fastrtps include/fastrtps/ src/cpp/fastrtps_deprecated/ 已弃用
OMG PSM 包装 dds:: include/dds/ src/cpp/dds/ ISO C++ DDS 薄封装

CMake 工程名仍为 fastrtps,ROS package 名也是 fastrtps,与产品名 Fast-DDS 并存——这是历史兼容设计。

2.1 分层调用关系

DomainParticipantDomainParticipantImplRTPSParticipant / RTPSParticipantImplBuiltinProtocolsNetworkFactoryMessageReceiverPDP 参与者发现EDP 端点发现WLP 存活协议

3. 目录结构

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
Fast-DDS/
├── include/
│ ├── fastdds/dds/ # 现代 DDS 公开 API(221 个头文件)
│ ├── fastdds/rtps/ # RTPS 公开 API
│ ├── fastrtps/ # Legacy API(201 个头文件)
│ └── dds/ # OMG PSM 包装(119 个头文件)
├── src/cpp/ # 全部实现(478 源文件,~20 万行)
│ ├── fastdds/ # DDS 层(~29K 行)
│ ├── rtps/ # RTPS 核心(~101K 行)★ 最大模块
│ ├── dynamic-types/ # XTypes 动态类型(~35K 行)
│ ├── security/ # 安全插件(~12K 行)
│ ├── statistics/ # 统计模块(可选,~14K 行)
│ ├── fastrtps_deprecated/ # Legacy 实现(~5K 行)
│ ├── dds/ # PSM 包装(~1.3K 行)
│ └── utils/ # 公共工具
├── test/ # unittest / blackbox / performance
├── tools/ # fastdds CLI、discovery server
├── examples/ # C++ 示例
├── cmake/ # 构建与打包
├── thirdparty/ # taocpp-pegtl 等
├── CMakeLists.txt
└── package.xml # ROS 包名 fastrtps

3.1 src/cpp/rtps/ 子模块

子目录 文件数 职责
builtin/ 60 PDP/EDP 发现、WLP、Discovery Server 数据库
transport/ 55 UDP/TCP/SHM 传输实现
history/ 25 CacheChange、Reader/Writer History、Payload Pool
messages/ 12 RTPS 报文组装/解析、子消息
DataSharing/ 11 进程内零拷贝共享内存
reader/ / writer/ 各 10 Stateful/Stateless Reader/Writer
participant/ RTPSParticipantImpl
security/ SecurityManager 框架
network/ NetworkFactory、收发资源
persistence/ SQLite3 持久化 Writer/Reader

4. DDS 层 — 现代 API

路径:include/fastdds/dds/src/cpp/fastdds/

4.1 核心实体

实体 公开头文件 实现
DomainParticipantFactory dds/domain/DomainParticipantFactory.hpp domain/DomainParticipantFactory.cpp
DomainParticipant dds/domain/DomainParticipant.hpp domain/DomainParticipant.cpp + DomainParticipantImpl.cpp
Publisher dds/publisher/Publisher.hpp publisher/PublisherImpl.cpp
DataWriter dds/publisher/DataWriter.hpp publisher/DataWriterImpl.cpp + DataWriterHistory.cpp
Subscriber dds/subscriber/Subscriber.hpp subscriber/SubscriberImpl.cpp
DataReader dds/subscriber/DataReader.hpp subscriber/DataReaderImpl.cpp + history/DataReaderHistory.cpp
Topic dds/topic/Topic.hpp topic/TopicImpl.cpp
TypeSupport dds/topic/TypeSupport.hpp topic/TypeSupport.cpp

设计模式:公开类(Entity 子类)+ Impl 类。公开类仅转发调用,Impl 持有 RTPS 对象和业务逻辑。

4.2 Participant 启用流程

DomainParticipantImpl::enable() 是 DDS 与 RTPS 的衔接点:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
ReturnCode_t DomainParticipantImpl::enable()
{
assert(rtps_participant_ == nullptr);

fastrtps::rtps::RTPSParticipantAttributes rtps_attr;
set_attributes_from_qos(rtps_attr, qos_);
rtps_attr.participantID = participant_id_;

RTPSParticipant* part = RTPSDomain::clientServerEnvironmentCreationOverride(...);

if (part == nullptr)
{
part = RTPSDomain::createParticipant(domain_id_, false, rtps_attr, &rtps_listener_);
// ...
}

guid_ = part->getGuid();
rtps_participant_ = part;

// autoenable: 依次 enable Topic → Publisher → Subscriber
if (qos_.entity_factory().autoenable_created_entities) { /* ... */ }

rtps_participant_->enable();

return ReturnCode_t::RETCODE_OK;
}

流程概要:

  1. DomainParticipantQos 映射出 RTPSParticipantAttributes
  2. 调用 RTPSDomain::createParticipant() 创建底层 RTPS 参与者
  3. autoenable_created_entities,依次 enable 已创建的 Topic/Publisher/Subscriber
  4. 调用 rtps_participant_->enable() 启动发现与传输

4.3 DataWriter 写数据路径

1
2
3
4
5
6
7
8
9
10
11
bool DataWriterImpl::write(
void* data)
{
if (writer_ == nullptr)
{
return false;
}

logInfo(DATA_WRITER, "Writing new data");
return ReturnCode_t::RETCODE_OK == create_new_change(ALIVE, data);
}

完整链路:

1
2
3
4
5
6
7
8
DataWriter::write(data)
→ DataWriterImpl::create_new_change()
→ TypeSupport::serialize() // Fast-CDR
→ WriterHistory 添加 CacheChange
→ RTPSWriter (StatefulWriter/StatelessWriter)
→ FlowController(流控,可选)
→ RTPSMessageGroup 组装 Data 子消息
→ NetworkFactory → Transport 发送

4.4 DataReader 读数据路径

1
2
3
4
5
6
7
Transport 接收 UDP/SHM 报文
→ MessageReceiver 解析 RTPS 子消息
→ StatefulReader / StatelessReader
→ ReaderHistory 存入 CacheChange
→ DataReaderImpl::read/take()
→ TypeSupport::deserialize() // Fast-CDR
→ 返回给应用 / RMW

4.5 其他 DDS 能力

模块 路径 说明
ContentFilteredTopic topic/ContentFilteredTopic*.cpp 内容过滤主题
DDSSQLFilter topic/DDSSQLFilter/ SQL-like 过滤表达式解析(pegtl)
TypeLookup fastdds/builtin/typelookup/ XTypes 远程类型查询
WaitSet / Condition fastdds/core/condition/ 同步等待机制
Log fastdds/log/ 可配置日志消费者

5. RTPS 层 — 协议核心

路径:include/fastdds/rtps/src/cpp/rtps/

5.1 RTPSParticipant

组件 路径
公开 API include/fastdds/rtps/participant/RTPSParticipant.h
实现 src/cpp/rtps/participant/RTPSParticipantImpl.cpp/.h
工厂 include/fastdds/rtps/RTPSDomain.hsrc/cpp/rtps/RTPSDomain.cpp

RTPSParticipantImpl 是 RTPS 层的中心对象,持有:

  • BuiltinProtocols — 发现与存活
  • NetworkFactory — 传输注册与 locator 选择
  • MessageReceiver — 入站报文分发
  • SecurityManager(可选)— 安全插件编排
  • Reader/Writer 集合

5.2 发现协议(BuiltinProtocols)

路径:src/cpp/rtps/builtin/BuiltinProtocols.cpp

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
bool BuiltinProtocols::initBuiltinProtocols(
RTPSParticipantImpl* p_part,
BuiltinAttributes& attributes)
{
// PDP — 按 discoveryProtocol 选择实现
switch (m_att.discovery_config.discoveryProtocol)
{
case DiscoveryProtocol_t::SIMPLE:
mp_PDP = new PDPSimple(this, allocation);
break;
case DiscoveryProtocol_t::CLIENT:
mp_PDP = new fastdds::rtps::PDPClient(this, allocation);
break;
case DiscoveryProtocol_t::SERVER:
mp_PDP = new fastdds::rtps::PDPServer(this, allocation, ...);
break;
// ...
}

mp_PDP->init(mp_participantImpl);

// WLP — Writer Liveliness Protocol
if (m_att.use_WriterLivelinessProtocol)
{
mp_WLP = new WLP(this);
mp_WLP->initWL(mp_participantImpl);
}

// TypeLookupManager
if (m_att.typelookup_config.use_client || m_att.typelookup_config.use_server)
{
tlm_ = new fastdds::dds::builtin::TypeLookupManager(this);
tlm_->init_typelookup_service(mp_participantImpl);
}

return true;
}

PDP — Participant Discovery Protocol

模式 用途
SIMPLE PDPSimple 默认:Multicast SPDP + 单播互发现
CLIENT PDPClient Discovery Server 客户端
SERVER PDPServer Discovery Server 服务端
SUPER_CLIENT PDPClient(..., true) 超级客户端
BACKUP PDPServer(..., TRANSIENT) 持久化 Discovery Server(需 SQLite3)

SPDP 通过内置 Writer/Reader 交换 ParticipantProxyData(GUID、locator、UserData 等)。

EDP — Endpoint Discovery Protocol

实现 路径 说明
EDPSimple rtps/builtin/discovery/endpoint/EDPSimple.cpp 动态端点发现(默认)
EDPStatic EDPStatic.cpp 静态端点配置(XML)
EDPClient/Server 配合 Discovery Server 中心化端点信息

EDP 在发现远端 Writer/Reader 的 WriterProxyData/ReaderProxyData 后,按 Topic 名 + QoS 兼容性做匹配,调用 matched_writer_add / matched_reader_add 建立通信关系:

1
2
3
4
5
6
7
publications_reader_.first->matched_writer_add(*temp_writer_proxy_data);
// ...
publications_writer_.first->matched_reader_add(*temp_reader_proxy_data);
// ...
subscriptions_reader_.first->matched_writer_add(*temp_writer_proxy_data);
// ...
subscriptions_writer_.first->matched_reader_add(*temp_reader_proxy_data);

WLP — Writer Liveliness Protocol

管理 DataWriter 存活断言(Automatic / Manual-by-Participant / Manual-by-Topic),超时后通知 DataReader。

5.3 Reader / Writer 类型

类型 可靠性 典型场景
StatelessWriter Best-Effort 不需 ACK 的发送
StatefulWriter Reliable 需 Heartbeat/AckNack 确认
StatelessReader Best-Effort 接收 BE 数据
StatefulReader Reliable 接收 REL 数据,回复 AckNack

StatefulWriter(~2100 行)维护 ReaderProxy 列表,负责:

  • 向每个 matched reader 发送 Heartbeat
  • 处理 AckNack,重传丢失的 CacheChange
  • 发送 Gap 通知不可用序列号
  • 与 FlowController 协作限流

5.4 History 与 CacheChange

路径:include/fastdds/rtps/common/CacheChange.hsrc/cpp/rtps/history/

1
2
3
4
5
6
7
8
9
10
CacheChange
├── writerGUID / sequenceNumber
├── serializedPayload(Fast-CDR 编码后的字节)
├── instanceHandle
└── kind(ALIVE / NOT_ALIVE_DISPOSED 等)

WriterHistory — _writer 侧变更缓存(发送队列)
ReaderHistory — _reader 侧变更缓存(接收队列)
CacheChangePool — 对象池,减少分配
TopicPayloadPool — payload 内存池(可共享)

5.5 报文处理

组件 路径 职责
MessageReceiver rtps/messages/MessageReceiver.cpp (~1500 行) 解析入站 RTPS 报文,分发 Data/Heartbeat/AckNack/Gap 子消息
RTPSMessageGroup RTPSMessageGroup.cpp 组装出站 RTPS 报文
RTPSMessageCreator RTPSMessageCreator.cpp 创建 RTPS 报文头
子消息模板 messages/submessages/ DataMsg、HeartbeatMsg、AckNackMsg、GapMsg

MessageReceiver 构造时根据是否启用安全选择不同的处理函数:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
MessageReceiver::MessageReceiver(
RTPSParticipantImpl* participant,
uint32_t rec_buffer_size)
: participant_(participant)
// ...
#if HAVE_SECURITY
, crypto_msg_(participant->is_secure() ? rec_buffer_size : 0)
// ...
#endif
{
#if HAVE_SECURITY
if (participant->is_secure())
{
process_data_message_function_ = std::bind(
&MessageReceiver::process_data_message_with_security, ...);
}
#endif
}

6. 传输层

路径:src/cpp/rtps/transport/include/fastdds/rtps/transport/

NetworkFactoryrtps/network/NetworkFactory.cpp)根据 RTPSParticipantAttributes 注册传输、选择 locator、创建收发资源。

传输 Descriptor 头文件 实现
UDPv4/v6 UDPv4TransportDescriptor.h UDPv4Transport.cppUDPTransportInterface.cpp
TCPv4/v6 TCPv4TransportDescriptor.h TCPv4Transport.cppTCPTransportInterface.cpp
TLS TCPChannelResourceSecure.cppSECURITY + OpenSSL)
Shared Memory shared_mem/SharedMemTransportDescriptor.h shared_mem/SharedMemTransport.cpp
Chaining ChainingTransportDescriptor.h 包装其他传输,做过滤/统计

6.1 默认传输配置

ROS 2 Humble 中 RMW 默认配置 UDP + SHM

  • UDP:跨进程/跨主机通信
  • SHM:同主机进程间高效通信(SHM_TRANSPORT_DEFAULT=ON

SHM 实现基于 MultiProducerConsumerRingBuffer 无锁环形缓冲区。

6.2 DataSharing(零拷贝)

路径:src/cpp/rtps/DataSharing/

与 SHM 传输不同,DataSharing 是 payload 级别的共享内存池

  • Writer 将 serialized payload 写入共享 segment
  • Reader 直接读取,避免 memcpy
  • 适用于同进程或配置了 DataSharing 的 Writer/Reader 对

7. 序列化与类型系统

7.1 Fast-CDR 集成

  • DDS 层通过 TopicDataType 接口定义 serialize() / deserialize() / getSerializedSizeProvider()
  • ROS 2 的 rosidl_typesupport_fastrtps_cpp 生成这些方法的实现,内部使用 eprosima::fastcdr::Cdr
  • RTPS 层只处理 SerializedPayload_t(原始字节 + length)

7.2 动态类型(XTypes)

路径:src/cpp/dynamic-types/include/fastrtps/types/

说明
DynamicTypeBuilderFactory 运行时构建类型
DynamicData 运行时数据实例
DynamicPubSubType 动态类型的 TopicDataType
TypeObjectFactory TypeObject 注册与查询
TypeLookupManager 远程类型发现服务

8. 安全模块(可选)

构建选项:option(SECURITY "Activate security" OFF),需 OpenSSL。

8.1 两层结构

  1. RTPS 框架src/cpp/rtps/security/

    • SecurityManager.cpp(~4300 行)— 编排认证、授权、加密
    • 插件接口:Authentication.hAccessControl.hCryptography.h
  2. 内置插件src/cpp/security/

子模块 实现 功能
authentication/ PKIDH.cpp PKI-DH 参与者认证
accesscontrol/ Permissions.cpp Governance/Permissions XML 解析与校验
cryptography/ AESGCMGMAC*.cpp AES-GCM 加密 + GMAC 完整性
artifact_providers/ FileProvider.cpp 证书/密钥文件加载

ROS 2 安全启用时,RMW 通过 rmw_dds_common 配置 Security 属性,Fast-DDS 加载对应插件。


9. 其他模块

9.1 Statistics(可选)

option(FASTDDS_STATISTICS "Enable Fast DDS Statistics Module" OFF)

路径:src/cpp/statistics/

提供 DDS 级统计 topic(如网络流量、延迟、丢包),通过 hook RTPS 层 StatisticsBase 收集数据。

9.2 持久化

option(SQLITE3_SUPPORT "Activate SQLITE3 support" ON)

路径:src/cpp/rtps/persistence/

  • SQLite3PersistenceService — 将 Writer/Reader 历史持久化到 SQLite
  • 用于 TRANSIENT/PERSISTENT durability 和 Discovery Server BACKUP 模式

9.3 XML 配置

路径:src/cpp/rtps/xmlparser/

  • XMLProfileManager — 加载 DEFAULT_FASTRTPS_PROFILES.xml 等配置文件
  • 可覆盖 Participant/Writer/Reader 的 QoS 和传输设置
  • ROS 2 可通过 FASTRTPS_DEFAULT_PROFILES_FILE 环境变量指定

9.4 工具

工具 路径 说明
fastdds tools/fastdds/ Python CLI:discovery、shm clean 等
fast-discovery-server tools/fds/ Discovery Server 独立进程

10. 构建与依赖

10.1 产物

CMake target fastrtps
库文件 libfastrtps.so
版本 2.6.11

10.2 依赖

依赖 用途
fastcdr CDR 序列化
foonathan_memory 内存分配器
Asio(bundled) 异步 I/O(TCP/UDP)
TinyXML2(bundled) XML 配置解析
OpenSSL(可选) SECURITY / TLS
SQLite3(可选,默认 ON) 持久化

10.3 重要 CMake 选项

选项 默认 说明
SECURITY OFF DDS Security 插件
SHM_TRANSPORT_DEFAULT ON 默认传输含 SHM
SQLITE3_SUPPORT ON SQLite 持久化
FASTDDS_STATISTICS OFF 统计模块
COMPILE_TOOLS ON 构建 fastdds CLI
BUILD_SHARED_LIBS ON 动态库
STRICT_REALTIME OFF 实时 API 行为

10.4 ROS / colcon 集成

1
2
3
4
5
6
7
<!-- package.xml -->
<name>fastrtps</name>
<version>2.6.11</version>
<depend>fastcdr</depend>
<depend>foonathan_memory_vendor</depend>
<depend>tinyxml2</depend>
<depend>libssl-dev</depend>

colcon.pkg 声明依赖:fastcdrFOONATHAN_MEMORYfoonathan_memory_vendor


11. ROS 2 RMW 集成

路径:ros2_humble/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/

11.1 核心包装结构

1
2
3
4
5
6
7
8
9
10
11
12
typedef struct CustomParticipantInfo
{
eprosima::fastdds::dds::DomainParticipant * participant_{nullptr};
ParticipantListener * listener_{nullptr};

eprosima::fastdds::dds::Publisher * publisher_{nullptr};
eprosima::fastdds::dds::Subscriber * subscriber_{nullptr};

mutable std::mutex entity_creation_mutex_;
bool leave_middleware_default_qos;
publishing_mode_t publishing_mode;
} CustomParticipantInfo;
RMW 结构 封装的 Fast-DDS 类型
CustomParticipantInfo DomainParticipant + 容器 Publisher/Subscriber
CustomPublisherInfo DataWriter + TypeSupport + Listener
CustomSubscriberInfo DataReader + Listener + ContentFilteredTopic

11.2 Participant 创建

rmw_fastrtps_shared_cpp/src/participant.cpp

  1. 构造 DomainParticipantQos(含 enclave、security 属性)
  2. 注册 UDPv4 + SharedMem 传输 descriptor
  3. DomainParticipantFactory::create_participant(domain_id, qos, listener)
  4. 创建容器 PublisherSubscriber
  5. ParticipantListener 监听 on_participant_discovery 更新 ROS graph

11.3 发布 / 订阅

RMW 操作 Fast-DDS 调用
rmw_publish CustomPublisherInfo::data_writer_->write(data)
rmw_take CustomSubscriberInfo::data_reader_->take(...)
QoS 映射 rmw_fastrtps_shared_cpp/qos.cppDataWriterQos/DataReaderQos
类型支持 TypeSupport.hpp 桥接 rosidl → TopicDataType

11.4 完整 ROS 2 通信链路

1
2
3
4
5
6
7
8
9
10
11
12
rclcpp::Publisher::publish(msg)
→ rcl_publish()
→ rmw_publish() [rmw_fastrtps_shared_cpp]
→ DataWriter::write(msg)
→ TypeSupport::serialize(msg) [rosidl_typesupport_fastrtps + Fast-CDR]
→ StatefulWriter::new_change()
→ RTPSMessageGroup → UDP/SHM
═══════ 网络 / 共享内存 ═══════
→ MessageReceiver → StatefulReader
→ TypeSupport::deserialize(msg)
→ rmw_take() → rcl_take()
→ rclcpp::Subscription callback

12. 测试

1
2
3
4
5
6
test/
├── unittest/ # 单元测试(rtps/history, transport, security 等)
├── blackbox/ # 端到端通信测试
├── performance/ # 吞吐量/延迟基准
├── communication/ # 多参与者互通
└── system/tools/ # fastdds CLI 测试

核心 API 覆盖:test/unittest/rtps/test/blackbox/


13. 设计特点小结

特点 说明
双 API 共存 现代 fastdds::dds + Legacy fastrtps + 底层 rtps
Impl 模式 所有 DDS Entity 均有对应 *Impl
RTPS 为核心 DDS 层是 RTPS 的薄封装;~50% 代码在 rtps/
插件化安全 Authentication/AccessControl/Cryptography 可替换
多传输并存 同一 Participant 可同时使用 UDP + SHM + TCP
Discovery 可扩展 Simple / Client-Server / Static EDP
内存优化 foonathan_memory + CacheChangePool + PayloadPool + DataSharing
ROS 2 QL1 声明 Quality Level 1(见 QUALITY.md

14. 与 Fast-CDR 的分工

层次 职责
应用语义 Fast-DDS DDS API Topic、QoS、匹配、History
Wire 协议 Fast-DDS RTPS 发现、可靠传输、子消息
Payload 编码 Fast-CDR CDR serialize/deserialize

Fast-DDS 不做序列化,只搬运 SerializedPayload_t;类型编码全部由 Fast-CDR + 生成的 TopicDataType 完成。


15. 推荐阅读顺序

  1. DDS 入口DomainParticipantImpl::enable() — 理解 DDS→RTPS 衔接
  2. 发现机制BuiltinProtocols::initBuiltinProtocols()PDPSimpleEDPSimple
  3. 发送路径DataWriterImpl::write()StatefulWriterRTPSMessageGroup
  4. 接收路径MessageReceiverStatefulReaderDataReaderImpl
  5. 传输选择NetworkFactory + UDPv4Transport / SharedMemTransport
  6. ROS 2 集成rmw_fastrtps_shared_cpp/src/participant.cpp + custom_participant_info.hpp
  7. 序列化rosidl_typesupport_fastrtps_cpp 生成代码 + Fast-CDR 分析
  8. 配置DEFAULT_FASTRTPS_PROFILES.xml + XMLProfileManager

16. API 速查

16.1 最小 DDS 发布示例

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
#include <fastdds/dds/domain/DomainParticipantFactory.hpp>
#include <fastdds/dds/publisher/Publisher.hpp>
#include <fastdds/dds/publisher/DataWriter.hpp>
#include <fastdds/dds/topic/Topic.hpp>
#include <fastdds/dds/topic/TypeSupport.hpp>

using namespace eprosima::fastdds::dds;

auto factory = DomainParticipantFactory::get_instance();
DomainParticipant* participant = factory->create_participant(0, PARTICIPANT_QOS_DEFAULT);

TypeSupport type(new MyTypePubSubType());
type.register_type(participant);

Topic* topic = participant->create_topic("MyTopic", "MyType", TOPIC_QOS_DEFAULT);
Publisher* publisher = participant->create_publisher(PUBLISHER_QOS_DEFAULT, nullptr);
DataWriter* writer = publisher->create_datawriter(topic, DATAWRITER_QOS_DEFAULT, nullptr);

MyMsg msg;
writer->write(&msg);

16.2 关键类一览

命名空间 层级
DomainParticipant fastdds::dds DDS
DataWriter / DataReader fastdds::dds DDS
RTPSParticipant fastrtps::rtps RTPS
StatefulWriter / StatefulReader fastrtps::rtps RTPS
PDPSimple / EDPSimple fastrtps::rtps 发现
NetworkFactory fastrtps::rtps 传输
MessageReceiver fastrtps::rtps 报文
CacheChange fastrtps::rtps 历史
SecurityManager fastrtps::rtps 安全

文档基于 ROS 2 Humble 工作区中的 Fast-DDS 2.6.11 源码分析生成。

iceoryx 整体框架与模块划分

iceoryx 整体框架与模块划分

源码根:ros2_humble/src/eclipse-iceoryx/iceoryx · 版本 2.0.6(见 VERSION
官方文档:doc/mkdocs.yml、各模块 README.md


1. 产品定位

iceoryx(Eclipse iceoryx™)是一个 进程间零拷贝共享内存通信中间件(IPC middleware),起源于 Bosch 车载领域,现由 Apex.AI 等维护。在 ROS 2 Humble 中它是 CycloneDDS / rmw_iceoryx 的零拷贝(Shared Memory)后端。

设计目标:

目标 实现手段
真零拷贝 payload 从生产到消费始终在同一块共享内存 Chunk 中,只传递”指针”(相对偏移)
无锁 数据面全部使用 lock-free 队列/free-list(LockFreeQueueLoFFLiSoFi
实时安全 关键路径无系统调用、无阻塞;最坏执行时间可预测
无动态内存分配 运行期不调用 new/malloc;一切容器定容量(cxx::vector<T, Capacity> 等),共享内存在 RouDi 启动时一次性划分
无异常 全库 noexcept,错误经 cxx::expected 返回

2. 分层架构(自上而下)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
┌──────────────────────────────────────────────────────────────┐
│ 应用层:ROS 2 (rmw_iceoryx / CycloneDDS-SHM) / 原生 iceoryx │
└──────────────┬───────────────────────────┬───────────────────┘
│ C++ API │ C API
┌──────────────▼──────────────┐ ┌─────────▼─────────────────┐
│ iceoryx_posh(POSIX SHM) │◄─┤ iceoryx_binding_c │
│ Publisher/Subscriber │ │ iox_pub_* / iox_sub_* │
│ RouDi · Runtime · WaitSet │ └───────────────────────────┘
│ mepoo(MemPool/Chunk)·CaPro │
└──────────────┬──────────────┘

┌──────────────▼──────────────────────────────────────────────┐
│ iceoryx_hoofs — 无异常/无堆分配 C++ 基础库 │
│ cxx 容器 · concurrent 无锁原语 · posix 封装 · relocatable ptr│
└──────────────┬──────────────────────────────────────────────┘

┌──────────────▼──────────────────────────────────────────────┐
│ platform 抽象层(linux/mac/qnx/unix/win) │
└──────────────────────────────────────────────────────────────┘

数据面主路径publisher.loan() → MemPool 取 Chunk(共享内存)→ 用户写入 → publish() → 把 Chunk 的相对指针推入各 Subscriber 的无锁队列 → subscriber.take() 直接读同一块内存。
控制面主路径:应用 Runtime 经 IPC channel(Unix Domain Socket) 向 RouDi 注册进程/创建端口 → RouDi 在共享内存管理段中分配端口数据结构并做 CaPro 服务发现/连接。


3. 模块总览

模块 路径 产物 职责
iceoryx_hoofs iceoryx_hoofs/ libiceoryx_hoofs “Handy Objects For Utilizing Files and Streams”:STL 替代容器、无锁并发原语、POSIX 封装、相对指针、platform 层
iceoryx_posh iceoryx_posh/ libiceoryx_posh + iox-roudi 可执行 POSIX SHM 通信核心:RouDi、Runtime、Pub/Sub、Client/Server(RPC)、mepoo、CaPro、WaitSet/Listener、gateway 框架
iceoryx_binding_c iceoryx_binding_c/ libiceoryx_binding_c C 语言绑定(iox_runtime_initiox_pub_loan_chunk 等)
iceoryx_dds iceoryx_dds/ iox-dds-gateway(本仓带 COLCON_IGNORE,ROS 构建中不编译) iceoryx ↔ DDS(Cyclone DDS)网关,跨主机桥接
iceoryx_meta iceoryx_meta/ CMake 元构建 统一构建入口、编译期常量配置(build_options.cmake 定义 IOX_MAX_PUBLISHERS 等)
tools tools/ iox-introspection-client 内省客户端(tools/introspection/)、构建/CI 脚本、docker
(示例/测试) iceoryx_examples/iceoryx_integrationtest/ icedelivery、waitset、iceperf 等示例与集成测试

4. 模块依赖关系

应用网关rmw_iceoryx / CycloneDDS-SHMC 应用iceoryx_dds gatewayiceoryx_binding_ciceoryx_posh<br/>RouDi · Runtime · Pub/Sub · mepooiceoryx_hoofs<br/>cxx · concurrent · posix · rpplatform 抽象层<br/>linux/qnx/mac/winiceoryx_meta<br/>编译期配置
  • posh 依赖 hoofs:所有端口数据结构建立在 cxx 容器 + rp::RelativePointer + concurrent 无锁原语之上
  • binding_c 只包一层 posh,不引入新机制
  • iceoryx_meta 不产生运行时代码,只负责把编译期常量(如 IOX_MAX_PUBLISHERS)传给 posh:
1
2
3
4
5
6
7
8
9
10
11
//--------- Communication Resources Start---------------------
// Publisher
constexpr uint32_t MAX_PUBLISHERS = build::IOX_MAX_PUBLISHERS;
constexpr uint32_t MAX_SUBSCRIBERS_PER_PUBLISHER = build::IOX_MAX_SUBSCRIBERS_PER_PUBLISHER;
constexpr uint32_t MAX_CHUNKS_ALLOCATED_PER_PUBLISHER_SIMULTANEOUSLY =
build::IOX_MAX_CHUNKS_ALLOCATED_PER_PUBLISHER_SIMULTANEOUSLY;
constexpr uint64_t MAX_PUBLISHER_HISTORY = build::IOX_MAX_PUBLISHER_HISTORY;
// Subscriber
constexpr uint32_t MAX_SUBSCRIBERS = build::IOX_MAX_SUBSCRIBERS;
constexpr uint32_t MAX_CHUNKS_HELD_PER_SUBSCRIBER_SIMULTANEOUSLY =
build::IOX_MAX_CHUNKS_HELD_PER_SUBSCRIBER_SIMULTANEOUSLY;

一切资源上限在编译期锁死——这正是”无动态内存分配”目标的体现(另有 MAX_PROCESS_NUMBER = 300 等,见同文件)。


5. 核心概念速览

概念 定义位置(相对 iceoryx 根) 一句话解释
RouDi iceoryx_posh/source/roudi/roudi.cppport_manager.cppprocess_manager.cpp Routing and Discovery 守护进程:创建共享内存段、管理进程注册、创建端口、撮合 Pub/Sub 连接。不在数据路径上
Runtime iceoryx_posh/include/iceoryx_posh/runtime/posh_runtime.hpp 每个应用进程一个单例,负责与 RouDi 的 IPC 会话,代理创建 Publisher/Subscriber 等端口
Publisher / Subscriber iceoryx_posh/include/iceoryx_posh/popo/publisher.hpp / subscriber.hpp(typed),untyped_*.hpp(untyped) 用户 API 层;底层是 internal/popo/ports/ 里的 PublisherPort/SubscriberPort
Chunk iceoryx_posh/include/iceoryx_posh/mepoo/chunk_header.hpp 一次消息的载体:ChunkHeader + 可选 user-header + user-payload,位于共享内存
MemPool iceoryx_posh/include/iceoryx_posh/internal/mepoo/mem_pool.hpp 同尺寸 Chunk 的定长池,free-list 用无锁 LoFFLi 管理
Segment iceoryx_posh/include/iceoryx_posh/internal/mepoo/mepoo_segment.hppsegment_manager.hpp 一个 POSIX 共享内存段 = 一组 MemPool + 读写访问组(按用户组授权)
CaPro iceoryx_posh/include/iceoryx_posh/capro/service_description.hppinternal/capro/capro_message.hpp Canonical Protocol 服务模型:(Service, Instance, Event) 三元组标识一个通信通道,RouDi 据此撮合
WaitSet / Listener iceoryx_posh/include/iceoryx_posh/popo/wait_set.hpp / listener.hpp 事件多路等待:WaitSet 是同步阻塞式(用户线程 wait),Listener 是异步回调式(自带线程)

CaPro 的服务三元组(本质是三个定容字符串 + 128 位类型哈希):

1
2
3
4
5
6
7
8
9
10
11
12
13
private:
/// @brief string representation of the service
IdString_t m_serviceString;
/// @brief string representation of the instance
IdString_t m_instanceString;
/// @brief string representation of the event
IdString_t m_eventString;

/// @brief 128-Bit class hash (32-Bit * 4)
ClassHash m_classHash{0, 0, 0, 0};

/// @brief How far this service should be propagated
Scope m_scope{Scope::WORLDWIDE};

ChunkHeader 是 Chunk 的头部元数据,含版本号以支持兼容性检测与 record&replay:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
struct ChunkHeader
{
using UserPayloadOffset_t = uint32_t;

/// @brief constructs and initializes a ChunkHeader
/// @param[in] chunkSize is the size of the chunk the ChunkHeader is constructed
/// @param[in] chunkSettings are the settings like user-payload size and user-header alignment
ChunkHeader(const uint32_t chunkSize, const ChunkSettings& chunkSettings) noexcept;

// copy/move ctors/assignment operators are deleted since the calculations for the user-header and user-payload
// alignment are dependent on the address of the this pointer
ChunkHeader(const ChunkHeader&) = delete;
ChunkHeader(ChunkHeader&&) = delete;

ChunkHeader& operator=(const ChunkHeader&) = delete;
ChunkHeader& operator=(ChunkHeader&&) = delete;

/// @brief From the 1.0 release onward, this must be incremented for each incompatible change, e.g.
/// - data width of members changes
/// - members are rearranged
/// - semantic meaning of a member changes
static constexpr uint8_t CHUNK_HEADER_VERSION{1U};

MemPool 的实现完全体现”共享内存 + 无锁”两大主题(成员是相对指针 + LoFFLi 无锁 free-list):

1
2
3
4
5
6
7
8
9
10
class MemPool
{
public:
using freeList_t = concurrent::LoFFLi;
static constexpr uint64_t CHUNK_MEMORY_ALIGNMENT = 8U; // default alignment for 64 bit

MemPool(const cxx::greater_or_equal<uint32_t, CHUNK_MEMORY_ALIGNMENT> chunkSize,
const cxx::greater_or_equal<uint32_t, 1> numberOfChunks,
posix::Allocator& managementAllocator,
posix::Allocator& chunkMemoryAllocator) noexcept;

6. 进程模型:RouDi 守护进程 + 应用进程 + IPC channel

RouDi 守护进程 iox-roudi应用进程 A应用进程 BProcessManagerPortManager<br/>CaPro 撮合RouDiMemoryManager<br/>创建共享内存段PoshRuntimePublisherPoshRuntimeSubscriber(共享内存<br/>管理段 + 用户段)

要点:

  1. RouDi 必须先启动:它 mmap 创建两类段——管理段(端口数据、队列、内省数据)和用户 payload 段(按 POSIX 用户组划分访问权限,见 roudi/memory/mepoo/segment_manager.hpp)。
  2. 应用启动即注册PoshRuntime::initRuntime("app_name") 通过 Unix Domain Socket 发送 REG,RouDi 回 REG_ACK 并附上共享内存布局信息;之后应用 mmap 同样的段。
  3. IPC channel 只走控制面,消息类型一目了然:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
enum class IpcMessageType : int32_t
{
BEGIN = -1,
NOTYPE = 0,
REG, // register app
REG_ACK,
CREATE_PUBLISHER,
CREATE_PUBLISHER_ACK,
CREATE_SUBSCRIBER,
CREATE_SUBSCRIBER_ACK,
CREATE_CLIENT,
CREATE_CLIENT_ACK,
CREATE_SERVER,
CREATE_SERVER_ACK,
CREATE_INTERFACE,
CREATE_INTERFACE_ACK,
CREATE_CONDITION_VARIABLE,
CREATE_CONDITION_VARIABLE_ACK,
CREATE_NODE,
CREATE_NODE_ACK,
KEEPALIVE,
TERMINATION,
TERMINATION_ACK,
PREPARE_APP_TERMINATION,
PREPARE_APP_TERMINATION_ACK,
ERROR,
APP_WAIT,
WAKEUP_TRIGGER,
REPLAY,
MESSAGE_NOT_SUPPORTED,
// etc..
END,
};
  1. Runtime 是应用侧的”RouDi 客户端”,所有端口创建都是向 RouDi 发请求:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
/// @brief The runtime that is needed for each application to communicate with the RouDi daemon
class PoshRuntime
{
public:
PoshRuntime(const PoshRuntime&) = delete;
PoshRuntime& operator=(const PoshRuntime&) = delete;
PoshRuntime(PoshRuntime&&) = delete;
PoshRuntime& operator=(PoshRuntime&&) = delete;
virtual ~PoshRuntime() noexcept = default;

/// @brief returns active runtime
///
/// @return active runtime
static PoshRuntime& getInstance() noexcept;

/// @brief creates the runtime with given name
///
/// @param[in] name used for registering the process with the RouDi daemon
///
/// @return active runtime
static PoshRuntime& initRuntime(const RuntimeName_t& name) noexcept;
  1. KEEPALIVE 心跳:RouDi 周期性检查应用存活,应用崩溃后回收其 Chunk 与端口(roudi/process_manager.cpp)。

7. 一次发布-订阅数据流总览

RouDiSubscriber(进程B)Subscriber无锁队列(共享内存)MemPool(共享内存)Publisher(进程A)RouDiSubscriber(进程B)Subscriber无锁队列(共享内存)MemPool(共享内存)Publisher(进程A)建链(控制面,仅一次)数据面(每条消息,RouDi 不参与)CREATE_PUBLISHER (CaPro: Service/Instance/Event)CREATE_SUBSCRIBER (同一 CaPro)PortManager 撮合,把 S 的队列挂到 P 的端口loan() → LoFFLi.pop() 取空闲 Chunk直接在 Chunk 上构造/写入数据publish() → Chunk 相对指针入队(引用计数++)take() → 弹出相对指针就地读 payload(零拷贝)释放 → 引用计数--,归零则 LoFFLi.push() 回池

关键性质:

  • 传的是”相对指针”不是数据——多订阅者时同一 Chunk 被引用计数共享,全程 0 次 memcpy。
  • 队列溢出策略可配:默认丢最旧(SoFi/LockFreeQueue 的 overflow push 语义),也可配置阻塞 publisher(ConsumerTooSlowPolicy::WAIT_FOR_CONSUMER)。
  • 指针跨进程有效的根基是 hoofs 的 RelativePointer(详见 01-hoofs基础库.md 第 6 节)。

8. 目录结构对照表

路径(相对 iceoryx 根) 内容
iceoryx_hoofs/include/iceoryx_hoofs/cxx/ STL 替代:expected/optional/variant/vector/string/function_ref
iceoryx_hoofs/include/iceoryx_hoofs/concurrent/ 公开无锁队列:lockfree_queue.hppresizeable_lockfree_queue.hpp
iceoryx_hoofs/include/iceoryx_hoofs/internal/concurrent/ sofi.hpploffli.hppsmart_lock.hpptrigger_queue.hpp
iceoryx_hoofs/include/iceoryx_hoofs/posix_wrapper/ + internal/posix_wrapper/ 信号量、互斥量、共享内存、UDS、ACL 等 POSIX 封装
iceoryx_hoofs/include/iceoryx_hoofs/internal/relocatable_pointer/ RelativePointer/relocatable_ptr(零拷贝的基石)
iceoryx_hoofs/platform/{linux,mac,qnx,unix,win}/ 平台抽象层头文件
iceoryx_posh/include/iceoryx_posh/popo/ 用户 API:Publisher/Subscriber/Client/Server/WaitSet/Listener
iceoryx_posh/include/iceoryx_posh/mepoo/ + internal/mepoo/ Memory Pool:ChunkHeader、MemPool、Segment、MemoryManager
iceoryx_posh/include/iceoryx_posh/capro/ CaPro 服务描述
iceoryx_posh/include/iceoryx_posh/roudi/ + source/roudi/ RouDi 应用框架、PortManager、ProcessManager、内存编排
iceoryx_posh/include/iceoryx_posh/runtime/ PoshRuntime、Node、ServiceDiscovery
iceoryx_posh/include/iceoryx_posh/gateway/ 网关基类(供 iceoryx_dds 等复用)
iceoryx_binding_c/include/iceoryx_binding_c/ C API 头文件
iceoryx_dds/ DDS 网关(本仓 COLCON_IGNORE)
iceoryx_meta/ 元构建 + build_options.cmake(编译期上限配置)
tools/introspection/ iox-introspection-client(ncurses 内省界面)
iceoryx_examples/ icedelivery / iceoptions / waitset / callbacks / iceperf 等示例

9. 文档阅读路线

  1. 本文 — 模块划分、进程模型与数据流
  2. 01-hoofs基础库.md — cxx 容器、无锁原语、POSIX 封装、相对指针(理解零拷贝的关键)

下一篇01-hoofs基础库.md

iceoryx_hoofs 基础库

iceoryx_hoofs 基础库

源码根:ros2_humble/src/eclipse-iceoryx/iceoryx/iceoryx_hoofs · 版本 2.0.6
hoofs = Handy Objects For Utilizing Files and Streams
上一篇:00-整体框架与模块划分.md


1. 定位:无异常、无堆分配的 C++ 基础库

iceoryx_posh 的一切数据结构都要放进共享内存并满足实时约束,因此不能用 STL(std::vectornewstd::string 有 SSO 之外的堆分配、容器内部存裸指针在别的进程地址空间无效)。hoofs 提供一套替代设施,遵循三条铁律:

铁律 落地方式
不抛异常 所有函数 noexcept;错误用 cxx::expected<T, E> 返回
不做堆分配 容器容量是模板参数(vector<T, Capacity>),内存全部内联在对象里
可放入共享内存 对象自包含(self-contained),指向自身内部用 relocatable_ptr,跨段引用用 RelativePointer

子模块一览(include/iceoryx_hoofs/):

子模块 路径 内容
cxx cxx/internal/cxx/ 词汇类型与容器(expected/optional/variant/vector/string/function_ref/list/stack …)
concurrent concurrent/internal/concurrent/ LockFreeQueue、IndexQueue、LoFFLi、SoFi、smart_lock、TriggerQueue …
posix_wrapper posix_wrapper/internal/posix_wrapper/ 共享内存、信号量、互斥量、UDS、消息队列、ACL、posixCall
relocatable_pointer internal/relocatable_pointer/ RelativePointer / relocatable_ptr / PointerRepository
platform platform/{linux,mac,qnx,unix,win}/ OS 差异抹平层
其他 log/error_handling/design_pattern/internal/units/ 日志、错误处理、Creation 模式、时间单位

2. cxx 容器与词汇类型:为何不用 STL

2.1 一览表

类型 文件(相对 iceoryx_hoofs/include/iceoryx_hoofs/) 对标 STL 差异关键点
cxx::expected<T, E> cxx/expected.hpp C++23 std::expected 无异常的错误传递主干;提供 and_then/or_else 链式调用
cxx::optional<T> cxx/optional.hpp std::optional 内联存储,无异常(访问空值走错误处理器而非 throw)
cxx::variant<...> cxx/variant.hpp std::variant 无异常、无 valueless_by_exception 状态
cxx::vector<T, Capacity> cxx/vector.hpp std::vector 容量编译期固定,内存内联,永不 realloc
cxx::string<Capacity> cxx/string.hpp std::string 定容字符数组,可安全放入共享内存
cxx::function_ref<Sig> cxx/function_ref.hpp C++26 function_ref 非拥有的可调用引用,无堆分配(std::function 可能堆分配)
cxx::function<Sig, Bytes> cxx/function.hpp std::function 定长内联存储版 function
cxx::list/forward_list/stack cxx/list.hpp 对应 STL 节点全部内联,定容
cxx::unique_ptr<T> cxx/unique_ptr.hpp std::unique_ptr 删除器为运行时 function_ref,用于 Chunk 归还等场景

不用 STL 的原因归纳:① STL 容器在增长时动态分配(违反实时/确定性);② 异常是 STL 的错误通道(违反无异常);③ STL 容器内部保存绝对指针,映射到另一个进程后失效(违反共享内存可用性);④ 各家 libc++/libstdc++ 的 ABI 与行为差异不可控(车规认证需要可控实现)。

2.2 expected:错误处理主干

1
2
3
4
5
6
7
8
9
10
11
12
13
14
template <typename... T>
class IOX_NO_DISCARD expected;

/// @brief expected implementation from the C++20 proposal with C++11. The interface
/// is inspired by the proposal but it has changes since we are not allowed to
/// throw an exception.
/// @param ErrorType type of the error which can be stored in the expected
///
/// @code
/// cxx::expected<int, float> callMe() {
/// bool l_errorOccured;
/// // ... do stuff
/// if ( l_errorOccured ) {
/// return cxx::error<float>(55.1f);

用法约定:成功返回 cxx::success<T>(value),失败返回 cxx::error<E>(err)IOX_NO_DISCARD 强制调用方检查结果。hoofs/posh 中几乎所有可失败 API(共享内存创建、UDS 收发、端口创建)都以它为返回类型,形成贯穿全库的错误管道。

2.3 vector / string:定容 + 内联

1
2
template <typename T, uint64_t Capacity>
class vector
1
2
template <uint64_t Capacity>
class string

容量进模板参数意味着:sizeof 编译期已知、无堆、memcpy 语义友好(配合 relocatable 设计),代价是超容 push_back 返回 false 而不是扩容——上层必须处理”满”这个正常状态,这正是 iceoryx 全库”资源上限显式化”哲学的缩影。


3. concurrent 并发原语

3.1 LockFreeQueue:MPMC 无锁定容队列

concurrent/lockfree_queue.hpp,多生产者多消费者(MPMC)、FIFO、定容、lock-free。内部结构是”两条索引队列 + 一个定长缓冲区”:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
protected:
using Queue = IndexQueue<Capacity>;
using BufferIndex = typename Queue::value_t;

// remark: actually m_freeIndices do not have to be in a queue, it could be another
// multi-push multi-pop capable lockfree container (e.g. a stack or a list)
Queue m_freeIndices;

// required to be a queue for LockFreeQueue to exhibit FIFO behaviour
Queue m_usedIndices;

Buffer<ElementType, Capacity, BufferIndex> m_buffer;

std::atomic<uint64_t> m_size{0u};

工作原理:push = 从 m_freeIndices 弹出一个空闲槽位索引 → 写 m_buffer[index] → 索引压入 m_usedIndicespop 反向。数据搬运与索引流转解耦,索引流转由底层 IndexQueueinternal/concurrent/lockfree_queue/index_queue.hpp)完成——一个基于 CyclicIndex(带循环计数防 ABA)+ std::atomic CAS 的 MPMC 索引环:

1
2
3
4
/// @brief lockfree queue capable of storing indices 0,1,... Capacity-1
template <uint64_t Capacity, typename ValueType = uint64_t>
class IndexQueue
{

特色 API:push() 满时必定成功,返回被挤掉的最旧元素(cxx::optional<T>)——这是订阅者队列”丢最旧”溢出策略的实现基础。另有 ResizeableLockFreeQueueconcurrent/resizeable_lockfree_queue.hpp)支持运行时在编译期上限内调整容量(订阅者 queue capacity 可配)。

3.2 SoFi:单生产者单消费者安全溢出 FIFO

Softer Fifo(internal/concurrent/sofi.hpp),SPSC、仅限 trivially copyable 类型、无锁,满时覆写最旧数据并把旧值交还给 push 方:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
/// @brief
/// Thread safe producer and consumer queue with a safe overflowing behavior.
/// SoFi is designed in a FIFO Manner but prevents data loss when pushing into
/// a full SoFi. When SoFi is full and a Sender tries to push, the data at the
/// current read position will be returned. SoFi is a Thread safe without using
/// locks. When the buffer is filled, new data is written starting at the
/// beginning of the buffer and overwriting the old.The SoFi is especially
/// designed to provide fixed capacity storage. When its capacity is exhausted,
/// newly inserted elements will cause elements either at the beginning
/// to be overwritten.The SoFi only allocates memory when
/// created , capacity can be is adjusted explicitly.
///
/// @param[in] ValueType DataType to be stored, must be trivially copyable
/// @param[in] CapacityValue Capacity of the SoFi
template <class ValueType, uint64_t CapacityValue>
class SoFi
{
static_assert(std::is_trivially_copyable<ValueType>::value, "SoFi can handle only trivially copyable data types");
/// @brief Check if Atomic integer is lockfree on platform
/// ATOMIC_INT_LOCK_FREE = 2 - is always lockfree
/// ATOMIC_INT_LOCK_FREE = 1 - is sometimes lockfree
/// ATOMIC_INT_LOCK_FREE = 0 - is never lockfree
static_assert(2 <= ATOMIC_INT_LOCK_FREE, "SoFi is not able to run lock free on this data type");

实现只用两个原子读写位置(读者只动 m_readPosition,写者只动 m_writePosition),内部容量比名义容量多 1 个槽位用于空/满判别。posh 中它经 cxx::VariantQueue 作为订阅者 Chunk 队列的一种可选实现(SoFi_SingleProducerSingleConsumer 策略)。

3.3 LoFFLi:无锁 free-list(MemPool 的心脏)

Lock-Free Free-List(internal/concurrent/loffli.hpp),管理”哪些 Chunk 索引空闲”。核心是把 next 索引和 ABA 计数打包进一个 8 字节原子量做 CAS:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
class LoFFLi
{
public:
using Index_t = uint32_t;

private:
struct alignas(8) Node
{
Index_t indexToNextFreeIndex;
uint32_t abaCounter;
};

static_assert(sizeof(Node) <= 8U,
"The size of 'Node' must not exceed 8 bytes in order to be lock-free on 64 bit systems!");

注意其索引数组指针是 rp::RelativePointer<Index_t> m_nextFreeIndex(同文件第 71 行)——因为 LoFFLi 本体和索引数组都活在共享内存里,必须用相对指针(见第 6 节)。这也解释了 MemPool 里”chunk 数只能 32 位”的注释:64 位 CAS 里要留 32 位给 ABA 计数。

3.4 smart_lock:给任意类型加锁的包装器

internal/concurrent/smart_lock.hpp。控制面(非实时路径)用它把任意对象变成线程安全对象:operator-> 返回一个持锁 Proxy,实现”每次调用自动加解锁”:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
/// @brief The smart_lock class is a wrapping class which can be used to make
/// an arbitrary class threadsafe by wrapping it with the help of the
/// arrow operator.
/// IMPORTANT: If you generate a threadsafe container with smart_lock,
/// only the container is threadsafe not the containing
/// elements!
/// @code
/// #include <algorithm>
/// #include <vector>
/// #include "smart_lock.hpp"
///
/// int main() {
/// concurrent::smart_lock<std::vector<int>> threadSafeVector;
/// threadSafeVector->push_back(123);
/// threadSafeVector->push_back(456);
/// threadSafeVector->push_back(789);
/// size_t vectorSize = threadSafeVector->size();
///
/// {
/// auto guardedVector = threadSafeVector.getScopeGuard();
/// auto iter = std::find(guardVector->begin(), guardVector->end(), 456);
/// if (iter != guardVector->end()) guardVector->erase(iter);
/// }
/// }
/// @endcode
template <typename T, typename MutexType = ::std::mutex>
class smart_lock

跨多次调用需要原子性时用 getScopeGuard()。RouDi 的进程表、服务注册表等控制面结构大量使用它。

3.5 其他并发设施

设施 文件 用途
FiFo internal/concurrent/fifo.hpp 最简 SPSC FIFO
TriggerQueue internal/concurrent/trigger_queue.hpp 可阻塞唤醒的队列
PeriodicTask internal/concurrent/periodic_task.hpp 周期任务线程(RouDi KEEPALIVE 监控等)
ActiveObject/TACO internal/concurrent/active_object.hpptaco.hpp 主动对象模式 / 线程间单槽交换

4. posix 封装

设计模式统一:类不可直接构造,经 DesignPattern::Creationdesign_pattern/creation.hpp)的 create() 工厂返回 cxx::expected,把”构造可能失败”显式化;所有系统调用经 posix::posixCallposix_wrapper/posix_call.hpp)包装,统一处理 errno、EINTR 重试与返回值检查。

4.1 SharedMemoryObject

internal/posix_wrapper/shared_memory_object.hpp = shm_open + mmap + 线性分配器三合一:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
class SharedMemoryObject : public DesignPattern::Creation<SharedMemoryObject, SharedMemoryObjectError>
{
public:
static constexpr void* NO_ADDRESS_HINT = nullptr;
SharedMemoryObject(const SharedMemoryObject&) = delete;
SharedMemoryObject& operator=(const SharedMemoryObject&) = delete;
SharedMemoryObject(SharedMemoryObject&&) noexcept = default;
SharedMemoryObject& operator=(SharedMemoryObject&&) noexcept = default;
~SharedMemoryObject() noexcept = default;

void* allocate(const uint64_t size, const uint64_t alignment) noexcept;
void finalizeAllocation() noexcept;

Allocator* getAllocator() noexcept;
const void* getBaseAddress() const noexcept;
void* getBaseAddress() noexcept;

uint64_t getSizeInBytes() const noexcept;
int getFileHandle() const noexcept;
bool hasOwnership() const noexcept;

注意 allocate()/finalizeAllocation():这是一个只进不出的 bump allocator(shared_memory_object/allocator.hpp)——RouDi 启动阶段在段内依次摆放管理结构和 MemPool,finalizeAllocation() 之后不允许再分配。这就是”无动态内存分配”在共享内存侧的执行机制:分配只发生在初始化阶段。

4.2 semaphore / mutex

  • posix_wrapper/semaphore.hpp:命名/匿名 POSIX 信号量;匿名版可放共享内存(pshared),是 WaitSet 的 ConditionVariableData 跨进程唤醒的底层。
  • internal/posix_wrapper/mutex.hpppthread_mutex_t 封装,支持跨进程 robust mutex(持有者死亡可恢复)。

4.3 IPC channel:UnixDomainSocket / MessageQueue / NamedPipe

Runtime↔RouDi 的控制通道有三种可选实现,统一在 internal/posix_wrapper/ipc_channel.hpp 的接口语义下(Linux 默认 UDS):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
/// @brief Wrapper class for unix domain socket
class UnixDomainSocket : public DesignPattern::Creation<UnixDomainSocket, IpcChannelError>
{
public:
struct NoPathPrefix_t
{
};
static constexpr NoPathPrefix_t NoPathPrefix{};
static constexpr uint64_t NULL_TERMINATOR_SIZE = 1U;
static constexpr uint64_t MAX_MESSAGE_SIZE = platform::IOX_UDS_SOCKET_MAX_MESSAGE_SIZE - NULL_TERMINATOR_SIZE;
/// @brief The name length is limited by the size of the sockaddr_un::sun_path buffer and the IOX_SOCKET_PATH_PREFIX
static constexpr size_t LONGEST_VALID_NAME = sizeof(sockaddr_un::sun_path) - 1;

using UdsName_t = cxx::string<LONGEST_VALID_NAME>;
using Message_t = cxx::string<MAX_MESSAGE_SIZE>;

提供 send/timedSend/receive/timedReceive,全部返回 cxx::expected

4.4 access_control:按用户组授权共享内存段

internal/posix_wrapper/access_control.hpp 封装 POSIX ACL(libacl)。posh 的每个用户 payload 段配置 reader/writer 用户组(mepoo/segment_config.hpp),RouDi 创建段时用 ACL 落实——这是 iceoryx 的安全模型:操作系统级用户组决定谁能读写哪个段


5. platform 抽象层

iceoryx_hoofs/platform/ 下每个 OS 一套同名头文件(linux/ mac/ qnx/ unix/ win/),如 platform/linux/include/iceoryx_hoofs/platform/ 里有 mman.hppsemaphore.hppsocket.hppacl.hpppthread.hpp 等 24 个头。上层代码一律 #include "iceoryx_hoofs/platform/xxx.hpp",由 CMake 选择平台目录;Windows 等缺失的 POSIX API 在对应目录里给出模拟实现,QNX 等 RTOS 只需薄转发。platform_settings.hpp 集中平台常量(UDS 路径前缀、最大消息长度等)。


6. relocatable pointer:共享内存中指针为何必须相对化(关键)

6.1 问题:同一块物理内存,各进程虚拟地址不同

mmap 同一个共享内存对象,进程 A 可能映射到 0x7f11...,进程 B 映射到 0x7f42...。若 A 在共享内存里存了一个裸指针(绝对虚拟地址),B 解引用它就是未定义行为。而 iceoryx 的端口结构、队列、free-list、Chunk 引用全部活在共享内存里且互相指来指去——所以”指针”必须改存与某个已知基址的偏移,读取时在本进程重算绝对地址。

iceoryx 给出两级方案:

方案 适用场景 额外依赖
relocatable_ptr<T> 指针与被指对象在同一内存块内(自包含对象) 无(存自身到目标的偏移)
RelativePointer<T> 指针与被指对象可在不同共享内存段 每进程的段注册表 PointerRepository

6.2 RelativePointer:(segment id, offset) 二元组

源码注释把场景画得很清楚:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
/// @brief pointer class to use when pointer and pointee are located in different shared memory segments
/// We can have the following scenario:
/// Pointer p is stored in segment S1 and points to object X of type T in segment S2.
///
/// Shared Memory S1: p S2: X
/// |___________________^
/// App1 a1 b1 c1 d1
/// App2 a2 b2 c2 d2
///
/// Now it is no longer true in general that both segments will be offset by the same difference in App2 and therefore
/// relocatable pointers are no longer sufficient.
/// Relative pointers solve this problem by incorporating the information from where they need to measure differences
/// (i.e. relative to the given address). This requires an additional registration mechanism to be used by all
/// applications where the start addresses and the size of all segments to be used are registered. Since these start
/// address may differ between applications, each segment is identified by a unique id, which can be provided upon
/// registration by the first application. In the figure, this means that the starting addresses of both segments(a1, a2
/// and c1, c2) would have to be registered in both applications.
/// Once this registration is done, relative pointers can be constructed from raw pointers similar to relocatable
/// pointers.
/// @note It should be noted that relocating a memory segment will invalidate relative pointers, i.e. relative pointers
/// are NOT relocatable. This is because the registration mechanism cannot be automatically informed about the copy of a
/// whole segment, such a segment would have to be registered on its own (and the original segment deregistered).

存储上只有两个 8 字节成员——段 id 与段内偏移,这就是它可以原样躺在共享内存里被任何进程读取的原因:

1
2
3
protected:
id_t m_id{NULL_POINTER_ID};
offset_t m_offset{NULL_POINTER_OFFSET};

编解码就是一次基址加减(基址查本进程的注册表):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
BaseRelativePointer::offset_t BaseRelativePointer::getOffset(const id_t id, const_ptr_t ptr) noexcept
{
if (id == NULL_POINTER_ID)
{
return NULL_POINTER_OFFSET;
}
auto* basePtr = getBasePtr(id);
return reinterpret_cast<offset_t>(ptr) - reinterpret_cast<offset_t>(basePtr);
}

BaseRelativePointer::ptr_t BaseRelativePointer::getPtr(const id_t id, const offset_t offset) noexcept
{
if (offset == NULL_POINTER_OFFSET)
{
return nullptr;
}
auto* basePtr = getBasePtr(id);
// NOLINTNEXTLINE(performance-no-int-to-ptr) reliance on integers for offset computation by design
return reinterpret_cast<ptr_t>(offset + reinterpret_cast<offset_t>(basePtr));
}

类型化外壳 RelativePointer<T>internal/relocatable_pointer/relative_pointer.hpp)在其上提供 operator* / operator-> / 隐式转 T*,用起来与裸指针无异。另有 AtomicRelativePointeratomic_relocatable_pointer.hpp)供无锁结构使用。

6.3 PointerRepository:每进程一份的段基址注册表

1
2
3
4
5
6
7
8
/// @brief Allows registration of memory segments with their start pointers and size.
/// This class is used to resolve relative pointers in the corresponding address space of the application.
/// Up to CAPACITY segments can be registered with MIN_ID = 1 to MAX_ID = CAPACITY - 1
/// id 0 is reserved and allows relative pointers to behave like normal pointers
/// (which is equivalent to measure the offset relative to 0).
template <typename id_t, typename ptr_t, uint64_t CAPACITY = 10000U>
class PointerRepository
{

要点:

  • 注册表是进程本地的静态单例(BaseRelativePointer::getRepository() 内的函数静态变量),不在共享内存里;每个进程 mmap 完各段后各自调用 registerPtr(id, 本进程基址, size)段 id 全局一致,基址各进程各异——这正是跨进程解引用成立的机制。
  • id 0 保留:偏移直接当裸指针用(相当于基址为 0),使同一套代码可同时处理进程内/共享内存两种情形。
  • 在 posh 中,应用向 RouDi 注册(REG/REG_ACK)后拿到各段的 id 与名字,mmap 后完成注册;此后端口结构里的 RelativePointer(如 MemPool 的 m_rawMemory、LoFFLi 的索引数组指针)在任何进程中都可正确解引用。
  • 注意注释的告警:RelativePointer 本身不可搬移(段整体 memcpy 到别处后失效),因为注册表无从得知搬移。

6.4 relocatable_ptr:自包含对象的可搬移指针

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
/// @brief Smart pointer type that allows objects using it to able to be copied by memcpy
/// without invalidating the pointer.
/// This applies only if it points to memory owned by the object itself
/// (i.e. not to memory outside of the object).
/// This is useful to improve copy-efficiency and allow the types build with relocatable
/// pointers only to be stored in shared memory.
/// It is useable like a raw pointer of the corresponding type and can be implicily
/// converted to one.
///
/// @tparam T the native type wrapped by the relocatable_ptr, i.e. relocatable_ptr<T>
/// has native type T and corresponds to a raw pointer of type T*.
///
/// @note It is advisable to use relocatable_ptr only for storage (e.g. member variables),
/// not to pass them around as function arguments or as return value.
/// There should be no need for this, since as pass-around type
/// regular pointers do the job just fine and do not incur
/// the slight runtime overhead of a relocatable_ptr.
/// There should be no memory overhead on 64 bit systems.
/// @note relocatable_ptr is not trivially copyable since in general the copy constructor
/// requires additional logic. Hence obects that contain it re not trivially
/// copyable in the C++ sense. However, if the pointees of a host object containing the
/// relocatable ptr are all located inside the object and the obect is otherwise trivially
/// copyable it can be safely copied by memcpy.
/// @todo specialize for another pointer class for this use case once it is fully defined/understood
template <typename T>
class relocatable_ptr

它存的是”从 this(指针自身地址)到目标的偏移”,因此整个对象连同指针一起 memcpy/重新映射后依然有效,且不需要任何注册表;限制是只能指向对象自身拥有的内存。

6.4.1 两者对比小结

relocatable_ptr<T> RelativePointer<T>
偏移基准 指针自身地址(this 段基址(查 PointerRepository)
可跨段指 否(仅对象内部)
需要注册表 是(每进程注册段基址)
段整体搬移后 仍有效 失效
典型用户 自包含容器 MemPool、LoFFLi、端口、Chunk 引用

6.5 为什么说这是理解 iceoryx 的关键

回看 00 篇的数据流:publish() 推给订阅者的”Chunk 指针”、订阅者队列本身、MemPool 的 free-list——全是建立在 RelativePointer 之上的结构(posh 中 ShmSafeUnmanagedChunk 等类型即是对”chunk 的相对引用”的封装,见 iceoryx_posh/include/iceoryx_posh/internal/mepoo/shm_safe_unmanaged_chunk.hpp)。零拷贝 = 只传偏移;没有指针相对化,跨进程共享数据结构无从谈起。


7. 内存与生命周期设计原则

  1. 分配只在初始化期:RouDi 启动时经 bump Allocator 一次性划分全部共享内存,finalizeAllocation() 后拒绝再分配;应用运行期唯一的”分配”是从 MemPool 无锁地借还 Chunk(O(1),free-list 操作)。
  2. 容量一律编译期封顶cxx::vector<T, Capacity>MAX_PUBLISHERSMAX_PROCESS_NUMBER = 300 等;”资源耗尽”是 API 层面的正常返回值(expected 错误 / push 返回 false),不是异常。
  3. 构造可失败 → 工厂化DesignPattern::Creation 把失败从构造函数挪到 create()expected 返回值,杜绝半初始化对象。
  4. RAII + 引用计数管理 Chunk:用户侧 popo::Sample/cxx::unique_ptr 析构自动归还;共享内存里 Chunk 由 ChunkManagement 引用计数(多订阅者共享一份 payload),归零回 MemPool。
  5. 崩溃回收:Chunk 的归属登记在共享内存中,RouDi 通过 KEEPALIVE 检测应用死亡后遍历回收其端口与 Chunk——生命周期的最终兜底者是 RouDi。
  6. 无锁优先,锁只留给控制面:数据面 LoFFLi/LockFreeQueue/SoFi;控制面(RouDi 内部进程表等)才用 smart_lock/mutex。

下一篇02-mepoo共享内存与内存池.md

02 mepoo:共享内存段与固定大小内存池

02 mepoo:共享内存段与固定大小内存池

源码锚点(iceoryx 2.0.6,根目录 /home/cp/work2/ros2Learn/ros2_humble/src/eclipse-iceoryx/iceoryx):
公共头:iceoryx_posh/include/iceoryx_posh/mepoo/chunk_header.hppchunk_settings.hppmepoo_config.hppsegment_config.hpp
内部头:iceoryx_posh/include/iceoryx_posh/internal/mepoo/mem_pool.hppmemory_manager.hppmepoo_segment.hpp/.inlsegment_manager.hpp/.inlchunk_management.hppshared_chunk.hpptyped_mem_pool.hpp
实现:iceoryx_posh/source/mepoo/;无锁 free-list:iceoryx_hoofs/.../concurrent/loffli.hpp

mepoo(Memory Pool)是 iceoryx 零拷贝通信的地基:RouDi 启动时创建共享内存段并在段内切好一组固定大小的 chunk 池;应用进程 attach 后,publisher 直接在共享内存里 loan chunk、写数据、投递 offset——全程没有 memcpy,也没有运行时 malloc

1. 总体结构

1
2
3
4
5
6
7
8
9
10
11
12
RouDi 进程
├─ 管理段(iceoryx_mgmt)
│ ├─ SegmentManager ← 所有段的元信息,应用通过它查询自己可映射的段
│ ├─ 各 MemPool 的 LoFFLi 空闲索引表(管理内存)
│ ├─ ChunkManagement 池(引用计数对象池)
│ └─ port 数据 / 条件变量等(见 03 篇)

└─ 用户数据段(每个 [reader组, writer组] 一个,shm 名 = writer 组名)
├─ MemPool #0: 128B × 10000 ┐
├─ MemPool #1: 1KB × 5000 │ 按 chunk 大小递增排列
├─ ... │ 的 bucket(桶)
└─ MemPool #6: 4MB × 10 ┘

两类内存的分离是刻意设计:管理数据(free-list、引用计数)只放在管理段,用户数据段里只有 chunk 本身。这样即使应用进程只有数据段的只读权限,也不影响引用计数的读写(管理段对所有应用可读写)。

2. MePooSegment 与 SegmentManager:段与用户组权限

2.1 段的创建(RouDi 侧)

MePooSegment 封装一个 POSIX 共享内存对象 + 一个 MemoryManager。构造时按 writer 组名创建 shm 并用 ACL 精细设置权限:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
template <typename SharedMemoryObjectType, typename MemoryManagerType>
inline MePooSegment<SharedMemoryObjectType, MemoryManagerType>::MePooSegment(
const MePooConfig& mempoolConfig,
posix::Allocator& managementAllocator,
const posix::PosixGroup& readerGroup,
const posix::PosixGroup& writerGroup,
const iox::mepoo::MemoryInfo& memoryInfo) noexcept
: m_sharedMemoryObject(std::move(createSharedMemoryObject(mempoolConfig, writerGroup)))
, m_readerGroup(readerGroup)
, m_writerGroup(writerGroup)
, m_memoryInfo(memoryInfo)
{
using namespace posix;
AccessController accessController;
if (!(readerGroup == writerGroup))
{
accessController.addPermissionEntry(
AccessController::Category::SPECIFIC_GROUP, AccessController::Permission::READ, readerGroup.getName());
}
accessController.addPermissionEntry(
AccessController::Category::SPECIFIC_GROUP, AccessController::Permission::READWRITE, writerGroup.getName());
...
m_memoryManager.configureMemoryManager(mempoolConfig, managementAllocator, *m_sharedMemoryObject.getAllocator());
m_sharedMemoryObject.finalizeAllocation();
}

要点:

  • 读写权限按 POSIX 用户组划分:writer 组 READWRITE、reader 组 READ、others NONE,通过文件 ACL(AccessController::writePermissionsToFile)落到 shm 文件描述符上。
  • shm 名字就是 writer 组名(SharedMemoryObjectType::create(writerGroup.getName(), ...)),大小 = MemoryManager::requiredChunkMemorySize(config)
  • 创建后调用 rp::BaseRelativePointer::registerPtr() 把「段基址 → segmentId」注册进 relative-pointer 机制,之后所有跨进程指针都以 (segmentId, offset) 形式存储。

2.2 SegmentManager:按用户查段

SegmentManager 持有所有 MePooSegment,本身也放在管理段里。应用注册时它按调用者的用户组算出该映射哪些段、以什么权限映射:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
template <typename SegmentType>
inline typename SegmentManager<SegmentType>::SegmentMappingContainer
SegmentManager<SegmentType>::getSegmentMappings(const posix::PosixUser& user) noexcept
{
// get all the groups the user is in
auto groupContainer = user.getGroups();
...
for (const auto& groupID : groupContainer)
{
for (const auto& segment : m_segmentContainer)
{
if (segment.getWriterGroup() == groupID)
{
// a user is allowed to be only in one writer group, as we currently only support one memory manager per
// process
if (!foundInWriterGroup)
{
mappingContainer.emplace_back(segment.getWriterGroup().getName(), ...);
foundInWriterGroup = true;
}
else
{
errorHandler(Error::kMEPOO__USER_WITH_MORE_THAN_ONE_WRITE_SEGMENT);
...

约束:一个用户最多属于一个 writer 组(一个进程只支持一个可写 MemoryManager);其余匹配 reader 组的段以只读方式映射。getSegmentInformationWithWriteAccessForUser() 则返回可写段的 MemoryManager 引用与 segmentId,供该进程的 publisher 分配 chunk 使用。

3. MemPool:固定大小块 + 无锁 free-list

3.1 为什么固定大小

  • O(1) 且无碎片:分配 = 从 free-list 弹出一个索引,addr = base + index * chunkSize;释放 = 压回索引。没有 first-fit 扫描、没有 external fragmentation,实时性可证。
  • 跨进程安全:free-list 是 32 位索引数组(LoFFLi),不含绝对指针,任何进程映射到不同基址都能用。
  • 崩溃可清理:chunk 的归属由索引可逆推(offset / chunkSize),RouDi 能在应用死亡后可靠回收。
  • 代价是内部碎片(申请 200B 会拿到 1KB bucket 的 chunk),因此 bucket 配置要贴合实际消息大小分布(见第 9 节)。

3.2 结构与分配路径

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
class MemPool
{
public:
using freeList_t = concurrent::LoFFLi;
static constexpr uint64_t CHUNK_MEMORY_ALIGNMENT = 8U; // default alignment for 64 bit

MemPool(const cxx::greater_or_equal<uint32_t, CHUNK_MEMORY_ALIGNMENT> chunkSize,
const cxx::greater_or_equal<uint32_t, 1> numberOfChunks,
posix::Allocator& managementAllocator,
posix::Allocator& chunkMemoryAllocator) noexcept;
...
void* getChunk() noexcept;
...
void freeChunk(const void* chunk) noexcept;

private:
...
rp::RelativePointer<uint8_t> m_rawMemory;

uint32_t m_chunkSize{0U};
/// needs to be 32 bit since loffli supports only 32 bit numbers
/// (cas is only 64 bit and we need the other 32 bit for the aba counter)
uint32_t m_numberOfChunks{0U};

std::atomic<uint32_t> m_usedChunks{0U};
std::atomic<uint32_t> m_minFree{0U};

freeList_t m_freeIndices;
};

注意两个 allocator:chunk 本体从 chunkMemoryAllocator(用户数据段)划出,LoFFLi 索引表从 managementAllocator(管理段)划出。

getChunk / freeChunksource/mepoo/mem_pool.cpp):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
void* MemPool::getChunk() noexcept
{
uint32_t l_index{0U};
if (!m_freeIndices.pop(l_index))
{
std::cerr << "Mempool [m_chunkSize = " << m_chunkSize << ...
return nullptr;
}
...
m_usedChunks.fetch_add(1U, std::memory_order_relaxed);
adjustMinFree();

return m_rawMemory + l_index * m_chunkSize;
}

void MemPool::freeChunk(const void* chunk) noexcept
{
cxx::Expects(m_rawMemory <= chunk
&& chunk <= m_rawMemory + (static_cast<uint64_t>(m_chunkSize) * (m_numberOfChunks - 1U)));

auto offset = static_cast<const uint8_t*>(chunk) - m_rawMemory;
cxx::Expects(offset % m_chunkSize == 0);

uint32_t index = static_cast<uint32_t>(offset / m_chunkSize);

if (!m_freeIndices.push(index))
{
errorHandler(Error::kPOSH__MEMPOOL_POSSIBLE_DOUBLE_FREE);
}

m_usedChunks.fetch_sub(1U, std::memory_order_relaxed);
}

3.3 LoFFLi:Lock-Free Free-List

concurrent::LoFFLi(hoofs)是单链式无锁空闲索引栈,head 是 64 位原子 Node{indexToNextFreeIndex, abaCounter}——32 位索引 + 32 位 ABA 计数器打包进一次 CAS,这就是 chunk 数被限制在 32 位的原因:

1
2
3
4
5
6
7
8
struct alignas(8) Node
{
Index_t indexToNextFreeIndex;
uint32_t abaCounter;
};

static_assert(sizeof(Node) <= 8U,
"The size of 'Node' must not exceed 8 bytes in order to be lock-free on 64 bit systems!");

push 还能检测重复归还(返回 false → kPOSH__MEMPOOL_POSSIBLE_DOUBLE_FREE),是 double-free 的最后防线。

4. ChunkHeader:真实内存布局

每个 chunk 的头部是一个 ChunkHeader(32 字节,8 字节对齐),后随可选的 user-header 与按需对齐的 user-payload:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
private:
// the order of these members must be changed carefully and if this happens, the m_chunkHeaderVersion
// needs to be adapted in order to be able to detect incompatibilities between publisher/subscriber
// or record&replay, m_chunkSize and m_chunkHeaderVersion should therefore neither changed the type,
// nor the position

// size of the whole chunk, including the header
uint32_t m_chunkSize{0U};
uint8_t m_chunkHeaderVersion{CHUNK_HEADER_VERSION};
// reserved for future functionality and used to indicate the padding bytes; currently not used and set to `0`
uint8_t m_reserved{0};
// currently just a placeholder
uint16_t m_userHeaderId{NO_USER_HEADER};
popo::UniquePortId m_originId{popo::InvalidPortId};
uint64_t m_sequenceNumber{0U};
uint32_t m_userHeaderSize{0U};
uint32_t m_userPayloadSize{0U};
uint32_t m_userPayloadAlignment{1U};
UserPayloadOffset_t m_userPayloadOffset{sizeof(ChunkHeader)};

字段说明:

字段 含义
m_chunkSize 整个 chunk 大小(含头),即所属 bucket 的 chunkSize
m_chunkHeaderVersion 布局版本号(record&replay 兼容性检测),当前为 1
m_userHeaderId user-header 类型 id;无 user-header 时为 NO_USER_HEADER(0x0000),2.0 中有 user-header 时统一填 UNKNOWN_USER_HEADER(0xFFFF)(占位,尚未开放自定义 id)
m_originId 发送方 publisher 的 UniquePortId,由 ChunkSender(friend)通过私有 setOriginId 写入
m_sequenceNumber 发送序号,ChunkSender::send 时递增写入
m_userPayloadOffset payload 相对 ChunkHeader 起始地址的偏移

4.1 对齐计算与 back-offset

payload 对齐由构造函数按三种情形计算(source/mepoo/chunk_header.cpp):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
if (userHeaderSize == 0U)
{
if (userPayloadAlignment <= alignof(mepoo::ChunkHeader))
{
// the most simple case with no user-header and the user-payload adjacent to the ChunkHeader
m_userPayloadOffset = sizeof(ChunkHeader);
}
else
{
// the second most simple case with no user-header but the user-payload alignment
// exceeds the ChunkHeader alignment and is therefore not necessarily adjacent
uint64_t addressOfChunkHeader = reinterpret_cast<uint64_t>(this);
uint64_t headerEndAddress = addressOfChunkHeader + sizeof(ChunkHeader);
uint64_t alignedUserPayloadAddress =
iox::cxx::align(headerEndAddress, static_cast<uint64_t>(userPayloadAlignment));
uint64_t offsetToUserPayload = alignedUserPayloadAddress - addressOfChunkHeader;
...
m_userPayloadOffset = static_cast<UserPayloadOffset_t>(offsetToUserPayload);

// this is safe since the alignment of the user-payload is larger than the one from the ChunkHeader
...
auto addressOfBackOffset = alignedUserPayloadAddress - sizeof(UserPayloadOffset_t);
auto backOffset = reinterpret_cast<UserPayloadOffset_t*>(addressOfBackOffset);
*backOffset = m_userPayloadOffset;
}
}

布局图(最复杂情形,带 user-header 与自定义对齐):

1
2
3
4
5
chunk 起始(8 字节对齐)
┌───────────────────┬──────────────┬─── padding ───┬────────────┬──────────────────┐
│ ChunkHeader (32B) │ user-header │ (对齐填充) │ backOffset │ user-payload │
└───────────────────┴──────────────┴───────────────┴─────4B─────┴──────────────────┘
↑ 总是紧邻 ChunkHeader ↑ 按 userPayloadAlignment 对齐

back-offset 是紧贴 payload 前面的 4 字节,存 payload→ChunkHeader 的偏移。这样 ChunkHeader::fromUserPayload() 只需读 payload 前 4 字节就能 O(1) 反查头部——这正是 Untyped API 里用户只持有 payload 指针也能 release/publish 的原因:

1
2
3
4
5
6
7
8
9
10
11
12
ChunkHeader* ChunkHeader::fromUserPayload(void* const userPayload) noexcept
{
if (userPayload == nullptr)
{
return nullptr;
}
uint64_t userPayloadAddress = reinterpret_cast<uint64_t>(userPayload);
// the back-offset is always stored in front of the user-payload, no matter if a user-header is used or not or if
// the user-payload has a custom alignment
auto backOffset = reinterpret_cast<UserPayloadOffset_t*>(userPayloadAddress - sizeof(UserPayloadOffset_t));
return reinterpret_cast<ChunkHeader*>(userPayloadAddress - *backOffset);
}

(无 user-header 且默认对齐时,m_userPayloadOffset 字段本身恰好位于 payload 前 4 字节,兼作 back-offset,不占额外空间。)

所需 chunk 大小由 ChunkSettings::create(userPayloadSize, userPayloadAlignment, userHeaderSize, userHeaderAlignment) 预先算出(mepoo/chunk_settings.hpp),并校验对齐是 2 的幂、user-header 对齐不超过 ChunkHeader 对齐等。

5. ChunkManagement 与 SharedChunk:跨进程引用计数

5.1 ChunkManagement

引用计数不放在 chunk 里,而是放在管理段的专用池中

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
struct ChunkManagement
{
using base_t = ChunkHeader;
using referenceCounterBase_t = uint64_t;
using referenceCounter_t = std::atomic<referenceCounterBase_t>;

ChunkManagement(const cxx::not_null<base_t*> chunkHeader,
const cxx::not_null<MemPool*> mempool,
const cxx::not_null<MemPool*> chunkManagementPool) noexcept;

iox::rp::RelativePointer<base_t> m_chunkHeader;
referenceCounter_t m_referenceCounter{1U};
/// @todo optimization: check if this can be replaced by an offset relative to the this pointer
iox::rp::RelativePointer<MemPool> m_mempool;
iox::rp::RelativePointer<MemPool> m_chunkManagementPool;
};

跨进程安全的三个要素:

  1. std::atomic<uint64_t> 计数器位于所有参与进程都以读写方式映射的管理段中,x86/ARM 上无锁原子(fetch_add/fetch_sub)跨进程有效——原子性由 CPU 缓存一致性保证,与进程无关,只要求同一物理内存。
  2. 所有指针都是 RelativePointer(segmentId + offset),各进程映射基址不同也能解引用。
  3. ChunkManagement 自身从 m_chunkManagementPool 分配,总数 = 所有 mempool 的 chunk 总数(generateChunkManagementPool,见下节),保证永不耗尽。

数据段可能对订阅方只读,但计数器在管理段——这就是”读者也能增减引用计数”的机制。

5.2 SharedChunk

SharedChunk 是持有 ChunkManagement* 的进程内智能句柄(类似 shared_ptr,但本身非线程安全,跨线程需各持副本):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
void SharedChunk::incrementReferenceCounter() noexcept
{
if (m_chunkManagement != nullptr)
{
m_chunkManagement->m_referenceCounter.fetch_add(1U, std::memory_order_relaxed);
}
}

void SharedChunk::decrementReferenceCounter() noexcept
{
if ((m_chunkManagement != nullptr)
&& (m_chunkManagement->m_referenceCounter.fetch_sub(1U, std::memory_order_relaxed) == 1U))
{
freeChunk();
}
}

void SharedChunk::freeChunk() noexcept
{
m_chunkManagement->m_mempool->freeChunk(m_chunkManagement->m_chunkHeader);
m_chunkManagement->m_chunkManagementPool->freeChunk(m_chunkManagement);
m_chunkManagement = nullptr;
}

计数归零时把 chunk 还给数据池、把 ChunkManagement 还给管理池,两次 freeChunk 都是无锁的 LoFFLi push。

派生形态 ShmSafeUnmanagedChunkshm_safe_unmanaged_chunk.hpp)把 ChunkManagement 的 RelativePointer 压缩进 64 位 RelativePointerData,可单周期写入、不会 torn write——队列(ChunkQueueData)、history、UsedChunkList 中存的都是它,即使应用写到一半崩溃,RouDi 也能读到一致值并完成回收。它「持有引用但不自动管理」:releaseToSharedChunk() 不增计数移交所有权,cloneToSharedChunk() 增计数复制。

6. MemoryManager:找最小合适 bucket

MemoryManager 管理一个段内的所有 MemPool。配置阶段要求 bucket 按 chunkSize 严格递增加入,最后生成 ChunkManagement 池并封锁再添加:

1
2
3
4
5
6
7
8
9
10
11
void MemoryManager::configureMemoryManager(const MePooConfig& mePooConfig,
posix::Allocator& managementAllocator,
posix::Allocator& chunkMemoryAllocator) noexcept
{
for (auto entry : mePooConfig.m_mempoolConfig)
{
addMemPool(managementAllocator, chunkMemoryAllocator, entry.m_size, entry.m_chunkCount);
}

generateChunkManagementPool(managementAllocator);
}

分配策略是线性扫描找第一个(即最小的)能装下 requiredChunkSize 的 bucket——因为有序,第一个命中即 best-fit;bucket 数很少(≤32),线性扫描比树查找更缓存友好:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
cxx::expected<SharedChunk, MemoryManager::Error> MemoryManager::getChunk(const ChunkSettings& chunkSettings) noexcept
{
void* chunk{nullptr};
MemPool* memPoolPointer{nullptr};
const auto requiredChunkSize = chunkSettings.requiredChunkSize();
...
for (auto& memPool : m_memPoolVector)
{
uint32_t chunkSizeOfMemPool = memPool.getChunkSize();
if (chunkSizeOfMemPool >= requiredChunkSize)
{
chunk = memPool.getChunk();
memPoolPointer = &memPool;
aquiredChunkSize = chunkSizeOfMemPool;
break;
}
}
...
else
{
auto chunkHeader = new (chunk) ChunkHeader(aquiredChunkSize, chunkSettings);
auto chunkManagement = new (m_chunkManagementPool.front().getChunk())
ChunkManagement(chunkHeader, memPoolPointer, &m_chunkManagementPool.front());
return cxx::success<SharedChunk>(SharedChunk(chunkManagement));
}
}

三种错误:NO_MEMPOOLS_AVAILABLE(没配任何池)、NO_MEMPOOL_FOR_REQUESTED_CHUNK_SIZE(消息比最大 bucket 还大,FATAL)、MEMPOOL_OUT_OF_CHUNKS(命中的 bucket 耗尽,MODERATE——注意不会降级到更大的 bucket,直接报错)。

内存需求计算(用于确定 shm 大小):

  • requiredChunkMemorySize = Σ chunkCount × (size + sizeof(ChunkHeader)),按 8 字节对齐;
  • requiredManagementMemorySize = Σ LoFFLi 索引表 + 总 chunk 数 × sizeof(ChunkManagement) + ChunkManagement 池自己的 LoFFLi。

7. TypedMemPool

TypedMemPool<T>internal/mepoo/typed_mem_pool.hpp/.inl)是给单一类型 T 用的独立小内存池:自带一个数据 MemPool(chunkSize 按 sizeof(T)+alignof(T)+ChunkHeader 算出)和一个配对的 ChunkManagement 池,createObject(args...) 在 chunk 里 placement-new 构造 T 并返回引用计数的 SharedPointer<T>

1
2
3
4
5
6
7
8
9
10
11
12
13
14
template <typename T>
template <typename... Targs>
inline cxx::expected<SharedPointer<T>, TypedMemPoolError> TypedMemPool<T>::createObject(Targs&&... args) noexcept
{
auto chunkManagement = acquireChunkManagementPointer();
if (chunkManagement.has_error())
{
return cxx::error<TypedMemPoolError>(chunkManagement.get_error());
}

auto newObject = SharedPointer<T>::create(SharedChunk(*chunkManagement), std::forward<Targs>(args)...);
...
return cxx::success<SharedPointer<T>>(newObject.value());
}

它不属于用户数据通路(pub/sub 用的是 MemoryManager),主要供 RouDi 在管理内存里放置需要共享的、带生命周期管理的对象。

8. 应用侧 attach 流程(PoshRuntime)

RouDi 与应用通过 IPC 通道(Unix Domain Socket / mq)握手,应用侧的 SharedMemoryUser 完成映射:

1
2
3
4
5
6
7
8
9
10
11
应用: PoshRuntime::initRuntime("app")
→ IpcRuntimeInterface: 发送 REG(runtimeName, pid, uid, ...) 给 RouDi
→ RouDi: 记录进程,回复 REG_ACK(shmTopicSize, segmentManagerOffset,
timestamp, mgmtSegmentId)
→ 应用: SharedMemoryUser 构造
① shm_open("iceoryx_mgmt", OPEN_EXISTING, READ_WRITE) 映射管理段
registerPtr(mgmtSegmentId, 基址) ← relative pointer 登记
② 由 (segmentId, offset) 解出 SegmentManager*
③ segmentManager->getSegmentMappings(当前进程用户)
④ 逐段 shm_open(组名, OPEN_EXISTING, 可写?RW:RO) 映射数据段
registerPtr(每段的 segmentId, 基址)

REG_ACK 的解析在 ipc_runtime_interface.cppwaitForRegAck 读出 shm 大小、SegmentManager 偏移与 segmentId),映射在:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
SharedMemoryUser::SharedMemoryUser(const size_t topicSize,
const uint64_t segmentId,
const rp::BaseRelativePointer::offset_t segmentManagerAddressOffset) noexcept
{
// create and map the already existing shared memory region
posix::SharedMemoryObject::create(roudi::SHM_NAME,
topicSize,
posix::AccessMode::READ_WRITE,
posix::OpenMode::OPEN_EXISTING,
posix::SharedMemoryObject::NO_ADDRESS_HINT)
.and_then([this, segmentId, segmentManagerAddressOffset](auto& sharedMemoryObject) {
rp::BaseRelativePointer::registerPtr(
segmentId, sharedMemoryObject.getBaseAddress(), sharedMemoryObject.getSizeInBytes());
...
this->openDataSegments(segmentId, segmentManagerAddressOffset);
...
})
.or_else([](auto&) { errorHandler(Error::kPOSH__SHM_APP_MAPP_ERR); });
}

void SharedMemoryUser::openDataSegments(const uint64_t segmentId,
const rp::BaseRelativePointer::offset_t segmentManagerAddressOffset) noexcept
{
auto ptr = rp::BaseRelativePointer::getPtr(segmentId, segmentManagerAddressOffset);
auto segmentManager = reinterpret_cast<mepoo::SegmentManager<>*>(ptr);

auto segmentMapping = segmentManager->getSegmentMappings(posix::PosixUser::getUserOfCurrentProcess());

各进程映射基址可以不同(NO_ADDRESS_HINT,由 OS 决定)——一切跨进程数据结构都只存 (segmentId, offset),这是 iceoryx 不要求相同虚拟地址的关键。

9. chunk 生命周期状态图

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
                 ┌────────────────────────────────────────────────┐
│ MemPool free-list (LoFFLi) │
└────────────────────────────────────────────────┘
│ getChunk() (pop index) ▲
▼ │ refCount 1→0:
[ALLOCATED / LOANED] │ freeChunk()
publisher loan(): placement-new ChunkHeader, │
ChunkManagement(refCount=1),登记进 │
publisher 的 UsedChunkList │
│ send() │
▼ │
[DELIVERED] │
ChunkDistributor 推入每个订阅者队列(每队列 +1 ref), │
写入 history(+1 ref),发送方 UsedChunkList 移除(-1) │
│ │ │
▼ ▼ │
[IN QUEUE] [IN HISTORY]────释放/被挤出─────┤
订阅者 take(): │
出队进入其 UsedChunkList │
│ │
▼ │
[HELD BY SUBSCRIBER] ──release()(Sample 析构)────────┘

队列溢出(DISCARD_OLDEST_DATA):被挤出的旧 chunk 直接 -1 ref
应用崩溃:RouDi 遍历该进程 port 的 UsedChunkList/队列/history 强制归还

引用计数的每个持有点:发送方 UsedChunkList、每个订阅者队列槽、distributor history、订阅者 UsedChunkList、以及进程内的每个 SharedChunk/Sample 副本。

10. 配置与容量参数

10.1 默认 MePooConfig

未提供配置时使用 MePooConfig::setDefaults()

1
2
3
4
5
6
7
8
9
10
11
12
MePooConfig& MePooConfig::setDefaults() noexcept
{
m_mempoolConfig.push_back({128, 10000});
m_mempoolConfig.push_back({1024, 5000});
m_mempoolConfig.push_back({1024 * 16, 1000});
m_mempoolConfig.push_back({1024 * 128, 200});
m_mempoolConfig.push_back({1024 * 512, 50});
m_mempoolConfig.push_back({1024 * 1024, 30});
m_mempoolConfig.push_back({1024 * 1024 * 4, 10});

return *this;
}
bucket(payload 字节) chunk 数 数据内存约
128 10000 1.5 MB
1 KB 5000 5.2 MB
16 KB 1000 16 MB
128 KB 200 25.6 MB
512 KB 50 25.6 MB
1 MB 30 30 MB
4 MB 10 40 MB

(实际每 chunk 另加 32B ChunkHeader;总计约 144 MB 数据段。)MePooConfig::optimize() 会排序并合并相同 size 的条目。

10.2 RouDi TOML 配置

RouDi 可用 -c 指定 TOML(示例 iceoryx_posh/etc/iceoryx/roudi_config_example.toml):

1
2
3
4
5
6
7
8
9
10
11
12
13
[general]
version = 1

[[segment]] # 可带 reader = "组名" / writer = "组名",缺省为启动 RouDi 的用户组

[[segment.mempool]]
size = 128
count = 10000

[[segment.mempool]]
size = 1024
count = 5000
# ... 与默认配置相同的 7 档

每个 [[segment]] 对应一个 SegmentConfig::SegmentEntry(reader 组、writer 组、一份 MePooConfig),见 mepoo/segment_config.hpp

10.3 编译期容量常量

来自 iceoryx_posh_types.hppcmake/IceoryxPoshDeployment.cmake(可用 -DIOX_MAX_* 覆盖):

常量 默认值 说明
MAX_NUMBER_OF_MEMPOOLS 32 每段最多 bucket 数
MAX_SHM_SEGMENTS 100 最多共享内存段数
IOX_MAX_PUBLISHERS 512 全系统 publisher port 上限
IOX_MAX_SUBSCRIBERS 1024 全系统 subscriber port 上限
IOX_MAX_CHUNKS_ALLOCATED_PER_PUBLISHER_SIMULTANEOUSLY 8 单 publisher 同时 loan 的 chunk 数
IOX_MAX_PUBLISHER_HISTORY 16 history 容量上限
IOX_MAX_CHUNKS_HELD_PER_SUBSCRIBER_SIMULTANEOUSLY 256 单 subscriber 同时持有的 chunk 数(=队列容量上限)
MAX_PROCESS_NUMBER 300 可注册进程数
CHUNK_DEFAULT_USER_PAYLOAD_ALIGNMENT 8 默认 payload 对齐

下一篇:03-popo发布订阅与通信原语.md

03 popo:发布订阅与通信原语

03 popo:发布订阅与通信原语

源码锚点(iceoryx 2.0.6,根目录 /home/cp/work2/ros2Learn/ros2_humble/src/eclipse-iceoryx/iceoryx):
用户 API:iceoryx_posh/include/iceoryx_posh/popo/publisher.hppsubscriber.hppsample.hppwait_set.hpplistener.hppclient.hppserver.hpp…)
Port 层:.../internal/popo/ports/;building blocks:.../internal/popo/building_blocks/
无锁队列:iceoryx_hoofs/.../concurrent/sofi.hppfifo.hppresizeable_lockfree_queue.hpp

popo(Port Port / publish-subscribe)在 mepoo 之上实现了完整的通信语义:typed/untyped 发布订阅、多订阅者分发、事件通知(WaitSet/Listener)以及 2.0 新增的 request/response。

1. 分层总览

1
2
3
4
5
6
7
8
9
10
11
12
用户 API 层     Publisher<T,H> / Subscriber<T,H>        UntypedPublisher / UntypedSubscriber
loan()/publish() → Sample<T> RAII loan(size)/publish(payload*)
│ │
Base 层 BasePublisher / BaseSubscriber(持 PortUser、TriggerHandle,接 WaitSet/Listener)

Port 用户侧 PublisherPortUser ── ChunkSender ── ChunkDistributor (发送路径)
SubscriberPortUser ── ChunkReceiver ── ChunkQueuePopper (接收路径)
│ 数据全部位于共享内存 ▼
共享内存数据 PublisherPortData{ChunkSenderData} SubscriberPortData{ChunkReceiverData=ChunkQueueData+UsedChunkList}

Port RouDi 侧 PublisherPortRouDi / SubscriberPortRouDi —— RouDi 处理 CaPro
消息(OFFER/SUB…),把订阅者队列指针挂进 publisher 的 ChunkDistributor

同一份 PortData 被两个进程以不同视图操作:应用进程用 *PortUser,RouDi 用 *PortRouDi——这是 iceoryx「端口数据放共享内存、逻辑放各自进程」的核心模式。

2. Typed / Untyped API 与 Sample RAII

2.1 Typed:loan / publish

1
2
3
4
5
6
template <typename T, typename H = mepoo::NoUserHeader>
class Publisher : public PublisherImpl<T, H>
{
public:
using PublisherImpl<T, H>::PublisherImpl;
};

PublisherImpl 提供三种发布方式(internal/popo/publisher_impl.hpp):

  • loan(Args&&...):按 sizeof(T)/alignof(T) 从端口 loan 一个 chunk 并就地构造 T,返回 cxx::expected<Sample<T,H>, AllocationError>
  • publish(Sample<T,H>&&):发布并交还所有权;
  • publishCopyOf(const T&) / publishResultOf(callable):便捷封装。

Sample<T,H> 是 RAII 句柄(基于 SmartChunk,内部是带自定义 deleter 的 cxx::unique_ptr<T>):析构未发布的 Sample 会自动 release chunk 回内存池;publish() 后释放所有权、不再触发 deleter

1
2
3
4
5
6
7
8
9
10
11
/// @brief The Sample class is a mutable abstraction over types which are written to loaned shared memory.
/// These samples are publishable to the iceoryx system.
template <typename T, typename H = cxx::add_const_conditionally_t<mepoo::NoUserHeader, T>>
class Sample : public SmartChunk<PublisherInterface<T, H>, T, H>
{
...
/// @brief Publish the sample via the publisher from which it was loaned and automatically
/// release ownership to it.
/// @details Only available for non-const type T.
template <typename S = T, typename = ForPublisherOnly<S, T>>
void publish() noexcept;

订阅侧对偶:Subscriber<T,H>::take() 返回 cxx::expected<Sample<const T, const H>, ChunkReceiveResult>——const T 保证订阅者不改共享数据,Sample 析构时自动 releaseChunk(引用计数 −1)。

2.2 Untyped

UntypedPublisher::loan(payloadSize, alignment...) 返回裸 void*(payload 指针),publish(void*) 内部用 ChunkHeader::fromUserPayload()(见 02 篇 back-offset)反查头部;UntypedSubscriber::take() 返回 const void*,需手动 release。适合网关等运行期才知道大小的场景。

user-header H(如内置的 RequestHeader、或用户自定义时间戳头)通过 ChunkSettings 参与 chunk 布局,用 Sample::getUserHeader() 访问。

3. Port 体系

3.1 PortData:放在共享内存里的状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
struct PublisherPortData : public BasePortData
{
PublisherPortData(const capro::ServiceDescription& serviceDescription,
const RuntimeName_t& runtimeName,
mepoo::MemoryManager* const memoryManager,
const PublisherOptions& publisherOptions,
const mepoo::MemoryInfo& memoryInfo = mepoo::MemoryInfo()) noexcept;

using ChunkQueueData_t = SubscriberPortData::ChunkQueueData_t;
using ChunkDistributorData_t =
ChunkDistributorData<DefaultChunkDistributorConfig, ThreadSafePolicy, ChunkQueuePusher<ChunkQueueData_t>>;
using ChunkSenderData_t =
ChunkSenderData<MAX_CHUNKS_ALLOCATED_PER_PUBLISHER_SIMULTANEOUSLY, ChunkDistributorData_t>;

ChunkSenderData_t m_chunkSenderData;

PublisherOptions m_options;

std::atomic_bool m_offeringRequested{false};
std::atomic_bool m_offered{false};
};

SubscriberPortData 对应持有 ChunkReceiverData_t(= ChunkQueueData + UsedChunkList)和 m_subscriptionStateSubscribeState 状态机:NOT_SUBSCRIBED → SUBSCRIBE_REQUESTED → SUBSCRIBED → …,见 iceoryx_posh_types.hpp)。这些 PortData 由 RouDi 在管理段分配,应用经 IPC 拿到 offset 后用 PublisherPortUser/SubscriberPortUser 包装。

3.2 User 侧与 RouDi 侧

PublisherPortUser 的 API 就是薄薄一层 ChunkSender 代理:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
/// @brief Allocate a chunk, the ownership of the SharedChunk remains in the PublisherPortUser for being able to
/// cleanup if the user process disappears
...
cxx::expected<mepoo::ChunkHeader*, AllocationError>
tryAllocateChunk(const uint32_t userPayloadSize,
const uint32_t userPayloadAlignment,
const uint32_t userHeaderSize = 0U,
const uint32_t userHeaderAlignment = 1U) noexcept;
...
void sendChunk(mepoo::ChunkHeader* const chunkHeader) noexcept;
...
void offer() noexcept;
...
bool hasSubscribers() const noexcept;

offer()/subscribe() 只是置位 m_offeringRequested/m_subscribeRequested 原子标志;RouDi 的发现循环通过 PublisherPortRouDi::tryGetCaProMessage() 轮询到变化,生成 CaPro(Canonical Protocol)消息做匹配,匹配成功后由 PublisherPortRouDi 调用 ChunkDistributor::tryAddQueue(订阅者的 ChunkQueueData*, historyRequest) 完成接线。订阅侧有两种 RouDi 策略类:SubscriberPortSingleProducer(1:n)与 SubscriberPortMultiProducer(n:m),由编译期 build::CommunicationPolicy 选择(默认 ManyToManyPolicy,见 cmake/iceoryx_posh_deployment.hpp.in)。

4. 数据路径 building blocks

4.1 ChunkSender:loan 记账 + 序号

ChunkSenderbuilding_blocks/chunk_sender.hpp/.inl)扩展 ChunkDistributortryAllocateMemoryManager 取 chunk(并做小优化:若上一个 chunk 仅自己持有且大小合适则复用 m_lastChunkUnmanaged),登记进 UsedChunkList(容量 = 每 publisher 同时 loan 上限,默认 8);发送时摘除并盖序号:

1
2
3
4
5
6
7
8
inline bool ChunkSender<ChunkSenderDataType>::getChunkReadyForSend(const mepoo::ChunkHeader* const chunkHeader,
mepoo::SharedChunk& chunk) noexcept
{
if (getMembers()->m_chunksInUse.remove(chunkHeader, chunk))
{
chunk.getChunkHeader()->setSequenceNumber(getMembers()->m_sequenceNumber++);
return true;
}

UsedChunkListinternal/popo/used_chunk_list.hpp)是为「应用随时可能死掉」设计的记账结构:定长数组 + 64 位 ShmSafeUnmanagedChunk 元素,写入单周期完成、无 torn write,RouDi 清理时可安全遍历。

4.2 ChunkDistributor:多订阅者分发与 history

ChunkDistributor 持有订阅者队列列表和 history 环(ChunkDistributorDataMAX_QUEUES = MAX_SUBSCRIBERS_PER_PUBLISHER = 256MAX_HISTORY_CAPACITY = 16)。deliverToAllStoredQueues 是发布的核心,实现了队列满两种策略:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
template <typename ChunkDistributorDataType>
inline uint64_t ChunkDistributor<ChunkDistributorDataType>::deliverToAllStoredQueues(mepoo::SharedChunk chunk) noexcept
{
uint64_t numberOfQueuesTheChunkWasDeliveredTo{0U};
typename ChunkDistributorDataType::QueueContainer_t remainingQueues;
{
typename MemberType_t::LockGuard_t lock(*getMembers());

bool willWaitForConsumer = getMembers()->m_consumerTooSlowPolicy == ConsumerTooSlowPolicy::WAIT_FOR_CONSUMER;
// send to all the queues
for (auto& queue : getMembers()->m_queues)
{
bool isBlockingQueue = (willWaitForConsumer && queue->m_queueFullPolicy == QueueFullPolicy::BLOCK_PRODUCER);

if (pushToQueue(queue.get(), chunk))
{
++numberOfQueuesTheChunkWasDeliveredTo;
}
else
{
if (isBlockingQueue)
{
remainingQueues.emplace_back(queue);
}
else
{
++numberOfQueuesTheChunkWasDeliveredTo;
ChunkQueuePusher_t(queue.get()).lostAChunk();
}
}
}
}

// busy waiting until every queue is served
while (!remainingQueues.empty())
{
std::this_thread::yield();
...
}

addToHistoryWithoutDelivery(chunk);

return numberOfQueuesTheChunkWasDeliveredTo;
}
  • 推送本质是拷贝一个 64 位 ShmSafeUnmanagedChunk(引用计数 +1),不拷贝数据
  • history 环满时先释放最旧(addToHistoryWithoutDelivery),新订阅者接入时 tryAddQueue 把最近 requestedHistory 条补发给它——即 ROS latched / DDS TRANSIENT_LOCAL 的等价物;
  • 加锁的是跨进程互斥量ThreadSafePolicybuilding_blocks/locking_policy.hpp),只保护队列列表/history 容器,真正的数据入队是无锁的。

队列满策略popo/port_queue_policies.hpp)由双方协商:

订阅者 QueueFullPolicy 发布者 ConsumerTooSlowPolicy 行为
DISCARD_OLDEST_DATA(默认) 任意 SoFi 溢出挤掉最旧样本,lostAChunk()m_queueHasLostChunks 供订阅者查询
BLOCK_PRODUCER WAIT_FOR_CONSUMER 发布者在上面的 while (!remainingQueues.empty()) 中 yield 忙等直至队列有空位(真正的背压,牺牲实时性)
BLOCK_PRODUCER DISCARD_OLDEST_DATA(默认) 不兼容,RouDi 匹配时拒绝连接(NACK)

4.3 ChunkQueue 与无锁队列(SoFi)

订阅者队列 ChunkQueueData 的存储是 cxx::VariantQueue

1
2
3
4
5
6
7
8
9
10
11
12
13
14
template <typename ChunkQueueDataProperties, typename LockingPolicy>
struct ChunkQueueData : public LockingPolicy
{
...
cxx::UniqueId m_uniqueId{};

static constexpr uint64_t MAX_CAPACITY = ChunkQueueDataProperties_t::MAX_QUEUE_CAPACITY;
cxx::VariantQueue<mepoo::ShmSafeUnmanagedChunk, MAX_CAPACITY> m_queue;
std::atomic_bool m_queueHasLostChunks{false};

rp::RelativePointer<ConditionVariableData> m_conditionVariableDataPtr;
cxx::optional<uint64_t> m_conditionVariableNotificationIndex;
const QueueFullPolicy m_queueFullPolicy;
};

VariantQueueiceoryx_hoofs/cxx/variant_queue.hpp)可在四种实现间选择:FiFo_SingleProducerSingleConsumerSoFi_SingleProducerSingleConsumerFiFo/SoFi_MultiProducerSingleConsumer(后两者映射到 ResizeableLockFreeQueue)。pub/sub 默认用 SoFiSafely overflowing FiFo)——无锁、溢出时返回被挤出的最旧元素而不是失败:

1
2
3
4
5
6
7
8
/// @brief
/// Thread safe producer and consumer queue with a safe overflowing behavior.
/// SoFi is designed in a FIFO Manner but prevents data loss when pushing into
/// a full SoFi. When SoFi is full and a Sender tries to push, the data at the
/// current read position will be returned. SoFi is a Thread safe without using
/// locks. ...
template <class ValueType, uint64_t CapacityValue>
class SoFi

ChunkQueuePusher::push 展示了溢出样本如何归还引用计数、以及入队后如何跨进程唤醒等待者:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
template <typename ChunkQueueDataType>
inline bool ChunkQueuePusher<ChunkQueueDataType>::push(mepoo::SharedChunk chunk) noexcept
{
auto pushRet = getMembers()->m_queue.push(chunk);
bool hasQueueOverflow = false;

// drop the chunk if one is returned by an overflow
if (pushRet.has_value())
{
pushRet.value().releaseToSharedChunk();
// tell the ChunkDistributor that we had an overflow and dropped a sample
hasQueueOverflow = true;
}

{
typename MemberType_t::LockGuard_t lock(*getMembers());
if (getMembers()->m_conditionVariableDataPtr)
{
ConditionNotifier(*getMembers()->m_conditionVariableDataPtr.get(),
*getMembers()->m_conditionVariableNotificationIndex)
.notify();
}
}

return !hasQueueOverflow;
}

4.4 ChunkReceiver

ChunkReceiver = ChunkQueuePopper + UsedChunkListtryGet() 出队后把 chunk 记入订阅者自己的 UsedChunkList(容量 = MAX_CHUNKS_HELD_PER_SUBSCRIBER_SIMULTANEOUSLY + 1 = 257,多出的 1 个允许用户在已满持有时先拿新再还旧,与 ara::com 对齐,见 chunk_receiver_data.hpp 注释);release() 移除并减引用。队列容量与持有上限被刻意设为相等(MAX_SUBSCRIBER_QUEUE_CAPACITY = MAX_CHUNKS_HELD_PER_SUBSCRIBER_SIMULTANEOUSLY),保证轮询用户一次能吞下整条满队列。

5. WaitSet 与 Listener:跨进程事件机制

5.1 ConditionVariable:共享内存”条件变量”

iceoryx 的跨进程通知原语不是 pthread condvar,而是POSIX 信号量 + 通知位图

1
2
3
4
5
6
7
8
9
10
struct ConditionVariableData
{
...
posix::Semaphore m_semaphore =
std::move(posix::Semaphore::create(posix::CreateUnnamedSharedMemorySemaphore, 0U)
...
RuntimeName_t m_runtimeName;
std::atomic_bool m_toBeDestroyed{false};
std::atomic_bool m_activeNotifications[MAX_NUMBER_OF_NOTIFIERS];
};

ConditionVariableData 由 RouDi 在管理段分配(每个 WaitSet/Listener 一个,上限 MAX_NUMBER_OF_CONDITION_VARIABLES = 1024)。生产者侧 ConditionNotifier(condVar, index).notify():先 m_activeNotifications[index] = truesem_post;消费者侧 ConditionListener::wait()sem_wait 醒来后扫描位图、收集并清零所有已置位的 index,返回排序后的通知索引向量(condition_listener.hpp/condition_notifier.hpp)。信号量创建在共享内存中(unnamed + pshared),因此天然跨进程。

订阅者队列通过 setConditionVariable(condVar, notificationIndex) 挂接(见 4.3 中 m_conditionVariableDataPtr),一个 condvar 用不同 index 区分最多 MAX_NUMBER_OF_NOTIFIERS = 256 个事件源。

5.2 WaitSet

1
2
3
4
5
6
7
8
/// @brief Logical disjunction of a certain number of Triggers
///
/// The WaitSet stores Triggers and allows the user to wait till one or more of those Triggers are triggered. It works
/// over process borders. With the creation of a WaitSet it requests a condition variable from RouDi and destroys it
/// with the destructor. Hence the lifetime of the condition variable is bound to the lifetime of the WaitSet.
/// @param[in] Capacity the amount of events/states which can be attached to the waitset
template <uint64_t Capacity = MAX_NUMBER_OF_ATTACHMENTS_PER_WAITSET>
class WaitSet

关键概念——事件(event)vs 状态(state)

  • attachEvent(subscriber, SubscriberEvent::DATA_RECEIVED):边沿触发。只在 notify() 发生时唤醒一次;若 wait 期间没有新 push,即使队列里还有旧数据也不会再报。
  • attachState(subscriber, SubscriberState::HAS_DATA):电平触发。每次 wait 返回前用 WaitSetIsConditionSatisfiedCallback(如 hasNewChunks())复查条件,只要队列非空就持续报告——不会因为”一次唤醒处理多条中只 take 了一条”而丢事件。

attach 时 WaitSet 生成 Trigger 并向源对象(subscriber 等)交付 TriggerHandle——包含 condvar 指针、唯一 trigger id 和 reset 回调;源对象触发即 TriggerHandle::trigger()ConditionNotifier::notify()TriggerHandle 析构时自动从 WaitSet 反注册(双向生命周期安全,popo/trigger_handle.hpp)。wait()/timedWait() 返回 NotificationInfoVector,可携带用户 id 与回调。

5.3 Listener

Listenerpopo/listener.hpp)= 一个 condvar + 内部线程threadLoopConditionListener::wait(),对每个激活 index 并发执行注册的回调(attachEvent(obj, event, createNotificationCallback(cb)))。与 WaitSet 的区别:Listener 是推模式、只支持 event(电平语义的 state 无法映射到”回调一次”模型)、完全线程安全;WaitSet 是拉模式,用户自己控制在哪个线程处理。两者共享 MAX_NUMBER_OF_NOTIFIERS = 256 个通知槽。

6. Request/Response(2.0 新增)

client/server 复用同一套 building blocks,每侧同时有发送与接收能力(internal/popo/ports/client_server_port_types.hpp):

1
2
3
4
5
6
7
8
9
10
11
12
13
using ClientChunkQueueData_t = ChunkQueueData<ClientChunkQueueConfig, ThreadSafePolicy>;

using ServerChunkQueueData_t = ChunkQueueData<ServerChunkQueueConfig, ThreadSafePolicy>;

using ClientChunkDistributorData_t =
ChunkDistributorData<ClientChunkDistributorConfig, ThreadSafePolicy, ChunkQueuePusher<ServerChunkQueueData_t>>;

using ServerChunkDistributorData_t =
ChunkDistributorData<ServerChunkDistributorConfig, ThreadSafePolicy, ChunkQueuePusher<ClientChunkQueueData_t>>;

using ClientChunkReceiverData_t = ChunkReceiverData<MAX_RESPONSES_PROCESSED_SIMULTANEOUSLY, ClientChunkQueueData_t>;

using ServerChunkReceiverData_t = ChunkReceiverData<MAX_REQUESTS_PROCESSED_SIMULTANEOUSLY, ServerChunkQueueData_t>;
  • ClientChunkSender(向 server 的请求队列推 request,distributor 只有 1 个队列)+ ChunkReceiver(响应队列,容量 16)。
  • ServerChunkReceiver(请求队列,容量 1024,可服务 MAX_CLIENTS_PER_SERVER = 256 个 client)+ ChunkSender(响应经 distributor 精确回投到发起请求的那个 client 队列)。

路由信息放在内置 user-header RequestHeader/ResponseHeaderpopo/rpc_header.hpp)里:

1
2
3
4
5
protected:
uint8_t m_rpcHeaderVersion{RPC_HEADER_VERSION};
uint32_t m_lastKnownClientQueueIndex{UNKNOWN_CLIENT_QUEUE_INDEX};
cxx::UniqueId m_uniqueClientQueueId;
int64_t m_sequenceId{0};

请求携带 client 响应队列的 UniqueId 与索引提示,server 发响应时调 ChunkSender::sendToQueue(chunkHeader, uniqueClientQueueId, lastKnownClientQueueIndex)getQueueIndex 先试提示索引、失败再线性查)。sequenceId 由用户设置/校验以匹配乱序响应;ResponseHeader 另有 setServerError() 标志。用户 API:typed Client<Req,Res>/Server<Req,Res>loan → send → take)与 untyped 版本,事件枚举 ClientEvent::RESPONSE_RECEIVED/ServerEvent::REQUEST_RECEIVED 可挂 WaitSet/Listener。

连接状态机为 ConnectionStateNOT_CONNECTED → CONNECT_REQUESTED → CONNECTED → …iceoryx_posh_types.hpp)。

7. 与 DDS QoS 概念对照

DDS QoS / 概念 iceoryx 2.0 对应 备注
RELIABILITY RELIABLE QueueFullPolicy::BLOCK_PRODUCER + ConsumerTooSlowPolicy::WAIT_FOR_CONSUMER 进程内不丢包,队列满时阻塞发布者(忙等)
RELIABILITY BEST_EFFORT DISCARD_OLDEST_DATA(默认) 溢出挤掉最旧样本,hasLostChunksSinceLastCall() 可检测
HISTORY KEEP_LAST(n) 订阅队列容量 SubscriberOptions::queueCapacity(≤256) 接收侧 history
DURABILITY TRANSIENT_LOCAL PublisherOptions::historyCapacity(≤16)+ SubscriberOptions::historyRequest 迟到订阅者补发最近 n 条
DURABILITY VOLATILE historyCapacity = 0(默认)
DEADLINE / LIVELINESS 无直接对应 RouDi 有进程级 keep-alive 监控,非 per-topic;待验证细节
PARTITION / DOMAIN ServiceDescription{Service, Instance, Event} 三元组 精确匹配,无通配 QoS 协商
OWNERSHIP EXCLUSIVE 无;OneToManyPolicy 编译期限制单发布者 多 publisher 同 topic 在 ManyToManyPolicy 下自由并存
LATENCY_BUDGET 无(零拷贝路径延迟本身为 µs 级)
RPC(DDS-RPC / ROS service) Client/Server port + RequestHeader/ResponseHeader 2.0 新增
Listener / WaitSet(DDS 实体) popo::Listener / popo::WaitSet 语义高度对应:StatusCondition ≈ attachState,事件回调 ≈ Listener

注意:这些策略在 RouDi 做订阅匹配时校验兼容性(如 BLOCK_PRODUCER 订阅者遇到 DISCARD_OLDEST_DATA 发布者会被拒绝),行为上类似 DDS 的 QoS RxO(requested vs offered)检查。


下一篇:04-RouDi守护进程与服务发现.md

04 RouDi 守护进程与服务发现

04 RouDi 守护进程与服务发现

RouDi:iceoryx_posh/source/roudi/;运行时:iceoryx_posh/source/runtime/;CaPro:iceoryx_posh/source/capro/
源码根:ros2_humble/src/eclipse-iceoryx/iceoryx(版本 2.0.6)

1. RouDi 的职责与定位

RouDi(Routing and Discovery)是 iceoryx 的中心守护进程,一台机器(或一个 domain)上只运行一个。它负责:

职责 实现
创建/销毁共享内存段 RouDiMemoryManager + MemoryProvider
进程注册与生命周期监控 ProcessManager(心跳超时清理)
端口(Publisher/Subscriber/Client/Server)的分配与销毁 PortManager + PortPool
服务发现路由(CaPro 协议状态机的”邮差”) PortManager::doDiscovery()
内省(introspection)数据发布 内置 topic

关键设计:RouDi 不在数据路径上。数据 chunk 从发布者到订阅者的传递完全发生在应用进程之间的共享内存里(见 05 篇);RouDi 只在”控制面”工作——建立/拆除连接、分配资源。RouDi 崩溃不会中断已建立连接上的数据传输(但无法再建新连接、新进程无法注册)。

2. 启动流程

2.1 入口:iox-roudi 可执行文件

roudi_main.cppiox-roudi 的 main:解析命令行 → 解析 TOML 配置 → 构造 IceOryxRouDiApprun()

1
2
3
4
5
6
7
8
9
10
11
12
13
14
int main(int argc, char* argv[]) noexcept
{
using iox::roudi::IceOryxRouDiApp;

iox::config::CmdLineParserConfigFileOption cmdLineParser;
auto cmdLineArgs = cmdLineParser.parse(argc, argv);
...
iox::config::TomlRouDiConfigFileProvider configFileProvider(cmdLineArgs.value());

auto roudiConfig = configFileProvider.parse();
...
IceOryxRouDiApp roudi(cmdLineArgs.value(), roudiConfig.value());
return roudi.run();
}

2.2 RouDiApp / IceOryxRouDiApp

基类 RouDiAppapplication/roudi_app.cpp)处理信号(SIGINT/SIGTERM 通过信号量唤醒主线程退出)、日志级别、monitoring 模式、配置校验(段/mempool 不能为空)。派生类 IceOryxRouDiApp::run() 创建两个核心对象后阻塞等待退出信号:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
uint8_t IceOryxRouDiApp::run() noexcept
{
if (m_run)
{
static cxx::optional<IceOryxRouDiComponents> m_rouDiComponents;
auto componentsScopeGuard = cxx::makeScopedStatic(m_rouDiComponents, m_config);

static cxx::optional<RouDi> roudi;
auto roudiScopeGuard =
cxx::makeScopedStatic(roudi,
m_rouDiComponents.value().rouDiMemoryManager,
m_rouDiComponents.value().portManager,
RouDi::RoudiStartupParameters{m_monitoringMode,
true,
RouDi::RuntimeMessagesThreadStart::IMMEDIATE,
m_compatibilityCheckLevel,
m_processKillDelay});
waitForSignal();
}
return EXIT_SUCCESS;
}

IceOryxRouDiComponents 的构造顺序值得注意:先清理遗留的 RouDi IPC channel,再创建并”宣布”共享内存,最后才构造 PortManager

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
IceOryxRouDiComponents::IceOryxRouDiComponents(const RouDiConfig_t& roudiConfig) noexcept
: rouDiMemoryManager(roudiConfig)
, portManager([&]() -> IceOryxRouDiMemoryManager* {
// this temporary object will create a roudi IPC channel
// and close it immediatelly
// if there was an outdated roudi IPC channel, it will be cleaned up
// if there is an outdated IPC channel, the start of the apps will be terminated
runtime::IpcInterfaceBase::cleanupOutdatedIpcChannel(roudi::IPC_CHANNEL_ROUDI_NAME);

rouDiMemoryManager.createAndAnnounceMemory().or_else([](RouDiMemoryManagerError error) {
LogFatal() << "Could not create SharedMemory! Error: " << error;
errorHandler(Error::kROUDI_COMPONENTS__SHARED_MEMORY_UNAVAILABLE, nullptr, iox::ErrorLevel::FATAL);
});
return &rouDiMemoryManager;
}())
{
}

2.3 内存体系:MemoryProvider / MemoryBlock / RouDiMemoryManager

RouDi 的内存创建采用两级抽象(roudi/memory/ 目录):

  • MemoryBlock:一段有 size/alignment 需求的逻辑内存块,如 PortPoolMemoryBlock(端口池)、MemPoolCollectionMemoryBlock(introspection mempool)、MemPoolSegmentManagerMemoryBlockSegmentManager,管理所有用户数据段)。
  • MemoryProvider:把若干 MemoryBlock 汇总,计算总大小并实际创建内存。POSIX 实现是 PosixShmMemoryProvider,基于 shm_open + mmap
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
cxx::expected<void*, MemoryProviderError> PosixShmMemoryProvider::createMemory(const uint64_t size,
const uint64_t alignment) noexcept
{
if (alignment > posix::pageSize())
{
return cxx::error<MemoryProviderError>(MemoryProviderError::MEMORY_ALIGNMENT_EXCEEDS_PAGE_SIZE);
}

posix::SharedMemoryObject::create(m_shmName, size, m_accessMode, m_openMode, nullptr)
.and_then([this](auto& sharedMemoryObject) {
sharedMemoryObject.finalizeAllocation();
m_shmObject.emplace(std::move(sharedMemoryObject));
});
...
return cxx::success<void*>(baseAddress);
}

DefaultRouDiMemory 组装出管理段(名字为 iceoryx_mgmt,见 iceoryx_posh_types.hppSHM_NAME),它容纳 introspection mempool 与 SegmentManager

1
2
3
4
5
6
7
8
9
10
11
12
13
14
DefaultRouDiMemory::DefaultRouDiMemory(const RouDiConfig_t& roudiConfig) noexcept
: m_introspectionMemPoolBlock(introspectionMemPoolConfig())
, m_segmentManagerBlock(roudiConfig)
, m_managementShm(SHM_NAME, posix::AccessMode::READ_WRITE, posix::OpenMode::PURGE_AND_CREATE)
{
m_managementShm.addMemoryBlock(&m_introspectionMemPoolBlock).or_else([](auto) {
errorHandler(
Error::kROUDI__DEFAULT_ROUDI_MEMORY_FAILED_TO_ADD_INTROSPECTION_MEMORY_BLOCK, nullptr, ErrorLevel::FATAL);
});
m_managementShm.addMemoryBlock(&m_segmentManagerBlock).or_else([](auto) {
errorHandler(
Error::kROUDI__DEFAULT_ROUDI_MEMORY_FAILED_TO_ADD_SEGMENT_MANAGER_MEMORY_BLOCK, nullptr, ErrorLevel::FATAL);
});
}

RouDiMemoryManager::createAndAnnounceMemory() 逐个调用 provider 的 create(),全部成功后 announceMemoryAvailable()roudi/memory/roudi_memory_manager.cpp 第 65~89 行)。用户数据段(payload segment)由 SegmentManager/MePooSegment 按 TOML 配置逐段创建,每段有 reader/writer POSIX 用户组权限。

所以磁盘上可见的共享内存文件通常是:/dev/shm/iceoryx_mgmt + 每个用户组一个数据段(如 /dev/shm/<group>)。

2.4 RouDi 对象:两个线程

RouDi 构造函数启动两个线程:

1
2
3
4
5
6
7
8
// run the threads
m_monitoringAndDiscoveryThread = std::thread(&RouDi::monitorAndDiscoveryUpdate, this);
posix::setThreadName(m_monitoringAndDiscoveryThread.native_handle(), "Mon+Discover");

if (roudiStartupParameters.m_runtimesMessagesThreadStart == RuntimeMessagesThreadStart::IMMEDIATE)
{
startProcessRuntimeMessagesThread();
}
线程 周期 职责
Mon+Discover DISCOVERY_INTERVAL = 100 ms ProcessManager::run() → 心跳监控 + PortManager::doDiscovery()
IPC-msg-process 阻塞收消息 处理应用发来的 IPC 请求(注册、建端口等)

IPC 消息处理线程创建名为 roudi 的 IPC channel 并打印著名的 “RouDi is ready for clients”:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
void RouDi::processRuntimeMessages() noexcept
{
runtime::IpcInterfaceCreator roudiIpcInterface{IPC_CHANNEL_ROUDI_NAME};

// the logger is intentionally not used, to ensure that this message is always printed
std::cout << "RouDi is ready for clients" << std::endl;

while (m_runHandleRuntimeMessageThread)
{
// read RouDi's IPC channel
runtime::IpcMessage message;
if (roudiIpcInterface.timedReceive(m_runtimeMessagesThreadTimeout, message))
{
auto cmd = runtime::stringToIpcMessageType(message.getElementAtIndex(0).c_str());
std::string runtimeName = message.getElementAtIndex(1);

processMessage(message, cmd, RuntimeName_t(cxx::TruncateToCapacity, runtimeName));
}
}
}

3. 进程注册:PoshRuntime 与 RouDi 的 IPC channel

3.1 IPC channel 的实体

应用与 RouDi 之间的控制通道是按平台选择的(iceoryx_hoofs/platform/*/platform_settings.hpp):Linux/QNX/macOS 上是 Unix Domain Socketusing IoxIpcChannelType = iox::posix::UnixDomainSocket;,Linux 见 iceoryx_hoofs/platform/linux/include/iceoryx_hoofs/platform/platform_settings.hpp 第 39 行),Windows 上是 NamedPipe。历史版本用的 POSIX message queue 在 2.0 中仍保留实现(iceoryx_hoofs/internal/posix_wrapper/message_queue.hpp),但默认平台配置不再使用。

消息本身是 ASCII 文本、以分隔符拼接的字段序列(IpcMessage),收发封装在 IpcInterfaceBasesource/runtime/ipc_interface_base.cpp)中。通道拓扑是每进程一条:RouDi 有名为 roudi 的接收通道;每个应用按自己的 runtime 名字创建一条接收应答的通道。

3.2 应用侧:PoshRuntime::initRuntime()

应用第一行代码通常是 iox::runtime::PoshRuntime::initRuntime("app_name"),它构造单例 PoshRuntimeImpl

1
2
3
4
5
6
7
8
9
10
11
12
13
14
PoshRuntimeImpl::PoshRuntimeImpl(cxx::optional<const RuntimeName_t*> name, const RuntimeLocation location) noexcept
: PoshRuntime(name)
, m_ipcChannelInterface(roudi::IPC_CHANNEL_ROUDI_NAME, *name.value(), runtime::PROCESS_WAITING_FOR_ROUDI_TIMEOUT)
, m_ShmInterface([&] {
// in case the runtime is located in the same process like RouDi the shm is already opened;
...
return location == RuntimeLocation::SAME_PROCESS_LIKE_ROUDI
? cxx::nullopt
: cxx::optional<SharedMemoryUser>({m_ipcChannelInterface.getShmTopicSize(),
m_ipcChannelInterface.getSegmentId(),
m_ipcChannelInterface.getSegmentManagerAddressOffset()});
}())
{
}

两步:① IpcRuntimeInterface 完成注册握手;② SharedMemoryUser 用握手拿到的信息 mmap 共享内存。

3.3 注册握手(REG / REG_ACK)

IpcRuntimeInterface 构造函数是一个状态机:WAIT_FOR_ROUDI → SEND_REGISTER_REQUEST → WAIT_FOR_REGISTER_ACK → FINISHED,默认最多等待 RouDi PROCESS_WAITING_FOR_ROUDI_TIMEOUT = 60 s。注册请求携带进程名、pid、uid、时间戳与版本信息:

1
2
3
4
5
6
7
8
9
10
11
// send IpcMessageType::REG to RouDi

IpcMessage sendBuffer;
int pid = getpid();
cxx::Expects(pid >= 0);
sendBuffer << IpcMessageTypeToString(IpcMessageType::REG) << m_runtimeName << cxx::convert::toString(pid)
<< cxx::convert::toString(posix::PosixUser::getUserOfCurrentProcess().getID())
<< cxx::convert::toString(transmissionTimestamp)
<< static_cast<cxx::Serialization>(version::VersionInfo::getCurrentVersion()).toString();

bool successfullySent = m_RoudiIpcInterface.timedSend(sendBuffer, 100_ms);

RouDi 侧 ProcessManager::addProcess() 做版本兼容性检查、把进程加入 m_processList(上限 MAX_PROCESS_NUMBER),并回 REG_ACK,内容是管理段大小 + SegmentManager 在管理段内的偏移 + 时间戳 + segment id

1
2
3
4
5
6
7
8
9
// send REG_ACK and BaseAddrString
runtime::IpcMessage sendBuffer;

auto offset = rp::BaseRelativePointer::getOffset(m_mgmtSegmentId, m_segmentManager);
sendBuffer << runtime::IpcMessageTypeToString(runtime::IpcMessageType::REG_ACK)
<< m_roudiMemoryInterface.mgmtMemoryProvider()->size() << offset << transmissionTimestamp
<< m_mgmtSegmentId;

m_processList.back().sendViaIpcChannel(sendBuffer);

应用拿到 ACK 后,SharedMemoryUserOPEN_EXISTING 方式 mmap iceoryx_mgmt,用相对指针机制 registerPtr(segmentId, baseAddress, size) 注册地址映射,再顺着 SegmentManager 打开自己有权限的所有数据段(source/runtime/shared_memory_user.cpp 第 29~85 行)。每个进程 mmap 的虚拟地址不同,这正是所有共享内存内部结构必须使用相对指针(RelativePointer)的原因

3.4 心跳 keep-alive 与进程监控

  • 应用侧:PoshRuntimeImpl 的定时器周期性调用 sendKeepAliveAndHandleShutdownPreparation() → 发 KEEPALIVE 消息(间隔 PROCESS_KEEP_ALIVE_INTERVAL = 3 × 100 ms = 300 ms)。
  • RouDi 侧:KEEPALIVE 分支调用 m_prcMgr->updateLivelinessOfProcess(runtimeName) 刷新时间戳;Mon+Discover 线程里的 monitorProcesses() 检查超时:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
void ProcessManager::monitorProcesses() noexcept
{
auto currentTimestamp = mepoo::BaseClock_t::now();

auto processIterator = m_processList.begin();
while (processIterator != m_processList.end())
{
if (processIterator->isMonitored())
{
auto timediff = units::Duration(currentTimestamp - processIterator->getTimestamp());
...
if (timediff > runtime::PROCESS_KEEP_ALIVE_TIMEOUT)
{
LogWarn() << "Application " << processIterator->getName() << " not responding (last response "
<< timediff.toMilliseconds() << " milliseconds ago) --> removing it";
...
m_portManager.deletePortsOfProcess(processIterator->getName());

m_processIntrospection->removeProcess(static_cast<int32_t>(processIterator->getPid()));

// delete application
processIterator = m_processList.erase(processIterator);

超时阈值 PROCESS_KEEP_ALIVE_TIMEOUT = 5 × 300 ms = 1.5 s(iceoryx_posh_types.hpp 第 266~268 行)。是否监控由 RouDi 启动时的 monitoring 模式 决定(MonitoringMode::ON/OFF,命令行 -m 切换;ON 时所有进程被监控)。

优雅退出走另一条路:应用析构 PoshRuntimeImpl 时发 TERMINATION,RouDi 清理端口后回 TERMINATION_ACK

4. PortManager 与端口分配

4.1 PortPool:为什么端口数据也在共享内存

所有端口的数据结构本体PublisherPortDataSubscriberPortDataConditionVariableDataNodeData……)都存放在管理段中的 PortPool(固定容量的 FixedPositionContainer 集合,roudi/port_pool.cpp)。原因:

  1. 端口是应用与 RouDi 的共享状态机。应用侧用 PublisherPortUser/SubscriberPortUser 视图操作它(写 m_offeringRequested、收发 chunk),RouDi 侧用 PublisherPortRouDi/SubscriberPortRouDi 视图操作同一块数据(处理 CaPro 状态迁移、把订阅者队列挂到发布者)。两个进程要同时看到同一对象,只能放在共享内存。
  2. 零拷贝路由需要。订阅时 RouDi 把订阅者的 ChunkQueueData 的指针(相对指针形式)直接塞进发布者的 ChunkDistributor 队列列表——发布者随后 push chunk 时完全不需要 RouDi 参与。
  3. 崩溃恢复。应用死掉后 RouDi 仍能访问其端口数据并回收 chunk(releaseAllChunks())。

端口创建后,RouDi 把端口对象在管理段内的相对指针偏移通过 IPC 回给应用(CREATE_PUBLISHER_ACK 携带 offset + segmentId),应用侧换算回本进程的绝对地址:

1
2
3
4
5
6
7
8
9
10
11
if (stringToIpcMessageType(IpcMessage.c_str()) == IpcMessageType::CREATE_PUBLISHER_ACK)

{
rp::BaseRelativePointer::id_t segmentId{0U};
cxx::convert::fromString(receiveBuffer.getElementAtIndex(2U).c_str(), segmentId);
rp::BaseRelativePointer::offset_t offset{0U};
cxx::convert::fromString(receiveBuffer.getElementAtIndex(1U).c_str(), offset);
auto ptr = rp::BaseRelativePointer::getPtr(segmentId, offset);
return cxx::success<PublisherPortUserType::MemberType_t*>(
reinterpret_cast<PublisherPortUserType::MemberType_t*>(ptr));
}

4.2 创建 Publisher 端口的完整链路

1
2
3
4
5
6
7
8
9
10
11
12
应用: iox::popo::Publisher<T> 构造
→ BasePublisher 构造: PoshRuntime::getInstance().getMiddlewarePublisher(service, options)
(posh_runtime_impl.cpp: 发 IpcMessageType::CREATE_PUBLISHER)
RouDi: RouDi::processMessage(CREATE_PUBLISHER) roudi.cpp:225
→ ProcessManager::addPublisherForProcess() process_manager.cpp:452
→ SegmentManager::getSegmentInformationWithWriteAccessForUser(user) // 找该用户可写的数据段
→ PortManager::acquirePublisherPortData() port_manager.cpp:851
→ 检查通信策略(ONE_TO_MANY 下同一 service 只许一个 publisher)
→ PortPool::addPublisherPort(...) // 在共享内存里构造 PublisherPortData
→ doDiscoveryForPublisherPort() // 立即跑一轮发现
→ 回 CREATE_PUBLISHER_ACK(offset, segmentId)
应用: 相对指针 → 绝对地址,构造 PublisherPortUser 视图

注意 addPublisherForProcess 里传入的 payloadDataSegmentMemoryManager按发起进程的 POSIX 用户挑选的数据段 memory manager——这决定了该 publisher 之后 loan() 从哪个 mempool 拿 chunk。

4.3 doDiscovery:发现主循环

Mon+Discover 线程每 100 ms 调 PortManager::doDiscovery(),轮询所有端口收集状态变化并互相路由:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
void PortManager::doDiscovery() noexcept
{
handlePublisherPorts();

handleSubscriberPorts();

handleServerPorts();

handleClientPorts();

handleInterfaces();

handleNodes();

handleConditionVariables();
}

5. CaPro 协议(Canonical Protocol)

5.1 ServiceDescription:service/instance/event 三元组

capro::ServiceDescriptioniceoryx_posh/capro/service_description.hpp 第 80、156~160 行)由三个字符串组成:m_serviceString / m_instanceString / m_eventString。ROS 2 的 rmw_iceoryx、cyclonedds 的 iceoryx 网关都会把 topic 名映射到这个三元组。两个端口能匹配的首要条件是三元组完全相等(PortManager::isCompatiblePubSub 第 483 行)。

5.2 CaproMessage 类型

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
enum class CaproMessageType : uint8_t
{
NOTYPE = 0,
FIND,
OFFER,
STOP_OFFER,
SUB,
UNSUB,
CONNECT,
DISCONNECT,
ACK,
NACK,
PUB,
REQ,
RES,
PING,
PONG,
MESSGAGE_TYPE_END
};

pub/sub 用到的核心是 OFFER/STOP_OFFER(发布者宣告)、SUB/UNSUB(订阅者请求)、ACK/NACK(应答);client/server 用 CONNECT/DISCONNECTCaproMessage 里最关键的字段是 m_chunkQueueData——订阅者接收队列在共享内存里的指针,它随 SUB 消息一路传给发布者。

5.3 订阅如何路由到发布者

双方端口各自是一个小状态机,RouDi 只是搬运消息:

  1. 应用调 Subscriber 构造(subscribeOnCreate 默认 true)→ 设置 m_subscribeRequested
  2. 下一轮 doDiscovery() 中,SubscriberPortSingleProducer::tryGetCaProMessage() 产出 SUB 消息并携带自己的队列指针
1
2
3
4
5
6
7
8
9
10
if (currentSubscribeRequest && (SubscribeState::NOT_SUBSCRIBED == currentSubscriptionState))
{
getMembers()->m_subscriptionState.store(SubscribeState::SUBSCRIBE_REQUESTED, std::memory_order_relaxed);

capro::CaproMessage caproMessage(capro::CaproMessageType::SUB, BasePort::getMembers()->m_serviceDescription);
caproMessage.m_chunkQueueData = static_cast<void*>(&getMembers()->m_chunkReceiverData);
caproMessage.m_historyCapacity = getMembers()->m_options.historyRequest;

return cxx::make_optional<capro::CaproMessage>(caproMessage);
}
  1. PortManager::sendToAllMatchingPublisherPorts()port_manager.cpp 第 500 行)遍历所有 publisher,对三元组匹配且 QoS 兼容者调用 dispatchCaProMessageAndGetPossibleResponse(SUB)
  2. 发布者侧把订阅者队列挂进自己的 ChunkSender(即 ChunkDistributor 的队列容器),成功则回 ACK:
1
2
3
4
5
6
7
8
9
10
if (capro::CaproMessageType::SUB == caProMessage.m_type)
{
const auto ret = m_chunkSender.tryAddQueue(
static_cast<PublisherPortData::ChunkQueueData_t*>(caProMessage.m_chunkQueueData),
caProMessage.m_historyCapacity);
if (!ret.has_error())
{
responseMessage.m_type = capro::CaproMessageType::ACK;
}
}
  1. ACK 被 RouDi 转回订阅者,订阅者状态迁移到 SUBSCRIBED。若没有匹配发布者则 RouDi 直接给订阅者发 NACK,订阅者进入 WAIT_FOR_OFFER,等以后收到 OFFER 再自动补发 SUB(subscriber_port_single_producer.cpp 第 73~82 行)。

反方向:发布者 OFFER 时 sendToAllMatchingSubscriberPorts()(第 539 行)把 OFFER 派发给所有等待的订阅者,触发它们回 SUB,随后同样走上面 3-5 步(一轮 discovery 内完成握手)。

QoS 兼容性检查(阻塞策略、history 支持):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
bool PortManager::isCompatiblePubSub(const PublisherPortRouDiType& publisher,
const SubscriberPortType& subscriber) const noexcept
{
if (subscriber.getCaProServiceDescription() != publisher.getCaProServiceDescription())
{
return false;
}

auto& pubOpts = publisher.getOptions();
auto& subOpts = subscriber.getOptions();

const bool blockingPoliciesAreCompatible =
!(pubOpts.subscriberTooSlowPolicy == popo::ConsumerTooSlowPolicy::DISCARD_OLDEST_DATA
&& subOpts.queueFullPolicy == popo::QueueFullPolicy::BLOCK_PRODUCER);

const bool historyRequestIsCompatible = !subOpts.requiresPublisherHistorySupport || pubOpts.historyCapacity > 0;

return blockingPoliciesAreCompatible && historyRequestIsCompatible;
}

另外 OFFER/STOP_OFFER 还会转发给所有 InterfacePortsendToAllMatchingInterfacePorts)——这是 gateway(如 iceoryx-dds、rmw 网关)监听全网服务变化的机制。

5.4 服务注册表(ServiceRegistry)

RouDi 在 PortManager 内维护一份 ServiceRegistryroudi/service_registry.cpp),OFFER 时 addPublisher、STOP_OFFER 时 removePublisher(server 同理)。每次变化都会把整个注册表作为一个 chunk 发布到内置 topic 上:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
void PortManager::publishServiceRegistry() const noexcept
{
...
PublisherPortUserType publisher(m_serviceRegistryPublisherPortData.value());
publisher
.tryAllocateChunk(sizeof(ServiceRegistry),
alignof(ServiceRegistry),
CHUNK_NO_USER_HEADER_SIZE,
CHUNK_NO_USER_HEADER_ALIGNMENT)
.and_then([&](auto& chunk) {
auto sample = static_cast<ServiceRegistry*>(chunk->userPayload());

// It's ok to copy as the modifications happen in the same thread and not concurrently
*sample = m_serviceRegistry;

publisher.sendChunk(chunk);
})
.or_else([](auto&) { LogWarn() << "Could not allocate a chunk for the service registry!"; });
}

对应的内置服务三元组是 {SERVICE_DISCOVERY_SERVICE_NAME, SERVICE_DISCOVERY_INSTANCE_NAME, SERVICE_DISCOVERY_EVENT_NAME}port_manager.cpp 第 71~74 行)。

6. 服务发现 API(2.0 的 ServiceDiscovery / findService)

2.0 把 1.0 里”应用发 FIND 消息、RouDi 应答”的模式改成了纯 pub/sub:应用侧 iox::runtime::ServiceDiscovery 内部就是一个订阅上述内置 topic 的 subscriber,findService()take() 最新注册表快照再本地过滤:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
void ServiceDiscovery::update()
{
// allows us to use update and hence findService concurrently
std::lock_guard<std::mutex> lock(m_serviceRegistryMutex);
m_serviceRegistrySubscriber.take().and_then([&](popo::Sample<const roudi::ServiceRegistry>& serviceRegistrySample) {
*m_serviceRegistry = *serviceRegistrySample;
});
}

void ServiceDiscovery::findService(const cxx::optional<capro::IdString_t>& service,
const cxx::optional<capro::IdString_t>& instance,
const cxx::optional<capro::IdString_t>& event,
const cxx::function_ref<void(const capro::ServiceDescription&)>& callableForEach,
const popo::MessagingPattern pattern) noexcept
{
...
update();

switch (pattern)
{
case popo::MessagingPattern::PUB_SUB:
{
m_serviceRegistry->find(
service, instance, event, [&](const roudi::ServiceRegistry::ServiceDescriptionEntry& serviceEntry) {
if (serviceEntry.publisherCount > 0)
{
callableForEach(serviceEntry.serviceDescription);
}
});
break;
}

三个参数都是 optional——nullopt 表示通配。还可以把 ServiceDiscovery 挂到 WaitSet/Listener 上监听 SERVICE_REGISTRY_CHANGED 事件(enableEvent,同文件第 82 行),实现事件驱动的发现。

ServiceRegistry::find() 本身是对最多 MAX_NUMBER_OF_SERVICES 条目的线性匹配(service_registry.cpp 第 141~165 行),每个条目带 publisherCount/serverCount 引用计数(multi-set 语义,同一 service 多个 publisher 只占一个条目)。

7. 内省(Introspection)服务

RouDi 自己作为一个”进程”注册(m_processIntrospection.addProcess(getpid(), IPC_CHANNEL_ROUDI_NAME)roudi.cpp 第 65 行),并用内部 publisher 端口周期发布以下内置 topic(服务描述常量定义在 iceoryx_posh_roudi_types.hpp / introspection_types.hpp):

内置服务 内容
IntrospectionMempoolService 各 mempool 的 chunk 使用量/最小剩余
IntrospectionProcessService 已注册进程列表(pid、名字、node)
IntrospectionPortService / IntrospectionPortThroughputService / IntrospectionSubscriberPortChangingDataService 端口清单、吞吐、订阅状态

这些端口在 PortManager 构造时通过 acquireInternalPublisherPortData() 创建(port_manager.cpp 第 80~96 行),chunk 从专用的 introspection mempool(管理段内,配置见 default_roudi_memory.cppintrospectionMemPoolConfig())分配。官方 iceoryx_introspectioniox-introspection-client)就是订阅这些 topic 的 ncurses 客户端。内置服务名以 Introspection/服务发现常量为 service 名,普通应用禁止创建同名 publisher(acquirePublisherPortDataWithoutDiscoveryisInternal() 检查,返回 INTERNAL_SERVICE_DESCRIPTION_IS_FORBIDDEN)。

8. RouDi 配置

8.1 TOML 配置文件

iox-roudi -c roudi_config.toml;不提供时先找默认路径(defaultConfigFilePath,即 /etc/iceoryx/roudi_config.toml),再退回编译期默认配置(TomlRouDiConfigFileProvider::parse()roudi_config_toml_file_provider.cpp 第 59~68 行)。格式(版本必须为 1):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[general]
version = 1

[[segment]] # 一个共享内存数据段;省略 reader/writer 则默认当前进程所在用户组
reader = "readergroup" # 拥有只读权限的 POSIX 用户组
writer = "writergroup" # 拥有读写权限的 POSIX 用户组

[[segment.mempool]] # 段内一个 mempool
size = 128 # chunk 的 payload 大小(字节)
count = 10000 # chunk 数量

[[segment.mempool]]
size = 1024
count = 5000

解析核心(段数上限 MAX_SHM_SEGMENTS、每段 mempool 上限 MAX_NUMBER_OF_MEMPOOLS):

1
2
3
4
5
6
7
8
9
10
11
for (auto mempool : *mempools)
{
auto chunkSize = mempool->get_as<uint32_t>("size");
auto chunkCount = mempool->get_as<uint32_t>("count");
...
mempoolConfig.addMemPool({*chunkSize, *chunkCount});
}
parsedConfig.m_sharedMemorySegments.push_back(
{iox::posix::PosixGroup::string_t(iox::cxx::TruncateToCapacity, reader),
iox::posix::PosixGroup::string_t(iox::cxx::TruncateToCapacity, writer),
mempoolConfig});

注意 mempool 的 size 是 payload 大小,实际 chunk 会加上 ChunkHeader 并按 MemPool::CHUNK_MEMORY_ALIGNMENT 对齐;mempool 必须按 chunk 大小递增排序(MemoryManager::addMemPool 强制检查)。

8.2 主要命令行参数

参数 作用
-c <file> TOML 配置路径
-m on/off monitoring 模式(cmd_line_args.hpp 第 31 行默认 MonitoringMode::ON
-l <level> 日志级别
-u <id> unique RouDi id(写入 UniquePortId,用于多 RouDi 隔离)
-k <sec> 关闭时等待应用退出的 kill delay(默认 PROCESS_DEFAULT_KILL_DELAY = 45 s)
-x <level> 版本兼容性检查级别

9. 崩溃行为

9.1 应用崩溃

  • monitoring ON:心跳超时 1.5 s 后,RouDi 主动 deletePortsOfProcess()——对每个端口执行有序关闭(先注入 STOP_OFFER/UNSUB 走完 CaPro 流程,通知对端),然后 releaseAllChunks() 回收该进程持有的所有 chunk(port_manager.cpp 第 796~848 行)。之后同名进程可重新注册。
  • monitoring OFF:RouDi 不会主动发现崩溃。同名进程重新注册时走”已存在则视为崩溃重注册”路径:
1
2
3
4
5
6
7
8
9
10
11
12
13
findProcess(name)
.and_then([&](auto& process) {
// process is already in list (i.e. registered)
// depending on the mode we clean up the process resources and register it again
// if it is monitored, we reject the registration and wait for automatic cleanup
// otherwise we remove the process ourselves and register it again
...
// process exists, we expect that the existing process crashed
LogWarn() << "Application " << name << " crashed. Re-registering application";

// remove the existing process and add the new process afterwards, we do not send ack to new process
constexpr TerminationFeedback TERMINATION_FEEDBACK{TerminationFeedback::DO_NOT_SEND_ACK_TO_PROCESS};
if (!this->searchForProcessAndRemoveIt(name, TERMINATION_FEEDBACK))

已知限制:如果应用恰好死在持有 ChunkDistributor 锁的瞬间,RouDi 清理时会遇到死锁风险并触发 FATAL(chunk_distributor.inlcleanup() 注释明确说明,见 05 篇 9.3 节)。

9.2 RouDi 崩溃 / 关闭

  • 已建立的 pub/sub 连接数据传输不受影响(数据路径不经过 RouDi),但:没有发现、没有新端口、进程无法注册/注销,keep-alive 发送失败会打印警告。共享内存文件仍在 /dev/shm 中,重启 RouDi 时以 PURGE_AND_CREATE 模式重建管理段(default_roudi_memory.cpp 第 30 行),旧应用必须重启(其映射的旧段已失效,且版本/会话不匹配)。
  • 正常关闭(SIGINT/SIGTERM):RouDi::shutdown() 先停 discovery 线程,然后向所有注册进程发 SIGTERM 并等待其发回 TERMINATION,超过 kill delay(默认 45 s)仍存活则 SIGKILL(roudi.cpp 第 102~133 行)。requestShutdownOfAllProcesses() 同时调用 m_portManager.unblockRouDiShutdown(),把所有 publisher 强制 STOP_OFFER,避免因 BLOCK_PRODUCER 策略卡死关闭流程。

10. 时序总览

共享内存PortManager(Mon+Discover线程)RouDi(IPC-msg线程)IPC channel(UDS)应用(PoshRuntime)共享内存PortManager(Mon+Discover线程)RouDi(IPC-msg线程)IPC channel(UDS)应用(PoshRuntime)启动: createAndAnnounceMemory()iceoryx_mgmt + 数据段loop[每 100ms]loop[每 300ms]REG(name,pid,uid,ts,version)processMessage(REG)ProcessManager::addProcessREG_ACK(size, segMgr offset, segId)mmap mgmt + 数据段, registerPtrCREATE_PUBLISHER(service,options)PortPool::addPublisherPortCREATE_PUBLISHER_ACK(offset,segId)doDiscovery(): OFFER/SUB/ACK 状态机路由ServiceRegistry 变更 → 发布内置topicmonitorProcesses() 心跳检查KEEPALIVE

下一篇

05 零拷贝数据路径详解

05 零拷贝数据路径详解

发布端:popo/building_blocks/chunk_sender.inlchunk_distributor.inl;订阅端:chunk_receiver.inlchunk_queue_popper.inl
内存:mepoo/memory_manager.cppmem_pool.cppshared_chunk.cpp
源码根:ros2_humble/src/eclipse-iceoryx/iceoryx(版本 2.0.6)

1. 传统中间件 vs iceoryx:拷贝次数对比

以一次进程间 pub/sub 为例,传统 socket 类中间件(UDP loopback / UDS)的路径:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
传统 (以 UDS/UDP loopback 为例, 至少 2~4 次拷贝 + 2 次系统调用):

发布进程 内核 订阅进程
┌────────┐ ①序列化拷贝 ┌────────┐ ③拷贝到 ┌────────┐
│ 用户数据 │──→ 发送缓冲 ──→│ 内核缓冲 │──→ 接收缓冲 ──→│ 反序列化 │
└────────┘ ②write() └────────┘ read() └────────┘

iceoryx (0 次拷贝, 数据本体不移动):

发布进程 共享内存 订阅进程
loan() ──────────→ ┌──────────────┐
直接原地构造 T │ chunk (T) │ ←────────── take()
publish() 只推送 ──→│ │ 拿到的是同一块内存的
8 字节相对指针 └──────────────┘ const T* 只读引用
└→ 订阅者无锁队列 ← ChunkDistributor

iceoryx 里”发送”一个 4 MB 的样本和发送 1 KB 的样本代价相同:真正跨进程传递的只是一个 8 字节的 ShmSafeUnmanagedChunk(相对指针),数据本体自始至终躺在共享内存 mempool 的同一个 chunk 里。这正是 iceperf 基准中 iceoryx 延迟与 payload 大小无关的原因(见第 11 节)。

参与者与所属文件(全部在 iceoryx_posh 中):

构件 文件 职责
Publisher<T> / Sample<T> include/iceoryx_posh/internal/popo/publisher_impl.inl 类型安全 API
PublisherPortUser source/popo/ports/publisher_port_user.cpp 端口用户视图
ChunkSender include/.../building_blocks/chunk_sender.inl 分配/发送 chunk
ChunkDistributor include/.../building_blocks/chunk_distributor.inl 推送到 N 个订阅队列 + history
MemoryManager / MemPool source/mepoo/memory_manager.cppmem_pool.cpp 共享内存池
ChunkQueuePusher/Popper include/.../building_blocks/chunk_queue_*.inl 无锁队列两端
ChunkReceiver include/.../building_blocks/chunk_receiver.inl 订阅端取 chunk
SharedChunk source/mepoo/shared_chunk.cpp 引用计数句柄

2. 发布端:loan() 的完整调用链

1
2
3
4
5
6
7
8
9
10
11
Publisher<T>::loan()                          publisher_impl.inl:39
→ PublisherImpl::loanSample() publisher_impl.inl:71
→ PublisherPortUser::tryAllocateChunk() publisher_port_user.cpp:42
→ ChunkSender::tryAllocate() chunk_sender.inl:100
├─ (快路径) 复用 m_lastChunkUnmanaged
└─ MemoryManager::getChunk() memory_manager.cpp:152
→ MemPool::getChunk() mem_pool.cpp:78 (LoFFLi 无锁空闲链表 pop)
→ placement-new ChunkHeader + ChunkManagement
→ 返回 SharedChunk (引用计数=1)
→ 在 chunk 的 userPayload 上 placement-new T(...)
→ 包装成 Sample<T,H> 返回

2.1 类型层:在共享内存上原地构造

1
2
3
4
5
6
7
template <typename T, typename H, typename BasePublisherType>
template <typename... Args>
inline cxx::expected<Sample<T, H>, AllocationError>
PublisherImpl<T, H, BasePublisherType>::loan(Args&&... args) noexcept
{
return std::move(loanSample().and_then([&](auto& sample) { new (sample.get()) T(std::forward<Args>(args)...); }));
}

loanSample()sizeof(T)/alignof(T) 交给端口层:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
template <typename T, typename H, typename BasePublisherType>
inline cxx::expected<Sample<T, H>, AllocationError> PublisherImpl<T, H, BasePublisherType>::loanSample() noexcept
{
static constexpr uint32_t USER_HEADER_SIZE{std::is_same<H, mepoo::NoUserHeader>::value ? 0U : sizeof(H)};

auto result = port().tryAllocateChunk(sizeof(T), alignof(T), USER_HEADER_SIZE, alignof(H));
if (result.has_error())
{
return cxx::error<AllocationError>(result.get_error());
}
else
{
return cxx::success<Sample<T, H>>(convertChunkHeaderToSample(result.value()));
}
}

2.2 ChunkSender::tryAllocate():快路径与 mempool 分配

ChunkSender 有个重要优化:m_lastChunkUnmanaged 缓存上一次发送的 chunk,若它已无其他持有者(所有订阅者都消费完、引用计数==1)且大小够用,就原地复用——发布循环稳定后基本不再触碰 mempool:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
auto& lastChunkUnmanaged = getMembers()->m_lastChunkUnmanaged;
mepoo::ChunkHeader* lastChunkChunkHeader =
lastChunkUnmanaged.isNotLogicalNullptrAndHasNoOtherOwners() ? lastChunkUnmanaged.getChunkHeader() : nullptr;

if (lastChunkChunkHeader && (lastChunkChunkHeader->chunkSize() >= requiredChunkSize))
{
auto sharedChunk = lastChunkUnmanaged.cloneToSharedChunk();
if (getMembers()->m_chunksInUse.insert(sharedChunk))
{
auto chunkSize = lastChunkChunkHeader->chunkSize();
lastChunkChunkHeader->~ChunkHeader();
new (lastChunkChunkHeader) mepoo::ChunkHeader(chunkSize, chunkSettings);
lastChunkChunkHeader->setOriginId(originId);
return cxx::success<mepoo::ChunkHeader*>(lastChunkChunkHeader);
}
...
}
else
{
// BEGIN of critical section, chunk will be lost if the process terminates in this section
// get a new chunk
auto getChunkResult = getMembers()->m_memoryMgr->getChunk(chunkSettings);

if (!getChunkResult.has_error())
{
auto& chunk = getChunkResult.value();

// if the application allocated too much chunks, return no more chunks
if (getMembers()->m_chunksInUse.insert(chunk))
{
// END of critical section
chunk.getChunkHeader()->setOriginId(originId);
return cxx::success<mepoo::ChunkHeader*>(chunk.getChunkHeader());
}

m_chunksInUseUsedChunkList)登记该进程当前借出的 chunk,超过上限报 TOO_MANY_CHUNKS_ALLOCATED_IN_PARALLEL;它也是 RouDi 崩溃清理的依据(第 9.3 节)。

2.3 MemoryManager::getChunk():按大小挑 mempool

MemoryManager 持有若干按 chunk 大小升序排列的 MemPool(数据段中)以及一个 ChunkManagement 管理池(管理段中)。分配时线性找第一个足够大的池:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
cxx::expected<SharedChunk, MemoryManager::Error> MemoryManager::getChunk(const ChunkSettings& chunkSettings) noexcept
{
void* chunk{nullptr};
MemPool* memPoolPointer{nullptr};
const auto requiredChunkSize = chunkSettings.requiredChunkSize();
...
for (auto& memPool : m_memPoolVector)
{
uint32_t chunkSizeOfMemPool = memPool.getChunkSize();
if (chunkSizeOfMemPool >= requiredChunkSize)
{
chunk = memPool.getChunk();
memPoolPointer = &memPool;
aquiredChunkSize = chunkSizeOfMemPool;
break;
}
}
...
else
{
auto chunkHeader = new (chunk) ChunkHeader(aquiredChunkSize, chunkSettings);
auto chunkManagement = new (m_chunkManagementPool.front().getChunk())
ChunkManagement(chunkHeader, memPoolPointer, &m_chunkManagementPool.front());
return cxx::success<SharedChunk>(SharedChunk(chunkManagement));
}
}

MemPool::getChunk() 本身是无锁的——空闲索引由 LoFFLi(Lock-Free Free List)维护:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
void* MemPool::getChunk() noexcept
{
uint32_t l_index{0U};
if (!m_freeIndices.pop(l_index))
{
std::cerr << "Mempool [m_chunkSize = " << m_chunkSize << ", numberOfChunks = " << m_numberOfChunks
<< ", used_chunks = " << m_usedChunks << " ] has no more space left" << std::endl;
return nullptr;
}
...
m_usedChunks.fetch_add(1U, std::memory_order_relaxed);
adjustMinFree();

return m_rawMemory + l_index * m_chunkSize;
}

每个 chunk 的布局:ChunkHeader(含 chunkSize、序列号、originId、userPayload 偏移等)+ 可选 user-header + 用户 payload。配套的 ChunkManagement(引用计数 + 指回 header/mempool 的相对指针)单独放在管理段的管理池里。

3. 发布端:publish() 如何把相对指针推入订阅队列

1
2
3
4
5
6
7
8
9
Sample<T>::publish() → PublisherImpl::publish()        publisher_impl.inl:87
→ PublisherPortUser::sendChunk() publisher_port_user.cpp:56
→ ChunkSender::send() chunk_sender.inl:184
→ getChunkReadyForSend(): 从 m_chunksInUse 移出、打序列号
→ ChunkDistributor::deliverToAllStoredQueues() chunk_distributor.inl:141
→ 对每个订阅队列 ChunkQueuePusher::push() chunk_queue_pusher.inl:47
→ VariantQueue<ShmSafeUnmanagedChunk>::push() (无锁 FIFO/SOFI)
→ ConditionNotifier::notify() (可选, 唤醒 WaitSet/Listener)
→ addToHistoryWithoutDelivery() (latched/history 支持)

PublisherImpl::publish() 只做指针换算,不碰数据:

1
2
3
4
5
6
7
template <typename T, typename H, typename BasePublisherType>
inline void PublisherImpl<T, H, BasePublisherType>::publish(Sample<T, H>&& sample) noexcept
{
auto userPayload = sample.release(); // release the Samples ownership of the chunk before publishing
auto chunkHeader = mepoo::ChunkHeader::fromUserPayload(userPayload);
port().sendChunk(chunkHeader);
}

核心的多播分发在 ChunkDistributor::deliverToAllStoredQueues()。注意队列列表 m_queues 就是 RouDi 在 CaPro SUB 握手时挂进来的订阅者接收队列(见 04 篇 5.3 节):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
template <typename ChunkDistributorDataType>
inline uint64_t ChunkDistributor<ChunkDistributorDataType>::deliverToAllStoredQueues(mepoo::SharedChunk chunk) noexcept
{
uint64_t numberOfQueuesTheChunkWasDeliveredTo{0U};
typename ChunkDistributorDataType::QueueContainer_t remainingQueues;
{
typename MemberType_t::LockGuard_t lock(*getMembers());

bool willWaitForConsumer = getMembers()->m_consumerTooSlowPolicy == ConsumerTooSlowPolicy::WAIT_FOR_CONSUMER;
// send to all the queues
for (auto& queue : getMembers()->m_queues)
{
bool isBlockingQueue = (willWaitForConsumer && queue->m_queueFullPolicy == QueueFullPolicy::BLOCK_PRODUCER);

if (pushToQueue(queue.get(), chunk))
{
++numberOfQueuesTheChunkWasDeliveredTo;
}
else
{
if (isBlockingQueue)
{
remainingQueues.emplace_back(queue);
}
else
{
++numberOfQueuesTheChunkWasDeliveredTo;
ChunkQueuePusher_t(queue.get()).lostAChunk();
}
}
}
}

// busy waiting until every queue is served
while (!remainingQueues.empty())

进入队列的元素是 ShmSafeUnmanagedChunk——一个被压缩到 8 字节 的 (segmentId, offset) 相对指针,指向 ChunkManagement。8 字节保证了写入的原子性(防 torn write),这是崩溃安全的关键:

1
2
3
4
5
6
7
8
9
// Torn writes are problematic since RouDi needs to cleanup all chunks when an application crashes. If the size is
// larger than 8 bytes on a 64 bit system, torn writes happens and the data is only partially written when the
// application crashes at the wrong time. RouDi would then read corrupt data and try to access invalid memory.
static_assert(sizeof(ShmSafeUnmanagedChunk) <= 8U,
"The ShmSafeUnmanagedChunk size must not exceed 64 bit to prevent torn writes!");
...
static_assert(std::is_trivially_copyable<ShmSafeUnmanagedChunk>::value,
"The ShmSafeUnmanagedChunk must be trivially copyable to prevent Frankenstein objects when the copy ctor "
"works on half dead objects!");

ChunkQueuePusher::push() 压入无锁队列后(若配置了条件变量)通知订阅者:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
template <typename ChunkQueueDataType>
inline bool ChunkQueuePusher<ChunkQueueDataType>::push(mepoo::SharedChunk chunk) noexcept
{
auto pushRet = getMembers()->m_queue.push(chunk);
bool hasQueueOverflow = false;

// drop the chunk if one is returned by an overflow
if (pushRet.has_value())
{
pushRet.value().releaseToSharedChunk();
// tell the ChunkDistributor that we had an overflow and dropped a sample
hasQueueOverflow = true;
}

{
typename MemberType_t::LockGuard_t lock(*getMembers());
if (getMembers()->m_conditionVariableDataPtr)
{
ConditionNotifier(*getMembers()->m_conditionVariableDataPtr.get(),
*getMembers()->m_conditionVariableNotificationIndex)
.notify();
}
}

return !hasQueueOverflow;
}

队列类型是 cxx::VariantQueuechunk_queue_data.hpp 第 46 行),订阅端口默认使用 SOFI(Safely Overflowing FIFO,SoFi_SingleProducerSingleConsumer/多生产者用 lock-free queue,iceoryx_hoofs/cxx/variant_queue.hpp 第 39~45 行):满时 push 会”挤出”最老的元素并返回给推送方释放——天然实现 KEEP_LAST 语义。

4. 订阅端:take() 调用链与相对指针→绝对地址

1
2
3
4
5
6
7
8
9
Subscriber<T>::take()                          subscriber_impl.inl:34
→ BaseSubscriber::takeChunk() base_subscriber.inl:89
→ SubscriberPortUser::tryGetChunk() subscriber_port_user.cpp:67
→ ChunkReceiver::tryGet() chunk_receiver.inl:74
→ ChunkQueuePopper::tryPop() chunk_queue_popper.inl:47
→ m_queue.pop() 得到 ShmSafeUnmanagedChunk
→ releaseToSharedChunk(): 相对指针 → 本进程绝对地址
→ m_chunksInUse.insert(chunk) (登记, 引用计数保持)
→ userPayload() 包成 Sample<const T> (只读)

tryPop() 中完成相对指针到绝对地址的转换(rp::RelativePointer 查本进程注册的 segment 基址表)并做 header 版本校验:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
template <typename ChunkQueueDataType>
inline cxx::optional<mepoo::SharedChunk> ChunkQueuePopper<ChunkQueueDataType>::tryPop() noexcept
{
auto retVal = getMembers()->m_queue.pop();

// check if queue had an element that was poped and return if so
if (retVal.has_value())
{
auto chunk = retVal.value().releaseToSharedChunk();

auto receivedChunkHeaderVersion = chunk.getChunkHeader()->chunkHeaderVersion();
if (receivedChunkHeaderVersion != mepoo::ChunkHeader::CHUNK_HEADER_VERSION)
{
...
return cxx::nullopt_t();
}
return cxx::make_optional<mepoo::SharedChunk>(chunk);
}

ShmSafeUnmanagedChunk::releaseToSharedChunk() 的实现(offset+id 还原为 ChunkManagement*):

1
2
3
4
5
6
7
8
9
10
SharedChunk ShmSafeUnmanagedChunk::releaseToSharedChunk() noexcept
{
if (m_chunkManagement.isLogicalNullptr())
{
return SharedChunk();
}
auto chunkMgmt = rp::RelativePointer<mepoo::ChunkManagement>(m_chunkManagement.offset(), m_chunkManagement.id());
m_chunkManagement.reset();
return SharedChunk(chunkMgmt);
}

上层 ChunkReceiver::tryGet() 把 chunk 登记进订阅者自己的 m_chunksInUse(持有上限 MAX_CHUNKS_HELD_PER_SUBSCRIBER_SIMULTANEOUSLY),最后 SubscriberImpl::take() 包装成只读 Sample<const T>

1
2
3
4
5
6
7
8
9
10
11
12
13
template <typename T, typename H, typename BaseSubscriberType>
inline cxx::expected<Sample<const T, const H>, ChunkReceiveResult>
SubscriberImpl<T, H, BaseSubscriberType>::take() noexcept
{
auto result = BaseSubscriberType::takeChunk();
if (result.has_error())
{
return cxx::error<ChunkReceiveResult>(result.get_error());
}
auto userPayloadPtr = static_cast<const T*>(result.value()->userPayload());
auto samplePtr = cxx::unique_ptr<const T>(userPayloadPtr, m_sampleDeleter);
return cxx::success<Sample<const T, const H>>(std::move(samplePtr));
}

5. 引用计数与释放回 MemPool

SharedChunk 是 chunk 的智能句柄,引用计数存在共享内存的 ChunkManagement 里(所有进程共享同一个计数)。持有者包括:发布者的 m_chunksInUse/m_lastChunkUnmanaged/history、每个订阅队列里的元素、每个订阅者 take() 后的 Sample。任何进程中最后一个 SharedChunk 析构时归还两块内存:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
SharedChunk::~SharedChunk() noexcept
{
decrementReferenceCounter();
}
...
void SharedChunk::decrementReferenceCounter() noexcept
{
if ((m_chunkManagement != nullptr)
&& (m_chunkManagement->m_referenceCounter.fetch_sub(1U, std::memory_order_relaxed) == 1U))
{
freeChunk();
}
}

void SharedChunk::freeChunk() noexcept
{
m_chunkManagement->m_mempool->freeChunk(m_chunkManagement->m_chunkHeader);
m_chunkManagement->m_chunkManagementPool->freeChunk(m_chunkManagement);
m_chunkManagement = nullptr;
}

MemPool::freeChunk() 把索引 push 回 LoFFLi 空闲链表,并带双重释放检测(mem_pool.cpp 第 96~112 行,kPOSH__MEMPOOL_POSSIBLE_DOUBLE_FREE)。订阅者侧 Sample 析构 → SampleDeleterSubscriberPortUser::releaseChunk()ChunkReceiver::release()m_chunksInUse 移除,触发上述计数递减。

6. 跨进程同步:无锁队列 + ConditionVariable

数据路径上没有互斥锁跨进程共享ChunkDistributor 的锁默认是同进程/RouDi 间的 ThreadSafePolicy 互斥量,push/pop 队列本身无锁)。订阅者等待新数据有两种方式:

  1. 轮询:循环 take() 直到 NO_CHUNK_AVAILABLE
  2. 事件驱动:WaitSet/Listener 把 ConditionVariableData(在共享内存中,含一个进程间 POSIX 信号量 + 通知位数组)挂到订阅端口;发布者 push 后 ConditionNotifier::notify()
1
2
3
4
5
6
7
8
9
10
void ConditionNotifier::notify() noexcept
{
if (m_notificationIndex < MAX_NUMBER_OF_NOTIFIERS)
{
getMembers()->m_activeNotifications[m_notificationIndex].store(true, std::memory_order_release);
}
getMembers()->m_semaphore.post().or_else([](auto) {
errorHandler(Error::kPOPO__CONDITION_NOTIFIER_SEMAPHORE_CORRUPT_IN_NOTIFY, nullptr, ErrorLevel::FATAL);
});
}

订阅进程在 ConditionListener::wait()sem_wait 阻塞,醒来后扫描通知位数组确定是哪个 trigger(condition_listener.cpp 第 91~117 行 waitImpl)。这是数据路径上唯一的系统调用点,且只在订阅者选择阻塞等待时发生。

7. 多订阅者:同一 chunk 只读共享

N 个订阅者的场景下,deliverToAllStoredQueues()同一个 chunk 的相对指针 push 进 N 个队列,每次 push 都通过 SharedChunk 拷贝构造把引用计数 +1。没有任何数据复制:

  • 所有订阅者 take() 得到的 Sample<const T> 指向同一物理内存,类型系统强制只读(const T*)。
  • 订阅者数据段通常以 READ_ONLY mmap(shared_memory_user.cpp 第 62 行按段权限决定),越权写会 SIGSEGV。
  • 各订阅者独立释放,最后一个释放者把 chunk 归还 mempool(第 5 节)。

这也带来约束:订阅者拿到样本后不能修改;需要修改就得自己 loan 新 chunk 拷贝(此时才有一次拷贝)。

8. history / latched topic 支持

对应 ROS/DDS 的 transient-local / latched 语义:

  • 发布侧 PublisherOptions::historyCapacity(上限 MAX_PUBLISHER_HISTORY = 编译期 CMake 变量 IOX_MAX_PUBLISHER_HISTORY,默认 16,见 iceoryx_posh/cmake/IceoryxPoshDeployment.cmake 第 5556 行)。每次发送后 chunk 也存入 m_history 环形缓存:deliverToAllStoredQueues() 末尾调用 addToHistoryWithoutDelivery(chunk)chunk_distributor.inl 第 212 行),容量满则释放最老的(第 296312 行)。
  • 订阅侧 SubscriberOptions::historyRequest。RouDi 转发 SUB 消息时携带该值,发布者 tryAddQueue() 在挂接队列的同时把 history 里最后 requestedHistory 个 chunk 立即补发给新订阅者:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
const auto currChunkHistorySize = getMembers()->m_history.size();

if (requestedHistory > getMembers()->m_historyCapacity)
{
LogWarn() << "Chunk history request exceeds history capacity! Request is " << requestedHistory
<< ". Capacity is " << getMembers()->m_historyCapacity << ".";
}

// if the current history is large enough we send the requested number of chunks, else we send the
// total history
const auto startIndex =
(requestedHistory <= currChunkHistorySize) ? currChunkHistorySize - requestedHistory : 0u;
for (auto i = startIndex; i < currChunkHistorySize; ++i)
{
pushToQueue(queueToAdd, getMembers()->m_history[i].cloneToSharedChunk());
}

另外未 offer 时 sendChunk() 不投递、只入 history(publisher_port_user.cpp 第 56~72 行),供 AUTOSAR field 之类”先设值后 offer”的用法。

9. 异常场景

9.1 订阅队列满 / 订阅者太慢

由发布者的 subscriberTooSlowPolicy 与订阅者的 queueFullPolicy 组合决定(不兼容组合在 RouDi 匹配时就被拒绝,见 04 篇 5.3 节):

组合 行为
DISCARD_OLDEST_DATA + DISCARD_OLDEST_DATA(默认) SOFI 队列挤掉最老样本,lostAChunk() 置丢失标记,订阅者可用 hasMissedData() 查询(chunk_queue_popper.inl 第 76 行)
WAIT_FOR_CONSUMER + BLOCK_PRODUCER 发布者在 deliverToAllStoredQueues() 中对满队列 busy-wait(std::this_thread::yield() 循环,chunk_distributor.inl 第 173~210 行),实现可靠背压;应用退出时靠 PREPARE_APP_TERMINATION 让 RouDi 强制 STOP_OFFER 解除阻塞

9.2 mempool 耗尽 / 样本过大

MemoryManager::getChunk() 区分三种错误(memory_manager.cpp 第 172~201 行):无 mempool(FATAL)、无足够大的 mempool(NO_MEMPOOL_FOR_REQUESTED_CHUNK_SIZE,样本大小超过配置上限)、池空(MEMPOOL_OUT_OF_CHUNKS)。上层映射为 AllocationError::RUNNING_OUT_OF_CHUNKS 等返回给 loan() 调用者。chunk 耗尽常见原因:订阅者长期持有 Sample 不释放、history 容量过大、queueCapacity × 订阅者数超过池容量。

9.3 chunk 泄漏检测与崩溃清理

  • 每个端口的 m_chunksInUseUsedChunkList)记录借出未还的 chunk。应用正常销毁端口或崩溃后 RouDi 清理端口时调用 releaseAllChunks()ChunkSender::releaseAll()
1
2
3
4
5
6
7
template <typename ChunkSenderDataType>
inline void ChunkSender<ChunkSenderDataType>::releaseAll() noexcept
{
getMembers()->m_chunksInUse.cleanup();
this->cleanup();
getMembers()->m_lastChunkUnmanaged.releaseToSharedChunk();
}
  • 应用把 Sample 的 payload 指针传错/重复释放会触发 kPOPO__CHUNK_SENDER_INVALID_CHUNK_TO_FREE_FROM_USER / kPOSH__MEMPOOL_POSSIBLE_DOUBLE_FREE
  • 源码注释明确标注了两处”临界区”:tryAllocate/send 中若进程恰好在拿到 chunk 与登记之间死亡,该 chunk 会泄漏(chunk_sender.inl 第 142、188 行注释);以及若应用死在 ChunkDistributor 持锁期间,RouDi 清理会死锁并 FATAL(chunk_distributor.inl 第 351~358 行 cleanup() 注释)。
  • 内省 mempool topic 展示每个池的 usedChunks 与历史最小空闲 minFreemem_pool.cppMemPoolInfo),是定位泄漏的第一工具。

10. 端到端时序图

订阅进程ConditionVariable(sem)订阅队列(无锁,共享内存)MemPool(共享内存)发布进程订阅进程ConditionVariable(sem)订阅队列(无锁,共享内存)MemPool(共享内存)发布进程placement-new T 原地构造(0拷贝)读 const T* (同一块物理内存)loan(): LoFFLi pop 空闲index → chunkpublish(): push 8字节相对指针 (refcount++)notify(): sem_postsem_wait 返回take(): pop → 相对指针转绝对地址Sample析构: refcount-- == 0 → freeChunk

11. 延迟特性与 iceperf 基准

iceoryx_examples/iceperf 是官方 ping-pong 延迟基准:leader/follower 两个进程对 1 KB~4096 KB 的 payload 各做 N 次 round trip,对比 POSIX MQ、UDS、iceoryx C++/C API(iceperf_leader.cppdoMeasurement())。README 中官方参考结果(Ubuntu 18.04, Xeon E3-1505M v5, 10 万次往返,单程平均延迟 µs):

Payload MQ UDS iceoryx
1 kB 3.1 4.3 0.73
64 kB 23 27 0.6
512 kB 160 210 0.61
4096 kB 1200 1700 0.61

关键观察:iceoryx 延迟 ≈ 0.6 µs 且与 payload 大小无关(只传指针),而 MQ/UDS 随大小线性增长(每字节都要过内核拷贝两次)。运行方式:

1
2
3
iox-roudi &
./iceperf-bench-follower &
./iceperf-bench-leader -n 100000

延迟构成(iceoryx 路径):LoFFLi pop/push 若干原子操作 + 队列 push/pop + (事件模式下)一次 sem_post/sem_wait。轮询模式可再省掉信号量,进入亚微秒级。

12. 使用约束

零拷贝的前提是”数据放哪儿都能被别的进程直接解释”,因此:

  1. POD / 无指针数据:类型必须可以直接放在共享内存——不能含裸指针、引用、虚函数表、std::string/std::vector 等堆分配成员(各进程堆地址不同、且堆不共享)。iceoryx 提供 iox::cxx::string/vector(定长、可放共享内存)替代。严格说要求可 relocatable,编译期没有强制的 is_trivially_copyable 校验(待验证:typed API 不做此 static_assert,需开发者自行保证)。
  2. 固定大小上限:chunk 来自预配置的 mempool,样本(含 ChunkHeader/user-header)不能超过最大 mempool 的 chunk 大小;动态大小需求要按最坏情况配置池(TOML,见 04 篇 8.1 节),或用 untyped API 自带 size 参数 loan。
  3. 同机限制:共享内存不跨主机。跨机需要网关(iceoryx-dds、cyclonedds/FastDDS 的 iceoryx 集成)把 chunk 内容再序列化上网络——那一跳自然不再是零拷贝。
  4. 其他实践约束:订阅者拿到的是 const 视图不可写;loan 出的 chunk 应尽快 publish 或释放(占用 m_chunksInUse 名额);单个 publisher 并行借出 chunk 数、单个 subscriber 并行持有 chunk 数均有编译期上限(iceoryx_posh_types.hppMAX_CHUNKS_ALLOCATED_PER_PUBLISHER_SIMULTANEOUSLY / MAX_CHUNKS_HELD_PER_SUBSCRIBER_SIMULTANEOUSLY)。

下一篇

06 C 绑定与 DDS 网关

06 C 绑定与 DDS 网关

本文重点iceoryx_binding_c 的 C API 设计(storage 模式、C↔C++ 映射、payload/user-header API),以及 iceoryx_dds 的 iox-dds-gateway(GatewayGeneric 框架、DDS↔iceoryx 双向桥接、Cyclone DDS 数据写入器)。
源码锚点:iceoryx_binding_c/source/c_publisher.cppiceoryx_binding_c/include/iceoryx_binding_c/*.hiceoryx_dds/include/iceoryx_dds/internal/gateway/*.inliceoryx_dds/source/gateway/main.cpp

源码根目录:/home/cp/work2/ros2Learn/ros2_humble/src/eclipse-iceoryx/iceoryx(v2.0.6)


1. 为什么需要 C 绑定

iceoryx 核心(iceoryx_posh)是 C++14 实现,大量使用模板(Publisher<T>)、cxx::expected、RAII。但下游一些用户是纯 C 代码——最典型的就是 CycloneDDS(C 语言实现的 DDS),其 ENABLE_SHM 零拷贝路径完全通过 iceoryx_binding_c 的 C API 调用 iceoryx(见 07-与CycloneDDS及ROS2集成.md)。

iceoryx_binding_c 提供的 C 对象一览:

C 句柄 头文件 对应 C++ 实体 用途
iox_pub_t publisher.h cpp2c_Publisher(包装 PublisherPortUser 发布(loan/publish chunk)
iox_sub_t subscriber.h cpp2c_Subscriber(包装 SubscriberPortUser 订阅(take/release chunk)
iox_ws_t wait_set.h cpp2c_WaitSet 阻塞等待多事件
iox_listener_t listener.h iox::popo::Listener 后台线程回调
iox_user_trigger_t user_trigger.h UserTrigger 用户自定义触发
iox_client_t / iox_server_t client.h / server.h UntypedClient / UntypedServer 请求-响应
iox_service_discovery_t service_discovery.h ServiceDiscovery 服务发现
iox_runtime_init() runtime.h PoshRuntime::initRuntime() 向 RouDi 注册进程

2. storage 模式:用户提供内存

C 没有构造函数,iceoryx 又刻意避免在 API 内部隐式堆分配,所以 C 绑定采用 “storage 结构体 + init 函数” 模式:调用者先在自己的栈/静态区准备一块 iox_xxx_storage_t,再交给 iox_xxx_init() 初始化并返回句柄。

1
2
3
4
5
typedef struct
{
// only size for pointer is necessary
uint64_t do_not_touch_me[1];
} iox_pub_storage_t;

注意注释 “only size for pointer is necessary”:在 v2.0 中这些 storage 结构体只需容纳一个指针——iox_pub_init 实际上仍会 new 一个 cpp2c_Publisher,并把指针存进 storage:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
iox_pub_t iox_pub_init(iox_pub_storage_t* self,
const char* const service,
const char* const instance,
const char* const event,
const iox_pub_options_t* const options)
{
if (self == nullptr)
{
LogWarn() << "publisher initialization skipped - null pointer provided for iox_pub_storage_t";
return nullptr;
}
// ... 校验 options 是否经过 iox_pub_options_init ...
auto* me = new cpp2c_Publisher();
self->do_not_touch_me[0] = reinterpret_cast<uint64_t>(me);

me->m_portData = PoshRuntime::getInstance().getMiddlewarePublisher(
ServiceDescription{
IdString_t(TruncateToCapacity, service),
IdString_t(TruncateToCapacity, instance),
IdString_t(TruncateToCapacity, event),
},
publisherOptions);
return me;
}

要点:

  • 服务寻址沿用 iceoryx 的 CaPro 三元组 {service, instance, event}(字符串),与 C++ 层 ServiceDescription 一一对应。
  • iox_pub_init 直接绑定到底层端口对象 PublisherPortDatagetMiddlewarePublisher),后续所有操作都是围绕该共享内存端口的薄包装,没有额外拷贝层。
  • storage 模式让 CycloneDDS 这类用户可以写 iox_pub_init(&(iox_pub_storage_t){0}, ...)(复合字面量),不关心生命周期细节;iox_pub_deinit 负责 delete

2.1 options 的防呆设计

C 结构体无法保证零初始化,为防止用户漏调 init,options 里埋了 magic number(initCheck):

1
2
3
4
5
6
7
8
9
10
11
12
13
constexpr uint64_t PUBLISHER_OPTIONS_INIT_CHECK_CONSTANT = 123454321;

void iox_pub_options_init(iox_pub_options_t* options)
{
// ...
PublisherOptions publisherOptions;
options->historyCapacity = publisherOptions.historyCapacity;
options->nodeName = nullptr;
options->offerOnCreate = publisherOptions.offerOnCreate;
options->subscriberTooSlowPolicy = cpp2c::consumerTooSlowPolicy(publisherOptions.subscriberTooSlowPolicy);

options->initCheck = PUBLISHER_OPTIONS_INIT_CHECK_CONSTANT;
}

若用户传入未 init 的 options,iox_pub_init 会走 errorHandler(Error::kBINDING_C__PUBLISHER_OPTIONS_NOT_INITIALIZED)(LogFatal 后终止)。


3. Publisher C API:loan / publish 与 user-header

发布侧核心 API(publisher.h)分三级,逐级增加控制粒度:

API payload 对齐 user-header 说明
iox_pub_loan_chunk 默认 8 字节 最常用
iox_pub_loan_aligned_chunk 自定义 大对齐类型(如 SIMD)
iox_pub_loan_aligned_chunk_with_user_header 自定义 自定义大小/对齐 CycloneDDS 用它挂 iceoryx_header_t

三者最终收敛到同一实现——直接调 C++ 端口层的 tryAllocateChunk

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
iox_AllocationResult iox_pub_loan_aligned_chunk_with_user_header(iox_pub_t const self,
void** const userPayload,
const uint32_t userPayloadSize,
const uint32_t userPayloadAlignment,
const uint32_t userHeaderSize,
const uint32_t userHeaderAlignment)
{
auto result = PublisherPortUser(self->m_portData)
.tryAllocateChunk(userPayloadSize, userPayloadAlignment, userHeaderSize, userHeaderAlignment)
.and_then([&userPayload](ChunkHeader* h) { *userPayload = h->userPayload(); });
if (result.has_error())
{
return cpp2c::allocationResult(result.get_error());
}

return AllocationResult_SUCCESS;
}

发布与释放同样是对 ChunkHeader 的直译(用户拿到的是 payload 指针,内部通过 ChunkHeader::fromUserPayload 反查头部):

1
2
3
4
5
6
7
8
9
void iox_pub_release_chunk(iox_pub_t const self, void* const userPayload)
{
PublisherPortUser(self->m_portData).releaseChunk(ChunkHeader::fromUserPayload(userPayload));
}

void iox_pub_publish_chunk(iox_pub_t const self, void* const userPayload)
{
PublisherPortUser(self->m_portData).sendChunk(ChunkHeader::fromUserPayload(userPayload));
}

其余管理 API:iox_pub_offer / iox_pub_stop_offer / iox_pub_is_offered / iox_pub_has_subscribers / iox_pub_get_service_description

3.1 payload ↔ user-header ↔ chunk-header 转换(chunk.h)

一个 chunk 的内存布局是 ChunkHeader | user-header(可选) | user-payload(详见 posh 的 mepoo::ChunkHeader)。C 侧通过 chunk.h 的 6 个函数在三者之间导航:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
/// @brief gets the user-payload from the chunk-header
void* iox_chunk_header_to_user_payload(iox_chunk_header_t* const chunkHeader);

const void* iox_chunk_header_to_user_payload_const(const iox_chunk_header_t* const chunkHeader);

/// @brief gets the user-header from the chunk-header
void* iox_chunk_header_to_user_header(iox_chunk_header_t* const chunkHeader);

const void* iox_chunk_header_to_user_header_const(const iox_chunk_header_t* const chunkHeader);

/// @brief gets the chunk-header from the user-payload
iox_chunk_header_t* iox_chunk_header_from_user_payload(void* const userPayload);

const iox_chunk_header_t* iox_chunk_header_from_user_payload_const(const void* const userPayload);

CycloneDDS 正是用 iox_chunk_header_from_user_payload + iox_chunk_header_to_user_header 把自己的 iceoryx_header_t(GUID、时间戳、keyhash 等 DDS 元数据)藏进 user-header(见 07 篇 §4)。


4. Subscriber C API

订阅侧与发布侧对称(subscriber.h):

API 说明
iox_sub_init / iox_sub_deinit storage 模式创建/销毁
iox_sub_take_chunk(sub, &userPayload) 取一个 chunk(零拷贝,拿到共享内存指针)
iox_sub_release_chunk 归还 chunk(引用计数减一,勿忘!否则 chunk 泄漏)
iox_sub_release_queued_chunks 清空队列中所有未取的 chunk
iox_sub_has_chunks / iox_sub_has_lost_chunks 有无数据 / 是否因队列溢出丢过数据
iox_sub_subscribe / iox_sub_unsubscribe / iox_sub_get_subscription_state 订阅状态管理

iox_sub_options_t 中值得注意的字段:queueCapacity(接收队列深度)、historyRequest(连接时补发历史样本数)、queueFullPolicy(队列满时丢最旧 or 阻塞发布者,对应 C++ 的 QueueFullPolicy)。


5. WaitSet 与 Listener C API

两者是 iceoryx 的两种事件通知机制在 C 侧的映射:

  • WaitSet(iox_ws_t:用户线程主动阻塞等待,语义类似 epoll
  • Listener(iox_listener_t:内部起后台线程,事件到达时调用用户回调,语义类似”信号槽”。

WaitSet 核心等待接口:

1
2
3
4
5
6
7
8
9
10
11
uint64_t iox_ws_timed_wait(iox_ws_t const self,
struct timespec timeout,
iox_notification_info_t* const notificationInfoArray,
const uint64_t notificationInfoArrayCapacity,
uint64_t* missedElements);

/// @brief waits until an event occurred
uint64_t iox_ws_wait(iox_ws_t const self,
iox_notification_info_t* const notificationInfoArray,
const uint64_t notificationInfoArrayCapacity,
uint64_t* missedElements);

可附着(attach)的对象包括 subscriber 的 state(如 SubscriberState_HAS_DATA,电平语义)和 event(如 SubscriberEvent_DATA_RECEIVED,边沿语义),以及 user-trigger、client、server、service-discovery。每个 attach 均有 _with_context_data 变体,把 void* 用户数据透传进回调:

1
2
3
4
5
6
ENUM iox_WaitSetResult iox_ws_attach_subscriber_state_with_context_data(iox_ws_t const self,
iox_sub_t const subscriber,
const ENUM iox_SubscriberState subscriberState,
const uint64_t id,
void (*callback)(iox_sub_t, void*),
void* const contextData);

Listener 的 attach 接口形态一致(无 id、无超时,事件直接驱动回调线程):

1
2
3
4
ENUM iox_ListenerResult iox_listener_attach_subscriber_event(iox_listener_t const self,
iox_sub_t const subscriber,
const ENUM iox_SubscriberEvent subscriberEvent,
void (*callback)(iox_sub_t));

CycloneDDS 的 shm_monitor.c 就是典型 Listener 用户:iox_listener_attach_subscriber_event_with_context_data(..., SubscriberEvent_DATA_RECEIVED, shm_subscriber_callback, &reader->m_iox_sub_context),在 iceoryx 后台线程里把 chunk 搬进 DDS reader cache(见 07 篇 §5)。

5.1 C ↔ C++ 映射层

internal/ 下的翻译层保证两种语言的枚举/对象一致:

文件 方向 内容
c2cpp_enum_translation.hpp/.cpp C → C++ iox_QueueFullPolicypopo::QueueFullPolicy
cpp2c_enum_translation.hpp/.cpp C++ → C AllocationErroriox_AllocationResult
cpp2c_publisher.hpp / cpp2c_subscriber.hpp C 句柄背后的包装类(持有 m_portData 端口指针)
cpp2c_waitset.hpp WaitSet 的 C 包装
c2cpp_binding.h CLASS/ENUM 宏:同一头文件在 C 里是 struct/裸枚举,在 C++ 里是 class/强类型

6. iceoryx_dds:iox-dds-gateway

6.1 用途与定位

iceoryx 只做同一台主机内的共享内存通信。当数据需要跨主机时,iceoryx_dds 提供可执行程序 iox-dds-gateway:它作为一个普通 iceoryx 应用挂到 RouDi 上,把本机 iceoryx topic 的数据经 DDS(网络) 转发出去,并把远端 DDS 数据注入本机 iceoryx,从而组成”机内零拷贝 + 机间 DDS 网络”的混合拓扑。

1
2
主机 A                                      主机 B
app --共享内存--> iox-dds-gateway ==DDS/UDP网络==> iox-dds-gateway --共享内存--> app

注意iceoryx_dds/ 目录下存在 COLCON_IGNORE 空标记文件(iceoryx_examples/ 同样有)。这意味着 ROS 2 Humble 用 colcon 构建工作区时该包被完全跳过、不参与编译。ROS 2 场景下跨主机通信由 CycloneDDS 自身的 UDP 路径完成(一份数据同时走 iceoryx 与网络,见 07 篇),并不需要 iox-dds-gateway。该网关面向的是”纯 iceoryx 应用 + 跨机扩展”的独立部署场景,需单独用 CMake(-DDDS_STACK=CYCLONE_DDS,见 iceoryx_dds/CMakeLists.txt)构建。

6.2 GatewayGeneric 框架

posh 提供了通用网关基类 gw::GatewayGeneric<channel_t>iceoryx_posh/gateway/gateway_generic.hpp):内部维护一组 Channel(iceoryx 终端 + 外部终端的配对),起两个线程分别周期执行:

  • discovery 线程(默认 1000ms):消费 RouDi 广播的 CaPro 发现消息,调用子类 discover() 动态建/拆 Channel;
  • forwarding 线程(默认 50ms):遍历所有 Channel 调用子类 forward() 搬运数据。

子类只需实现 loadConfiguration() / discover() / forward() 三个纯虚函数。iceoryx_dds 基于它派生出两个方向的网关:

1
2
3
4
5
6
7
8
9
10
11
12
/// @brief DDS Gateway implementation for the iceoryx to DDS direction.
template <typename channel_t = gw::Channel<popo::UntypedSubscriber, dds::data_writer_t>,
typename gateway_t = gw::GatewayGeneric<channel_t>>
class Iceoryx2DDSGateway : public gateway_t
{
public:
/// @brief Creates a gateway with DDS set as interface
Iceoryx2DDSGateway() noexcept;

void loadConfiguration(const config::GatewayConfig& config) noexcept;
void discover(const capro::CaproMessage& msg) noexcept;
void forward(const channel_t& channel) noexcept;

Channel 的两端:iceoryx 侧是 UntypedSubscriber(iox→DDS 方向)或 UntypedPublisher(DDS→iox 方向);外部侧是 dds::data_writer_t / data_reader_t(编译期通过 dds_types.hpp 绑定到 Cyclone 实现)。

6.3 iceoryx → DDS 方向

发现:收到 RouDi 的 OFFER CaPro 消息就自动为该服务建 Channel(STOP_OFFER 则拆除),内省服务被排除:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
switch (msg.m_type)
{
case capro::CaproMessageType::OFFER:
{
if (!this->findChannel(msg.m_serviceDescription).has_value())
{
popo::SubscriberOptions options;
options.queueCapacity = SUBSCRIBER_CACHE_SIZE;
IOX_DISCARD_RESULT(setupChannel(msg.m_serviceDescription, options));
}
break;
}
case capro::CaproMessageType::STOP_OFFER:
{
if (this->findChannel(msg.m_serviceDescription).has_value())
{
IOX_DISCARD_RESULT(this->discardChannel(msg.m_serviceDescription));
}
break;
}

转发:从 untyped subscriber take 出 chunk,把 ChunkHeader 的元信息填进 IoxChunkDatagramHeader,连同 user-header、payload 交给 DDS writer:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
template <typename channel_t, typename gateway_t>
inline void Iceoryx2DDSGateway<channel_t, gateway_t>::forward(const channel_t& channel) noexcept
{
auto subscriber = channel.getIceoryxTerminal();
while (subscriber->hasData())
{
subscriber->take().and_then([&](const void* userPayload) {
auto dataWriter = channel.getExternalTerminal();
auto chunkHeader = iox::mepoo::ChunkHeader::fromUserPayload(userPayload);
iox::dds::IoxChunkDatagramHeader datagramHeader;
datagramHeader.userHeaderId = chunkHeader->userHeaderId();
datagramHeader.userHeaderSize = chunkHeader->userHeaderSize();
datagramHeader.userPayloadSize = chunkHeader->userPayloadSize();
datagramHeader.userPayloadAlignment = chunkHeader->userPayloadAlignment();
dataWriter->write(datagramHeader,
static_cast<const uint8_t*>(chunkHeader->userHeader()),
static_cast<const uint8_t*>(chunkHeader->userPayload()));
subscriber->release(userPayload);
});
}
}

IoxChunkDatagramHeader 是跨网络传输 chunk 的线格式头(版本号 + 端序 + user-header/payload 尺寸与对齐):

1
2
3
4
5
6
7
8
9
10
11
static constexpr uint8_t DATAGRAM_VERSION{1U};

/// @note This must always be the first member and always 1 bytes in order to prevent issues with endianess when
/// deserialized or incorrectly detected versions due to different size
uint8_t datagramVersion{DATAGRAM_VERSION};
/// @note This must always be 1 byte in order to prevent issues with endianess when deserialized
Endianess endianness{Endianess::UNDEFINED};
uint16_t userHeaderId{0xFFFF};
uint32_t userHeaderSize{0U};
uint32_t userPayloadSize{0U};
uint32_t userPayloadAlignment{0U};

6.4 CycloneDataWriter:DDS 侧如何写

cyclone_data_writer.cpp 使用 CycloneDDS 的 C++ 绑定(cyclonedds-cxx)。topic 名由 CaPro 三元组拼接,类型是 IDL 定义的字节序列 Mempool::Chunkiceoryx_dds/msg/Mempool.idl):

1
2
3
4
5
6
7
8
void iox::dds::CycloneDataWriter::connect() noexcept
{
m_publisher = ::dds::pub::Publisher(CycloneContext::getParticipant());
auto topic = "/" + std::string(m_serviceId) + "/" + std::string(m_instanceId) + "/" + std::string(m_eventId);
m_topic = ::dds::topic::Topic<Mempool::Chunk>(CycloneContext::getParticipant(), topic);
m_writer = ::dds::pub::DataWriter<Mempool::Chunk>(m_publisher, m_topic);
LogDebug() << "[CycloneDataWriter] Connected to topic: " << topic;
}

write()序列化的 datagramHeader | user-header | payload 三段依次拷贝进 Mempool::Chunk 的字节 vector 再 m_writer.write(chunk)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
datagramHeader.endianness = getEndianess();

auto serializedDatagramHeader = iox::dds::IoxChunkDatagramHeader::serialize(datagramHeader);
auto datagramSize =
serializedDatagramHeader.size() + datagramHeader.userHeaderSize + datagramHeader.userPayloadSize;

auto chunk = Mempool::Chunk();
chunk.payload().reserve(datagramSize);

std::copy(serializedDatagramHeader.data(),
serializedDatagramHeader.data() + serializedDatagramHeader.size(),
std::back_inserter(chunk.payload()));
// ... 追加 userHeaderBytes、userPayloadBytes ...
m_writer.write(chunk);

显然这一步是有拷贝、有序列化的——网关做的是”共享内存 → 网络报文”的边界转换,零拷贝只存在于机内 iceoryx 段。

6.5 DDS → iceoryx 方向

DDS2IceoryxGateway::forward() 是镜像流程:先 peek DDS 侧的 datagram 头,按其中的 size/alignment 向 iceoryx loan 一个 chunk,再把网络数据 takeNext 直接解到 chunk 的 user-header/payload 里后 publish

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
while (reader->hasSamples())
{
reader->peekNextIoxChunkDatagramHeader().and_then([&](auto datagramHeader) {
constexpr uint32_t USER_HEADER_ALIGNMENT{1U};
publisher
->loan(datagramHeader.userPayloadSize,
datagramHeader.userPayloadAlignment,
datagramHeader.userHeaderSize,
USER_HEADER_ALIGNMENT)
.and_then([&](auto userPayload) {
auto chunkHeader = iox::mepoo::ChunkHeader::fromUserPayload(userPayload);
reader
->takeNext(datagramHeader,
static_cast<uint8_t*>(chunkHeader->userHeader()),
static_cast<uint8_t*>(chunkHeader->userPayload()))
.and_then([&]() { publisher->publish(userPayload); })
// ... or_else: release + LogWarn ...

局限:DDS2IceoryxGateway::discover() 是空实现(源码注释 “requires dds discovery which is currently not implemented in the used dds stack”),因此 DDS→iox 方向只能靠 TOML 静态配置服务列表,无法自动发现远端 DDS topic。

6.6 网关主程序

iox-dds-gateway 可执行文件同时跑两个方向的网关,配置来自 TOML(默认 /etc/iceoryx/gateway_config.toml,解析失败则退回”全自动发现”默认配置):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
iox::runtime::PoshRuntime::initRuntime("iox-dds-gateway");

iox::config::GatewayConfig gatewayConfig;
iox::dds::Iceoryx2DDSGateway<> iox2ddsGateway;
iox::dds::DDS2IceoryxGateway<> dds2ioxGateway;

iox::config::TomlGatewayConfigParser::parse()
.and_then([&](auto config) { gatewayConfig = config; })
.or_else([&](auto err) {
iox::dds::LogWarn() << "[Main] Failed to parse gateway config with error: "
<< iox::config::TOML_GATEWAY_CONFIG_FILE_PARSE_ERROR_STRINGS[err];
iox::dds::LogWarn() << "[Main] Using default configuration.";
gatewayConfig.setDefaults();
});

iox2ddsGateway.loadConfiguration(gatewayConfig);
dds2ioxGateway.loadConfiguration(gatewayConfig);

iox2ddsGateway.runMultithreaded();
dds2ioxGateway.runMultithreaded();

7. 小结

  • iceoryx_binding_c 是对 posh 端口层的零抽象成本直译:storage 模式规避隐式分配 API 的观感(内部仍有一次 new),loan/publish/take/release 与 C++ 完全同语义,是 CycloneDDS SHM 集成的地基。
  • iceoryx_dds 展示了 GatewayGeneric 的扩展方式:discovery + forwarding 双线程模板,把机内共享内存数据桥接到 DDS 网络;但在 ROS 2 Humble 构建中被 COLCON_IGNORE 排除,属独立部署组件。

下一篇

07-与CycloneDDS及ROS2集成.md:CycloneDDS 如何用这些 C API 实现同机零拷贝,以及 ROS 2 Humble 的完整启用链路。

07 与 CycloneDDS 及 ROS 2 集成

07 与 CycloneDDS 及 ROS 2 集成

本文重点:CycloneDDS 如何借助 iceoryx 实现同机零拷贝(ENABLE_SHM),以及 ROS 2 Humble 的完整链路 rmw_cyclonedds_cpp → cyclonedds → iceoryx
源码锚点(cyclonedds 0.10.x,随 ROS 2 Humble):ddsi_shm_transport.c/.hshm_monitor.cdds_loan.cdds_write.cq_init.cddsi_cfgelems.h

cyclonedds 根目录:/home/cp/work2/ros2Learn/ros2_humble/src/eclipse-cyclonedds/cyclonedds
rmw 根目录:/home/cp/work2/ros2Learn/ros2_humble/src/ros2/rmw_cyclonedds/rmw_cyclonedds_cpp


1. 整体链路

1
2
3
4
5
6
7
8
9
10
11
12
13
ROS 2 应用
│ rclcpp::LoanedMessage / publish

rmw_cyclonedds_cpp (rmw_node.cpp: borrow_loaned_message → dds_loan_sample)


cyclonedds ddsc 层 (dds_write.c: dds_write_impl_iox / dds_loan.c)
│ iox_pub_loan_aligned_chunk_with_user_header / iox_pub_publish_chunk

iceoryx_binding_c → iceoryx_posh 共享内存段

│ iox_listener 回调 (shm_monitor.c) → iox_sub_take_chunk → rhc_store
cyclonedds 读端 ← ← ← ← ← ┘

前提条件汇总(缺一不可,任一不满足则静默退回 UDP loopback):

条件 层次
cyclonedds 以 -DENABLE_SHM=ON 编译(定义 DDS_HAS_SHM 编译期
CYCLONEDDS_URISharedMemory/Enable=true 配置
iox-roudi 守护进程已运行 运行时
读写双方在同一台主机(同一 RouDi 域) 拓扑
QoS:KEEP_LAST + VOLATILE/TRANSIENT_LOCAL 等(§6) QoS
消息类型固定大小(POD)才能”真零拷贝”,否则序列化进共享内存 类型

2. 编译与配置开关

2.1 编译期:ENABLE_SHM

CMake 选项 ENABLE_SHM 打开后定义 DDS_HAS_SHMSharedMemory 配置组才会出现在 schema 中:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
static struct cfgelem shmem_cfgelems[] = {
BOOL("Enable", NULL, 1, "false",
MEMBER(enable_shm),
FUNCTIONS(0, uf_boolean, 0, pf_boolean),
DESCRIPTION("<p>This element allows to enable shared memory in Cyclone DDS.</p>")),
STRING("Locator", NULL, 1, "",
MEMBER(shm_locator),
FUNCTIONS(0, uf_string, ff_free, pf_string),
DESCRIPTION(
"<p>Explicitly set the Iceoryx locator used by Cyclone to check whether "
"a pair of processes is attached to the same Iceoryx shared memory. The "
"default is to use one of the MAC addresses of the machine, which should "
"work well in most cases.</p>"
)),
STRING("Prefix", NULL, 1, "DDS_CYCLONE",
MEMBER(iceoryx_service),
FUNCTIONS(0, uf_string, ff_free, pf_string),
DESCRIPTION(
"<p>Override the Iceoryx service name used by Cyclone.</p>"
)),
ENUM("LogLevel", NULL, 1, "info",
MEMBER(shm_log_lvl),
...

四个配置项:Enable(默认 false)、Locator(同机判定地址,默认取 MAC)、Prefix(iceoryx CaPro service 名,默认 DDS_CYCLONE)、LogLevel

2.2 运行时:CYCLONEDDS_URI

1
2
3
4
5
6
7
8
9
# ROS 2 Humble 启用零拷贝的典型环境
export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
export CYCLONEDDS_URI='<CycloneDDS><Domain><SharedMemory><Enable>true</Enable><LogLevel>info</LogLevel></SharedMemory></Domain></CycloneDDS>'

# 终端 1:先起 RouDi(管理共享内存段与 mempool)
iox-roudi

# 终端 2/3:正常跑 ROS 2 节点
ros2 run demo_nodes_cpp talker

ROS 2 Humble 官方仓库 rmw_cyclonedds 附带 shared_memory_support.md,其配置方式与此一致。


3. 域初始化:iceoryx 作为”虚拟传输”

q_init.cenable_shm 为真时执行 iceoryx_init():注册 iceoryx runtime(进程名 iceoryx_rt_<pid>_<启动时间>),并通过 vnet(虚拟传输工厂) 挂一个 kind 为 NN_LOCATOR_KIND_SHEM(值 16,q_protocol.h:110)的传输:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
static int iceoryx_init (struct ddsi_domaingv *gv)
{
shm_set_loglevel(gv->config.shm_log_lvl);

char *sptr;
ddsrt_asprintf (&sptr, "iceoryx_rt_%"PRIdPID"_%"PRId64, ddsrt_getpid (), gv->tstart.v);
GVLOG (DDS_LC_SHM, "Current process name for iceoryx is %s\n", sptr);
iox_runtime_init (sptr);
free(sptr);

// FIXME: this can be done more elegantly when properly supporting multiple transports
if (ddsi_vnet_init (gv, "iceoryx", NN_LOCATOR_KIND_SHEM) < 0)
return -1;
ddsi_factory_find (gv, "iceoryx")->m_enable = true;

SHEM locator 的地址如何来:未显式配置 SharedMemory/Locator 时取本机第一个非 loopback 网卡的 MAC 地址——它唯一标识”这台机器”,用于后续同机判定:

1
2
3
4
5
6
7
8
memset (gv->loc_iceoryx_addr.address, 0, sizeof (gv->loc_iceoryx_addr.address));
if (ddsrt_eth_get_mac_addr (gv->interfaces[if_index].name, gv->loc_iceoryx_addr.address))
{
GVERROR ("Unable to get MAC address for iceoryx\n");
return -1;
}
gv->loc_iceoryx_addr.kind = NN_LOCATOR_KIND_SHEM;
gv->loc_iceoryx_addr.port = 0;

4. 发现:如何判定”同机可用 SHM”

  1. 发布:端点 QoS 未屏蔽 SHEM 时,SEDP 报文的 locator 列表里额外带上本机的 SHEM locator(q_ddsi_discovery.c:1190 起,判断 !(xqos->ignore_locator_type & NN_LOCATOR_KIND_SHEM))。
  2. 接收:建立 proxy 端点时,遍历对端 locator,若发现 kind 为 SHEM 且 地址(MAC)与本机 loc_iceoryx_addr 相同,即认定对端与本进程挂在同一个 iceoryx 共享内存上:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
static void has_iceoryx_address_helper (const ddsi_xlocator_t *n, void *varg)
{
struct has_iceoryx_address_helper_arg *arg = varg;
if (n->c.kind == NN_LOCATOR_KIND_SHEM && memcmp (arg->loc_iceoryx_addr->address, n->c.address, sizeof (arg->loc_iceoryx_addr->address)) == 0)
arg->has_iceoryx_address = true;
}

static bool has_iceoryx_address (struct ddsi_domaingv * const gv, struct addrset * const as)
{
if (!gv->config.enable_shm)
return false;
else
{
struct has_iceoryx_address_helper_arg arg = {
.loc_iceoryx_addr = &gv->loc_iceoryx_addr,
.has_iceoryx_address = false
};
addrset_forall (as, has_iceoryx_address_helper, &arg);
return arg.has_iceoryx_address;
}
}

判定结果存入 pwr->is_iceoryx / prd->is_iceoryxddsi_proxy_endpoint.c:240、562)。匹配(match)时进一步与本地端点 QoS 合成最终决策,例如 writer 侧:

1
2
3
4
5
#ifdef DDS_HAS_SHM
const bool use_iceoryx = prd->is_iceoryx && !(wr->xqos->ignore_locator_type & NN_LOCATOR_KIND_SHEM);
#else
const bool use_iceoryx = false;
#endif

use_iceoryx 为真时,该 reader 视作”已确认所有 Heartbeat”(可靠性交给 iceoryx 队列,跳过 RTPS 重传机制)。发送端 nn_xpack_send1跳过对 SHEM locator 的真实网络发送

1
2
3
4
5
6
#ifdef DDS_HAS_SHM
// SHM_TODO: We avoid sending packet while data is SHMEM.
// I'm not sure whether this is correct or not.
if (!gv->mute && loc->c.kind != NN_LOCATOR_KIND_SHEM)
#else
if (!gv->mute)

5. 端点创建与数据路径

5.1 writer/reader 是否挂 iceoryx:QoS 闸门

创建 writer 时先过 dds_writer_support_shm(),不满足则把 ignore_locator_type |= NN_LOCATOR_KIND_SHEM(即该端点彻底退出 SHM 路径):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
static bool dds_writer_support_shm(const struct ddsi_config* cfg, const dds_qos_t* qos, const struct dds_topic *tp)
{
if (NULL == cfg ||
false == cfg->enable_shm)
return false;

// check necessary condition: fixed size data type OR serializing into shared
// memory is available
if (!tp->m_stype->fixed_size && (!tp->m_stype->ops->get_serialized_size ||
!tp->m_stype->ops->serialize_into)) {
return false;
}

// only VOLATILE or TRANSIENT LOCAL
if(!(qos->durability.kind == DDS_DURABILITY_VOLATILE ||
qos->durability.kind == DDS_DURABILITY_TRANSIENT_LOCAL)) {
return false;
}

// only KEEP LAST
if(qos->history.kind != DDS_HISTORY_KEEP_LAST) {
return false;
}

reader 侧 dds_reader_support_shm()dds_reader.c:510)条件类似,另加 ignorelocal == DDS_IGNORELOCAL_NONE。通过后创建 iceoryx 端点,CaPro 三元组 = {Prefix 配置(默认 DDS_CYCLONE), 类型名, topic 名}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
#ifdef DDS_HAS_SHM
if (wr->m_wr->has_iceoryx)
{
DDS_CLOG (DDS_LC_SHM, &wr->m_entity.m_domain->gv.logconfig, "Writer's topic name will be DDS:Cyclone:%s\n", wr->m_topic->m_name);
iox_pub_options_t opts = create_iox_pub_options(wqos);

// NB: This may fail due to icoeryx being out of internal resources for publishers
// In this case terminate is called by iox_pub_init.
// it is currently (iceoryx 2.0 and lower) not possible to change this to
// e.g. return a nullptr and handle the error here.
wr->m_iox_pub = iox_pub_init(&(iox_pub_storage_t){0}, gv->config.iceoryx_service, wr->m_topic->m_stype->type_name, wr->m_topic->m_name, &opts);
memset(wr->m_iox_pub_loans, 0, sizeof(wr->m_iox_pub_loans));
}
#endif

5.2 iceoryx_header_t:chunk 里的 DDS 元数据

DDS 语义(GUID、时间戳、statusinfo、keyhash、数据状态)通过 iceoryx 的 user-header 随 chunk 传递:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
typedef enum {
IOX_CHUNK_UNINITIALIZED,
IOX_CHUNK_CONTAINS_RAW_DATA,
IOX_CHUNK_CONTAINS_SERIALIZED_DATA
} iox_shm_data_state_t;

struct iceoryx_header {
struct ddsi_guid guid;
dds_time_t tstamp;
uint32_t statusinfo;
uint32_t data_size;
unsigned char data_kind;
ddsi_keyhash_t keyhash;
iox_shm_data_state_t shm_data_state;
};

shm_data_state 区分 chunk 里是原始结构体(POD 零拷贝)还是 CDR 序列化字节(非固定类型的退化模式)。

5.3 shm_create_chunk:借 chunk

分配封装在 shm_create_chunk(),调用第 06 篇讲的 iox_pub_loan_aligned_chunk_with_user_header,user-header 即 iceoryx_header_t;拿不到 chunk 时非阻塞重试 10 次(每次 sleep 1ms)后放弃

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
void *shm_create_chunk(iox_pub_t iox_pub, size_t size) {
iceoryx_header_t *ice_hdr;
void *iox_chunk;

int32_t number_of_tries =
10; // try 10 times over at least 10ms, considering the wait time below

while (true) {
enum iox_AllocationResult alloc_result =
iox_pub_loan_aligned_chunk_with_user_header(
iox_pub, &iox_chunk, (uint32_t)size,
IOX_C_CHUNK_DEFAULT_USER_PAYLOAD_ALIGNMENT,
sizeof(iceoryx_header_t), 8);

if (AllocationResult_SUCCESS == alloc_result)
break;

if (--number_of_tries <= 0) {
return NULL;
}

dds_sleepfor(DDS_MSECS(1));
}

iox_chunk_header_t *iox_chunk_header =
iox_chunk_header_from_user_payload(iox_chunk);
ice_hdr = iox_chunk_header_to_user_header(iox_chunk_header);
ice_hdr->data_size = (uint32_t)size;
ice_hdr->shm_data_state = IOX_CHUNK_UNINITIALIZED;
return iox_chunk;
}

5.4 写路径:dds_write_impl_iox

dds_write() 在 writer 有 iceoryx publisher 时走专用分支(dds_write.c:593-600)。核心逻辑:

  1. 拿 chunk:数据若本来就是 loan 出来的 chunk(deregister_pub_loan 命中)直接复用;否则新借 chunk 并填充——固定大小类型 memcpy(零序列化),否则 ddsi_sertype_serialize_into 序列化进共享内存
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
static bool fill_iox_chunk(dds_writer *wr, const void *sample, void *iox_chunk, size_t sample_size)
{
bool has_fixed_size_type = wr->m_topic->m_stype->fixed_size;
bool ret = true;
iceoryx_header_t *iox_header = iceoryx_header_from_chunk(iox_chunk);
if (has_fixed_size_type) {
memcpy(iox_chunk, sample, sample_size);
iox_header->shm_data_state = IOX_CHUNK_CONTAINS_RAW_DATA;
} else {
size_t size = iox_header->data_size;
ret = ddsi_sertype_serialize_into(wr->m_wr->type, sample, iox_chunk, size);
if(ret) {
iox_header->shm_data_state = IOX_CHUNK_CONTAINS_SERIALIZED_DATA;
}
...
  1. 决定是否只走 iceoryx:没有网络 reader、VOLATILE、没有本地非 iceoryx 快路径 reader 时,可完全跳过 CDR 序列化与 RTPS:
1
2
3
4
const bool use_only_iceoryx =
no_network_readers &&
ddsi_wr->xqos->durability.kind == DDS_DURABILITY_VOLATILE &&
num_fast_path_readers == 0;
  1. 投递deliver_data_via_iceoryx() 把 GUID/时间戳等填进 iceoryx_header_tiox_pub_publish_chunk;混合场景(同时有远端 reader)则序列化一份走 UDP,chunk 照样发给同机 iceoryx reader(dds_write.c:277-286)。

5.5 读路径:shm_monitor + Listener 回调

每个 domain 持有一个 shm_monitorshm__monitor.h),内部是一个 iceoryx Listener。reader 创建后 attach:

1
2
3
4
5
6
7
8
9
10
11
12
dds_return_t shm_monitor_attach_reader(shm_monitor_t* monitor, struct dds_reader* reader)
{

if(iox_listener_attach_subscriber_event_with_context_data(monitor->m_listener,
reader->m_iox_sub,
SubscriberEvent_DATA_RECEIVED,
shm_subscriber_callback,
&reader->m_iox_sub_context) != ListenerResult_SUCCESS) {
DDS_CLOG(DDS_LC_SHM, &reader->m_rd->e.gv->logconfig, "error attaching reader\n");
return DDS_RETCODE_OUT_OF_RESOURCES;
}
++monitor->m_number_of_attached_readers;

数据到达时,iceoryx 后台线程执行 receive_data_wakeup_handler:循环 iox_sub_take_chunk → 按 chunk 头里的 GUID 反查(proxy)writer → 构造 ddsi_serdata_from_iox(serdata 直接引用 chunk,不拷贝)→ 塞进 reader history cache:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
while (true)
{
shm_lock_iox_sub(rd->m_iox_sub);
enum iox_ChunkReceiveResult take_result = iox_sub_take_chunk(rd->m_iox_sub, (const void** const)&chunk);
shm_unlock_iox_sub(rd->m_iox_sub);

// NB: If we cannot take the chunk (sample) the user may lose data.
// Since the subscriber queue can overflow and will evict the least recent sample.
...
const iceoryx_header_t* ice_hdr = iceoryx_header_from_chunk(chunk);

// Get writer or proxy writer
struct ddsi_entity_common * e = entidx_lookup_guid_untyped (gv->entity_index, &ice_hdr->guid);
...
// Create struct ddsi_serdata
struct ddsi_serdata* d = ddsi_serdata_from_iox(rd->m_topic->m_stype, ice_hdr->data_kind, &rd->m_iox_sub, chunk);
d->timestamp.v = ice_hdr->tstamp;
d->statusinfo = ice_hdr->statusinfo;
...
ddsi_make_writer_info(&wrinfo, e, xqos, d->statusinfo);
(void)ddsi_rhc_store(rd->m_rd->rhc, &wrinfo, d, tk);

同一 iox_sub 会被应用线程(take/return loan)和 listener 线程并发访问,因此所有操作都包在 shm_lock_iox_sub/shm_unlock_iox_subiox_sub_context_t.mutexddsi_shm_transport.c:33-43)里。


6. dds_loan_sample:应用级零拷贝 API

dds_loan_api.h 暴露三个关键函数(实现在 dds_loan.c):

API 语义
dds_is_shared_memory_available(entity) 该 reader/writer 是否挂上了 iceoryx(m_iox_sub/m_iox_pub != NULL
dds_is_loan_available(entity) 在前者基础上还要求 类型 fixed_size —— 真零拷贝的判定
dds_loan_sample(writer, &sample) 借一个共享内存 sample,用户就地填数据后 dds_write
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
dds_return_t dds_loan_sample(dds_entity_t writer, void **sample) {
#ifndef DDS_HAS_SHM
(void)writer;
(void)sample;
return DDS_RETCODE_UNSUPPORTED;
#else
dds_return_t ret;
dds_writer *wr;

if (!sample)
return DDS_RETCODE_BAD_PARAMETER;

if ((ret = dds_writer_lock(writer, &wr)) != DDS_RETCODE_OK)
return ret;

// the loaning is only allowed if SHM is enabled correctly and if the type is
// fixed
if (wr->m_iox_pub && wr->m_topic->m_stype->fixed_size) {
*sample = dds_writer_loan_chunk(wr, wr->m_topic->m_stype->iox_size);
if (*sample == NULL) {
ret = DDS_RETCODE_ERROR; // could not obtain a sample
}
} else {
ret = DDS_RETCODE_UNSUPPORTED;
}
...

writer 内部有一个 m_iox_pub_loans[MAX_PUB_LOANS] 池登记未归还的 loan;dds_writederegister_pub_loan 命中即知道”这块数据本来就在共享内存里”,直接发布,全程零拷贝。未 write 的 loan 可用 dds_return_writer_loan 归还(否则 chunk 泄漏,见 §8)。


7. ROS 2 Humble 链路:rmw_cyclonedds_cpp

7.1 loan 能力探测

创建 publisher/subscription 时,rmw 层把”类型是否 POD(fixed type)”与 cyclonedds 的 loan 能力合并成 can_loan_messages

1
2
3
4
dds_delete_listener(listener);
pub->type_supports = *type_supports;
pub->is_loaning_available = is_fixed_type && dds_is_loan_available(pub->enth);
pub->sample_size = sample_size;

(subscription 同理,rmw_node.cpp:2817;随后赋给 rmw_publisher->can_loan_messages / rmw_subscription->can_loan_messages。)

7.2 借出与发布

rclcppLoanedMessage 最终落到 rmw_borrow_loaned_messageinit_and_alloc_sample(内部走 dds_data_allocator / iceoryx chunk):

1
2
3
4
5
6
7
8
9
10
// if the publisher can loan
if (cdds_publisher->is_loaning_available) {
auto sample_ptr = init_and_alloc_sample(cdds_publisher, cdds_publisher->sample_size);
RET_NULL_X(sample_ptr, return RMW_RET_ERROR);
*ros_message = sample_ptr;
return RMW_RET_OK;
} else {
RMW_SET_ERROR_MSG("Borrowing loan for a non fixed type is not allowed");
return RMW_RET_ERROR;
}

rmw_publish 对 loan 出来的消息直接构造 serdata 并把 chunk 标记为 RAW 后 dds_writecdr

1
2
3
4
5
6
7
8
// if the publisher allow loaning
if (cdds_publisher->is_loaning_available) {
auto d = new serdata_rmw(cdds_publisher->sertype, ddsi_serdata_kind::SDK_DATA);
d->iox_chunk = ros_message;
// since we write the loaned chunk here, set the data state to raw
shm_set_data_state(d->iox_chunk, IOX_CHUNK_CONTAINS_RAW_DATA);
if (dds_writecdr(cdds_publisher->enth, d) >= 0) {
return RMW_RET_OK;

非 POD 类型不能 loan,但 SHM 可用时序列化结果仍可写进共享内存(退化为”一次序列化 + 零网络拷贝”):

1
2
3
4
5
6
7
if (dds_is_shared_memory_available(pub->enth)) {
auto sample_ptr = init_and_alloc_sample(pub, serialized_message->buffer_length);
RET_NULL_X(sample_ptr, return RMW_RET_ERROR);
memcpy(sample_ptr, serialized_message->buffer, serialized_message->buffer_length);
shm_set_data_state(sample_ptr, IOX_CHUNK_CONTAINS_SERIALIZED_DATA);
d->iox_chunk = sample_ptr;
}

7.3 应用层写法(LoanedMessage)

1
2
3
4
// 消息须是 POD/固定大小(如自定义的定长数组图像消息)
auto loaned = publisher->borrow_loaned_message(); // rmw_borrow_loaned_message
loaned.get().data = ...; // 直接写共享内存
publisher->publish(std::move(loaned)); // rmw_publish(loan 路径)

“真零拷贝”的类型条件:ROS 消息经 rosidl 映射后必须固定大小——不能含 string、动态数组/sequence。像 sensor_msgs/msg/Image(含 string encoding 和动态 data)不满足,只能走”序列化进共享内存”的退化路径;工程上常用定长自定义消息规避。


8. 限制与常见坑

现象 源码依据 / 对策
QoS 不满足 静默退回网络路径,性能与预期不符 dds_writer_support_shm/dds_reader_support_shm:必须 KEEP_LAST、VOLATILE 或 TRANSIENT_LOCAL(且 TRANSIENT_LOCAL 的 durability_service 深度受 iceoryx history 上限约束,dds_writer.c:331 起);reader 不能 ignore_local
非固定大小类型 can_loan_messages == false,loan 接口报错 dds_is_loan_available 要求 fixed_size;数据仍可能经 serialize_into 进共享内存,但多一次序列化
iox-roudi 没起 进程启动时 iceoryx runtime 注册失败:反复打印等待 RouDi 的日志,超时后 iceoryx errorHandler 直接终止进程(不会优雅退回 UDP) iox_runtime_initq_init.c:1098)无失败返回路径;iceoryx 2.0 的 PoshRuntime 等待 RouDi 超时即 terminate(默认等待时长约 60s,待验证)。对策:SharedMemory/Enable=true 时必须保证 RouDi 先行启动
mempool 尺寸不匹配 dds_loan_sample / dds_write 返回 DDS_RETCODE_ERROR/OUT_OF_RESOURCES,iceoryx 日志报无合适 chunk shm_create_chunk 重试 10 次后返回 NULL;需在 RouDi 的 TOML 里配置足够大/足够多的 mempool(chunk 大小 ≥ payload + sizeof(iceoryx_header_t) + ChunkHeader 开销)
订阅队列溢出丢数据 高频发布 + 消费慢时旧样本被驱逐 shm_monitor.c:106-109 注释明示;iceoryx 队列深度上限(MAX_SUBSCRIBER_QUEUE_CAPACITY=256)也约束了 reader 的有效 history 深度
持有 chunk 过多 日志 TOO_MANY_CHUNKS_HELD_IN_PARALLEL 应用长期持有 take 出的 loaned message 导致;及时归还(dds_return_loan / 析构 LoanedMessage
跨主机误判 无(按 MAC 判定) 容器/虚拟网卡环境 MAC 可能相同或取不到,必要时用 SharedMemory/Locator 显式指定
一写多读混合 同 topic 既有同机 SHM reader 又有远端 reader 时仍会序列化 use_only_iceoryx 条件苛刻(无网络 reader + VOLATILE + 无快路径本地 reader),属预期行为

9. 快速验证清单

1
2
3
4
5
6
7
8
9
# 1. 确认 cyclonedds 编译带 SHM(ROS 2 Humble 二进制默认已启用 iceoryx 支持)
ldd $(ros2 pkg prefix rmw_cyclonedds_cpp)/lib/librmw_cyclonedds_cpp.so | grep iceoryx

# 2. 打开 SHM 日志观察是否真的走了共享内存
export CYCLONEDDS_URI='<CycloneDDS><Domain><SharedMemory><Enable>true</Enable><LogLevel>verbose</LogLevel></SharedMemory><Tracing><Verbosity>config</Verbosity><OutputFile>stdout</OutputFile></Tracing></Domain></CycloneDDS>'
# 日志中应出现 "My iceoryx address: ..."、"Writer's topic name will be DDS:Cyclone:<topic>"

# 3. 用 iceoryx 内省工具确认端口已注册(见 08 篇)
iox-introspection-client --all

下一篇

08-示例与性能工具.md:iceoryx_examples 代表性示例走读、iceperf 基准方法与排障工具。