WiredTiger Memory Model Test 实战用两线程微基准实测 ARM64 与 x86_64 的内存重排序【免费下载链接】mongoThe MongoDB Database项目地址: https://gitcode.com/GitHub_Trending/mo/mongoWiredTiger 工具链中的 Memory Model Test 是一个独立编译运行的 C 微基准程序通过两组精心设计的两线程共享变量访问模式直观演示处理器乱序执行out-of-order memory access在不同内存模型下的真实表现。本文完整讲解它的测试原理、构建与运行方式、命令行参数并结合 MongoDB 仓库中的实际源码memory_model_test.cpp与 CI 配置evergreen.yml给出 M1 Max 与 Graviton ARM64 实例上的实测输出对比。读完后你能掌握如何在 ARM64/x86_64 平台上自行构建并运行该测试、如何解读输出中的乱序比例以及为什么“一个 barrier”与“两个 barrier”在多核环境下会产生本质不同的结果。这个工具是什么技术演示而非通过/失败测试Memory Model Test 位于 MongoDB 仓库的 WiredTiger 第三方存储引擎工具目录 src/third_party/wiredtiger/tools/memory-model-test/目录下仅有三个文件说明文档 README.md、主程序 memory_model_test.cpp 以及 C17 回退实现 basic_semaphore.h。按 README.md 的定义它不是一个 pass or fail 类型的测试而是一个技术演示technology demo和测试床testbed它执行一系列可能取决于处理器产生乱序内存访问的操作序列它的核心价值在于以具体形式清晰展示ARM64 与 x86_64 两种内存模型的行为差异该测试的灵感部分来自 Preshing 博客上那篇经典的 “Memory reordering caught in the act” 文章该文章也是理解本类测试的推荐读物可在仓库 README 的 References 一节找到出处说明。之所以这个工具对数据库引擎有现实意义WiredTiger 这类高并发存储引擎大量依赖原子操作与内存屏障来保证多线程下的可见性与一致性而不同架构处理器对“程序员看到的指令顺序”与“内存中实际生效顺序”的承诺强度不同这类差异只有在真实硬件上用微基准实测才能感知。测试原理拆解两组线程访问模式每个测试都是两个线程访问若干共享变量且所有共享变量在每个测试的每次迭代开始前都会被重置为 0源码中 perform_test 的主循环 对x、y、r1、r2逐一清零后才放行工作线程。测试分为两大组Group 1每线程一写一读store-load 交错线程 1 先向共享变量x写 1然后读取对方的变量y存入r1线程 2 先向y写 1然后读取x存入r2变体覆盖三种同步强度无屏障、一个屏障或原子、两个屏障或原子若发生乱序内存访问两个读的结果可能都是 0两个线程都在对方写入“生效”前完成了读取。判断“乱序”的回调在 源码 L218auto out_of_order_check_code_for_write_then_read []() { return r1 0 r2 0; };即r1 0 r2 0时判定本轮出现乱序。Group 1 各变体的线程代码片段源码 L187-L216// 无屏障仅编译器屏障 auto thread_1_code_write_then_read []() { x 1; COMPILER_BARRIER; r1 y; }; // 写与读之间插入一条 CPU 内存屏障 auto thread_1_code_write_then_barrier_then_read []() { x 1; MEMORY_BARRIER; r1 y; }; // 使用 seq_cst 原子自增代替普通写 auto thread_1_atomic_increment_and_read []() { __atomic_add_fetch(x, 1, __ATOMIC_SEQ_CST); r1 y; };Group 2一线程两写、另一线程两读store-store 与 load-load 重排线程 1 依次向x写 2、向y写 3线程 2 依次读取y存入r1、读取x存入r2同样提供无屏障、单屏障、双屏障/原子的变体若发生乱序内存访问可能出现r1 3而r2 0——即读到了y的新值 3却没看到x的新值 2仍读到初始值 0。判断逻辑在 源码 L252auto out_of_order_check_code_for_write_then_write []() { return r1 3 r2 0; };Group 2 是区分两种内存模型的关键在 x86_64 的强内存模型下这类 store-store 重排不会让该乱序可见而在 ARM64 的弱内存模型下则可能观察到。源码用编译期宏检测架构L32-L38#if defined(x86_64) || defined(__x86_64__) #define BARRIER_INSTRUCTION mfence const bool is_arm64 false; #elif defined(__aarch64__) #define BARRIER_INSTRUCTION dmb ish const bool is_arm64 true; #endif随后每个 Group 2 测试配置都把这个is_arm64作为“允许乱序”的标志传入L299-L303 等意味着在 x86_64 上 Group 2 的乱序被视为不允许发生——一旦发生就会触发错误输出。两种“屏障”的区别编译器屏障 vs CPU 内存屏障源码 L40-L41 定义了两个关键宏这是理解全部测试变体的钥匙#define MEMORY_BARRIER asm volatile(BARRIER_INSTRUCTION ::: memory) #define COMPILER_BARRIER asm volatile( ::: memory)COMPILER_BARRIER是一条空的内联汇编带memoryclobber只阻止编译器在优化时重排内存访问对 CPU 乱序执行没有任何约束MEMORY_BARRIER发射架构相关的完整屏障指令——x86_64 上为mfenceARM64 上为dmb ish——约束的是处理器对内存访问的排序。所以“无屏障”变体并非完全不约束所有变体都保留了COMPILER_BARRIER观察到的乱序全部来自 CPU 层面。执行框架信号量握手、随机延迟与缓存行隔离从源码结构看测试还包含三个容易被忽略但影响结果可复现性的设计逐迭代信号量握手。thread_pair 为两个工作线程各配一对二进制信号量start_semaphore1/2、end_semaphore1/2。主循环每轮迭代先清零共享变量再释放两个 start 信号量放行工作线程随后等待两个 end 信号量确认本轮执行完毕L101-L114。默认实现使用 C20 的std::binary_semaphore若定义AVOID_CPP20_SEMAPHORE宏则改用项目自带的 basic_semaphore.h——一个基于std::mutexstd::condition_variable的简易计数信号量L17-L38。伪随机忙等延迟。工作线程在获得放行后先执行while (rng() % 8 ! 0);L48-L53——用std::mt19937忙转到能整除 8 为止平均约 8 次循环的随机时延让两个线程的读写在时间轴上“错位”提高撞见乱序窗口的概率。缓存行隔离。共享变量被刻意放入一个 128 字节间隔的结构体L166-L182const int space 128 - sizeof(int); struct var_holder { int x 0; char spacer1[space]; int y 0; char spacer2[space]; int r1 0; char spacer3[space]; int r2 0; };源码注释明确说明把x、y、r1、r2推到不同缓存行上能大幅提高在 ARM64 Evergreen 实例上观察到 Group 2 乱序的概率——避免同缓存行false sharing 一类的缓存一致性流量干扰观察。另外从源码结构看还有一个值得留意的细节代码里定义了test_writes_then_reads_one_barrier_one_atomic配置L287-L293但它并未出现在perform_test的实际调用序列中L338-L372 共 10 次调用Group 1 五项 Group 2 五项因此该变体当前不会实际运行。构建前提条件继承自原文档ARM64 或 x86_64 CPU支持 C17或更好的 C20的 C 编译器可选CMake 和 ninja。默认情况下该工具使用 C20 的std::binary_semaphore。遗憾的是并非所有自称支持 C20 的编译器都提供这个类必要时定义AVOID_CPP20_SEMAPHORE宏即可改用项目自带的basic_semaphore。两种构建方式方式一CMake。README 给出的流程为mkdir build cd build cmake -G Ninja ../. ninja需要注意当前仓库快照中该目录仅包含README.md、memory_model_test.cpp、basic_semaphore.h三个文件并未附带CMakeLists.txt因此实操中应以 g 直接编译为准这也是 CI 中真实使用的构建方式。方式二直接用 g推荐CI 同款。在 EvergreenUbuntu 环境上先确保 MongoDB 工具链的 C 编译器在 PATH 中export PATH/opt/mongodbtoolchain/v5/bin:$PATH然后在 Mac 或 Evergreen 上用 g 编译C20g -o memory_model_test -O2 memory_model_test.cpp -lpthread -stdc20 -Wall -WerrorC17 本地basic_semaphoreg -o memory_model_test -O2 memory_model_test.cpp -lpthread -stdc17 -Wall -Werror -DAVOID_CPP20_SEMAPHORE部分测试使用了编译器屏障COMPILER_BARRIER来防止编译器在优化阶段重排内存访问。若编译时出现与#include semaphore或信号量相关的错误请检查编译器是否正确以及是否按 C20 编译或使用AVOID_CPP20_SEMAPHORE宏改用本地basic_semaphore。选择 C 信号量库的原因是它在 Mac 和 Ubuntu 上都有支持。CI 中如何构建与调度WiredTiger 的 Evergreen 配置src/third_party/wiredtiger/test/evergreen.yml定义了两个任务exec_timeout_secs均设为 86400memory-model-testLinux/ARM64工作目录wiredtiger/tools/memory-model-test按 C20 编译后执行./memory_model_test -n 100000000memory-model-test-mac先打印 CPU 型号sysctl -n machdep.cpu.brand_string和 macOS 版本sw_vers以便结果可溯源然后按 C17 -DAVOID_CPP20_SEMAPHORE编译并执行./memory_model_test -n 100000000。两个任务在任务组中的调度周期为batchtime: 40320即28 天见 evergreen.yml L1628-L1629 与 evergreen_develop.yml L135-L137。由于它属于低频执行的“观察型”任务配置中特意关闭了 stepbackevergreen_develop.yml L129-L134 注释解释了原因这类测试运行不频繁失败时往往前面已有许多提交会以同样方式失败回退检查只会产生大量冗余工作。运行与命令行参数./memory_model_test使用默认循环次数1,000,000、单线程对运行通常需要几十秒./memory_model_test -n loop count自定义迭代次数./memory_model_test -p pair count自定义线程对数量。这是实验模式允许多个线程对同时执行。但 README 明确指出该选项似乎并不能增加乱序事件的次数价值有限且各线程对的消息目前会相互交错输出。参数解析在 main 函数 L384-L399 中通过getopt(argc, argv, n:p:)实现默认值loop_count 1000000、num_thread_pairs 1L381-L382。CI 使用-n 1000000001 亿次迭代以换取更稳定的统计比例。程序启动时会打印环境信息包括 C 标准号与所用的二进制信号量实现std::binary_semaphore或basic_semaphore以及“Running on ARM64 / Running on x86”——注意架构判定来自编译期预处理宏而非运行时探测。结果解读哪些测试允许乱序哪些必须为零每个测试都会报告是否/何时观察到乱序操作。按 README.md 的说明结果分三类理论上永远不应出现乱序的测试因为屏障/原子使用正确或受处理器设计约束如 Group 1 的双屏障/双原子变体以及 Group 2 中读侧带屏障的变体、x86_64 上的全部 Group 2 测试。这些测试一旦报告乱序就是错误——源码会在计数大于 0 且_out_of_order_allowed false时打印醒目的错误行L140-L143******** ERROR out of order operations were not allowed, but did occur. ********可能但未必出现乱序的测试乱序能否被抓到取决于两个线程间时序的随机性以及某些测试/硬件组合上可测量乱序效应本身发生率低。架构相关的预期差异所有 x86_64 与 ARM64 处理器在 Group 1 的部分测试上都会出现一些乱序但受内存模型差异影响只有 ARM 会在 Group 2 中显示乱序。实测输出示例一Mac StudioM1 MaxARM64以下为在 M1 Max 上以 1,000,000 次迭代运行时的输出。注意在 ARM64 上所有“可能乱序”的场景全部出现了乱序且比例相当可观WiredTiger Memory Model Test Running on ARM64 with 1 thread pairs(s) and loop count 1000000 -- Group 1: Tests that have a read and a write in each thread -- Test name: Test writes then reads Test description: Each thread writes then reads. Out of orders ARE POSSIBLE. Total of 176874 out of orders detected out of 1000000 iterations (17.6874%) in test Test writes then reads Test name: Test writes then reads with one barrier Test description: Each thread writes then reads, with one barrier between the write and read on thread 2. Out of orders ARE POSSIBLE. Total of 6898 out of orders detected out of 1000000 iterations (0.6898%) in test Test writes then reads with one barrier Test name: Test writes then reads with two barriers Test description: Each thread writes then reads, with a barrier between the write and read on each thread. Out of orders are NOT POSSIBLE. Total of 0 out of orders detected out of 1000000 iterations (0%) in test Test writes then reads with two barriers Test name: Test writes then reads with one atomic Test description: Each thread writes then reads, with one atomic increment used for one write. Out of orders ARE POSSIBLE. Total of 65025 out of orders detected out of 1000000 iterations (6.5025%) in test Test writes then reads with one atomic Test name: Test writes then reads with two atomics Test description: Each thread writes then reads, with atomic increments used for both writes. Out of orders are NOT POSSIBLE. Total of 0 out of orders detected out of 1000000 iterations (0%) in test Test writes then reads with two atomics -- Group 2: Tests that have two reads in one thread, and two writes in the other thread -- Test name: Test writes and reads Test description: One thread has two writes, the other has two reads. Out of orders ARE POSSIBLE on ARM64. Total of 9892 out of orders detected out of 1000000 iterations (0.9892%) in test Test writes and reads Test name: Test writes and reads, with barrier between writes Test description: One thread has two writes with a barrier between them, the other has two reads. Out of orders ARE POSSIBLE on ARM64. Total of 2671 out of orders detected out of 1000000 iterations (0.2671%) in test Test writes and reads, with barrier between writes Test name: Test writes and reads, with barrier between reads Test description: One thread has two writes, the other has two reads with a barrier between them. Out of orders are NOT POSSIBLE. Total of 0 out of orders detected out of 1000000 iterations (0%) in test Test writes and reads, with barrier between reads Test name: Test writes and reads, with barrier between writes and between reads Test description: One thread has two writes with a barrier between them, the other has two reads with a barrier between them. Out of orders are NOT POSSIBLE. Total of 0 out of orders detected out of 1000000 iterations (0%) in test Test writes and reads, with barrier between writes and between reads Test name: Test writes and reads, with atomics Test description: One thread has two writes using atomic increments, the other has two reads. Out of orders are ARE POSSIBLE on ARM64. Total of 810 out of orders detected out of 1000000 iterations (0.081%) in test Test writes and reads, with atomics这份输出值得逐条对照原理解读Group 1 首个测试 17.7% 的乱序率说明仅靠“程序员顺序”在 M1 Max 上完全不可靠“一个 barrier”把 17.7% 压到 0.69%但仍非零——单侧屏障只约束了一个线程的写读顺序另一个线程依然可能先读后写“两个 barriers”与“两个 atomics”__ATOMIC_SEQ_CST原子自增均为 0与理论预期一致Group 2 中写在写之间加屏障store-store 顺序被约束仍出现 0.27% 乱序——因为乱序发生在读线程的 load-load 一侧而把屏障放在读侧则降为 0。实测输出示例二Evergreen ubuntu2004-arm64-smallGravitonARM64以下为在 Evergreen Graviton ARM64 实例上以 100,000,000 次迭代运行的输出Evergreen 移除了空行WiredTiger Memory Model Test Running on ARM64 with 1 thread pairs(s) and loop count 100000000 -- Group 1: Tests that have a read and a write in each thread -- Test name: Test writes then reads Test description: Each thread writes then reads. Out of orders ARE POSSIBLE. Total of 27147 out of orders detected out of 100000000 iterations (0.027147%) in test Test writes then reads Test name: Test writes then reads with one barrier Test description: Each thread writes then reads, with one barrier between the write and read on thread 2. Out of orders ARE POSSIBLE. Total of 359 out of orders detected out of 100000000 iterations (0.000359%) in test Test writes then reads with one barrier Test name: Test writes then reads with two barriers Test description: Each thread writes then reads, with a barrier between the write and read on each thread. Out of orders are NOT POSSIBLE. Total of 0 out of orders detected out of 100000000 iterations (0%) in test Test writes then reads with two barriers Test name: Test writes then reads with one atomic Test description: Each thread writes then reads, with one atomic increment used for one write. Out of orders ARE POSSIBLE. Total of 996 out of orders detected out of 100000000 iterations (0.000996%) in test Test writes then reads with one atomic Test name: Test writes then reads with two atomics Test description: Each thread writes then reads, with atomic increments used for both writes. Out of orders are NOT POSSIBLE. Total of 0 out of orders detected out of 100000000 iterations (0%) in test Test writes then reads with two atomics -- Group 2: Tests that have two reads in one thread, and two writes in the other thread -- Test name: Test writes and reads Test description: One thread has two writes, the other has two reads. Out of orders ARE POSSIBLE on ARM64. Total of 360 out of orders detected out of 100000000 iterations (0.00036%) in test Test writes and reads Test name: Test writes and reads, with barrier between writes Test description: One thread has two writes with a barrier between them, the other has two reads. Out of orders ARE POSSIBLE on ARM64. Total of 0 out of orders detected out of 100000000 iterations (0%) in test Test writes and reads, with barrier between writes Test name: Test writes and reads, with barrier between reads Test description: One thread has two writes, the other has two reads with a barrier between them. Out of orders are NOT POSSIBLE. Total of 0 out of orders detected out of 100000000 iterations (0%) in test Test writes and reads, with barrier between reads Test name: Test writes and reads, with barrier between writes and between reads Test description: One thread has two writes with a barrier between them, the other has two reads with a barrier between them. Out of orders are NOT POSSIBLE. Total of 0 out of orders detected out of 100000000 iterations (0%) in test Test writes and reads, with barrier between writes and between reads Test name: Test writes and reads, with atomics Test description: One thread has two writes using atomic increments, the other has two reads. Out of orders are ARE POSSIBLE on ARM64. Total of 41 out of orders detected out of 100000000 iterations (4.1e-05%) in test Test writes and reads, with atomics与 M1 Max 相比Graviton 上的乱序表现为频率低约 3 个数量级如 Group 1 首个测试从 17.69% 降至 0.0271%在 ARM64 上所有“可能乱序”的场景中除一个外均出现了乱序唯一例外是 Group 2 的 “with barrier between writes” 测试按内存模型它本应可能出现乱序却报告为 0。README 对此的态度是诚实的尚不清楚原因一种可能是该类乱序在该实例上稀见到 1 亿次迭代内未被抓到。小结WiredTiger Memory Model Test 用最少的代码量单个约 400 行的翻译单元加一个信号量回退头文件搭建了一个可自持运行的内存模型观察台COMPILER_BARRIER与MEMORY_BARRIERmfence/dmb ish区分了编译器重排与 CPU 重排两种来源is_arm64标志把“乱序是否被允许”编译进每个测试的预期中而 128 字节缓存行隔离与随机忙等延迟则专门提升弱内存模型下乱序的可见率。对从事高并发 C 开发的读者它的结论非常实用在弱内存模型平台上单侧屏障与单个 seq_cst 原子都不足以保证跨线程的写读可见性顺序只有两侧同时约束双屏障或双原子才让 Group 1 的乱序稳定归零而 Group 2 进一步说明约束必须加在“被观察的那一侧”读侧 load-load才真正有效。若需复现实测数据可参照上文 g 编译命令与 CI 中-n 100000000的运行参数在当前仓库对应路径下构建运行。相关参考仓库内路径说明文档README.mdReferences 一节另列出了 Preshing 博客文章与 ARM 官方内存屏障示例文档供延伸阅读主程序memory_model_test.cppC17 信号量回退实现basic_semaphore.hCI 任务定义evergreen.yml、evergreen_develop.yml【免费下载链接】mongoThe MongoDB Database项目地址: https://gitcode.com/GitHub_Trending/mo/mongo创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考