向量检索边缘化:在香港服务器上落地 Milvus / pgvector 的冷热分层与 NVMe Pool + RDMA 直通

发布人:A5数据发布时间:2025-09-24 19:25阅读量:863

那天晚上 23:40,香港到东京的 RTT 稳定在 45ms 左右,但真实用户的向量召回链路一旦绕行到核心区的“统一大库”,尾延迟就像被人拽住脚踝。我们的召回服务在云上,embedding 在云上,向量索引也在云上,唯独用户在香港。A/B 面板给我看,一半流量在 95 分位卡到 350ms——这不是我能接受的数字。

我盯着机柜里两台新上的 NVMe 盒子发呆:一组满血的 U.2 盘,旁边是两张亮着蓝灯的 ConnectX 网卡。我脑子里只有一句话:把检索推到边缘,把热数据钉在 NVMe 上,用 RDMA 把吞吐和延迟都榨出来。于是,就有了下面这套方案与踩坑记录。


1. 目标与路线图

业务目标

  • P95 向量检索延迟 ≤ 80ms(端到端,香港用户,TopK=50,查询向量维度 768/1024)

  • 热数据 100% 命中边缘,冷数据回落不超过 300ms

  • 单集群吞吐 ≥ 8k QPS,峰值≥ 12k QPS(只读阶段)

  • 故障演练覆盖:单盘/单网卡/单节点/单交换机/单可用区失效

技术路线

  • 存储层:NVMe Pool 以 NVMe-oF (RDMA/RoCEv2) 暴露卷;本地/远端混合池化

  • 引擎层:Milvus(2.x)负责“热检索 + 冷分层至对象存储”,PostgreSQL + pgvector 负责元数据/回查与一部分通用相似度检索(以 SQL 能力补齐)

  • 冷热分层:Milvus 的 segment/索引放 热层 NVMe,冷层 MinIO(S3 兼容);pgvector 通过 Tablespace 做表/索引的冷热 tablespace 迁移

  • 网络:25/100GbE RoCEv2 RDMA,PFC + ECN,Jumbo Frame;业务网与存储网物理/逻辑隔离

  • 编排:Kubernetes + Helm 部署 Milvus 组件;PostgreSQL 裸机+Patroni(或单机起步,后期再切)

  • 操作系统:CentOS 7.9(内核升级至 ELRepo kernel-ml 以便更好的 RDMA/NVMe 驱动支持)


2. 机房与硬件清单(落地可复用)

注:以下为我最终上线时的 实际可行 配置,不是实验室“神机”。预算友好、可替换。

2.1 角色与节点规格

角色 数量 CPU 内存 本地存储 网络 备注
NVMe-oF Target(存储池) 2 2×Xeon Silver 4310 256GB 8×3.84TB U.2 NVMe(RAID0 by SPDK/VOL) 2×100GbE ConnectX-6(RoCEv2) 负责 RDMA 暴露卷
Milvus Query/Index Node 3 2×Xeon Silver 4310 256GB 2×1.92TB NVMe(本地缓存) 2×25GbE ConnectX-5 可选 1×A10 GPU(可选)
Milvus Meta / etcd / Pulsar / MinIO 3 1×Xeon Silver 4210 128GB 2×960GB SSD(RAID1) 2×25GbE 可合并/拆分
PostgreSQL + pgvector 2(主备) 1×Xeon Gold 6230 256GB 系统盘+2×3.84TB NVMe(Tablespace 热层) 2×25GbE Patroni/keepalived
汇聚/监控 2 任意 64GB 2×480GB SSD 2×10GbE Prom+Grafana+Loki

交换机:

  • TOR:Mellanox SN2010(或等价 25/100GbE),启用 PFC/ECN

  • 核心:任意兼容 100GbE,上行 LAG

2.2 操作系统与固件

  • CentOS 7.9(注意 EOL,生产请上内核新版本):

     
    <span class="hljs-comment"># 升级到 ELRepo 主线内核,提升 RDMA/NVMe 驱动可用性</span> yum install https://www.elrepo.org/elrepo-release-7.el7.elrepo.noarch.rpm -y yum --enablerepo=elrepo-kernel install kernel-ml -y grub2-set-default 0 && reboot
  • 固件:升级 NVMe 固件、MLX 网卡固件到供应商建议的稳定版本(机房窗口操作)


3. 网络拓扑与关键参数

3.1 逻辑拓扑(简化)

 
<span class="hljs-selector-attr">[Clients(HK)]</span> │ 业务网(外部) <span class="hljs-selector-attr">[Edge Gateways/LBs]</span> │ 业务网(内部 VLAN <span class="hljs-number">110</span>) ┌───▼───────────────────────────────────────────────────────────────┐ │ Kubernetes(Milvus 集群,VLAN <span class="hljs-number">110</span>/<span class="hljs-number">120</span>) │ │ Proxy/Query/Index ────► NVMe-oF Initiators (RDMA VLAN <span class="hljs-number">120</span>) │ │ │ │ │ │ │S3/对象存储(MinIO,VLAN <span class="hljs-number">130</span>) │RDMA │ │ ▼ ▼ │ │ <span class="hljs-selector-attr">[MinIO]</span> ◄────── RDMA ──────► <span class="hljs-selector-attr">[NVMe-oF Targets]</span> │ └──────────────────────────────────────────────────────────────────┘ │ <span class="hljs-selector-attr">[PostgreSQL 主/备,VLAN 110/130]</span>

3.2 网络参数(RoCEv2 推荐)

项 值/建议
MTU 9000(端到端一致:主机、交换机、容器 CNI)
RDMA RoCEv2(基于 UDP 的 L3 可路由)
流控 PFC(优先级 3)+ ECN(交换机开启,DiffServ 映射)
优先级 DSCP=26(示例),网卡/交换机/主机一致映射到 RoCE 队列
中断亲和 irqbalance 关闭,手动绑核(见后文)
NUMA QueryNode/IndexNode 绑到与 RDMA 网卡同一 NUMA 域
监控 mlx5 exporter / rdma 计数器、PFC pause 统计

主机侧 RDMA 简要配置

 
<span class="hljs-comment"># MLX 网卡 RoCEv2 开启</span> mlxconfig -d mlx5_0 s ROCE_CC_MODE=ECN ROCE_CC_RP=1 ROCE_CC_NP=1 ethtool -A eth2 rx on tx on <span class="hljs-comment"># PFC 由交换机 / DCB 配置保证</span> <span class="hljs-comment"># Jumbo & DSCP 保持一致</span> ip <span class="hljs-built_in">link</span> <span class="hljs-built_in">set</span> dev eth2 mtu 9000 tc qdisc replace dev eth2 root fq <span class="hljs-comment"># 关闭 irqbalance,手动中断绑核(示例)</span> systemctl stop irqbalance <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> /proc/irq/*/mlx5_comp*/smp_affinity_list; <span class="hljs-keyword">do</span> <span class="hljs-built_in">echo</span> 8 > <span class="hljs-string">"<span class="hljs-variable">$i</span></span>"; <span class="hljs-keyword">done</span> <span class="hljs-comment"># 绑CPU核 3(示例)</span>

4. NVMe Pool + RDMA 直通(NVMe-oF + SPDK)

4.1 NVMe-oF Target 部署

前置:Hugepage & DPDK

 
<span class="hljs-comment"># Hugepage</span> <span class="hljs-built_in">echo</span> 8192 > /proc/sys/vm/nr_hugepages <span class="hljs-built_in">echo</span> <span class="hljs-string">'vm.nr_hugepages=8192'</span> >> /etc/sysctl.conf <span class="hljs-comment"># 绑定 NVMe-oF Target 使用的网卡到 DPDK(视 SPDK 版本而定)</span> modprobe vfio-pci dpdk-devbind.py --<span class="hljs-built_in">bind</span>=vfio-pci 0000:af:00.0 <span class="hljs-comment"># 示例 PCI 地址</span>

编译/安装 SPDK(精简示例)

 
yum groupinstall <span class="hljs-string">"Development Tools"</span> -y yum install numactl-devel CUnit-devel libaio-devel openssl-devel rdma-core-devel -y git <span class="hljs-built_in">clone</span> https://github.com/spdk/spdk.git <span class="hljs-built_in">cd</span> spdk && git submodule update --init ./configure --with-rdma --enable-debug make -j && make install

SPDK NVMe-oF Target 配置(nvmf_target.json 片段)

 
<span class="hljs-punctuation">{</span> <span class="hljs-attr">"subsystems"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span> <span class="hljs-punctuation">{</span> <span class="hljs-attr">"subsystem"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"nvmf"</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"config"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span> <span class="hljs-punctuation">{</span> <span class="hljs-attr">"method"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"nvmf_set_config"</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"params"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span> <span class="hljs-attr">"acceptor_poll_rate"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">10000</span> <span class="hljs-punctuation">}</span> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span> <span class="hljs-punctuation">{</span> <span class="hljs-attr">"method"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"nvmf_create_transport"</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"params"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span> <span class="hljs-attr">"trtype"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"RDMA"</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"max_queue_depth"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">1024</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"io_unit_size"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">131072</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"max_io_qpairs_per_ctrlr"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">64</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"in_capsule_data_size"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">8192</span> <span class="hljs-punctuation">}</span> <span class="hljs-punctuation">}</span> <span class="hljs-punctuation">]</span> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span> <span class="hljs-punctuation">{</span> <span class="hljs-attr">"subsystem"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"bdev"</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"config"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span> <span class="hljs-punctuation">{</span> <span class="hljs-attr">"method"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"bdev_nvme_attach_controller"</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"params"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span> <span class="hljs-attr">"name"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"Nvme0"</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"trtype"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"PCIe"</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"traddr"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"0000:b1:00.0"</span> <span class="hljs-punctuation">}</span> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span> <span class="hljs-punctuation">{</span> <span class="hljs-attr">"method"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"bdev_split_create"</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"params"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span> <span class="hljs-attr">"name"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"Nvme0n1"</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"split_count"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">8</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"split_size_mb"</span><span class="hljs-punctuation">:</span> <span class="hljs-number">512000</span> <span class="hljs-punctuation">}</span> <span class="hljs-punctuation">}</span> <span class="hljs-punctuation">]</span> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span> <span class="hljs-punctuation">{</span> <span class="hljs-attr">"subsystem"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"nvmf"</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"config"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span> <span class="hljs-punctuation">{</span> <span class="hljs-attr">"method"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"nvmf_create_subsystem"</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"params"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span> <span class="hljs-attr">"nqn"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"nqn.2025-09.hk.edge:nvme0"</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"allow_any_host"</span><span class="hljs-punctuation">:</span> <span class="hljs-literal"><span class="hljs-keyword">true</span></span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"serial_number"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"EDGE0001"</span> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span> <span class="hljs-punctuation">{</span> <span class="hljs-attr">"method"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"nvmf_subsystem_add_ns"</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"params"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span> <span class="hljs-attr">"nqn"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"nqn.2025-09.hk.edge:nvme0"</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"namespace"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"Nvme0n1p0"</span> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span> <span class="hljs-punctuation">{</span> <span class="hljs-attr">"method"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"nvmf_subsystem_add_listener"</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"params"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span> <span class="hljs-attr">"nqn"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"nqn.2025-09.hk.edge:nvme0"</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"trtype"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"RDMA"</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"traddr"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"10.120.0.10"</span><span class="hljs-punctuation">,</span> <span class="hljs-attr">"trsvcid"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"4420"</span> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">}</span> <span class="hljs-punctuation">]</span> <span class="hljs-punctuation">}</span> <span class="hljs-punctuation">]</span> <span class="hljs-punctuation">}</span>

Target 运行

 
sudo -E /usr/local/bin/nvmf_tgt -m 0xF --json /etc/spdk/nvmf_target.json

4.2 Initiator(Milvus/PG 节点)挂载卷

 
<span class="hljs-comment"># 发现目标</span> nvme discover -t rdma -a 10.120.0.10 -s 4420 <span class="hljs-comment"># 连接</span> nvme connect -t rdma -n nqn.2025-09.hk.edge:nvme0 -a 10.120.0.10 -s 4420 <span class="hljs-comment"># 磁盘将以 /dev/nvmeXnY 显示,创建文件系统</span> mkfs.xfs -f /dev/nvme1n1 <span class="hljs-built_in">mkdir</span> -p /data/nvme_hot && <span class="hljs-built_in">echo</span> <span class="hljs-string">"/dev/nvme1n1 /data/nvme_hot xfs noatime,nodiscard,allocsize=2M 0 0"</span> >> /etc/fstab && mount -a

注意

  • NVMe-oF target 双机做 一致性与多路径:使用 nvme multipath -i on 并在业务低峰做 failover 演练。

  • 观察 nvme reconnect 行为,必要时设置 ctrl loss tmo。


5. Milvus:热在 NVMe,冷在 MinIO(S3)

5.1 部署(Kubernetes + Helm,关键 values)

存储与分层核心参数(示例):

 
<span class="hljs-attr">common:</span> <span class="hljs-attr">storageType:</span> <span class="hljs-string">remote</span> <span class="hljs-comment"># 远端对象存储(MinIO/S3)</span> <span class="hljs-comment"># 分层:热段缓存与索引本地落盘</span> <span class="hljs-attr">localStorage:</span> <span class="hljs-attr">enabled:</span> <span class="hljs-literal">true</span> <span class="hljs-attr">path:</span> <span class="hljs-string">/data/nvme_hot/milvus</span> <span class="hljs-attr">cacheSizeGB:</span> <span class="hljs-number">512</span> <span class="hljs-comment"># 本地段缓存,配合 QueryNode</span> <span class="hljs-attr">minio:</span> <span class="hljs-attr">enabled:</span> <span class="hljs-literal">true</span> <span class="hljs-attr">resources:</span> { <span class="hljs-string">...</span> } <span class="hljs-attr">mode:</span> <span class="hljs-string">distributed</span> <span class="hljs-attr">persistence:</span> <span class="hljs-attr">size:</span> <span class="hljs-string">8Ti</span> <span class="hljs-comment"># 冷层容量(与生命周期策略联动)</span> <span class="hljs-attr">etcd:</span> { <span class="hljs-string">...</span> } <span class="hljs-attr">pulsar:</span> { <span class="hljs-string">...</span> } <span class="hljs-attr">queryNode:</span> <span class="hljs-attr">resources:</span> <span class="hljs-attr">limits:</span> <span class="hljs-attr">cpu:</span> <span class="hljs-string">"20"</span> <span class="hljs-attr">memory:</span> <span class="hljs-string">"120Gi"</span> <span class="hljs-attr">config:</span> <span class="hljs-attr">queryNode:</span> <span class="hljs-attr">cache:</span> <span class="hljs-attr">cacheSize:</span> <span class="hljs-number">120000</span> <span class="hljs-comment"># 约 120GB,段/向量缓存(内存)</span> <span class="hljs-attr">maxParallelism:</span> <span class="hljs-number">32</span> <span class="hljs-attr">persistence:</span> <span class="hljs-attr">mountPath:</span> <span class="hljs-string">/data/nvme_hot/milvus/query</span> <span class="hljs-attr">indexNode:</span> <span class="hljs-attr">resources:</span> <span class="hljs-attr">limits:</span> <span class="hljs-attr">cpu:</span> <span class="hljs-string">"24"</span> <span class="hljs-attr">memory:</span> <span class="hljs-string">"160Gi"</span> <span class="hljs-attr">config:</span> <span class="hljs-attr">indexNode:</span> <span class="hljs-attr">indexSliceSize:</span> <span class="hljs-number">2147483648</span> <span class="hljs-comment"># 2GB 切片</span> <span class="hljs-attr">build:</span> <span class="hljs-attr">hnsw:</span> <span class="hljs-attr">M:</span> <span class="hljs-number">48</span> <span class="hljs-attr">efConstruction:</span> <span class="hljs-number">320</span> <span class="hljs-attr">ivf_pq:</span> <span class="hljs-attr">nlist:</span>
&amp

本文由A5数据原创发布,转载请注明出处:https://www.a5idc.com/helpcontent/1170.html