AI推理低时延:在香港机房用 A10/L40S + TensorRT + Triton 动态批处理窗口,把 p95 从 120ms 打到 48ms 的完整部署与优化指南

发布人:A5数据发布时间:2025-09-26 10:58阅读量:940

那天是周五凌晨两点,香港荔枝角机房空的Grafana 大屏还在顽固地显示:p95=120ms。白天新增的 A/B 流量压上来之后,延迟抖成了“锯齿”。我站在两台刚上架的服务器前——一台插着 2×L40S,旁边是 4×A10 的推理节点,心里只想把“锯齿”磨平。
接下来 6 个小时,我从 BIOS 到驱动再到 TensorRT 引擎、Triton 动态批处理窗口一路抠参数,直到清晨 7 点,延迟曲线终于从锯齿变成了平线,p95 落在 48ms。这篇文章,就是那次上线的完整复盘和“可复制”的手册。

目标与场景

目标:在香港落地低时延推理集群,兼顾 稳定的 p95 与 合理吞吐。

典型业务:多模型混部(图像分类/检索 + 轻量文本推断),请求负载不均,流量高峰/低谷交替。

关键技术点:

  • A10/L40S 混部适配(不同 GPU 的并发与批大小策略不同)
  • TensorRT FP16/INT8 引擎(形状动态 + 校准)
  • Triton Inference Server 的 dynamic_batching + max_queue_delay_microseconds(批处理窗口)
  • 客户端微批聚合 与 自适应窗口(把队列等待控制在“刚刚好”)

机房与硬件拓扑(实配)

节点 服务器 CPU / 内存 GPU 存储 / 网络 备注
trt-l40s-01 2U 服务器(双路) 2×Xeon(Ice Lake)/ 512GB 2×L40S(48GB) 2×7.68TB NVMe;2×25GbE 主承载低延迟实时接口
trt-a10-01 2U 服务器(单/双路) 1×或2×Xeon / 256GB 4×A10(24GB) 2×3.84TB NVMe;2×10GbE 承载吞吐与离线/次要在线流

选型心得:L40S 主要抗突发、抗抖动;A10 单卡性价比高,适合并发实例多开增吞吐。两者混部能把成本与时延都拉到更优平衡。

版本矩阵(我当时的组合,供你参考)

组件 版本
OS Ubuntu 22.04 LTS(文末附 CentOS 7 兼容做法)
NVIDIA 驱动 535/550 系列(与下方 CUDA/TensorRT 匹配)
CUDA Toolkit 12.x
TensorRT 8.6+/10.x
Triton Inference Server 24.x(容器)
Docker 24.x + nvidia-container-toolkit
监控 DCGM Exporter + Prometheus + Grafana

BIOS / 内核 / 系统前置优化(“把地基打牢”)

BIOS

  • Above 4G Decoding:开启
  • Resizable BAR:开启(部分平台)
  • NUMA:确认 GPU 与 NIC 在同一 NUMA 域优先插槽
  • C-States:适当限制深度,降低抖动

电源/频率

 
nvidia-smi -pm 1 <span class="hljs-comment"># Persistence Mode</span> nvidia-smi -pl 300 <span class="hljs-comment"># 视卡而定,给足但别越保修限制</span> <span class="hljs-comment"># 某些卡支持锁频,若支持:</span> <span class="hljs-comment"># nvidia-smi -lgc <min,max></span>

网络栈(减少队列积压 + 提升突发接入能力)

<span class="hljs-built_in">cat</span> <<<span class="hljs-string">EOF | sudo tee /etc/sysctl.d/99-lowlatency.conf net.core.somaxconn=1024 net.core.netdev_max_backlog=16384 net.ipv4.tcp_tw_reuse=1 net.ipv4.tcp_fin_timeout=15 net.core.rmem_max=134217728 net.core.wmem_max=134217728 net.ipv4.tcp_rmem=4096 87380 134217728 net.ipv4.tcp_wmem=4096 65536 134217728 EOF</span> sudo sysctl --system

IRQ/亲和性

  • 保证 NIC / NVMe 中断与对应 NUMA 亲和
  • Triton 进程绑核,避免与高噪声服务抢同一 L3

驱动/容器:安装与校验

<span class="hljs-comment"># 1) 驱动(略)</span> nvidia-smi <span class="hljs-comment"># 2) NVIDIA Container Toolkit</span> distribution=$(. /etc/os-release;<span class="hljs-built_in">echo</span> $ID<span class="hljs-variable">$VERSION_ID</span>) curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg curl -fsSL https://nvidia.github.io/libnvidia-container/<span class="hljs-variable">$distribution</span>/libnvidia-container.list | \ sed <span class="hljs-string">'s#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g'</span> | \ sudo <span class="hljs-built_in">tee</span> /etc/apt/sources.list.d/nvidia-container-toolkit.list sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit sudo nvidia-ctk runtime configure --runtime=docker sudo systemctl restart docker <span class="hljs-comment"># 3) DCGM Exporter(监控)</span> docker run -d --gpus all --<span class="hljs-built_in">rm</span> --name=dcgm-exporter -p 9400:9400 nvidia/dcgm-exporter:latest

TensorRT 引擎构建(ONNX→TRT,动态形状 + FP16/INT8)

快速路径:trtexec

trtexec --onnx=model.onnx \ --saveEngine=model_fp16.plan \ --explicitBatch --minShapes=input:1x3x224x224 \ --optShapes=input:4x3x224x224 \ --maxShapes=input:8x3x224x224 \ --fp16 --workspace=4096

INT8 校准(Python 骨架)

注:校准集必须能代表线上分布;INT8 下延迟更低、吞吐更高,但需校准谨慎。

<span class="hljs-comment"># build_int8_engine.py</span> <span class="hljs-keyword">import</span> tensorrt <span class="hljs-keyword">as</span> trt <span class="hljs-keyword">import</span> numpy <span class="hljs-keyword">as</span> np <span class="hljs-keyword">class</span> <span class="hljs-title class_">MyEntropyCalibrator</span>(trt.IInt8EntropyCalibrator2): <span class="hljs-keyword">def</span> <span class="hljs-title function_">__init__</span>(<span class="hljs-params">self, batch_iter, input_shape, cache_file</span>): <span class="hljs-built_in">super</span>().__init__() self.batch_iter = batch_iter self.d_input = <span class="hljs-literal">None</span> self.input_shape = input_shape self.cache_file = cache_file <span class="hljs-keyword">def</span> <span class="hljs-title function_">get_batch_size</span>(<span class="hljs-params">self</span>): <span class="hljs-keyword">return</span> self.input_shape[<span class="hljs-number">0</span>] <span class="hljs-keyword">def</span> <span class="hljs-title function_">get_batch</span>(<span class="hljs-params">self, names</span>): <span class="hljs-keyword">try</span>: batch = <span class="hljs-built_in">next</span>(self.batch_iter) <span class="hljs-comment"># ndarray [N,C,H,W]</span> <span class="hljs-keyword">except</span> StopIteration: <span class="hljs-keyword">return</span> <span class="hljs-literal">None</span> <span class="hljs-keyword">if</span> self.d_input <span class="hljs-keyword">is</span> <span class="hljs-literal">None</span>: <span class="hljs-keyword">import</span> pycuda.driver <span class="hljs-keyword">as</span> cuda self.d_input = cuda.mem_alloc(batch.nbytes) cuda.memcpy_htod(self.d_input, batch) <span class="hljs-keyword">return</span> [<span class="hljs-built_in">int</span>(self.d_input)] <span class="hljs-keyword">def</span> <span class="hljs-title function_">read_calibration_cache</span>(<span class="hljs-params">self</span>): <span class="hljs-keyword">try</span>: <span class="hljs-keyword">return</span> <span class="hljs-built_in">open</span>(self.cache_file, <span class="hljs-string">"rb"</span>).read() <span class="hljs-keyword">except</span>: <span class="hljs-keyword">return</span> <span class="hljs-literal">None</span> <span class="hljs-keyword">def</span> <span class="hljs-title function_">write_calibration_cache</span>(<span class="hljs-params">self, cache</span>): <span class="hljs-built_in">open</span>(self.cache_file, <span class="hljs-string">"wb"</span>).write(cache) <span class="hljs-keyword">def</span> <span class="hljs-title function_">build_engine</span>(<span class="hljs-params">onnx_path, plan_path, min_shape, opt_shape, max_shape, int8=<span class="hljs-literal">False</span></span>, cache=<span class="hljs-string">"calib.cache"</span>): logger = trt.Logger(trt.Logger.WARNING) builder = trt.Builder(logger) network = builder.create_network(<span class="hljs-number">1</span><<<span class="hljs-built_in">int</span>(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)) parser = trt.OnnxParser(network, logger) <span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(onnx_path, <span class="hljs-string">"rb"</span>) <span class="hljs-keyword">as</span> f: <span class="hljs-keyword">assert</span> parser.parse(f.read()), parser.get_error(<span class="hljs-number">0</span>) config = builder.create_builder_config() config.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, <span class="hljs-number">4</span><<<span class="hljs-number">30</span>) <span class="hljs-comment"># 4GB</span> profile = builder.create_optimization_profile() input_tensor = network.get_input(<span class="hljs-number">0</span>) profile.set_shape(input_tensor.name, <span class="hljs-built_in">min</span>=min_shape, opt=opt_shape, <span class="hljs-built_in">max</span>=max_shape) config.add_optimization_profile(profile) config.set_flag(trt.BuilderFlag.FP16) <span class="hljs-keyword">if</span> int8: config.set_flag(trt.BuilderFlag.INT8) calib_iter = (np.random.randn(*opt_shape).astype(np.float32) <span class="hljs-keyword">for</span> _ <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(<span class="hljs-number">100</span>)) config.int8_calibrator = MyEntropyCalibrator(calib_iter, opt_shape, cache) engine = builder.build_engine(network, config) <span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(plan_path, <span class="hljs-string">"wb"</span>) <span class="hljs-keyword">as</span> f: f.write(engine.serialize()) <span class="hljs-comment"># 用法:</span> <span class="hljs-comment"># build_engine("model.onnx","model_int8.plan",(1,3,224,224),(4,3,224,224),(8,3,224,224),int8=True)</span>

Triton 模型仓库与动态批处理(把“窗口”掐准)

目录结构

models/ resnet_trt/ <span class="hljs-number">1</span>/model.plan config.pbtxt

config.pbtxt 示例(FP16/INT8,动态批处理窗口)

name: "resnet_trt" platform: "tensorrt_plan" max_batch_size: 8 input [ { name: "input" data_type: TYPE_FP16 dims: [3,224,224] } ] output [ { name: "logits" data_type: TYPE_FP16 dims: [1000] } ] instance_group [ { kind: KIND_GPU, gpus: [0], count: 2 } # L40S:更大批次可更少实例;A10:更小批次多实例 ] dynamic_batching { preferred_batch_size: [ 1, 2, 4, 8 ] max_queue_delay_microseconds: 800 # **批处理窗口**(核心旋钮) } response_cache { enable: true } # 适合重复查询的场景

经验:

  • L40S:可把 preferred_batch_size 值做大一点(如 2/4/8/16);
  • A10:更偏向 1/2/4;
  • max_queue_delay_microseconds 是“窗口”,起点 500–1000μs;后续让监控驱动自适应。

Triton 容器启动(Docker Compose)

<span class="hljs-comment"># docker-compose.yml</span> <span class="hljs-attr">services:</span> <span class="hljs-attr">triton:</span> <span class="hljs-attr">image:</span> <span class="hljs-string">nvcr.io/nvidia/tritonserver:24.06-py3</span> <span class="hljs-attr">restart:</span> <span class="hljs-string">always</span> <span class="hljs-attr">shm_size:</span> <span class="hljs-string">1g</span> <span class="hljs-attr">deploy:</span> <span class="hljs-attr">resources:</span> <span class="hljs-attr">reservations:</span> <span class="hljs-attr">devices:</span> <span class="hljs-bullet">-</span> <span class="hljs-attr">capabilities:</span> [<span class="hljs-string">gpu</span>] <span class="hljs-attr">command:</span> <span class="hljs-string">> tritonserver --model-repository=/models --exit-on-error=false --http-port=8000 --grpc-port=8001 --metrics-port=8002 --rate-limit=execution_count </span> <span class="hljs-attr">volumes:</span> <span class="hljs-bullet">-</span> <span class="hljs-string">/data/models:/models</span> <span class="hljs-attr">ports:</span> <span class="hljs-bullet">-</span> <span class="hljs-string">"8000:8000"</span> <span class="hljs-bullet">-</span> <span class="hljs-string">"8001:8001"</span> <span class="hljs-bullet">-</span> <span class="hljs-string">"8002:8002"</span>

客户端微批聚合:把“小水滴”凑成“刚好一杯”

服务器端 max_queue_delay_microseconds 只是上限,客户端侧再做 ≤1ms 的微批窗口,能稳定把 GPU 吞吐榨干而不拉高抖动。

<span class="hljs-comment"># client_microbatch.py</span> <span class="hljs-keyword">import</span> time, threading, queue <span class="hljs-keyword">import</span> tritonclient.grpc <span class="hljs-keyword">as</span> grpcclient <span class="hljs-keyword">import</span> numpy <span class="hljs-keyword">as</span> np Q = queue.Queue() client = grpcclient.InferenceServerClient(<span class="hljs-string">"trt-l40s-01:8001"</span>) WINDOW_US = <span class="hljs-number">600</span> <span class="hljs-comment"># 初始窗口,后续自适应</span> MAX_BATCH = <span class="hljs-number">8</span> <span class="hljs-keyword">def</span> <span class="hljs-title function_">worker</span>(): <span class="hljs-keyword">global</span> WINDOW_US <span class="hljs-keyword">while</span> <span class="hljs-literal">True</span>: start = time.time() batch = [] <span class="hljs-keyword">while</span> (time.time() - start)*<span class="hljs-number">1e6</span> < WINDOW_US <span class="hljs-keyword">and</span> <span class="hljs-built_in">len</span>(batch) < MAX_BATCH: <span class="hljs-keyword">try</span>: batch.append(Q.get(timeout=WINDOW_US/<span class="hljs-number">1e6</span>)) <span class="hljs-keyword">except</span> queue.Empty: <span class="hljs-keyword">pass</span> <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> batch: <span class="hljs-keyword">continue</span> inp = np.stack([x <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> batch], axis=<span class="hljs-number">0</span>).astype(np.float16) inputs = [grpcclient.InferInput(<span class="hljs-string">"input"</span>, inp.shape, <span class="hljs-string">"FP16"</span>)] inputs[<span class="hljs-number">0</span>].set_data_from_numpy(inp, binary_data=<span class="hljs-literal">True</span>) res = client.infer(model_name=<span class="hljs-string">"resnet_trt"</span>, inputs=inputs) <span class="hljs-comment"># 记录延迟到本地指标,供自适应模块读取</span> latency_ms = (time.time() - start)*<span class="hljs-number">1000</span> record_latency(latency_ms, <span class="hljs-built_in">len</span>(batch)) <span class="hljs-keyword">def</span> <span class="hljs-title function_">record_latency</span>(<span class="hljs-params">lat, bsz</span>): <span class="hljs-comment"># 自行写入 Prometheus Pushgateway 或本地滑窗</span> <span class="hljs-keyword">pass</span> threading.Thread(target=worker, daemon=<span class="hljs-literal">True</span>).start() <span class="hljs-keyword">def</span> <span class="hljs-title function_">submit</span>(<span class="hljs-params">req</span>): Q.put(req) <span class="hljs-comment"># 生产者线程塞请求</span>

自适应批处理窗口(让窗口“自己会变大/变小”)

核心思路:以 p95 目标 为约束(如 50ms),如果稳定低于目标,就 每 10s 把 WINDOW_US+50;如果超过,就 -50;在 [200, 1500]us 内夹紧。

TARGET_P95_MS = <span class="hljs-number">50</span> WINDOW_US = <span class="hljs-number">600</span> <span class="hljs-keyword">def</span> <span class="hljs-title function_">tune_loop</span>(): <span class="hljs-keyword">global</span> WINDOW_US <span class="hljs-keyword">while</span> <span class="hljs-literal">True</span>: p95 = get_recent_p95() <span class="hljs-comment"># 从 Prometheus 或本地滑窗取</span> <span class="hljs-keyword">if</span> p95 < TARGET_P95_MS * <span class="hljs-number">0.9</span> <span class="hljs-keyword">and</span> WINDOW_US < <span class="hljs-number">1500</span>: WINDOW_US += <span class="hljs-number">50</span> <span class="hljs-keyword">elif</span> p95 > TARGET_P95_MS <span class="hljs-keyword">and</span> WINDOW_US > <span class="hljs-number">200</span>: WINDOW_US -= <span class="hljs-number">50</span> time.sleep(<span class="hljs-number">10</span>)

实测数据(节选)

测于香港机房,开 FP16;INT8 在此模型上另有 10–20% 的延迟优化(准确性通过校准保障)。

L40S 节点(单卡,实例数=2)

批大小 p50 (ms) p95 (ms) QPS(稳态)
1 14 29 650
2 16 31 980
4 19 36 1520
8 24 44 1850

A10 节点(单卡,实例数=3)

批大小 p50 (ms) p95 (ms) QPS(稳态)
1 18 36 420
2 21 41 700
4 27 55 980

组合上线后,整体 p95 从 120ms → 48ms,QPS 提升 ~1.8×。关键在于:客户端微批 + Triton 动态批处理窗口 + 按 GPU 定制的实例数/批大小。

监控与告警要点(最有用的那几个)

  • DCGM:sm_util, mem_copy_util, pstate, power_usage
  • Triton:nv_inference_request_duration_us, nv_inference_queue_duration_us, nv_inference_exec_count
  • 自研:客户端 发送→返回 全链路延迟,分 p50/p95,并记录 batch 实际大小分布(看窗口是否有效)
  • 告警:p95 连续 5 分钟高于阈值 + queue_duration 上升,自动把窗口收紧 100–200μs

线上“坑”与解决(都是半夜踩的)

GPU 负载不满但延迟高:

排查发现 NIC 与 GPU 不在同一 NUMA,跨 NUMA 引发额外拷贝;调整插槽 + 进程绑核,p95 立刻下降 ~8ms。

INT8 推精度掉点:

校准集太干净;换真实线上样本 + 叠加噪声增强,重新校准后精度回到期望,延迟仍优于 FP16。

A10 上批太大抖动:

A10 多开实例、批保持在 1/2/4,吞吐反而更好;L40S 则倾向更大批。

GPU 频率时不时降档:

机房某时段温度波动,pstate 频繁切换;加风道塑形 + 合理 -pl,锁定负载区间。

Triton 冷启动慢:

首次加载引擎占用时间;上线前做 预热(warmup),并开启 response_cache 对热门查询生效。

网络路径 MTU 不一致:

香港—内地线路部分链路 MTU 较小,观察到分片/重传;在出口侧对 PMTU 做探测与限 MTU,抖动立降。

本文由A5数据原创发布,转载请注明出处:https://www.a5idc.com/helpcontent/1188.html