vllm.models.minimax_m3.amd.mono.kernels.common
¶
Device-side building blocks shared by the mono kernels.
Ported from the FlyDSL GLM-5 layer kernel (ROCm/FlyDSL branch
glm5-mla-moe-layer-monokernel @ 1e28fd8a,
kernels/mla_moe_layer/shared_reuse_moe_kernel.py): wave reductions, tagged
mailboxes and the MFMA GEMV loop. Only published flydsl APIs and aiter's vendored
buffer_ops / dpp_utils are used, so nothing here needs the FlyDSL repo.
Every function but kernel_symbol is traced inside a @flyc.kernel body.
Raw boundaries the kernels keep, for want of an equivalent on the pinned FlyDSL:
buffer_ops loads / stores (the kernels take raw Int64 addresses, no tensors
for make_buffer_tensor); raw MFMA intrinsics (hand-laid operands: permlane
transposes, fp8 packing, MXFP4 scales); the LLVM intrinsics here (expect,
s_memrealtime, the approximate rsq / rcp / exp2) and the struct unpacking of
permlane*_swap; inttoptr for the timeline's global stores.
Classes:
-
Mailbox–Tagged-pair hand-off between CTAs (device scope) and GPUs (system scope).
Functions:
-
ar_block_sum–Sum of the 768 per-pack partials of a 6144-wide bf16 row in the order the
-
ar_block_sums–ar_block_sumof several rowsparts = [(p0, p1), ...]behind one set -
ar_pack_sumsq–One logical thread of the custom all-reduce epilogue: its 8 elements'
-
bf16_pair–Two f32 -> one f32-typed word holding (bf16(a), bf16(b)).
-
bf2_f32–Packed bf16 pair word -> (f32 low, f32 high).
-
block_max–Max of v over the CTA's WAVES waves;
redneeds WAVES words. -
butterfly–Butterfly reduction over the given xor offsets, in the given order.
-
div_rn–x / drounded to nearest, bit-identical to the IEEE division, given -
fp8_pack4–Four f32 (already in range) -> one dword of four E4M3 bytes, RNE.
-
fp8x8_bf16_pk–Eight fp8 of two dwords -> bf16 MFMA operand (exact) by
-
hw_rsq–v_rsq_f32 (~1 ulp, no libm range fixups).
-
kernel_symbol–Kernel symbol that profilers demangle to
vllm::<stem>_<params>. The -
lane_gather–Lane
src_lane's value of v, a source lane a lane (ds_bpermute). -
memrealtime–s_memrealtime: the 100 MHz device clock (no FlyDSL wrapper).
-
permlane_swap–v_permlane{32,16}_swap on registers (x, y): off 32 trades x's upper-half
-
readlane–Lane
src_lane's value of v (wave-uniform). -
traced–Apply flydsl's AST rewriting to a helper, so its
while/ifover -
unlikely–condmarked cold (llvm.expect): LLVM lays the branch out of the hot -
xred–op(v, lane ^ off) for a symmetric op; 32 / 16 take both halves of one swap.
-
xshfl–Value of lane
lane ^ off: offsets 32 / 16 via v_permlane*_swap, in-row
Mailbox
¶
Tagged-pair hand-off between CTAs (device scope) and GPUs (system scope).
Every 32-bit value is stored next to this launch's epoch tag; a consumer
polls the payload until all tags match, so a hand-off costs one round trip
(no store drain, no separate flag). The epoch is step * LAYER_SLOTS +
layer + 1 with step a device counter, so the sequence is graph-safe.
Methods:
-
poll–Poll
(base_addr, pair index, npairs)specs in one batch. -
put_bf–Store 2 or 4 elements as aligned packed bf16 pairs.
-
put_words–Raw 32-bit words (1 or 2) at pairs i.. as (word, tag). Integer payloads
Source code in vllm/models/minimax_m3/amd/mono/kernels/common.py
265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 | |
poll(specs, scope='agent', batch=POLL_MAX)
¶
Poll (base_addr, pair index, npairs) specs in one batch.
All pairs are re-loaded together while any tag is stale, so a batch costs
one round trip after its last producer lands. scope: "agent" for pairs
another CTA of this GPU wrote, "system" for pairs a peer GPU wrote. Returns
the value words.
Source code in vllm/models/minimax_m3/amd/mono/kernels/common.py
put_bf(base_addr, i, vs, cm=CM_DEV)
¶
Store 2 or 4 elements as aligned packed bf16 pairs.
Source code in vllm/models/minimax_m3/amd/mono/kernels/common.py
put_words(base_addr, i, words, cm=CM_DEV)
¶
Raw 32-bit words (1 or 2) at pairs i.. as (word, tag). Integer payloads (fp8 words) must come this way: through an f32 value a NaN bit pattern is not kept.
Source code in vllm/models/minimax_m3/amd/mono/kernels/common.py
_spin(load_all, pending)
¶
Re-issue load_all until pending clears; a side-effecting op in the
loop keeps the loads from being hoisted.
Source code in vllm/models/minimax_m3/amd/mono/kernels/common.py
ar_block_sum(p0, p1, lane, wave, red)
¶
Sum of the 768 per-pack partials of a 6144-wide bf16 row in the order the 1-stage fused all-reduce + RMSNorm reduces them: 24 warps of 32, a 16..1 butterfly per warp, then the 24 warp sums butterflied again in one warp.
Logical warp 3 * wave + lane // 32 holds p0; logical warp
3 * wave + 2 holds p1 in lanes < 32 (WAVES = 8). red needs
24 words. Matching this order keeps rstd bit-exact with the original path.
Source code in vllm/models/minimax_m3/amd/mono/kernels/common.py
ar_block_sums(parts, lane, wave, red)
¶
ar_block_sum of several rows parts = [(p0, p1), ...] behind one set
of barriers; red needs 24 words per row.
Source code in vllm/models/minimax_m3/amd/mono/kernels/common.py
ar_pack_sumsq(vals)
¶
One logical thread of the custom all-reduce epilogue: its 8 elements'
squares accumulated in order (hipcc contracts acc += v * v into fma).
Source code in vllm/models/minimax_m3/amd/mono/kernels/common.py
bf16_pair(a, b)
¶
Two f32 -> one f32-typed word holding (bf16(a), bf16(b)).
bf2_f32(w)
¶
block_max(v, lane, wave, red)
¶
Max of v over the CTA's WAVES waves; red needs WAVES words.
Source code in vllm/models/minimax_m3/amd/mono/kernels/common.py
butterfly(v, offsets, op=_add)
¶
Butterfly reduction over the given xor offsets, in the given order.
The order is the numerical contract: the aiter per-head norm reduces 32 lanes with offsets 16, 8, 4, 2, 1, and matching it keeps rstd bit-exact.
Source code in vllm/models/minimax_m3/amd/mono/kernels/common.py
div_rn(x, d, r)
¶
x / d rounded to nearest, bit-identical to the IEEE division, given
r = 1.0 / d from one true division per divisor (Markstein: a quotient
within 1 ulp plus its exact fma residual times RN(1/d) rounds correctly).
Source code in vllm/models/minimax_m3/amd/mono/kernels/common.py
fp8_pack4(a, b, c, d)
¶
Four f32 (already in range) -> one dword of four E4M3 bytes, RNE.
Source code in vllm/models/minimax_m3/amd/mono/kernels/common.py
fp8x8_bf16_pk(words)
¶
Eight fp8 of two dwords -> bf16 MFMA operand (exact) by v_cvt_scalef32_pk_bf16_fp8 (scale 1), two values an instruction.
Source code in vllm/models/minimax_m3/amd/mono/kernels/common.py
hw_rsq(x)
¶
kernel_symbol(stem, **params)
¶
Kernel symbol that profilers demangle to vllm::<stem>_<params>. The
amdhsa assembler rejects :: in a symbol, so it is Itanium-mangled, as
aiter's _ZN5aiter...E kernels are.
Source code in vllm/models/minimax_m3/amd/mono/kernels/common.py
lane_gather(v, src_lane)
¶
Lane src_lane's value of v, a source lane a lane (ds_bpermute).
memrealtime()
¶
s_memrealtime: the 100 MHz device clock (no FlyDSL wrapper).
permlane_swap(off, x, y)
¶
v_permlane{32,16}_swap on registers (x, y): off 32 trades x's upper-half lanes with y's lower-half lanes; off 16 trades x's odd 16-lane rows with y's even rows (each within its half). -> (x', y').
Source code in vllm/models/minimax_m3/amd/mono/kernels/common.py
readlane(v, src_lane)
¶
Lane src_lane's value of v (wave-uniform).
Source code in vllm/models/minimax_m3/amd/mono/kernels/common.py
traced(fn)
¶
Apply flydsl's AST rewriting to a helper, so its while / if over
traced values lower to scf ops. Kernels rewrite only their own body.
unlikely(cond)
¶
cond marked cold (llvm.expect): LLVM lays the branch out of the hot
path. A rare path's code inline among the hot stages costs them even when it
never runs (the long-context selection: S = 16, 3k context +1.4 us).
Source code in vllm/models/minimax_m3/amd/mono/kernels/common.py
xred(v, off, op)
¶
op(v, lane ^ off) for a symmetric op; 32 / 16 take both halves of one swap.
Source code in vllm/models/minimax_m3/amd/mono/kernels/common.py
xshfl(v, off)
¶
Value of lane lane ^ off: offsets 32 / 16 via v_permlane*_swap, in-row
offsets via DPP (VALU latency, not LDS).