Common AI Infra Terminology - Cat · Code

Start by memorizing the layers

Hardware/interconnect: GPU, SM, HBM, host memory, PCIe, NVLink, RDMA; determines available compute, bandwidth, and communication paths.
runtime: organizes model weights, operators, scheduling, batching, the KV cache, and GPU workers so they can execute; examples include vLLM, TensorRT-LLM, SGLang, and the Triton backend.
gateway/router: access authentication, rate limiting, routing, load balancing, retries, and streaming pass-through. It typically doesn't run the Transformer forward pass, and it doesn't own the runtime's full scheduling state.
platform: Kubernetes places and maintains objects like Pods and Services; Ray schedules tasks/Actors and logical resources; KubeRay uses CRDs to manage Ray clusters and jobs.
observability and benchmarking: Prometheus handles scrappable metric time series, and OpenTelemetry unifies the collection semantics for traces/metrics/logs; the load-testing tool is the one that actually measures the workload.

1. Hardware, CUDA, and Performance

Term	Explanation	Learning boundary
Host / Device	The CPU and its memory are the host; the GPU and its memory are the device. CPU code can launch GPU kernels, copy memory, and wait for completion.	"Visible to the GPU" doesn't mean the program is already doing its work on the GPU; you also have to look at kernels, data, and synchronization.
SM / CUDA core / kernel	An SM is the hardware unit in the GPU that executes thread blocks; a kernel is a function launched on the device. Thread blocks are scheduled onto SMs.	Kernel launches, synchronization, and memory access all affect end-to-end time, so you can't just look at theoretical FLOPS.
HBM / global memory / shared memory / register	HBM typically holds model weights and the KV cache; global memory is accessible by all SMs; shared memory is on-chip shared space within a thread block; registers are private to a thread.	Compute-bound and bandwidth/memory-access-bound are different bottlenecks; decode often tends to expose weight/KV memory-access and synchronization costs more readily.
Occupancy	The degree to which warps/thread blocks that can reside on an SM fill up the hardware limit.	High occupancy doesn't guarantee high performance; you also have to look at memory bandwidth, instruction efficiency, memory coalescing, and actual parallelism.
FLOPS, bandwidth, arithmetic intensity	FLOPS is arithmetic throughput, memory bandwidth is how much data is moved per unit time, and arithmetic intensity is the amount of computation per byte moved.	Roofline is a performance-upper-bound analysis tool, not a measured utilization number; you should measure kernel, communication, scheduling, and API overhead separately.
CUDA stream / asynchronous	A stream is an ordered queue of GPU work; many CUDA operations can be launched asynchronously, and you establish dependencies with events or synchronization when needed.	"The function returned" doesn't mean the GPU has finished; timing must be properly synchronized.

Source: CUDA Programming Model, CUDA C++ Programming Guide.

2. Distributed Training and Communication

Term	Explanation	Don't confuse it with
rank / world size / process group	rank is a process's number within a communication group; world size is the number of processes in the group; a process group defines which communications it takes part in.	rank isn't a GPU number, and the mapping needs to be confirmed explicitly; different groups can have different members.
collective	A communication operation that requires all ranks in the group to participate in a consistent order. Common ones are Broadcast, AllReduce, AllGather, ReduceScatter, and AlltoAll.	AllReduce is a communication semantic, not an algorithm; Ring/Tree are implementations, and NCCL is a communication library.
AllReduce	Each rank contributes data, and after a reduction such as a sum, every rank receives the same result. DDP commonly uses it to synchronize gradients.	You must ensure count, dtype, call order, and rank participation are consistent, otherwise you may get a hang, a crash, or incorrect data.
AllGather / ReduceScatter	AllGather concatenates each rank's data and sends it to all ranks; ReduceScatter reduces first, then sends a different shard to each rank.	ReduceScatter + AllGather can semantically make up an AllReduce, but the memory footprint and communication cost still need to be measured.
DDP / FSDP	DDP typically holds the full model per process and synchronizes gradients; FSDP shards parameters, gradients, and optimizer state, and aggregates/releases them as needed.	Both are PyTorch training abstractions, not equivalent to Kubernetes/Ray job scheduling.
DP (Data Parallel)	Different replicas process different data, and the parameter replicas stay consistent through gradient synchronization; in inference serving it often means multiple independent engines/replicas handling different requests.	Training DP and serving DP have similar goals but different state; the KV cache in serving DP is usually independent per instance.
TP (Tensor Parallel)	Splits the weights/compute tensors of a single layer across multiple GPUs, which then need to communicate frequently to complete one forward pass together.	TP is not "just launching more replicas"; it forms one logical model instance, and the communication topology is critical.
PP (Pipeline Parallel)	Puts different layers/stages on different GPUs, with micro-batches flowing between stages.	The bubbles, uneven stage load, and activation retention of PP differ from TP's intra-layer communication.
EP (Expert Parallel)	A MoE's experts are spread across different ranks, tokens are routed to the corresponding expert, and the typical communication is AlltoAll.	EP only describes how experts are parallelized; attention layers may still use TP/DP, so EP can't be treated as a synonym for DP or TP.

Source: PyTorch Distributed, NCCL Overview, NCCL Collective Operations.

3. Model Execution and LLM Inference

Term	Explanation	Engineering tip
training vs inference	training has forward, loss, backward, gradients, and the optimizer update; inference usually only does forward and sampling, without updating weights.	Training throughput is usually read as samples/tokens per second and step time; serving also has to look at queueing, first token, and streaming interval.
prefill	Feeds a batch of tokens from the user's input prompt into the model, computes context representations, and builds the KV cache.	It tends to lean toward large matrix computations; a long prompt mainly affects TTFT and memory usage.
decode	Autoregressively generates new tokens step by step; each step uses a new query to access the KV cache of the existing context.	It's often a small amount of compute plus a lot of weight/KV memory access; it's affected by bandwidth, batch, cache layout, and scheduling.
KV cache	Stores the Key/Value of processed tokens to avoid recomputing historical tokens during decode.	It isn't the model weights, and it isn't "caching the complete response"; its size grows with context tokens, layer count, KV heads, head dimension, and dtype.
MHA / GQA / MQA	In MHA each query head has its own corresponding KV heads; in GQA multiple query heads share a set of KV heads; MQA shares even further down to very few KV heads.	Reducing KV heads usually lowers the KV cache and decode memory access, but it isn't the same as reducing query heads or the model's total computation.
continuous / in-flight batching	Requests don't have to wait for the whole batch to finish; the scheduler can add/remove requests between iterations, letting the prefill/decode of different requests share the GPU.	Batch size, concurrency, queue wait, and token budget together determine throughput/latency; don't just look at the static batch.
paged KV cache / prefix caching	Manages the KV cache in blocks, reducing the need for large contiguous chunks of memory; prefix caching reuses the already-computed KV for identical prefixes.	Hit rate, block size, lease/eviction policy, and multi-replica routing all change the payoff; a cache hit doesn't mean the request returns immediately.
quantization / speculative decoding	quantization stores or computes at lower precision; speculative decoding has a draft generate candidates first, then a target verifies them.	Quantization mainly affects weights/compute/bandwidth, and doesn't necessarily shrink the KV cache automatically; the speedup depends on the model, hardware, and acceptance rate.

Source: vLLM Architecture Overview, vLLM Data Parallel Deployment, vLLM Serve/KV cache options, SGLang docs, TensorRT-LLM docs.

4. LLM Inference Serving and Platform Operations

model execution runtime: loads weights, executes operators, manages GPU workers, and schedules requests and the KV cache. vLLM's official architecture explicitly describes the Engine Core's responsibilities as coordinating the scheduler, KV cache, and GPU workers.
inference server: provides the protocol, model instances, batching, health checks, and metrics for one or more models; Triton's dynamic batching combines requests within a queueing window and can be configured with a preferred batch size / queue delay.
gateway / router: handles authentication, routing, rate limiting, retries, failover, and streaming proxy between clients and the backend runtime. It can pick a backend by load or KV-cache locality, but that doesn't make it an inference runtime.
Kubernetes Pod / Deployment / Service: a Pod is a schedulable unit of execution; a Deployment manages the desired state of stateless replicas; a Service provides a stable service-discovery/access endpoint. Kubernetes handles resource orchestration and lifecycle, not the TP/PP computation semantics of the Transformer.
Ray task / actor / logical resource: a Ray task is a schedulable function call, and an Actor is a stateful compute entity; Ray expresses scheduling requirements with logical CPU/GPU/custom resources. Ray Train can set up a training worker group and hook into PyTorch Distributed.
KubeRay: an operator that manages Ray on Kubernetes; the core CRDs are RayCluster, RayJob, and RayService. It connects platform lifecycle with the Ray runtime, but doesn't replace PyTorch/NCCL collectives.
autoscaling / admission / backpressure: autoscaling changes capacity, admission controls whether requests are accepted, and backpressure lets upstreams sense queuing or a resource shortage; none of the three is about "computing a single model faster."

Source: Triton Dynamic Batching, Triton Metrics, Kubernetes Concepts, Ray Resources, KubeRay on Kubernetes.

5. Benchmarking and Observability Metrics

Metric	Meaning	Boundary
throughput	The number of requests, input tokens, or output tokens completed per unit time.	You must state clearly whether it's request/s, input tok/s, or output tok/s, along with concurrency, input/output length, batch, and whether queueing is included.
latency	The time a request takes from some starting point to an endpoint.	client latency, gateway latency, server request latency, and model compute latency are not the same quantity.
TTFT	Time to First Token, the time from when a request is sent/received to the first output token.	It's affected by the network, queue, prefill, and sending the first packet, and isn't equal to pure prefill kernel time.
ITL / TPOT	The time between adjacent output tokens/streaming responses, often used to describe the decode experience; the exact naming and denominator depend on the tool's definition.	Don't treat the average ITL as the whole-request latency; output length and streaming aggregation affect the result.
p50 / p95 / p99	The median, 95th, and 99th percentiles of the latency distribution.	tail latency isn't the average; you should state the time window, sample size, stable phase of the load test, and the quantile calculation method.
queue time / compute time	The time a request waits in the scheduling queue / the backend's execution and related computation time.	A larger total latency may be a capacity or admission problem rather than the kernel getting slower.
Prometheus counter/gauge/histogram/summary	counter accumulates monotonically, gauge can go up or down, histogram buckets the distribution, and summary exposes client-side quantiles directly.	For latency distributions, histogram is usually the preferred choice; don't simply average a summary's quantile across instances.
trace / span / metric / log	A trace describes one cross-service causal chain; a span describes one operation within it; a metric is an aggregatable number; a log is an event/record.	OTel is a collection and semantic standard, not a backend store; you should correlate by trace id, request id, and model/version labels.
SLO / error budget	An SLO is a service target (such as availability, TTFT p99, or success rate); the error budget is the allowed amount of failure/violation.	Metrics are observed facts, while SLOs are user-facing targets; high GPU utilization doesn't mean the SLO is met.

Source: Prometheus Metric Types, Prometheus Histograms and Summaries, OpenTelemetry Signals, OpenTelemetry Semantic Conventions, Triton GenAI-Perf Metrics.

6. The Most Confusing Boundaries: A One-Page Review

Training ≠ Inference: training has backpropagation and parameter updates; inference usually doesn't. The DP/TP/PP on the serving side is execution-deployment semantics, and you can't directly apply the gradient synchronization explanation from training.
runtime ≠ gateway: the runtime executes the model and manages scheduling/the KV cache; the gateway handles the entrance and traffic governance. One gateway can proxy multiple runtimes, and one runtime can also be accessed by multiple entrances.
throughput ≠ latency: raising concurrency and batching may increase token/s while making queueing and p99 worse; you must report both together under the same workload.
DP/TP/PP/EP are different dimensions: DP replicates and processes different samples/requests; TP splits a single layer's tensor; PP splits layers; EP splits MoE experts. In a combined deployment, you need to state which layers and communication groups each dimension acts on.
prefill ≠ decode: prefill processes the existing prompt and builds the cache; decode generates token by token and reuses the cache. The two can be scheduled together, or split across different instances in P/D disaggregation.
KV cache ≠ prefix cache ≠ response cache: the KV cache is a model intermediate state; prefix cache is the KV reuse for identical prefixes; response cache is caching the final result/response, and the hit semantics, invalidation conditions, and correctness boundaries differ.
Model parallelism ≠ multiple replicas: TP/PP/EP together form one cross-GPU execution instance; DP is mostly multiple replicas/engines. The replica count in Kubernetes alone can't describe the model-parallelism approach.
GPU utilization ≠ effective throughput: a high utilization sample may come from communication, copies, or inefficient kernels; you should judge it alongside tokens/s, queue time, TTFT/ITL, memory usage, NCCL time, and error rate.

7. A Suggested Engineering Learning Path

Start with CUDA to understand host/device, the memory hierarchy, streams, synchronization, and bandwidth/compute bottlenecks.
Then use PyTorch Distributed + NCCL to practice rank, process group, and AllReduce/AllGather/ReduceScatter, and watch how a wrong order causes a hang.
Read one runtime's request path: API → scheduler → prefill/decode → KV cache → model forward → sampling → stream response.
Use Triton or vLLM to load-test with a fixed model, fixed input/output lengths, and fixed concurrency, recording throughput, TTFT, ITL, p50/p95/p99, and queue/compute separately.
Finally, put the runtime into Kubernetes/Ray, clarify what each layer is responsible for—objects, state, failures, and autoscaling boundaries—and then use OTel/Prometheus to correlate end-to-end requests.