15. Command and metric reference

Commands are templates. Check --help because Nsight options and report names evolve.

Baseline inventory

Record the hardware and tool versions before any profiling run:

nvidia-smi -L
nvidia-smi --query-gpu=name,driver_version,pstate,clocks.sm,clocks.mem,\
temperature.gpu,power.draw,memory.total --format=csv
nsys --version
ncu --version

Nsight Systems

Capture a bounded CUDA/NVTX timeline:

nsys profile --trace=cuda,nvtx,osrt \
  --sample=process-tree --cpuctxsw=process-tree \
  --output=timeline ./application

Expose nodes inside CUDA Graphs when necessary:

nsys profile --trace=cuda,nvtx \
  --cuda-graph-trace=node --output=graph_nodes ./application

Sample GPU metrics on supported systems:

nsys profile --trace=cuda,nvtx \
  --gpu-metrics-device=all \
  --gpu-metrics-frequency=10000 \
  --output=gpu_metrics ./application

Generate summaries:

nsys stats --report cuda_gpu_kern_sum,cuda_gpu_mem_time_sum timeline.nsys-rep
nsys stats --report cuda_api_sum,nvtx_sum timeline.nsys-rep
nsys stats --report cuda_gpu_trace:base timeline.nsys-rep

Nsight Compute

List sets and sections:

ncu --list-sets
ncu --list-sections
ncu --query-metrics-mode suffix --metrics sm__throughput

Profile exactly one matching launch (replay makes each additional launch costly, and a later launch may not represent the shape or state you actually want to characterize):

ncu --set detailed \
  --kernel-name-base function \
  --kernel-name 'regex:kernel_name' \
  --launch-count 1 \
  --output=kernel_report \
  ./application

Reach the sixth matching launch:

ncu --set full --kernel-name 'regex:kernel_name' \
  --launch-skip 5 --launch-count 1 -o kernel_sixth ./application

Collect selected sections:

ncu --section SpeedOfLight \
  --section LaunchStats \
  --section Occupancy \
  --section SchedulerStats \
  --section WarpStateStats \
  --section MemoryWorkloadAnalysis \
  --section ComputeWorkloadAnalysis \
  --section SourceCounters \
  -k 'regex:kernel_name' -c 1 -o focused ./application

Export details:

ncu --import focused.ncu-rep --page details --csv

Compilation and disassembly

Build with line info, not full debug info (-G disables optimizations and changes the generated code), and inspect the generated code directly:

nvcc -O3 -lineinfo -Xptxas=-v kernel.cu -o kernel
cuobjdump --dump-resource-usage ./kernel
cuobjdump --dump-sass ./kernel
nvdisasm --print-line-info kernel.cubin

Metric map

Use section labels in the UI before relying on raw metric names. Raw names are architecture-specific, but their prefixes are useful:

Minimal interpretation worksheet

Fill this in, in order, for any kernel under investigation:

Workload and shape:
End-to-end metric and distribution:
Pass / family share:
Selected launch (grid, block, regs, smem):
Compute vs memory throughput:
Theoretical / achieved occupancy:
Active / eligible / issued warps:
Dominant stalls (only if issue-starved):
Bytes, hit rates, sectors, local traffic, conflicts:
Dominant execution pipe and expected instructions:
Source/SASS location:
Amdahl ceiling:
Next one-variable experiment: