16. Troubleshooting and counter access

ERR_NVGPUCTRPERM or no permission for counters

Recent NVIDIA drivers restrict access to GPU performance counters by default. On a workstation, enable access through the NVIDIA settings UI:

  • NVIDIA App: System → Advanced → Developer → Manage GPU Performance Counters, then allow access for all users.
  • Legacy NVIDIA Control Panel, run as administrator: enable Developer Settings, open Developer → Manage GPU Performance Counters, and allow access.

On Linux, an administrator can instead grant only the profiling process the required capability or configure the driver's counter-access policy. Follow NVIDIA's current platform-specific instructions rather than copying an old module option blindly. Containers must pass through the profiling capability as well as the GPU.

Verify with a one-launch basic capture:

ncu --set basic --kernel-name 'regex:known_kernel' \
  --launch-count 1 ./application

On WSL, restart the WSL VM after changing Windows driver settings if access does not update immediately:

wsl --shutdown

Nsight Compute cannot connect to the CUDA driver

Check that the process sees the real driver library rather than the CUDA stub:

nvidia-smi
ldconfig -p | grep libcuda

Containers need GPU device access and profiling permissions. A sandbox may intentionally block the driver even when ordinary shell CUDA programs work.

System-wide CPU sampling fails

Linux perf_event_paranoid, container capabilities, or WSL support can block system-wide samples. Prefer process-tree sampling when it answers the question:

nsys profile --sample=process-tree --cpuctxsw=process-tree ./application

Lowering a system security control is an administrative decision, not a routine profiling step.

Reports disagree on duration

Expected causes include warm-up, clock variation, ncu multi-pass replay, memory backup/restore, cache-state changes, graph tracing mode, and profiler overhead. Use:

  • unprofiled wall time for final performance;
  • Systems for pass/family contribution and gaps;
  • Compute for per-kernel character;
  • a microbenchmark with stable clocks for precise kernel comparisons.

Achieved occupancy looks impossible

Very short kernels, periodic sampling, new architectures, or tool/driver version mismatches can produce counter anomalies such as achieved occupancy slightly above the nominal maximum. Cross-check static theoretical occupancy, active-warps counters, duration, and release notes. Do not anchor a rewrite on one implausible sampled value.

Source correlation is missing

Rebuild the production-equivalent optimized binary with -lineinfo, retain JIT artifacts, and ensure Nsight Compute can locate source paths. Avoid -G for performance characterization. If only SASS is available, profile by PC and opcode class.

Profiling takes forever

Full-application replay-based profiling scales with what the kernel touches, not just how long it runs. Narrow the scope before accepting a long capture:

  • filter one exact kernel occurrence;
  • begin with basic or selected sections;
  • shorten the workload after warm-up;
  • use NVTX/start-stop ranges;
  • consider application/range replay for large touched memory or stateful code;
  • disable source/PC sampling sections until needed.

The GUI cannot open a remote report

Capture with CLI on the target, copy the .nsys-rep or .ncu-rep, and open it with a compatible or newer desktop UI. Preserve reports alongside version and command metadata.