14. External practice exercises

Use these small workloads to learn one phenomenon at a time before profiling a large application.

1. Vector addition: bandwidth and launch overhead

Vector addition is close to the simplest kernel that can exist: one load, one add, one store, no data reuse, no branching. That is precisely why it is the right starting point — any launch-overhead or bandwidth-roof effect seen here cannot be blamed on algorithmic complexity, so it isolates the profiling phenomenon itself rather than an artifact of the workload.

Profile C[i] = A[i] + B[i] across sizes from 1 KiB to 1 GiB.

Predict and then verify:

  • small sizes are launch/latency dominated;
  • large sizes approach a DRAM bandwidth roof;
  • effective bytes are two reads plus one write;
  • occupancy beyond what hides memory latency does not raise the roof.

Start from the official CUDA Samples repository's introductory vector example or write a one-dimensional grid-stride loop.

2. Matrix transpose: coalescing and bank conflicts

Vector addition isolated a single memory pass in one dimension. Transpose adds a second dimension and a shared-memory path, which is what makes it possible to tell coalescing and bank-conflict effects apart.

Implement three versions:

  1. direct transpose with strided stores;
  2. shared-memory tiled transpose;
  3. tiled transpose with a padded shared leading dimension.

Use Systems to confirm equal launch structure, then Compute to compare global sector efficiency, shared conflict wavefronts, and effective bandwidth. This is the cleanest exercise for separating DRAM transactions from shared-memory bank mapping.

3. Reduction: synchronization and tail behavior

Transpose kept every thread's work independent until the final store. Reduction removes that independence: threads must combine results, so synchronization and tail behavior become the new variable to isolate.

Sweep power-of-two and awkward sizes. Compare a block reduction with a warp-shuffle version. Inspect barrier stalls, branch efficiency, waves, and the cost of a second reduction launch. Then test whether fusion into the consumer beats a locally faster standalone reduction.

4. GEMM: Tensor eligibility and waves

The first three exercises are memory-bound by construction. GEMM moves the ceiling to the Tensor pipe, where eligibility and wave efficiency replace DRAM throughput as the limiting question.

Use the CUTLASS Profiler or cuBLAS benchmark to sweep neighboring M,N,KM,N,K values, dtypes, and layouts. Record selected kernel, Tensor-pipe utilization, tile count, wave efficiency, and padding efficiency. Include small skinny matrices; square peak-TFLOP charts do not represent every application.

5. Fused softmax or layer normalization: the fusion roof

GEMM isolated one large compute-bound kernel. Fusion goes the other way, asking how many of the smaller memory-bound passes from exercises 1 and 3 can be collapsed into one kernel instead.

Compare a framework composition with the official Triton fused-softmax or layer-normalization tutorial. Count launches and bytes avoided. A successful fusion should show both a simpler Systems timeline and less memory traffic, not merely fewer source lines.

6. Attention: resource trade-offs

Attention is where the earlier lessons compound: Tensor Core eligibility (exercise 4) and kernel fusion (exercise 5) now share one kernel with the register and occupancy limits from the transpose and reduction exercises.

Use the Triton fused-attention tutorial across head dimensions and sequence lengths. Track register count, shared memory, occupancy, Tensor utilization, and numerical correctness. Observe how the best tile changes with shape and how an occupancy decrease can accompany a speedup.

Each exercise should produce a one-page ledger: baseline, hypothesis, one change, relevant counters, selected-kernel timing, and end-to-end timing. This is the same ledger discipline from chapter 12, applied to workloads small enough to iterate on in minutes rather than hours.