Step-by-Step Shader Debugging Workflow with NVIDIA Shader Debugger

How to Use NVIDIA Shader Debugger to Fix GLSL and HLSL Shaders

Debugging shaders can be one of the toughest parts of graphics development. NVIDIA Shader Debugger (part of NVIDIA Nsight Visual Studio Edition / Nsight Graphics depending on your setup) gives you the ability to step through shader execution, inspect variables, view GPU state, and find the exact cause of rendering errors. This guide shows a practical workflow to debug and fix GLSL and HLSL shaders efficiently.

Prerequisites

• NVIDIA GPU with up-to-date drivers.
• Nsight Graphics or Nsight Visual Studio Edition installed (match your development environment).
• A working project that compiles and runs your shaders (GLSL for OpenGL/Vulkan or HLSL for Direct3D/Vulkan).
• Debug build with symbols enabled (compile shaders with debug info if supported).

1. Choose the right Nsight tool

• Use Nsight Graphics for frame capture and shader debugging across Vulkan, OpenGL, DirectX.
• Use Nsight Visual Studio Edition when integrated into Visual Studio for Direct3D/HLSL workflows. Pick the tool that matches your API and developer environment.

2. Enable shader debug information

• HLSL (Direct3D): Compile with debug flags (e.g., /Zi and /Od for FXC, or -Zi for DXC) to retain source mapping. For DXC, use -Qstrip_reflect only when you no longer need reflection.
• GLSL (OpenGL/Vulkan): Use compiler options that preserve debug info (GLSL compiler or glslang for Vulkan: -g to include source mapping). In Vulkan, include SPIR-V debug info when compiling (glslangValidator -g).

3. Capture a frame

1. Launch your application through Nsight (attach to process or start from Nsight).
2. Reproduce the rendering issue in the app.
3. Capture a frame at the moment the problem appears. In Nsight Graphics, use the Capture Frame button; in Nsight Visual Studio, use the frame capture shortcut.

4. Locate the problematic draw / dispatch

• Open the captured frame.
• Use the Event List / API Calls view to find the draw call or dispatch that produced the incorrect output.
• Inspect the rendered target and mesh preview to confirm which draw call is incorrect.

5. Open shader debug for the relevant stage

• In the Event Details, find the shader you want to inspect (vertex, fragment/pixel, geometry, compute).
• Click “Debug” or “Debug Shader” for that specific shader stage. Nsight will prompt for a pixel/sample or a thread invocation (for compute).

6. Select an invocation to debug

• For fragment/pixel shaders: pick the pixel location or sample on the render target where the visual error appears.
• For vertex/geometry: select a vertex index or primitive.
• For compute shaders: choose a workgroup and thread (x,y,z) coordinates. Selecting the right invocation is critical—choose one that demonstrates the bug.

7. Step through shader execution

• Use Step Into, Step Over, and Step Out to follow execution.
• Watch the instruction pointer, control flow, and executed lines in your shader source (source mapping requires debug info).
• Inspect registers and temporary variables that the debugger exposes.
• Observe changes in input/output variables across stages.

8. Inspect variables and GPU state

• View local variables, uniforms, textures, samplers, and UBO/SSBO contents.
• For samplers/textures, check bound resource IDs, formats, and actual sampled values.
• For vertex inputs, confirm positions, normals, UVs, and other attributes are as expected.
• For compute shaders, inspect buffers and shared memory to find synchronization or indexing bugs.

9. Common issues and how to find them

• Incorrect interpolation: Check interpolation qualifiers and input semantics; inspect values at the pixel stage.
• Out-of-bounds buffer access: Validate indices before reading; inspect buffer sizes and index calculations.
• Precision/rounding errors: Inspect variable types (float vs. double vs. half) and intermediate conversions.
• Undefined behavior from uninitialized variables: Look for variables that are never written before use.
• Wrong resource binding: Verify descriptor/binding slots and root/signature tables; inspect the actual bound resource in the debugger.
• Coordinate-space mismatches: Trace transforms through vertex and varying outputs to pixel shader to confirm consistent spaces.

10. Reproduce, edit, and validate fixes

• Make an edit in your shader source (locally or via your editor).
• Recompile and run the application in debug mode.
• Capture a new frame and re-run the same invocation through the debugger to confirm the fix