When modern HPC systems—comprised of CPUs and GPUs—perform numerical computations, floating-point exceptional quantities such as NaN and infinity in the GPU context remains, at present, insufficiently handled. This is because commonly used GPUs and the CUDA language have no inherent exception detection capabilities. Existing compiler-based approaches to this problem are tied to a given compiler (thus handicapped when dealing with precompiled libraries) and cannot detect exceptions generated by optimized binaries. In this paper, we contribute BinFPE, a unique tool that addresses these challenges. BinFPE uses the NVBit dynamic binary instrumentation framework to check the machine registers after each calculation to recognize exceptions, and conveys this information via GPU-to-CPU channels to the CPU for final reporting. We demonstrate the effectiveness of BinFPE on 42 CUDA programs, reporting previously unreported exceptions. We also present the limitations of BinFPE and our perspective on building GPU tools via binary instrumentation.