Today, I tested on an old machine with NVIDIA GPU k80 installed, but found strange issue, such as creating stream failed. Using nvidia-smi command and find the 3rd card’s ECC is 2, not 0:
It seems there is no better method. Just restart the server to reset GPU, and it worked normally.