The relevant part of my code had looked like this:
$norm_u = sqrt($norm_u);
$norm_v = sqrt($norm_v);
$angle = arccos($prod/$norm_u /$norm_v );
Once the problem was brought to my attention, I was able to fix it, by doing this:
if ($prod>0 and $prod*$prod == $norm_u *$norm_v ) {$angle=0;}
elsif ($prod<0 and $prod*$prod == $norm_u * $norm_v ){$angle=pi;}
else {$angle = arccos($prod/sqrt($norm_u) /sqrt($norm_v) ); }
$norm_u = sqrt($norm_u);
$norm_v = sqrt($norm_v);
But my first attempts at a fix also failed to work, like putting the if-clause after I had taken the square roots (the if-clause wouldn't trigger and the same error would occur).
I am running an older version of WebWork (2.10), but this still seems like a weird problem to have.