The relevant part of my code had looked like this:

$norm_u = sqrt($norm_u);

$norm_v = sqrt($norm_v);

$angle = arccos($prod/$norm_u /$norm_v );

Once the problem was brought to my attention, I was able to fix it, by doing this:

if ($prod>0 and $prod*$prod == $norm_u *$norm_v ) {$angle=0;}

elsif ($prod<0 and $prod*$prod == $norm_u * $norm_v ){$angle=pi;}

else {$angle = arccos($prod/sqrt($norm_u) /sqrt($norm_v) ); }

$norm_u = sqrt($norm_u);

$norm_v = sqrt($norm_v);

But my first attempts at a fix also failed to work, like putting the if-clause after I had taken the square roots (the if-clause wouldn't trigger and the same error would occur).

I am running an older version of WebWork (2.10), but this still seems like a weird problem to have.