"NaN" or Not a Number is generally a result of an invalid equation. i.e. You are trying to divide a number by 0.
The way the math script works, if you leave a field blank, for the user to enter a value it will initially view that field as having no value and will use 0 on page load which results in a NaN result.
If you want to replace that value with something else you can easily do so by adding this to the end of your formula in the results field:
|| 0
Example:(field1 / field2) || 0