Calculation: Correct input, wrong output. why?

Question & Answer
, , , ,
1

image

My calculated field (test) is this formula:

 (sum({PLAN_RSRC_USAGE},[{BATCH_ID}])/sum({PLAN_RSRC_QTY_ROUTING_UOM},[{BATCH_ID}]))* sum({ACTUAL_QTY_IN_ROUTING_UOM},[{BATCH_ID}])

from the image above, the formula is (A/B)*C. (55.14/17920)*17920 If you calculate it with any calculator the result should be 55.14. but here in QuickSight it is coming to 53.76 I could not figure out why!

If I substitute the numbers directly (55.14/17920)*17920 I get the correct result (test2 column)

HOWEVER,

I tried a little experiment, I changed the formula in test2 to the following:

55.14/17920

and the result was this

0.0031000000

but If you enter the same numbers in any calculator you will get this:

0.0030770089285714

what is going on here? how can I get the correct result?

2

It has to do with rounding.

1 Like
3

Thanks, this solved the issue. using round() did not. I hope this gets resolved by Amazon rather quickly.

1 Like