QS is doing the math wrong, Significant digit issue? Increase precision of calculated fields?

Q&A
, ,
1

OK, I have a calculated field in my dashboard.

There are two fields, I’ll Just call them field1 and Field 2 here:

The formula is sum(field1)/sum(field2)*12

The value of sum(field1): 31,096,666.12 (checked this against external sources and it’s correct)

The Value of sum(field2): 7,645,221,183.47 (checked this against external sources and it’s correct)

The value that my calculated field gives me is: 0.0480

The problem is that this number is wrong! it should be 0.0488 I’ve done the math using a calculator, I plugged it into google, and I did it in excel. I’m positive that 0.0480 is not the correct answer.

Is there a way to increase the precision of the math?
2

Hi @TRube ,

Can you check if the data type of the fields in your dataset is set to decimal-fixed or decimal-float?

1 Like
3

It was set to Decimal Fixed… I had that same thought after I posted.

I’ve changed it in the data set but it’s probably going to process for an hour while it reloads everything :expressionless: So I will have to check in the morning if that fixed it.