Rainfall data - cumulative counter - by date and time

Hi All.

I have a very simple 2 column dataset: 1) date and 2) count. The count column is rainfall data and is cumulative - that is, each millimetre of rain counted yesterday, remains part of today’s and future rain fall data.

I would like to display the data in tables, rain fall for the last 6 hrs, rain fall yesterday, rainfall in last 7 days, rain fall total for the last months, and so on.

I would like some assistance please. I have tried many Functions, including the various Period over Period functions, Difference, Lag, etc, etc. I see that I am likely to need a syntax combination or a set of Calculated Fields perhaps for each data/time based calculation.

I understand the general concept to be: calculate each row value minus the previous row value.

As suggested by the forum, the link below is to the Arena sandbox, and displays the 2 column simple data.

Assistance is greatly appreciated.

Updated - date issue.
data6

Hello @Rodger, welcome to the Quicksight community!

I tried a few things against your arena before I had to jump to another task. A few things I came up with in that time are using parameters and Lag to get the appropriate counts. Could you try troubleshooting the following calculated fields against your original dataset?

Previous 6 hours:

sum(Count)
-
lag(sum(Count), [Date ASC],1)

Previous Month:

sum(ifelse(
Date = ${EndOfLast2Months},
Count,
NULL
))
-
sum(ifelse(
Date = ${EndPreviousMonth},
Count,
NULL
))

For Previous Month, same would happen for previous week, you could create parameters to pass the date value of the end of the previous month, May, and the last value of 2 months previous, April, and just use minus to get the correct cumulative value. I would also suggest setting the granularity of your date parameters to HOUR and using relative dates rather than hard coding. Let me know if you run into any errors or the values are wrong.

Hi @duncan,

Thanks for your help. I have updated the Arena, unfortunately I can only share dashboards and not the analysis page.

Thank you. I was able to get your Previous 6 hours: Lag to work. Its results were the same for a Calculated Field i created using Diff by Date. I see it effectively generates the cell value = cell minus previous cell. i understand this is required to deal with the original data set, being a cumulative counter.

difference(min(Count), [Date ASC], -1)

However, I was not able to get your Previous Month: (or week) to work. As it errored out. I am definitely open to additional help to improve your calculation formula.

My issue is within your ifelse - i think is it errors on the expression “${EndOfLast2Months}”. I am websearching to understand this, but failing at the moment.

The updated Arena displays my clunky calculation to deliver daily, weekly, and it could be used for monthly also. However, in this format the filters will not work, and each period (day, week, month, etc) will be a stand only visual, which is less than great. I used the simple:

periodOverPeriodDifference(max(Count),Date, DAY, 1)

data7

Thanks

Hi folks and @duncan,

I have resolved my previous response mis-understanding and have created 2 x Parameters {EndOfLast2Months} {EndPreviousMonth} - actually but for weeks.

I have only been able to use 1 parameter at a time within a data Filter - relative dates. I have not been able to use the Parameters against the calculated field " Previous Month: you provided.

All table tweaks using the Previous Month: calculated field resolve empty. The calculated field syntax is basically the same as yours but with weeks, as reflected in the names. I swapped out sum and tried average.

Any other guidance would be great.
See updated Arena -
8

sum(ifelse(
Date = ${EndOfLast2Weeks},
Count,
NULL
))
-
sum(ifelse(
Date = ${EndPreviousWeek},
Count,
NULL
))
1 Like

Hi @Rodger, it seems you’re encountering challenges with using parameters within calculated fields. I’ll try to provide some highlevel guidance to help you get started, the syntax is pseudocode, please check the documentation and use the Amazon Q to help derive the correct syntax.

Steps to Achieve the Desired Output:

  1. Create Parameters:
  • Ensure you have created two parameters: EndOfLast2Weeks and EndPreviousWeek.
  1. Create Calculated Fields:
  • You need to create two separate calculated fields that will use these parameters. For weeks, your logic will be similar but adjusted for weekly granularity.

Calculated Field for Last 2 Weeks:

Pseudocode, verify syntax.

ifelse(
    {Date} >= dateAdd('WK', -2, truncDate('WK', now())) AND
    {Date} <= dateAdd('WK', -1, truncDate('WK', now())),
    {Count},
    null
)

Calculated Field for Previous Week:

Pseudocode, verify syntax.

ifelse(
    {Date} >= dateAdd('WK', -1, truncDate('WK', now())) AND
    {Date} < truncDate('WK', now()),
    {Count},
    null
)
  1. Create Aggregated Calculated Fields:
  • Now, create aggregated calculated fields using sum or average as needed.

Aggregated Field for Last 2 Weeks:

Pseudocode, verify syntax.

sum(
    ifelse(
        {Date} >= dateAdd('WK', -2, truncDate('WK', now())) AND
        {Date} <= dateAdd('WK', -1, truncDate('WK', now())),
        {Count},
        null
    )
)

Aggregated Field for Previous Week:

Pseudocode, verify syntax.

sum(
    ifelse(
        {Date} >= dateAdd('WK', -1, truncDate('WK', now())) AND
        {Date} < truncDate('WK', now()),
        {Count},
        null
    )
)
  1. Difference Calculation:
  • Create a final calculated field to calculate the difference between these two aggregated fields.

Difference Field:

Pseudocode, verify syntax.

sum(
    ifelse(
        {Date} >= dateAdd('WK', -2, truncDate('WK', now())) AND
        {Date} <= dateAdd('WK', -1, truncDate('WK', now())),
        {Count},
        null
    )
)
-
sum(
    ifelse(
        {Date} >= dateAdd('WK', -1, truncDate('WK', now())) AND
        {Date} < truncDate('WK', now()),
        {Count},
        null
    )
)
  1. Use Parameters in Filters:
  • Ensure your dataset is correctly filtered using the created parameters. These parameters should dynamically adjust your date ranges.

Verification and Troubleshooting:

  • Verify Parameter Values: Ensure that the parameter values are set correctly and match the expected date ranges.
  • Check Date Field Formatting: Ensure your date fields in the dataset are properly formatted as dates.
  • Debug Step-by-Step: Test each calculated field individually to ensure they return the expected values before combining them.
  • Data Availability: Ensure that your data for the specified weeks actually exists in the dataset. Sometimes, empty results occur due to no data matching the date range.

If these steps do not resolve your issue, try using custom SQL or pre-processing at the data source.

Did this solution work for you? I am marking this reply as, “Solution,” but let us know if this is not resolved. Thanks for posting your questions on the QuickSight Community!

Hi Xclipse.

Thanks for the help. I created a new Arena using actual government rain data (many years to current) to test the situation. I still have issues.

The key problem, is the cumulative data is not displaying/aggregating each days daily rainfall to display totals: daily, week, monthly, year, previous in comparison of etc. Due to the cumulative dataset, the values returned are compounding.

Your Pseudocode threw an error on ‘dateAdd’ - as not supported in Quicksight. I took this as being addDateTime. Was this correct?

To properly form the addDateTime i had to swap the amount , period as outlined in the documentation, and following your Calculated Field for Last 2 Weeks, it looks like this:

ifelse(
Date >= addDateTime(-2, ‘WK’, truncDate(‘WK’, now())) AND
Date <= addDateTime(-1, ‘WK’, truncDate(‘WK’, now())),
counter1,
null
)

I created a set of Parameters based on Day, Week, Month, Quarter, Year.
I created a set of Calculated Fields based on Day, Week, Month. But Year errored.
The robot was not able to assist - Amazon Q.
I tried both sum and average in the Calculated Field as suggested.

The Parameters were used to filter the Date field - data and time range.

Help would be great.

Xclipse - DD WK MM

Hi @Rodger, besides using custom SQL – I’m out of ideas. Check out some of these calculations: Period Over Period and Other Date Comparison Calculations (YoY, MoM, YTD, Rolling 90 days, etc) and table-calculation-functions documentation.