QuickSight Dataset limit issue solutions

Question & Answer
, , , ,
1

A few datasets used in our project are failing to refresh errors of DATA_SET_SIZE_LIMIT_EXCEEDED and ROW_SIZE_LIMIT_EXCEEDED.
The immediate solution is to apply filters to the dataset to reduce the size, and we did so, but our requirement is to have at least year-to-date data.

We have applied filters for including around 10 months of data. Are there any other ways to ensure that we can get this without resorting to filters? There is no option to increase any quota size for dataset size and row count limit.

More information: Our datasets are powered by views set up in AWS Athena using certain tables based on use case queries, in which data is ingested based on monthly, weekly and quarterly cadence.

2

Hi @Shriniket_Wagh,

Since you are running into hard limits for row and dataset size, you can consider options asides filtering by:

  • Aggregate data at source: You can pre-aggregate data at the athena level since your datasets is usng AWS Athena views. Create views that summarize data by day, week, or month depending on your requirement.

  • Incremental Refresh Optimization: You can optimize your Athena views to only include the most granular data for recent periods and progressively aggregate older data.

  • SPICE Capacity Management: Monitor and manage your SPICE capacity usage to identify which datasets consume the most capacity and optimize accordingly.

Kind regards,
Demola