QuickSight Data Ingestion Reduces After 1M Rows (Failure)

Hi,

I’m exporting legacy CUR data daily to S3, and it appends properly. In QuickSight, I refresh the data source every day .

Every month, after the row count reaches ~1M (around the 19th of every month), it unexpectedly drops to half (e.g., 900k → 450k).

The S3 data remains intact, and refresh settings are correct. What could be causing this?

Thanks,
Satyaraj

Hi @Satyaraj,
What type of refresh are you using; are you using an incremental refresh by chance and it’s only bringing in part of the data?

1 Like

Hi ,

Thanks for your response! I’m not using an incremental refresh; it’s a full refresh scheduled daily. The data in the S3 file is overwritten every day, and I ensure the file is complete before the refresh. However, once the row count reaches around 1M (typically around the 19th of the month), QuickSight consistently reduces the row count by half during the refresh.

The S3 data remains intact, and the refresh settings are configured for a full import. Could this behavior be related to any limitations or issues with QuickSight’s data ingestion process?

Thanks,
Satyaraj

Hello @Satyaraj,

The limits of SPICE are 1 billion (1,000,000,000) rows or 1 TB for each dataset

so I do not think the number of rows is the problem here, unless you have columns with a lot of data where you might be hitting the 1TB before the 1B rows, but as you mention this is related to CUR, I do not think this will be the problem.

Can you confirm that when you query the data in Athena you get the 900k rows instead of the 450k you see in QuickSight?

Please let us know.

Hi ,

Not using Athena here. ingesting data directly from s3

Please find Attached Screenshot

Thanks,
Satyaraj Jajala

Can you create a table in Athena pointing to the S3 bucket to check if the number of rows match?

Yes , Created table and checked rows in Athena there are 1703557 rows

But in Quicksight it is showing 813315 rows