I have an S3 dataset, generated by AWS Firehose, with data aggregated in 15 minute chunks. This doesn’t take long to exceed the 1,000 file limit imposed by the JSON manifest format for SPICE imports.
My request has two parts: 1) If there are more than 1,000 items, could SPICE at least import the 1,000 most recent items (based on the AWS Firehose sharding format, organized by year / month / hour). 2) Have an option to suppress error emails for this specific error.
With this feature in place, I would at least be able to view my most recent data in QuickSight. Currently, nothing has been refreshed since May, because that’s when I hit the 1,000 item limit.
The best workaround would be to use Amazon Athena , as this would allow you to have as many files in your S3 bucket as you may need and thereafter you will still be able to upload your data to QuickSight via Athena and apply the necessary filter conditions to restrict the data .
Please check another reference blog for the similar use-case .
Did this solution work for you? I am marking this as, “Solution,” but let us know if this is not resolved. Thanks for posting your questions on the QuickSight Community Q&A Forum!