Cross join bigger than 1GB - SPICE ingestion error

Q&A
,
1

Hello,

I started getting this error on one of my datasets refresh in PSICE.

Untitled
Untitled580×155 15.9 KB

My secondary table has been bigger than 1 GB from the start and I also found this article saying that it was increased to 20 GB.

Amazon Quick Sight launches a 20x higher limit for SPICE JOIN - AWS

Is this a Quick Sight error that will be fixed? or do I need to work around and not use a secondary table for my data?

2

Hi @kgdata,

Thank you for posting.

There are few recommendations I will propose;

  • Reduce the table size of your secondary table by filtering or aggregating data before ingestion into SPICE

  • Explore direct query mode if you can’t reduce your table size

    If the two options can’t resolve it, please contact AWS support

Regards,
Demola

3

Thank you for the quick response and suggestions.

After some trial and error, I narrowed down the issue. I had to create separate SPICE instances and then import and link them. Previously, the dataset had a mix of direct queries, SPICE datasets, and files directly loaded.

After creating separate SPICE datasets for each file, the dataset started refreshing again.

However, I still don’t understand why it stopped working in the first place. As mentioned earlier, it worked fine up to that point, and the data has always been over 1GB.

Adding a visual in case anyone else comes across this problem.

solution
solution1523×328 123 KB

4

@kgdata ,

It’s the combined size of all those cross joined data sources . in your scenario, the small file is part of the secondary tables. what about the remaining 2 datasets ?

Kind regards,
Koushik

5
  • top left is the main table - 29.4GB (Quick Sight specifies this is the main table in the original error)
  • top right - 10.5GB
  • middle right - 6.4 MB
  • bottom right - 9.3KB