could some one help on this please, this is very weird, how come join limit only 1GB? now we can’t reduce of join, or else if we remove it will breaks whole dashboard, how to increase join size to 10 GB, please
error:
QuickSight ran out of join memory while processing your request. To join between data sources, the smaller source tables combined can’t exceed the join memory limit of (1GB). The largest one, “accounts_users”, doesn’t count toward the memory limit. To alleviate this problem, you can start by reducing the size of the initial source tables in the join
I have tried, even direct query will not work, since we have 54 fields on one table, we do required all the fields so i can’t trim the fields. can’t be possible to increase the join limitation to at least 5GB?
can’t appear direct query, please check screenshot, because it’s (use in a new dataset) from main dataset. BTW, direct queries are very slow for large data,so i usually import into SPICE.
I misread the documentation here: Joining data - Amazon QuickSight. There is just a 1GB data prep limit. I’ll mark this as a feature request. Thank you
Within Athena you can create a table that points to location of S3 bucket (S3 , Athena and QuickSight have to be in the same region ).
Amazon QuickSight can connect to Athena as a data source. You can use either use Custom SQL or joins on your different tables and then import the data into SPICE.
Any update on this please: Memory capacity limitation exceeded for data sources in the join configuration.Breakdown of data source memory capacity in bytes by LogicalTableId at the point of failure.
I have already 3 child datasets, so i can’t create more then 3 as well.
Can the QuickSight team add a feature to the Data Prep tool that shows you the cross-join table size?
It’s not clear whether same datasource tables are included in the cross-join table size calculation or if its only the sum of the size of tables that have a different source than the largest table. This forces the user to spend hours trying to reconfigure the datasets to bring the size back down below the 1GB limit.
Showing how close we are to the cross-join size limit would help reduce risk of exceeding this limit when curating datasets.
This is affecting us too, and very much by surprise. We’ve just added 10% more data, and now all of our datasets are failing, and there is no recourse, fix or documented solution.
We’ve invested months into this design, and now we are constrained by this undocumented and surprising limitation.
On top of that, the docs state that it only affects “cross-source”. Yet we do not have any cross-source datasets. All our sources are actually other SPICE datasets, which all originated from a single source (RDS Aurora).
Same-source datasets have no restrictions on data size.
And we only have LEFT joins, no crazy cross joins or anything like that.
QuickSight considers the parent datasets as their own separate sources even if the underlying data source in each one is the same.
When you view a child dataset, you can see the parent datasets are listed under “Sources”. In my screenshot below all my sources are SPICE datasets that are connected to the same database.
So, to be clear, each child dataset count as a whole separate data source?
I’d imagine QS would count all SPICE to be the same source, given that it most likely using a single storage and query engine. But perhaps makes sense if it is using something like Athena for cache, where each query would result in entirely separate result files, and thereby they are not sharing the query plan and thus count as different sources.
Hi all, we are going to soon launch the capability to JOIN SPICE tables with much higher limit. Can you please message me offline, we can arrange preview if you would like to test. Thanks!
