Yes, when will this be released? We are currently running into this issue and highly frustrating in terms of breaking our automation and workflows.
Hi All, we are going to soon release the new feature to increase the SPICE JOIN limit for secondary table size. I would like to ask for your feedback here, what size do you need to unblock your use case? 5GB, 10GB, 20GB?
Please be noted for two things:
1/ the new feature is only applicable to SPICE dataset join. If you have data from different data source that you want to join, the limit is still 1GB. You can ingestion those data tables into SPICE to get unblocked by the new size limit.
2/ the size limit is only applied to all the secondary tables adding together, the primary (left) table doesn’t have size limit. Current limit is 1GB.
Thanks!
2 Likes
Hi @m0ltar, previously we are treating all SPICE datasets same behavior as cross-source datasets. When users request to do a join with SPICE datasets, we basically have to load the secondary data into memory and run the join. This approach is similar to how we deal with cross-source data. That’s why we have the 1GB limit.
We are going to soon launch SPICE JOIN feature. With the new capability, we are going to move data in to the same node, and run the JOIN all in SPICE infrastructure. Meaning we are now regarding SPICE datasets single source. That’s why now we will support much higher limit beyond 1GB.
Can you share your use case? If we increase the limit from 1GB, what’s the new limit can unblock your use case? Thanks!
1 Like
How much do you need @Viacheslav ?
@Ashokquicksight We are going to increase the limit for all secondary table size in a SPICE JOIN. What is the new size limit (current limit is 1GB) is good for you?
We no longer have the use case, as we had to refactor everything and move the JOINs back to the database. Otherwise, I am not sure. How can I measure the JOIN size? I think it depends on the data storage, compression and how QS uses it.
Our workaround was to use custom SQL to merge the second largest dataset with the largest one to decrease the total size of the remaining datasets to less than 1 GB. Normally we would want to re-use that second largest dataset in other child datasets, but with this approach, we lose that re-usability.
If the total size of the remaining datasets outside of the largest one can be up to 20 GB, that would work for our use case.
Thanks @David_Wong and @m0ltar for the feedback! We are actually in the phase of preview for the SPICE JOIN capability with limited increases to 20GB. Do you want to try the preview feature?
@Viacheslav, @evanbetterc and @Ashokquicksight - also for your information.
Feel free to message me offline about your account ID and region, we can enable it for you.
Thanks!
When do we plan to launch this feature for everyone ? And where can we view the current configured join limits ?
Hi @VAMSI1, we will launch the feature in a few months, if you wish to preview it, please message me offline. Thanks!
while i’m combining different datasources to make a join, due to memory size limitations we are facing issues.
Error: Memory capacity limitation exceeded for data sources in the join configuration.Breakdown of data source memory capacity in bytes by LogicalTableId at the point of failure.
Can you enable me the preview feature which has spice join capability of 20GB.
Account ID- 884515407868
Region - India
Hi emilyzhu, I also want to try the preview. How can I contact you offline?
Best regards,
Mattias
Hi Emily, I’m also not sure how to message offline but would love to be included in a preview for this feature, as we’re suddenly hitting join limits.
Thanks
Hi All, thanks for your reaching out! We are planning to GA the SPICE JOIN feature very soon in the first week of June. Please stand by, when we launch the feature, we will post announcements!
3 Likes
Thanks for the update! Do you know what the limit will be set to? We’re launching in June, so need to know if it’ll be enough or if we need to go down a different route… TIA
hi all, thanks for all the attention for this post! We are fully aware of the importance of relaxing the join limitation to your use cases. I am happy to share that the feature is about to be GA soon to all QS users (end of June). The new limitation will be 20GB (comparing with the current 1GB limit). Noted this will be applied to only SPICE join, meaning you will need to have all the parent tables already in SPICE to enjoy the 20GB new limit for join.
Please pay attention to the launch announcement. And feel free to keep providing feedback in this post let us know how you like the feature and any other enhancement we can work on SPICE!
Thanks!
2 Likes
hi @emilyzhu, is the new SPICE join available for all QS users already? The SPICE join limit of 20GB already added to this doc, so I assume they’re available.
But I’ve also just hit with the join memory error. I’ve tried to update all the Sources datasets to use SPICE cache, but the dataset import still failed with the same memory error. So,
- When you said ‘parent tables’, do you mean ‘Sources’ in the dataset UI (I attached the UI)?
- Is there a specific SPICE JOIN type that I need to set in the dataset UI?
