Need some urgent help with the new data experience page. I am joining two Direct Query tables. I’ve selected and renamed my columns and can preview the data without issues. However, QuickSight will not let me “Save & Publish.”
Is there a known issue or limitation with joining Direct Query tables in the new interface? The preview works, but the final publish fails every time.
Are both of the DQ datasets that you’re trying to join from the same datasource or different?
Are both of the DQ datasets created in the new data prep experience? (I believe this is a current limitation)
In scenarios where I’ve needed to utilize Direct Query, I’ve been continuing to use the legacy experience for the time being while there are still limitations for the new data prep. I’ll include add’l documentation below relating to the limitations still in place.
Yes, both Direct Query (DQ) datasets are from the same data source, and both were created using the new data prep experience.
To provide some background on what I did previously: I first uploaded Dataset A as a Direct Query dataset with Athena as the source. I then uploaded Dataset B, also as a Direct Query dataset, again using Athena as the data source.
Next, I opened Dataset A, added a few calculated fields, and renamed some columns. After that, I clicked Create new dataset, which opened the new data prep experience. From there, I selected Add data, added Dataset A and Dataset B, performed a left join, and applied the transformations shown in the first image I shared earlier. The preview was successful; however, I am unable to save and publish the dataset — that is the issue.
My question is:
I uploaded two separate Direct Query datasets into Quick and then created a new dataset by clicking Add data and joining Dataset A and Dataset B. Is this supported, or is there a limitation? Could the issue be related to performing transformations on Dataset A first and then joining it with Dataset B? Is there any known limitation that would prevent the dataset from being saved and published?
I’d like to provide some background on the error below and would appreciate it if you could let me know whether I’m approaching this correctly or if I may be missing something.
Here is what I did:
I added Dataset 1 into QuickSight.
I added Dataset 2 into QuickSight.
I then created a third dataset by joining Dataset 1 and Dataset 2 (both already existing in QuickSight).
In Dataset 1, I performed some transformations such as:
Converting milliseconds to minutes
Renaming columns
I performed similar transformations in Dataset 2 as well (fyi I am using new data experience page for all datasets).
After that, I created the third dataset by joining these two transformed datasets. I do not see any validation errors during the join, but when I attempt to publish the dataset, it fails with a “cannot publish dataset” message.
My question is:
If I apply transformations separately in Dataset 1 and Dataset 2, publish them, and then create a new dataset that joins those two — is that supported? Or could the issue be caused by performing transformations in the base datasets and then performing additional transformations again in the joined dataset?
For example:
Dataset 1 → Convert Queue Wait Time (ms) to minutes
Dataset 2 → Convert Agent Talk Time (ms) to minutes
Dataset 3 → Join Dataset 1 and 2, then create additional calculated fields
Is this layering of transformations potentially causing the publish error? Any thoughts around this?
Please let me know if I am misunderstanding how QuickSight handles transformed datasets and joins.
Hi @shravya, so sorry for missing your first response!
To me, your approach seems valid since the conversions you’re applying are not relying or affecting too many other fields in the dataset, correct?
And the columns being renamed aren’t being used in the join process to match like fields?
Did Dataset 1 and 2 publish correctly without the errors?
How complex are the calc. fields being built off the joined dataset? Have you tried publishing the joined dataset prior to creating the additional calc. fields? I wonder if that is causing the error.
From experience though, in relation to the new data prep, I’ve encountered some publishing errors as well where it’s had issues with certain types of fields and their formatting, so this process may work better in the legacy version as well.
Just checking back in since we haven’t heard from you in a bit. I wanted to see if the guidance shared earlier helped resolve your question, or if you found a solution in the meantime.
If you still have any additional questions related to your initial post, feel free to share them. Otherwise, any update you’re able to provide within the next 3 business days would be helpful for the community.
Hello @shravya, I think it may be in your best interests to either manage the dataset joins in the old view (there are less bugs and everything you are trying to accomplish should be more straight forward), or you can join the datasets within Athena. Since you are pulling from the same data source, I always recommend managing the joins directly in SQL. You can make a view in Athena and pull that into Quick, or you can build the SQL to run the joins in custom SQL within Quick.
@DylanMorozowski - eah, I understand the approach you’re suggesting. However, my tables in Athena are already cleaned, and the customer requirement was to have all the tables available directly in QuickSight so that users can join any two tables as needed. That said, what you mentioned is absolutely correct, and I personally prefer using SQL as well.
I am okay with using old data prep page always but just wondering when this gets sunset.
@Brett - Sorry i was late in answering but these are my answers fo ryour questions
Yes Datasets saved correctly individually when i renames or perform any other small transformations.
Yes, the columns that got renamed are not used for joining. I have the name as is for both datasets.
Calculated fields are not too complex they are just time conversions (milliseconds to hh:mm:ss)
Yes i tried publishing joined datasets before making any transformations on any datasets and it was successfull.
@shravya I have been working from the assumption that they won’t sunset the old layout until the new one has all of the proper functionality, but I guess we can prepare a post if that happens too soon! I totally understand you being hesitant though. I believe we still need a way to manage the geospatial fields from the new layout as well!
To concluded this, I was able to join the datasets in new data preparation page and perform transformations on top of that rather than performing transformations on individual datasets.
Also as you said alternative way is to use SQL for better joins.
