With every upload, it is only uploading 75% of whatever the amount. For example when we uploaded a file with 45m rows, it only uploaded 39, when it was an upload of 31m, it uploaded only 23. But QS keeps saying upload is 100%, even though it is not the case
Hello @besmithc, welcome to the QuickSight community! When you say with every upload, are these uploads querying a database to update a current dataset within QuickSight in SPICE or is this something you are seeing when loading in data from a dataset on Direct Query?
Also is it directly querying a view or table in your database or is custom SQL creating the query? It is possible rows are being filtered or excluded based on the SQL that has been written or the update is an incremental refresh within spice that is only pulling the latest changes and adding them to the current dataset.
hey @DylanM taking over from Ben as this is my issue but he posted it for me.
the answer to your questions here-
We are performing a manual upload of 4 datasets. During the upload, no issues or errors are shown.
We then link the datasets based on foreign keys.
The structure is as follows
Dataset A (survey_participant) has a many2many relation to Dataset B (survey_compund_answers)
Dataset B (survey_compound_answers) has a one2many relation to Dataset C (survey_compound_questions)
Dataset A,B, and C are present in the XLS file as different tabs (survey_participant, survey_compound_answers, survey_compuound_questions)
We facilitate the many2many relation by adding a foreign key Dataset (which we call response), which is uploaded in CSV format
We’re linking all rows of Dataset A to all rows in Dataset B, so the foreign key table is quite large (10K in survey_participant and 3.6K in survey_compound_answers ==> ±36M rows)
In the XLS file, you can ignore the tab “survey_questions” and “survey_answers”
We don’t execute any SQL or add any filters on the data
also sharing the source-
Hello @Rainasensi, the best thought I have at the moment is that the joins that are being completed within QuickSight wouldn’t necessary join every single row within the individual datasets. If the value you are joining on does not match the parent table on the joining value, then it would probably be skipped. I could be wrong but is it possible anything is repeated between 2 tables so it wouldn’t join or excluded based on an incorrect matching value?
Hello @Rainasensi, are you still having issues with this data source in QuickSight? If you resolved it feel free to post your solution or send over a follow-up question. Otherwise I can archive this question and you can post a new topic if another problem appears.
Hi Dylan Raina is currently on leave so I am picking this up, thank you for chasing! I will have an answer by the end of the week!
1 Like
Hello @besmithc, thank you for the follow-up! I’ll make sure to check in towards the end of the week.
Hello @besmithc, I am going to go ahead and archive this ticket. If you still need assistance with this problem or something else, post a new question in the community so you will be at top priority for a response from one of our experts. Good luck!