Hello,
I’ve made an incremental refresh to some of my datasets. The date field that defines the incremental window is a string field with the parse date function I convert to a date field in the SQL query. Does it affect the query performance and the cost of the query? The chargers are very high.
1 Like
Hello @miap, the chargers associated with a query against your database are related to many factors. Are you converting the string field to a date field within QuickSight or on the database itself? Maybe there is something where it isn’t matching correctly in the database and it is running as a full refresh on the source data anyway.
Other things to consider, resource costs are also impacted by the type of database you are using, the amount of data you are querying, the complexity of the query you are running, as well as the frequency. Are you using a complex SQL statement with joins to many tables? It may be worth also considering reducing the amount of data you are querying with where clauses in your SQL to query fewer rows. There are many options to consider here, so I may need a little more information about how your dataset is built to guide you further. Thank you!
Hi @DylanM,
Yes, I’m converting the date field within QuickSight, and the SQL query is complex with join to multiply tables.
I suspect that the issue is the date parse, is it meaningful?
1 Like
Hello @miap, I don’t imagine the date parse SQL function in QuickSight would be the problem associated with higher costs on the database side. I think the more likely issue would be how you are joining tables, the number of select statements and inner joins being utilized, as well as the quantity of data overall. I’ve run multiple date altering functions within a custom SQL dataset and have not had this same experience.
I don’t imagine there would be an issue utilizing the parse_date to manage your date field and run incremental refreshes off of it. I really think the price issue is going to be related to a different aspect of your query.
The parse date field is the field that I use in the configure incremental refresh, if the date is a string in the query that converts to a date, how does the ‘window size’ know where to start/continue from?
Hello @miap, if you think that is causing the issue, what if you also imported the the date field as a 2nd column without the parse date function and changed the the settings for your incremental refresh to run off that field. Then you could check the logs in your database to see if the cost of the refresh is improved. That is going to be the best way to determine if that is the issue or not.
I would also be interested to see if you ran a full refresh, if the number of rows returned would be different than the incremental refresh. That would also be another way to test the cost difference between running one over the other.
Hello @miap, since we have not heard back from you, I wanted to reach out and see if you found any new information from testing. Were you able to find any further information about what is causing the cost of the query to run high? Please let me know and I can guide you further. If we do not hear back from you in 3 days, I will archive this topic. Thank you!
Hello @miap, since we have not heard back, I will archive this topic. If you still need assistance with this issue, please post a new topic in the community and link to this question. That will ensure you are at the top of the priority list for a response from one of our QuickSight experts. Thank you!