Hi, we own an enterprise QS account used by ~18K users with ~50K (Analysis & Dashboards), 8K DataSets and ~450 DataSources (Redshift, RDS and adhoc[excel,csv, etc]). I am trying to make API calls using boto3 SDK to fetch QS metadata around (list_data_sets,list_data_sources,list_analyses,list_dashboards) and their permissions and other details. I have referred to blog and created script to get details as per my use-case.
Lately, I am seeing three major issues as mentioned below and seek support from experts to unblock myself.
-
list_data_sources, list_analyses are taking very long (1+ hours) to execute and sometime running into timeouts due to token expiry. While I understand I could increase session time, but this will put me in the eyes of security and governance teams. So, is there something, i could do to speed up the process and complete list actions in short timeframe.
-
Post list activity, I execute describe API calls to fetch asset details and ownership. When executing describe and describe_permissions on multiple assets, I noticed details around some assets are returning blank in output file. However, when I execute describe and describe_permissions individual on assets which had blank data, I am able to fetch asset details. So, is this something to do with API throttling? I have even applied ratelimiter to control API calls, but still see this happening. What causes this to happen and why?
-
While I understand QS API have thresholds and these numbers vary for each API. Is there a place where I can find the thresholds of each API and if needed can these numbers be adjusted?
FYI. The account type we use is enterprise and is in us-east-1