Hi, we are trying to use boto3 to fetch/create/refresh quicksight metadata/data from glue job. Following are the boto3 calls made from glue job…
describe_data_source
create_data_set
create_ingestion
describe_ingestion
The issue is that any of these boto3 calls from glue job timeout. However, the boto3 calls to S3, STS and SSM are successful (we have endpoints created for these services). I have considered using VPC Endpoints for quicksight. However, I do not see it as an available service while creating an endpoint. Furthermore, per AWS documentation, the Quicksight VPC connection nor Quicksight VPC Endpoint would allow to make api calls to quicksight. Hence, I was wondering what would it take to have these boto3 calls to quicksight from glue job successful. Any help/direction on this issue is highly appreciated. Thank you.
Info: The glue job is configured to use a data connection and assigned an IAM role that has full access to Quicksight, EC2.
Check the endpoint (VPC->Endpoints) for Glue service (Example : service name: “com.amazonaws.eu-west-1.glue”), this one was assigned to the same Subnet and Security Group as the Glue Connection which was used in the Glue Python Shell Job.
Ensure that your Glue job’s network configuration is set to use the VPC where the QuickSight interface endpoint is created.
Verify that the IAM role associated with the Glue job has the necessary permissions to interact with QuickSight.
Once the interface VPC endpoint is set up, retry the boto3 calls from your Glue job. The calls should now go through the VPC endpoint without timing out.
Please refer the below documentation this might be helpful for you.
If the problem still persists, I would recommend filing a case with AWS Support where we can dive into the details so that we can help you further. Here are the steps to open a support case. If your company has someone who manages your AWS account, you might not have direct access to AWS Support and will need to raise an internal ticket to your IT team or whomever manages your AWS account. They should be able to open an AWS Support case on your behalf.
@Xclipse Thank you for the clarifications. The Glue endpoint, security group and the IAM role are all in-line with your suggestions. However, since we are on gov cloud, we are unable to create Quicksight endpoint. Have created a case with AWS Support. Will respond back with a working solution once thru (for benefit of others who might reach this thread with a similar issue).