S3 Connector loading 10% of Rows into Quicksight

Question & Answer
,
1

Hello! I am using the s3 connector to get my data from Gainsight (a customer success platform) into a viz tool. Quicksight has an S3 connector which I am using. However, Quicksight only holds 1400/11000 records. Is there a reason this is the case?

2

Hello @bmike4

When Amazon QuickSight only loads a portion of the records from an S3 connector, there can be several potential reasons for this. Here are some common issues and solutions to investigate:

  1. Sampling Limitations

Amazon QuickSight sometimes samples data to improve performance, especially during the initial data import process. Ensure that you have configured the import to load the full dataset.

  • Solution: When you create or edit your data set in QuickSight, check the data import options and make sure the sampling is set to load all records.
  1. File Format and Structure

The structure or format of your files in S3 might cause QuickSight to misinterpret the data, leading to incomplete loading.

  • Solution: Ensure that your data files are in a format supported by QuickSight (e.g., CSV, JSON, Parquet). Verify that the files are well-structured and free of errors.
  1. Permissions and Quotas

Insufficient permissions or hitting service quotas can restrict the number of records QuickSight can access.

  • Solution: Check your AWS IAM permissions to ensure QuickSight has full access to the S3 bucket. Also, review your AWS service quotas to make sure you haven’t exceeded any limits.
  1. Data Preparation Settings

QuickSight’s data preparation settings might inadvertently limit the number of records being imported.

  • Solution: Review your data preparation settings in QuickSight. Ensure there are no filters or transformations that limit the number of records being processed.
  1. Connectivity Issues

Network or connectivity issues between QuickSight and S3 might cause incomplete data loading.

  • Solution: Verify that there are no network issues affecting the connection between QuickSight and your S3 bucket. You might want to try reloading the data set.
  1. Data Set Refresh Configuration

QuickSight might be using a cached version of the data set, not reflecting the full number of records available in S3.

  • Solution: Refresh the data set in QuickSight to ensure it’s loading the latest data from S3.

Steps to Diagnose and Resolve:

  1. Check Data Import Settings:
  • Go to the data set in QuickSight.
  • Edit the data set and ensure the import settings are configured to load all data without sampling.
  1. Verify File Format and Structure:
  • Open the files in your S3 bucket and verify they are correctly formatted.
  • Ensure there are no structural issues that might cause QuickSight to stop reading records.
  1. Check Permissions and Quotas:
  • In the AWS Management Console, verify IAM permissions for QuickSight.
  • Check service quotas for QuickSight and S3.
  1. Review Data Preparation Settings:
  • In QuickSight, review any filters or transformations applied during data preparation.
  • Ensure there are no conditions that might limit the number of records.
  1. Refresh Data Set:
  • In QuickSight, refresh the data set to reload the data from S3.
  1. Monitor Network Connectivity:
  • Ensure stable network connectivity between QuickSight and S3.
  • Check for any network issues or disruptions.

By following these steps, you should be able to identify and resolve the issue preventing QuickSight from loading all records from your S3 connector.

1 Like
3

Hi @bmike4

It’s been a while since we last heard from you, did you have any additional or follow up questions regarding your initial topic? Were you able to try out the approach that was shared?

If we do not hear back within the next 3 business days, I’ll go ahead and mark the solution to close out the topic.

Thank you!

4

Hi @bmike4

Since we have not heard back from you with any further information or follow-up questions, I will archive this topic. If you still need assistance, please post a new question in the community and link to this topic to provide relevant information. That will ensure you are at the top of the priority list for a response from one of our QuickSight experts.

Thank you!