Creating quicksight dataset using a s3 file

Question & Answer
,
1

Hi All,
I want to create DataSet using S3 file via CreateDatSet API. But the challenge is , there are lot many S3 files(999+) inside a folder and I have created the manifest file at this folder level using folder prefix manually, since it’s not possible to add each s3 file URI to the manifest . In this case , do we have to create a separate manifest for each file or there is some other workaround to achieve this?

Thanks

1 Like
2

Hi @Shreya – You can just give the prefix. All files in a specified bucket or folder are imported. Amazon QuickSight recursively retrieves files from child folders. Refer Supported formats for Amazon S3 manifest files - Amazon QuickSight

The following example identifies two buckets that contain .clf files for import. One is in the same AWS Region as the Amazon QuickSight account, and one in a different AWS Region. The delimiter, textqualifier, and containsHeader fields are skipped because they are not applicable to log files.

{
“fileLocations”: [
{
“URIPrefixes”: [
“https://awsexamplebucket.your-s3-url.com”,
“s3://awsexamplebucket2/”
]
}
],
“globalUploadSettings”: {
“format”: “CLF”
}
}

1 Like
3

Thanks for your response.

But what if I want to generate the dataset using only one of the files inside the folder at which manifest is built. Is there any parameter in the dataset request where we can directly pass the s3 file location suing which we can create a dataset

4

Hi @Shreya - Yes, you can use the “URIs” section of the manifest file for choosing the specific file within a folder. The following example shows a manifest file that identifies two specific .csv files for import. These files use double quotes for text qualifiers.

{
“fileLocations”: [
{
“URIs”: [
“https://yourBucket.s3.amazonaws.com/data-file.csv”,
“https://yourBucket.s3.amazonaws.com/data-file-2.csv”
]
}
],
“globalUploadSettings”: {
“textqualifier”: “"”
}
}

The following example shows a manifest file that identifies one specific .tsv file for import. This file also includes a bucket in another AWS Region that contains additional .tsv files for import.

{
“fileLocations”: [
{
“URIs”: [
“https://s3.amazonaws.com/awsexamplebucket/data.tsv”
]
},
{
“URIPrefixes”: [
“https://s3-us-east-1.amazonaws.com/awsexamplebucket/”
]
}
],
“globalUploadSettings”: {
“format”: “TSV”,
“delimiter”: “\t”
}
}

2 Likes