When connecting file from S3 There are missing columns

Question & Answer
,
1

Hi,

I have a csv file in S3 with 98 columns.
When I am uploading this file through the upload file option to Quicksight, all columns are available.
However, when connecting the same file through S3 URI, only 10 columns are available.

What can be the issue?

2

Hi @Annab - Welcome to AWS QuickSight community and thanks for posting the question. This is strange, is it possible to share the manifest file details so that we can have a look on this.

Tagging some experts for their advise as well, @Max @David_Wong @Kellie_Burton

Regards - Sanjeeb

3

@Sanjeeb2022 Thank you for the quick response.
This is the content of the manifest file (I changed the bucket name to XXXX):
{
“fileLocations”: [
{
“URIs”: [
“us3://XXXXX/main_dataset.csv”,
“us3://XXXXX/working_hours.csv”
]
}
],
“globalUploadSettings”: {
“format”: “CSV”,
“delimiter”: “,”,
“textqualifier”: “"”,
“containsHeader”: “true”
}
}

4

Hi @Annab - Why there are 2 files in the URIs? Are these 2 files have same layout. Can you please test with one file.

Regards - Sanjeeb

1 Like
5

Hello,

Are you sure the files you are loading have the same number of columns, from the documentation:

Files that you select for import must be delimited text (for example, .csv or .tsv), log (.clf), or extended log (.elf) format, or JSON (.json). All files identified in one manifest file must use the same file format. Plus, they must have the same number and type of columns.

Kind regards,
Andres.

6

Hi @Sanjeeb2022

There are few files in the bucket, that’s why I added all of them to the manifest file.

I tried to follow your advise and leave only one URI in the manifest file, but I am getting an error that the manifest file cannot be parsed.
I cannot find anything wrong with it.

{
“fileLocations”: [
{
[“URIs”: “us3://XXXXX/main_dataset.csv”]

}],
"globalUploadSettings": {
    "format": "CSV",
    "delimiter": ",",
    "textqualifier": "\"",
    "containsHeader": "true"
}

}

7

Hi @andres007 ,

I am sure that the files are the same.
I have downloaded the file from the S3 and uploaded it to Quicksight.

8

Hi,

I mean if they are have the same number of columns and the same type of fields of all the files in the same manifest file.

To use a manifest file, you can load multiple files but they need to be have the same number of columns and format.

About the manifest file, please try an online json fomatter, normally the double quotes you put in your example are “open” and “close”, please use all the same double quotes like in this example.

{
    "fileLocations": [
        {
            "URIs": [
                "https://xxxxxx.s3.us-west-2.amazonaws.com/andres_radar_chart_datatest.csv"
            ]
        }
    ],
    "globalUploadSettings": {
        "format": "CSV",
        "delimiter": ","
    }
}
1 Like
9

Hi @andres007 ,

It worked! Thank you :slight_smile:

The two files indeed do not have the same structure.

1 Like