Duplicate rows when loading data in S3

Q&A
,
1

Hi All,
I have a csv data file in my s3 bucket. I have connected it as a quicksight dataset source using manifest file. The problem I am facing is that it is creating double the number of rows that I have in my S3 file every time I refresh it. This is affecting my sum and other counts.
Any help? Do I need changes in my JSON file below?

{
“fileLocations”: [
{
“URIs”: [
“s3://bucket/Controllership_SLA_output.csv”
]
},
{
“URIPrefixes”: [
“s3://bucket/”
]
}
],
“globalUploadSettings”: {
“format”: “CSV”,
“delimiter”: “,”,
“containsHeader”: “true”
}
}

Data from S3 source having a duplicate row with only difference in date format of column
2

@sakshisl ,

Based on the above you just have one file which is being used to load to SPICE and used in your dashboard.
Check the number of rows after the refresh the completes. Post a screenshot of what you mean with double number of rows ?

Kind regards,
Koushik

3

image
image1565×211 18.8 KB

I only have 4 rows in my S3 file but it doubles in row count when I load it.

4

@sakshisl ,
in the Quick Sight analysis ? Do you see the duplicated rows as well ?

1 Like
5

Hello @sakshisl and @Koushik_Muthanna!

@sakshisl were you able to find a solution for this issue, or if you still need help could you follow up on @Koushik_Muthanna 's questions above?

7

Hi, no it is not yet solved. I am seeing duplicates in my analysis even when I do count.
I have to select distinct in these cases.
My S3 file has the appropriate records but when I connect to Quicksight it is duplicating each record and doubling the output.

9

Hello @sakshisl !

Have you tried the suggested solutions in this post?

10

This is not useful for me

11

@sakshisl ,

Can you
1/post a screenshot of your s3 bucket ? Is there only 1 file with 4 records and you see 8 records when it is finally ingested into SPICE ?
2/post a screenshot of the data prep ( we should be seeing only 4 records )

I would like to validate the above 2 before asking you to log a support ticket for further assistance.

Kind regards,
Koushik

12

Screenshot (103)
Screenshot (103)1594×229 30.3 KB

Input/ingested file

Screenshot (104)
Screenshot (104)1920×234 20.9 KB

When loaded in Quicksight the count goes to more than double.

13

@sakshisl ,

please remove the URIPrefixes from the manifest file and test the ingestion process. Let us know if that solves the issue , if not then would recommend to open a support ticket.

Kind regards,
Koushik

2 Likes
14

Hi Koushik,
That worked for me. Thanks alot.

1 Like