How to create dataset from data source of S3 using CLI

Question & Answer
1

How to create the dataset using the CLI command from the data source of s3,
I have made the Datasource as S3 and for that, I have create the manifest filed and I have uploaded and I got the arn.
when run this command
aws quicksight list-data-sources --aws-account-id xxxxx
I got this arn
arn:aws:quicksight:ap-northeast-1:xxxxxxxxx:datasource/xxxxxxxxx

My goal is I need to create the dataset using the CLI , data source as s3.
please help me with that, How I can create it.

1 Like
2

Hi @suresh1 ,

To create a dataset you can refer to the following API call : create-data-set — AWS CLI 2.9.10 Command Reference
I would suggest the following :

Create the dataset initially through the UI . To get an understanding of the structure ,you can then run describe-data-set .

aws quicksight describe-data-set --data-set-id XXXXX --aws-account-id XXXXX

To create then through the cli : extract the cli skeleton which is required for the final structure ( creates a file create_dataset.json in my downloads folder )

aws quicksight create-data-set --generate-cli-skeleton ./Downloads/create_dataset.json

Create the structure by updating the required parameters ( example of my create_dataset.json , I have deleted the other parameters which are not required. )

{
    "AwsAccountId": "XXXXX",
    "DataSetId": "k_anonymity_dataset",
    "Name": "k_anonymity_dataset",
    "PhysicalTableMap": {
        "s3physicaltable": {
            "S3Source": {
                "DataSourceArn": "arn:aws:quicksight:eu-central-1:XXXXX:datasource/k_anonymityds",
                "UploadSettings": {
                    "Format": "CSV",
                    "StartFromRow": 1,
                    "ContainsHeader": true,
                    "TextQualifier": "DOUBLE_QUOTE",
                    "Delimiter": ","
                },
                "InputColumns": [{
                        "Name": "customer",
                        "Type": "STRING"
                    },
                    {
                        "Name": "cars",
                        "Type": "STRING"
                    },
                    {
                        "Name": "location",
                        "Type": "STRING"
                    }
                ]
            }
        }
    },
    "LogicalTableMap": {
        "s3physicaltable": {
            "Alias": "k_anonymity_dataset",
            "DataTransforms": [{
                "CastColumnTypeOperation": {
                    "ColumnName": "customer",
                    "NewColumnType": "INTEGER"
                },
                "CastColumnTypeOperation": {
                    "ColumnName": "cars",
                    "NewColumnType": "INTEGER"
                }
            }],
            "Source": {
                "PhysicalTableId": "s3physicaltable"
            }
        }
    },
    "ImportMode": "SPICE",
    "Permissions": [{
        "Principal": "arn:aws:quicksight:eu-central-1:XXXXX:user/default/usera",
        "Actions": [
            "quicksight:PassDataSet",
            "quicksight:DescribeIngestion",
            "quicksight:CreateIngestion",
            "quicksight:UpdateDataSet",
            "quicksight:DeleteDataSet",
            "quicksight:DescribeDataSet",
            "quicksight:CancelIngestion",
            "quicksight:DescribeDataSetPermissions",
            "quicksight:ListIngestions",
            "quicksight:UpdateDataSetPermissions"
        ]
    }],
    "DataSetUsageConfiguration": {
        "DisableUseAsDirectQuerySource": true,
        "DisableUseAsImportedSource": true
    }
}

My example file

customer cars location
1 1 Brussels
2 1 Brussels
3 1 Brussels
4 1 Brussels
5 1 Brussels
6 1 Brussels
7 1 Brussels
8 1 Brussels

Kind Regards,
Koushik