How to create a dataset from IoT Analytics using API

Hi.

I need to create a dataset from IoT Analytics using Java API. The steps I’ve done are the following:

  • Created an IoT Analytics datastore, using CreateDatastore api call in analytics client.
  • Created an IoT Analytics dataset, using CreateDataset api call in analytics client.
  • Create a QuickSight datasource, using CreateDataSource api call in quicksight client. This datasource refers to analytics dataset.

The next step would be to create a QuickSight dataset (CreateDataSet api call in quicksight client), but from api documentation I can’t infer how to do it.
I’ve been able to create all this from UI and I’ve tried to describe the dataset from AWS CLI, but it complains about this is not supported for this kind of datasets yet.

Any help would be appreciated.

Thanks in advance.

Hi Juan,

May I get the screenshot of the complete error message you are getting when you ran command in CLI.
Maybe it could be because CreateDataSet API doesn’t support datasets that include uploaded files as a source. You need to create dataset from scratch just like the JSON I am pasting here:

POST /accounts/AwsAccountId

/data-sets HTTP/1.1 Content-type: application/json { β€œColumnGroups”: [ { β€œGeoSpatialColumnGroup”: { β€œColumns”: [ "

string

" ], β€œCountryCode”: "

string

", β€œName”: "

string

" } } ], β€œColumnLevelPermissionRules”: [ { β€œColumnNames”: [ "

string

" ], β€œPrincipals”: [ "

string

" ] } ], β€œDataSetId”: "

string

", β€œDataSetUsageConfiguration”: { β€œDisableUseAsDirectQuerySource”:

boolean

, β€œDisableUseAsImportedSource”:

boolean

}, β€œFieldFolders”: { "

string

" : { β€œcolumns”: [ "

string

" ], β€œdescription”: "

string

" } }, β€œImportMode”: "

string

", β€œLogicalTableMap”: { "

string

" : { β€œAlias”: "

string

", β€œDataTransforms”: [ { β€œCastColumnTypeOperation”: { β€œColumnName”: "

string

", β€œFormat”: "

string

", β€œNewColumnType”: "

string

" }, β€œCreateColumnsOperation”: { β€œColumns”: [ { β€œColumnId”: "

string

", β€œColumnName”: "

string

", β€œExpression”: "

string

" } ] }, β€œFilterOperation”: { β€œConditionExpression”: "

string

" }, β€œProjectOperation”: { β€œProjectedColumns”: [ "

string

" ] }, β€œRenameColumnOperation”: { β€œColumnName”: "

string

", β€œNewColumnName”: "

string

" }, β€œTagColumnOperation”: { β€œColumnName”: "

string

", β€œTags”: [ { β€œColumnDescription”: { β€œText”: "

string

" }, β€œColumnGeographicRole”: "

string

" } ] }, β€œUntagColumnOperation”: { β€œColumnName”: "

string

", β€œTagNames”: [ "

string

" ] } } ], β€œSource”: { β€œDataSetArn”: "

string

", β€œJoinInstruction”: { β€œLeftJoinKeyProperties”: { β€œUniqueKey”:

boolean

}, β€œLeftOperand”: "

string

", β€œOnClause”: "

string

", β€œRightJoinKeyProperties”: { β€œUniqueKey”:

boolean

}, β€œRightOperand”: "

string

", β€œType”: "

string

" }, β€œPhysicalTableId”: "

string

" } } }, β€œName”: "

string

", β€œPermissions”: [ { β€œActions”: [ "

string

" ], β€œPrincipal”: "

string

" } ], β€œPhysicalTableMap”: { "

string

" : { β€œCustomSql”: { β€œColumns”: [ { β€œName”: "

string

", β€œType”: "

string

" } ], β€œDataSourceArn”: "

string

", β€œName”: "

string

", β€œSqlQuery”: "

string

" }, β€œRelationalTable”: { β€œCatalog”: "

string

", β€œDataSourceArn”: "

string

", β€œInputColumns”: [ { β€œName”: "

string

", β€œType”: "

string

" } ], β€œName”: "

string

", β€œSchema”: "

string

" }, β€œS3Source”: { β€œDataSourceArn”: "

string

", β€œInputColumns”: [ { β€œName”: "

string

", β€œType”: "

string

" } ], β€œUploadSettings”: { β€œContainsHeader”:

boolean

, β€œDelimiter”: "

string

", β€œFormat”: "

string

", β€œStartFromRow”:

number

, β€œTextQualifier”: "

string

" } } } }, β€œRowLevelPermissionDataSet”: { β€œArn”: "

string

", β€œFormatVersion”: "

string

", β€œNamespace”: "

string

", β€œPermissionPolicy”: "

string

", β€œStatus”: "

string

" }, β€œRowLevelPermissionTagConfiguration”: { β€œStatus”: "

string

", β€œTagRules”: [ { β€œColumnName”: "

string

", β€œMatchAllValue”: "

string

", β€œTagKey”: "

string

", β€œTagMultiValueDelimiter”: "

string

" } ] }, β€œTags”: [ { β€œKey”: "

string

", β€œValue”: "

string

" } ] }

Hi Juan, This blog should help you.

It demonstrates and gives guidelines on AWS IoT Analytics to Prepare Data for QuickSight.

Hi Tanaya.

Thank you very much for your answer.

When I make all the steps via AWS Console and later, I try to describe the dataset with AWS CloudShell I get the following error:

Regarding the post, I’ve already gone through it, but my need is to create the dataset via api.

I’m afraid that api does not support creating this kind of datasets yet, as api documentation says the following for CreateDataSet and DescribeDataSet operations:

This operation doesn’t support datasets that include uploaded files as a source

May be some engineer from QuickSight team could confirm this point.

Best regards,

1 Like

Hi @Juan_Carlos_Gonzalez ,

Yes creating/describing a dataset for files as source is not supported , noted as part of the documentation.

Kind regards,
Koushik

Hi Koushik.

Thanks for your answer. I think that this limitation of the api is not clear enough as we’re talking a IoT Analytics origin. Anyway, do you think this api methods will support this kind of datasets in the future?.

Best regards,

Hi @Juan_Carlos_Gonzalez ,

In your previous screenshot there is a dataset β€œqs_assets_dataset” , is this created from the UI where the datasource is AWS IOT Analytics ? . Have you then loaded this data into SPICE ?
The operation which you are unable to now perform is describe this IOT Analytics Origin dataset ?

Kind regards,
Koushik

Hi @Koushik_Muthanna,

Yes to all your questions. Our need isn’t to describe the asset. This was just a way to get its definition once it was created via UI. Our real need is to create it via API.

Our solution is ingesting sensor data in SiteWise. We’ve embedded QuickSight console into our app and want to provide a minimal infrastructure in order to ease using this data. So, our intent is the following:

  • Configure SiteWise coldtier storage.
  • Create an IoT Analytics datastore, using SiteWise cold tier storage bucket.
  • Create an IoT Analytics dataset from previous datastore, using the proper query.
  • Create a QuickSight datasource with IoT Analytics source.
  • Create a QuickSight dataset with IoTAnalytics source. This is what we don’t know how to do via api.

Thanks in advance.

Best regards,

Hi @Juan_Carlos_Gonzalez ,

Thanks for providing the details, at the moment this functionality is not available.

I will mark this as a feature request.

Kind Regards,
Koushik