Recommendation for Appropriate Data Pipeline Services

Question & Answer
,
1

Hello :wave: ,

As a relatively new user to the AWS ecosystem, I have some questions about a specific use case for QuickSight and was hoping some experts would be able to provide some recommendations.

I have a fairly large database, currently stored within Amazon RDS. This data is structured, but would require processing before it was in the state that I would want to import it into QuickSight. What would be the most efficienct and/or cost effective services to allow me to perform SQL transformations against this data and then load the results into QuickSight for visualisation? This pipeline should be able to handle the datasets updating and have the queries continue to refresh the data that will then result in updated QuickSight dashboards.

Currently, I have been able to query the RDS data within MySQL and export the resultant data, and then import them into SPICE within QuickSight. This has allowed me to build familiarity with the tools, but am now looking for the best solution for automation of these processes.

From basic research, it looks like some combination of RedShift, Glue, Athena, and potentially a few others will get me what I need, but I was wondering if anyone with a bit more expertise had a recommendation? Happy to talk through any questions on my use case or take any advice if any of my current understanding is incorrect.

Thank you.

2

Hello Jack,

First of all Welcome to the QuickSight Community.

Based on the contraints mentioned, In my opinion I would say. Glue > Athena > QuickSight

See some reference here

Hope this helps.

Cheers,
Deep

3

Hi Deep,

Thanks for the resources and the recommendation!

I’ll have a read through and work with our DevOps team to see if this setup will work for our organisation, but it looks promising from what you’ve said. Thanks for the swift response!

1 Like
4

Hi Deep,

Sorry to come back to this question, but I wondered if you’d have any more insight now that I understand my use case a little better.

I ideally need the data to be refreshed as close to “Live” as possible, with the tables in the MySQL database receiving multiple updates per minute. I’d like to see these with only a minute or two delay within QuickSight.

Since I’ve fully understood these requirements, I’ve done some more research and was wondering if you had any further insight. It looks like change tracking can be done using DMS CDC and this could then feed those changes to an S3 bucket, or into Kinesis, ready for querying in Athena which can then be read from QuickSight.

Do you believe either of these solutions would be cost effective for my use case?

Thanks