Hi All,
I am looking to build a real-time visual dashboard fed by Kinesis Data Streams. The data is in JSON with a consistent schema.
I already have a near real-time solution where the Kinesis Data Stream is connected to Firehose there by sending the data to S3 and visualized in QuickSight. However, we are not meeting our desired latency for the visuals to be updated.
Are there any recommendations to do this?
A couple things we do to get to closest to real time.
Use Athena to query & partition the data to query it faster. And query it by the smallest amount of data you can (i.e to the latest minute / second).
Have you looked into federated queries? This will hit a lambda directly and transfer the data through Athena. It’s not real time but it gets pretty close.
Have a refresh button in quicksight that triggers a direct query. What we have done before is have a table that has a navigation action that stays on the same page & changes a parameter to a set value. This parameter does not affect anything. But that fact that it refreshes will trigger a direct query.
Hope any of these help
This would be a really cool feature though to have a direct connect or something with Kinesis
Hi Max,
Thanks for your recommendations.
It for sure would be a cool feature to have a direct connection - something that would have wide use-cases. We are looking for the lowest latency. Of the three approaches you mentioned, would you agree that method 2 or 3 will have the lowest latency?
-Ross
Option 3 is more of a work around in QuickSight that will allow you to easily query the data without having to refresh the page.
In regards to option two, I think that will depend on how the lambda is set up / how much data you are sending through it.
It could help, but again it depends.
Hi Max,
Do you have a link to an example of how to implement option 3?
Regards,
-Ross
First create a calculated field called refresh data.
Then create a table with Refresh Data in the group by.
You can get rid of headers / title to make it look cleaner.
Then create a parameter called Refresh Data with a default value of ‘Refresh Data’.
Next add a filter to all of the visuals you want to refresh on a click to be equal to the new parameter you just created.
Finally, add an action to the Refresh Button visual that will update the Refresh Data parameter to ‘Refresh Data’ (this will trigger and update).
The target sheet will be the same sheet as your visual. And you will link it to the Refresh Data field.
Let me know if that helps!
Thank you @Max this helped a lot!
