Use Case and Problem
This article shows how to use Amazon QuickSight and Amazon Athena to analyze CloudWatch Logs.
Prerequisites
- Amazon Account with necessary permissions to create and access the required services.
Ex: Amazon S3 , Athena, Amazon QuickSight & CloudWatch access. - Amazon CloudWatch logs are already configured and have data to analyze.
Step by Step Solution
Step 1: Set up Amazon CloudWatch Logs
- If you haven’t already, set up Amazon CloudWatch Logs to collect and store your log data. You can follow the official AWS documentation to configure your log groups and streams.
Step 2: Create an Athena Table for CloudWatch Logs
- Go to the AWS Management Console and navigate to the Amazon Athena service.
- In the Athena Query Editor, click on the “Settings” icon and ensure that the query result location is set to a valid S3 bucket (e.g., “s3://your-bucket-name/athena-results/”).
- Create a new table that points to your CloudWatch logs data. In the Athena Query Editor, execute the following SQL query to create the table.
CREATE EXTERNAL TABLE cloudwatch_logs (
`timestamp` timestamp,
`message` string
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
'serialization.format' = '1',
'input.regex' = '^(\\S+)\\s+(.*)$'
)
LOCATION 's3://s3bucketname/';
Step 3: Query CloudWatch Logs Using Athena
-
Now that your table is set up, you can query your CloudWatch logs data using Athena.
For example:- SELECT timestamp, log_group, log_stream, message
FROM my_cloudwatch_logs
WHERE log_group = ‘access_log’
LIMIT 10
- SELECT timestamp, log_group, log_stream, message
Step 4: Set up Amazon QuickSight
- Navigate to the Amazon QuickSight service in the AWS Management Console.
- Click on Datasets from left navigation and then choose New dataset.
- Choose “Athena” as the data source.
- Select your Athena database and the table you created in Step 2.
- Click “Edit/Preview Data” to review and customize the data if necessary.
Step 5: Create Visualizations in Amazon QuickSight
- Once your dataset is ready, create a new analysis.
- Choose the dataset you created earlier.
- Start building visualizations by selecting the appropriate chart types and dimensions.
- Customize the visuals and add filters as needed.
Step 6: Share and Publish Dashboards (Optional)
- After creating your visualizations, you can share the dashboard with others within your organization, or publish it to the web for wider access.
Code sample: Git Repository
Conclusion
That’s it! You now have a step-by-step guide on how to analyze AWS CloudWatch logs using Amazon Athena and Amazon QuickSight.
Author Bio
Lasantha Sanjeewa Silva is an AWS Community Builder and Cloud DevOps Enthusiast.