In my previous blog post I have explained how to automatically create AWS Athena Partitions for cloudtrail logs between two dates. That script will help us to create the partitions till today. But cloudtrail will generate log on everyday. So I was thinking to automate this process too. For this automation I have used Lambda which is a serverless one. This will automate AWS Athena create partition on daily basis. Your Lambda function needs Read permisson on the cloudtrail logs bucket, write access on the query results bucket and execution permission for Athena.
Here is my AWS CloudTrail Log path in S3.
Create the table with Partitions
IAM Policy for the Lambda
Parameters for S3 bucket and Athena
s3_buckcet – Bucket name where your cloudtrail logs stored.
s3_prefix – Path for your cloudtrail logs (give the prefix before the regions. For eg: s3://bucket/AWSLogs/AccountID/Cloudtrail/regions/year/month/day/log_files. So you need to use path: AWSLogs/AccountID/Cloudtrail/ ).
s3_ouput – Path for where your Athena query results need to be saved.
database – Name of the DB where your cloudwatch logs table located.
table_name – Nanme of the table where your cloudwatch logs table located.
Main Function for create the Athena Partition on daily
NOTE: I have created this script to add partition as current date +1(means tomorrow’s date). Because its always better to have one day additional partition, so we don’t need wait until the lambda will trigger for that particular date.
Before schedule it, you need to create partition for till today. Refer this link will help you to create partitions between any particular date.