Pull in S3 file name as metadata

In case anyone is hitting the same issue, I ended up solving it by adding a captureTime field to my source data. This calculated the UTC time, then used the jq utility to add it to each json record.

CUR_UTC=$(date --utc +%FT%TZ)
echo “$(jq ‘.[] += {“captureTimeUTC”: "’”$CUR_UTC"‘"}’ data/$THE_FILE)" > data/$THE_FILE

2 Likes