Using incron to automatically copy data to S3🔗

Overview🔗

This is not strictly related to Matillion ETL, however it is a common task to get data easily into S3 from a premise or system. This approach is especially used in a micro-batching scenario.

The below instructions should be enough to get you started with incron, however this is a powerful tool that is capable of much more than the scope of this article can cover.

Installation🔗

To begin, incron must be installed. This daemon will set and watch a directory for changes. To get this on Amazon Linux, run:

sudo yum install incron

Once installed, delete the /etc/incron.allow file if it exists. This is used to allow who can use the tool.

sudo rm /etc/incron.allow

Alternatively, allow your user in the file.

Setting up aws-cli🔗

Assuming that the aws-cli is already installed, run as it is on an Amazon Linux machine. If not, use the instructions here. Run:

aws configure

Follow the on-screen instructions. Once installed, create the script that will copy your file. Run:

incrontab -e

To edit the inotify configuration, add a line as so:

<path to local watched directory> IN_CLOSE_WRITE /usr/local/bin/aws s3 cp $@/$# s3://<bucket name><prefix path>

Example:

/home/ed/lntest IN_CLOSE_WRITE /usr/local/bin/aws s3 cp $@/$# s3://matillion/delme/

Now to test, add a file to the watched directory and it will appear in S3. If anything is wrong, it is written to /var/log/syslog.

The full incron documentation is worth a read and it's available here.