Amazon S3 Logstash Configuration

Logs

Install Integration

Create S3 Policy

In the top left corner of your aws console you will notice a services drop down arrow. Open it and from that menu choose IAM.

Now in the left hand menu you want to select policies. Once you have reached the policies page you want to hit the Create Policy that appears towards the top of the page.

On the create policy screen choose the json tab and enter the following:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Read",
      "Effect": "Allow",
      "Action": ["s3:GetObject"],
      "Resource": ["arn:aws:s3:::your-bucket/*"]
    },
    {
      "Sid": "List",
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": ["arn:aws:s3:::your-bucket"]
    }
  ]
}

At the bottom of the page select Next.

On the next page you need to give your policy a name.

Finally scroll to the bottom of the page and now hit create policy.

Create User

You are going to need to create a new user. While on the IAM page, in the left hand menu choose users.

Now at the top of the page select Create User.

Enter a username and continue onto the next page of creation.

In the next section you want to attach an exisiting policy. Highlight attach exisiting policies.

Search the policy list below for your newly created policy.

Continue onwards to the User Review step, you can choose to set any Tags here but they aren't necessary. Check all settings are correct and select create user.

This will take you to the Users page where you will see the new user that has just been created. Click on the new user to go to the user details page.

On the user details page click the Create access key link.

Here we need to select the "Third-party service" option shown below.

Scroll down to the very bottom of the page and click the confirmation checkbox and then click the "Next" button.

Enter a description and then click the Create access key option.

On the next screen you will be given your Access Key ID and Secret Access Key. You will need to make a note of these or alternatively download the .csv file provided.

Confirm S3 Bucket

Ensure your logs are being sent to an S3 bucket. The following Getting Started guide will help you achieve this if you are not doing so already:

Getting started creating an Amazon S3 bucket and sending data to Logit.io.

Configure Logstash for Amazon S3

To start sending logs and metrics from AWS to your Stack you need to configure an AWS Input on your Logit.io Stack.

Check Logit.io for your logs

Data should now have been sent to your Stack.

If you don't see take a look at How to diagnose no data in Stack below for how to diagnose common issues.