Amazon S3 Logstash Configuration

Pull logs from a S3 Bucket to Logstash

Install Integration

Please click on the Install Integration button to configure your stack for this source.

Create S3 Policy

In the top left corner of your aws console you will notice a services drop down arrow. Open it and from that menu choose IAM.

List of IAM policies

Now in the left hand menu you want to select policies. Once you have reached the policies page you want to hit the Create Policy that appears towards the top of the page.

On the create policy screen choose the json tab and enter the following:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Read",
      "Effect": "Allow",
      "Action": ["s3:GetObject"],
      "Resource": ["arn:aws:s3:::your-bucket/*"]
    },
    {
      "Sid": "List",
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": ["arn:aws:s3:::your-bucket"]
    }
  ]
}

Create policy editor in JSON

At the bottom of the page select Next.

Create policy editor next button

On the next page you need to give your policy a name.

Create new policy form fields

Finally scroll to the bottom of the page and now hit create policy.

Create new policy form create policy button

Create User

You are going to need to create a new user. While on the IAM page, in the left hand menu choose users.

Now at the top of the page select Create User.

Add User from Users page

Enter a username and continue onto the next page of creation.

Fill in set user details form

In the next section you want to attach an exisiting policy. Highlight attach exisiting policies.

Highlight existing policies box

Search the policy list below for your newly created policy.

Find and select policy

Continue onwards to the User Review step, you can choose to set any Tags here but they aren't necessary. Check all settings are correct and select create user.

This will take you to the Users page where you will see the new user that has just been created. Click on the new user to go to the user details page.

Select new user

On the user details page click the Create access key link.

Select create access key

Here we need to select the "Third-party service" option shown below.

Select third party service

Scroll down to the very bottom of the page and click the confirmation checkbox and then click the "Next" button.

Confirm recommendation

Enter a description and then click the Create access key option.

Enter description

On the next screen you will be given your Access Key ID and Secret Access Key. You will need to make a note of these or alternatively download the .csv file provided.

Success message with your Access Key ID

Confirm S3 Bucket

Ensure your logs are being sent to an S3 bucket. The following Getting Started guide will help you achieve this if you are not doing so already:

Getting started creating an Amazon S3 bucket and sending data to Logit.io.

Configure Logstash for Amazon S3

To start sending logs and metrics from AWS to your Stack you need to configure an AWS Input on your Logit.io Stack.

Go to Dashboard

Logit.io will verify your input before it is applied, we will contact you to confirm when this has been completed.

Check Logit.io for your logs

Data should now have been sent to your Stack.

View My Data

If you don't see take a look at How to diagnose no data in Stack below for how to diagnose common issues.

S3 Logging Overview

Sending data from Amazon S3 to Logit.io is a crucial aspect of efficient log management and analysis for businesses utilizing AWS infrastructure.

By efficiently integrating these two services, organizations can gain valuable insights from their log data and enhance their operational efficiency.

Logit.io offers an intuitive solution that allows users to ship, analyze, and visualize logs from Amazon S3, ensuring that vital information is readily available for troubleshooting and monitoring. Via Logit.io, you can also take advantage of our Amazon ELB Application (Elastic Load Balancer) to ingest further data into our centralised platform. In addition, you can link your CloudWatch metrics to Logit.io for comprehensive log data analysis and real-time monitoring. By leveraging these services in tandem, you can enhance your operational intelligence and proactively address issues within your AWS infrastructure.

Logit.io's AWS logging (opens in a new tab) is known for its end-to-end integration of these components.