Logstash Input for CloudWatch Logs

Stream events from CloudWatch Logs.

Update Jan 2024

Now it can suppot CloudWatch cross-account observability.

• logstash should be 8.4.0+
• default includeLinkedAccounts is true, add aws_account_id to specify the list of accounts to search. You can specify as many as 20 account IDs in the array. Default it is null and returns all log groups in the monitoring account and all log groups in all source accounts that are linked to the monitoring account.
• if set log_group_prefix to 'false', log_group should be loggroups arn

Parameters

Parameter	Input Type	Required	Default
aws_account_id	string or Array of strings	No
log_group	string or Array of strings	Yes
log_group_prefix	boolean	No	false
start_position	beginning, end, or an Integer	No	beginning
sincedb_path	string	No	$HOME/.sincedb*
interval	number	No	60
aws_credentials_file	string	No
access_key_id	string	No
secret_access_key	string	No
session_token	string	No
region	string	No	us-east-1
codec	string	No	plain

Update July 2023

Hello, this project has not received any maintenance since May of 2018. It is no longer a path I am using professionally, and I can not devote the time to maintaining it.

Purpose

Specify an individual log group or array of groups, and this plugin will scan all log streams in that group, and pull in any new log events.

Optionally, you may set the log_group_prefix parameter to true which will scan for all log groups matching the specified prefix(s) and ingest all logs available in all of the matching groups.

Usage

Parameters

Parameter	Input Type	Required	Default
log_group	string or Array of strings	Yes
log_group_prefix	boolean	No	false
start_position	beginning, end, or an Integer	No	beginning
sincedb_path	string	No	$HOME/.sincedb*
interval	number	No	60
aws_credentials_file	string	No
access_key_id	string	No
secret_access_key	string	No
session_token	string	No
region	string	No	us-east-1
codec	string	No	plain

start_position

The start_position setting allows you to specify where to begin processing a newly encountered log group on plugin boot. Whether the group is 'new' is determined by whether or not the log group has a previously existing entry in the sincedb file.

Valid options for start_position are:

• beginning - Reads from the beginning of the group (default)
• end - Sets the sincedb to now, and reads any new messages going forward
• Integer - Number of seconds in the past to begin reading at

Logstash Default config params

Other standard logstash parameters are available such as:

• add_field
• type
• tags

Example

input {
    cloudwatch_logs {
        log_group => [ "/aws/lambda/my-lambda" ]
        access_key_id => "AKIAXXXXXX" 
        secret_access_key => "SECRET"
    }
}

Development

The default logstash README which contains development directions and other information has been moved to DEVELOPER.md.

Contributing

All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.

Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here.

It is more important to the community that you are able to contribute.

For more information about contributing, see the CONTRIBUTING file.