Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SNMP input with tables appear to split some events into separate table rows for the same index. #16164

Closed
DanSheps opened this issue May 16, 2024 · 3 comments
Closed

SNMP input with tables appear to split some events into separate table rows for the same index. #16164

DanSheps opened this issue May 16, 2024 · 3 comments
Labels
bug

Comments

@DanSheps
Copy link

Logstash information:

Please include the following information:

  1. Logstash version: 8.12.0
  2. Logstash installation source: RPM
  3. How is Logstash being run: systemd

Plugins installed:

ogstash-codec-avro (3.4.1)
logstash-codec-cef (6.2.7)
logstash-codec-collectd (3.1.0)
logstash-codec-dots (3.0.6)
logstash-codec-edn (3.1.0)
logstash-codec-edn_lines (3.1.0)
logstash-codec-es_bulk (3.1.0)
logstash-codec-fluent (3.4.2)
logstash-codec-graphite (3.0.6)
logstash-codec-json (3.1.1)
logstash-codec-json_lines (3.1.0)
logstash-codec-line (3.1.1)
logstash-codec-msgpack (3.1.0)
logstash-codec-multiline (3.1.1)
logstash-codec-netflow (4.3.2)
logstash-codec-plain (3.1.0)
logstash-codec-rubydebug (3.1.0)
logstash-filter-aggregate (2.10.0)
logstash-filter-anonymize (3.0.7)
logstash-filter-cidr (3.1.3)
logstash-filter-clone (4.2.0)
logstash-filter-csv (3.1.1)
logstash-filter-date (3.1.15)
logstash-filter-de_dot (1.0.4)
logstash-filter-dissect (1.2.5)
logstash-filter-dns (3.2.0)
logstash-filter-drop (3.0.5)
logstash-filter-elasticsearch (3.16.1)
logstash-filter-fingerprint (3.4.3)
logstash-filter-geoip (7.2.13)
logstash-filter-grok (4.4.3)
logstash-filter-http (1.5.0)
logstash-filter-json (3.2.1)
logstash-filter-kv (4.7.0)
logstash-filter-memcached (1.2.0)
logstash-filter-metrics (4.0.7)
logstash-filter-mutate (3.5.8)
logstash-filter-prune (3.0.4)
logstash-filter-ruby (3.1.8)
logstash-filter-sleep (3.0.7)
logstash-filter-split (3.1.8)
logstash-filter-syslog_pri (3.2.0)
logstash-filter-throttle (4.0.4)
logstash-filter-translate (3.4.2)
logstash-filter-truncate (1.0.6)
logstash-filter-urldecode (3.0.6)
logstash-filter-useragent (3.3.5)
logstash-filter-uuid (3.0.5)
logstash-filter-xml (4.2.0)
logstash-input-azure_event_hubs (1.4.5)
logstash-input-beats (6.7.2)
└── logstash-input-elastic_agent (alias)
logstash-input-couchdb_changes (3.1.6)
logstash-input-dead_letter_queue (2.0.0)
logstash-input-elastic_serverless_forwarder (0.1.4)
logstash-input-elasticsearch (4.19.1)
logstash-input-exec (3.6.0)
logstash-input-file (4.4.6)
logstash-input-ganglia (3.1.4)
logstash-input-gelf (3.3.2)
logstash-input-generator (3.1.0)
logstash-input-graphite (3.0.6)
logstash-input-heartbeat (3.1.1)
logstash-input-http (3.8.0)
logstash-input-http_poller (5.5.1)
logstash-input-imap (3.2.1)
logstash-input-jms (3.2.2)
logstash-input-pipe (3.1.0)
logstash-input-redis (3.7.0)
logstash-input-snmp (1.3.3)
logstash-input-snmptrap (3.1.0)
logstash-input-stdin (3.4.0)
logstash-input-syslog (3.7.0)
logstash-input-tcp (6.4.1)
logstash-input-twitter (4.1.1)
logstash-input-udp (3.5.0)
logstash-input-unix (3.1.2)
logstash-integration-aws (7.1.6)
 ├── logstash-codec-cloudfront
 ├── logstash-codec-cloudtrail
 ├── logstash-input-cloudwatch
 ├── logstash-input-s3
 ├── logstash-input-sqs
 ├── logstash-output-cloudwatch
 ├── logstash-output-s3
 ├── logstash-output-sns
 └── logstash-output-sqs
logstash-integration-elastic_enterprise_search (3.0.0)
 ├── logstash-output-elastic_app_search
 └──  logstash-output-elastic_workplace_search
logstash-integration-jdbc (5.4.6)
 ├── logstash-input-jdbc
 ├── logstash-filter-jdbc_streaming
 └── logstash-filter-jdbc_static
logstash-integration-kafka (11.3.3)
 ├── logstash-input-kafka
 └── logstash-output-kafka
logstash-integration-logstash (1.0.1)
 ├── logstash-input-logstash
 └── logstash-output-logstash
logstash-integration-rabbitmq (7.3.3)
 ├── logstash-input-rabbitmq
 └── logstash-output-rabbitmq
logstash-output-csv (3.0.10)
logstash-output-elasticsearch (11.22.2)
logstash-output-email (4.1.3)
logstash-output-file (4.3.0)
logstash-output-graphite (3.1.6)
logstash-output-http (5.6.0)
logstash-output-lumberjack (3.1.9)
logstash-output-nagios (3.0.6)
logstash-output-null (3.0.5)
logstash-output-pipe (3.0.6)
logstash-output-redis (5.0.0)
logstash-output-stdout (3.1.4)
logstash-output-tcp (6.1.2)
logstash-output-udp (3.2.0)
logstash-output-webhdfs (3.1.0)
logstash-patterns-core (4.3.4)

JVM: Bundled

OS version (uname -a if on a Unix-like system): 3.10.0-1160.49.1.el7.x86_64 #1 SMP Tue Nov 30 15:51:32 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

I am using the SNMP input plugin with SNMP tables querying the following mibs:

                    "1.3.6.1.4.1.14179.2.2.1.1.1",
                    "1.3.6.1.4.1.14179.2.2.1.1.2",
                    "1.3.6.1.4.1.14179.2.2.1.1.3",
                    "1.3.6.1.4.1.14179.2.2.1.1.4",
                    "1.3.6.1.4.1.14179.2.2.1.1.6",
                    "1.3.6.1.4.1.14179.2.2.1.1.8",
                    "1.3.6.1.4.1.14179.2.2.1.1.9",
                    "1.3.6.1.4.1.14179.2.2.1.1.13",
                    "1.3.6.1.4.1.14179.2.2.1.1.16",
                    "1.3.6.1.4.1.14179.2.2.1.1.17",
                    "1.3.6.1.4.1.14179.2.2.1.1.19",
                    "1.3.6.1.4.1.14179.2.2.1.1.22",
                    "1.3.6.1.4.1.14179.2.2.1.1.26",
                    "1.3.6.1.4.1.14179.2.2.1.1.27",
                    "1.3.6.1.4.1.14179.2.2.1.1.28",
                    "1.3.6.1.4.1.14179.2.2.1.1.33",
                    "1.3.6.1.4.1.14179.2.2.1.1.37",
                    "1.3.6.1.4.1.9.9.513.1.1.1.1.32",
                    "1.3.6.1.4.1.9.9.513.1.1.1.1.104",
                    "1.3.6.1.4.1.9.9.513.1.1.1.1.106",
                    "1.3.6.1.4.1.9.9.513.1.1.1.1.105",
                    "1.3.6.1.4.1.9.9.513.1.1.1.1.38"

The full table config is:

        tables => [
            {
                "name" => "aps"
                "columns" => [
                    "1.3.6.1.4.1.14179.2.2.1.1.1",
                    "1.3.6.1.4.1.14179.2.2.1.1.2",
                    "1.3.6.1.4.1.14179.2.2.1.1.3",
                    "1.3.6.1.4.1.14179.2.2.1.1.4",
                    "1.3.6.1.4.1.14179.2.2.1.1.6",
                    "1.3.6.1.4.1.14179.2.2.1.1.8",
                    "1.3.6.1.4.1.14179.2.2.1.1.9",
                    "1.3.6.1.4.1.14179.2.2.1.1.13",
                    "1.3.6.1.4.1.14179.2.2.1.1.16",
                    "1.3.6.1.4.1.14179.2.2.1.1.17",
                    "1.3.6.1.4.1.14179.2.2.1.1.19",
                    "1.3.6.1.4.1.14179.2.2.1.1.22",
                    "1.3.6.1.4.1.14179.2.2.1.1.26",
                    "1.3.6.1.4.1.14179.2.2.1.1.27",
                    "1.3.6.1.4.1.14179.2.2.1.1.28",
                    "1.3.6.1.4.1.14179.2.2.1.1.33",
                    "1.3.6.1.4.1.14179.2.2.1.1.37",
                    "1.3.6.1.4.1.9.9.513.1.1.1.1.32",
                    "1.3.6.1.4.1.9.9.513.1.1.1.1.104",
                    "1.3.6.1.4.1.9.9.513.1.1.1.1.106",
                    "1.3.6.1.4.1.9.9.513.1.1.1.1.105",
                    "1.3.6.1.4.1.9.9.513.1.1.1.1.38"
                ]
            }
        ]

When querying some of our access points, a few of the mib entries overflow into a new "index" object (same index for the table, but a new object within the table array).

This means that if I split {} aps, it results in 2 partial events instead of 1 full event. I would expect it would keep all indexed mibs together.

This may be related to data size perhaps, however I cannot confirm. This only happens on 4 of the Wireless Access Points in the table list our of 489.

Steps to reproduce:

  1. Configure an SNMP input with the above table setting
  2. split {} the table into separate events
  3. Output the event
@edmocosta
Copy link
Contributor

Hi @DanSheps!

Thank you for reporting! I'm not sure if I fully understood this issue, could you please provide more context on the expected behaviour and the used pipeline's split config?

The SNMP table is mapped as an array by the snmp plugin, so If you're splitting the event by the table's name split { field => "aps" }, the split filter will generated one event per received OID, without including the other array's elements (OIDs) on it.

For example, the following snmp event would be split into two events, one per per "aps" item:

"aps" => [
        [0] {
            "index" => "1.2.1",
            "1.3.6.1.2.1.1.9.1.2.1" => "1.3.6.1.6.3.11.2.3.1.1"
        },
        [1] {
            "index" => "1.2.2",
            "1.3.6.1.2.1.1.9.1.2.2" => "1.3.6.1.6.3.15.2.1.1"
        }
]

Nevertheless, we've an open issue with the "index" value that may be related to this case. My suggestion for now is to try it out with the new SNMP integration plugin, which uses the latest underline SNMP libraries and maybe already fixed the problem.

Thank you!

@DanSheps
Copy link
Author

DanSheps commented May 17, 2024 •
edited
Loading

Thanks @edmocosta that looks like the exact same issue. I did capture some data from the output and what you get is something like this

    {
      "index": "<MAC Address #1 in decimal format>",
      "1.3.6.1.4.1.9.9.513.1.1.1.1.38": <Wireless Admin Status>,
      "1.3.6.1.4.1.9.9.513.1.1.1.1.105": "<RF Tag>"
    },
    {
      "index": "<MAC Address #2 in decimal format>",
      "1.3.6.1.4.1.9.9.513.1.1.1.1.38": <Wireless Admin Status>,
      "1.3.6.1.4.1.9.9.513.1.1.1.1.105": "<RF Tag>"
    },
    {
      "index": "<MAC Address #3 in decimal format>",
      "1.3.6.1.4.1.9.9.513.1.1.1.1.38": <Wireless Admin Status>,
      "1.3.6.1.4.1.9.9.513.1.1.1.1.105": "<RF Tag>"
    },
    {
      "index": "<MAC Address #4 in decimal format>",
      "1.3.6.1.4.1.9.9.513.1.1.1.1.38": <Wireless Admin Status>,
      "1.3.6.1.4.1.9.9.513.1.1.1.1.105": "<RF Tag>"
    },
	
    {
      "1.3.6.1.4.1.14179.2.2.1.1.1.17": "<SERIALNUMBER>",
      "index": "<MAC Address #1 in decimal format>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.1": "<MAC ADDRESS>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.3": "<AP Name>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.33": "<Ethernet MAC Address>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.6": <Status>,
      "1.3.6.1.4.1.14179.2.2.1.1.1.8": "<Software Version>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.9": "<Boot Loader Version>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.4": "<Location Name>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.2": <RADIO SLOTS>,
      "1.3.6.1.4.1.9.9.513.1.1.1.1.106": "<Policy Tag>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.16": "<Model>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.28": "<Static IP Address>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.37": <Admin Status>,
      "1.3.6.1.4.1.14179.2.2.1.1.1.26": "<Subnet Mask>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.19": "<IP Address>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.13": <Port Number>,
      "1.3.6.1.4.1.9.9.513.1.1.1.1.104": "<Site Tag>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.22": <AP Type>,
      "1.3.6.1.4.1.14179.2.2.1.1.1.27": "<IP Gateway>",
      "1.3.6.1.4.1.9.9.513.1.1.1.1.32": "<Location Name>"
    },
    {
      "1.3.6.1.4.1.14179.2.2.1.1.1.17": "<SERIALNUMBER>",
      "index": "<MAC Address #2 in decimal format>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.1": "<MAC ADDRESS>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.3": "<AP Name>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.33": "<Ethernet MAC Address>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.6": <Status>,
      "1.3.6.1.4.1.14179.2.2.1.1.1.8": "<Software Version>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.9": "<Boot Loader Version>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.4": "<Location Name>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.2": <RADIO SLOTS>,
      "1.3.6.1.4.1.9.9.513.1.1.1.1.106": "<Policy Tag>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.16": "<Model>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.28": "<Static IP Address>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.37": <Admin Status>,
      "1.3.6.1.4.1.14179.2.2.1.1.1.26": "<Subnet Mask>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.19": "<IP Address>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.13": <Port Number>,
      "1.3.6.1.4.1.9.9.513.1.1.1.1.104": "<Site Tag>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.22": <AP Type>,
      "1.3.6.1.4.1.14179.2.2.1.1.1.27": "<IP Gateway>",
      "1.3.6.1.4.1.9.9.513.1.1.1.1.32": "<Location Name>"
    },
    {
      "1.3.6.1.4.1.14179.2.2.1.1.1.17": "<SERIALNUMBER>",
      "index": "<MAC Address #3 in decimal format>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.1": "<MAC ADDRESS>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.3": "<AP Name>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.33": "<Ethernet MAC Address>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.6": <Status>,
      "1.3.6.1.4.1.14179.2.2.1.1.1.8": "<Software Version>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.9": "<Boot Loader Version>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.4": "<Location Name>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.2": <RADIO SLOTS>,
      "1.3.6.1.4.1.9.9.513.1.1.1.1.106": "<Policy Tag>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.16": "<Model>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.28": "<Static IP Address>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.37": <Admin Status>,
      "1.3.6.1.4.1.14179.2.2.1.1.1.26": "<Subnet Mask>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.19": "<IP Address>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.13": <Port Number>,
      "1.3.6.1.4.1.9.9.513.1.1.1.1.104": "<Site Tag>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.22": <AP Type>,
      "1.3.6.1.4.1.14179.2.2.1.1.1.27": "<IP Gateway>",
      "1.3.6.1.4.1.9.9.513.1.1.1.1.32": "<Location Name>"
    },
    {
      "1.3.6.1.4.1.14179.2.2.1.1.1.17": "<SERIALNUMBER>",
      "index": "<MAC Address #4 in decimal format>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.1": "<MAC ADDRESS>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.3": "<AP Name>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.33": "<Ethernet MAC Address>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.6": <Status>,
      "1.3.6.1.4.1.14179.2.2.1.1.1.8": "<Software Version>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.9": "<Boot Loader Version>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.4": "<Location Name>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.2": <RADIO SLOTS>,
      "1.3.6.1.4.1.9.9.513.1.1.1.1.106": "<Policy Tag>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.16": "<Model>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.28": "<Static IP Address>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.37": <Admin Status>,
      "1.3.6.1.4.1.14179.2.2.1.1.1.26": "<Subnet Mask>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.19": "<IP Address>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.13": <Port Number>,
      "1.3.6.1.4.1.9.9.513.1.1.1.1.104": "<Site Tag>",
      "1.3.6.1.4.1.14179.2.2.1.1.1.22": <AP Type>,
      "1.3.6.1.4.1.14179.2.2.1.1.1.27": "<IP Gateway>",
      "1.3.6.1.4.1.9.9.513.1.1.1.1.32": "<Location Name>"
    }
}

And my appologies I was unaware there was a separate repo for plugins for logstash, which likely explains why I was unable to find any issues here.

You could likely close this since I suspect the bug linked in the plugin repo is the same bug.

@edmocosta
Copy link
Contributor

Duplicate of logstash-plugins/logstash-integration-snmp#30

@edmocosta edmocosta closed this as not planned Won't fix, can't repro, duplicate, stale May 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@roaksoax @DanSheps @edmocosta