Skip to content

3.3.0
Compare
Choose a tag to compare
@aaronkaplan aaronkaplan released this 01 Mar 18:22
· 86 commits to develop since this release
3.3.0
This tag was signed with the committer’s verified signature.
aaronkaplan AaronK
GPG key ID: 1AED8B672DD4C9B1
Learn about vigilant mode.
6d3bdbc
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.

3.3.0 (2024-03-01)

Configuration

  • Add new optional configuration parameters for intelmq.bots.collectors.stomp.collector
    and intelmq.bots.outputs.stomp.output (PR#2408 by Jan Kaliszewski):
    • auth_by_ssl_client_certificate (Boolean, default: true; if false then
      ssl_client_certificate and ssl_client_certificate_key will be ignored);
    • username (STOMP authentication login, default: "guest"; to be used only
      if auth_by_ssl_client_certificate is false);
    • password (STOMP authentication passcode, default: "guest"; to be used only
      if auth_by_ssl_client_certificate is false).
  • Add the possibility to set the ssl_ca_certificate configuration parameter for
    intelmq.bots.collectors.stomp.collector and/or intelmq.bots.outputs.stomp.output
    to an empty string - which means that the SSL machinery used for STOMP communication
    will attempt to load the system’s default CA certificates (PR#2414 by Jan Kaliszewski).

Core

  • intelmq.lib.message: For invalid message keys, add a hint on the failure to the exception: not allowed by configuration or not matching regular expression (PR#2398 by Sebastian Wagner).
  • intelmq.lib.exceptions.InvalidKey: Add optional parameter additional_text (PR#2398 by Sebastian Wagner).
  • Change the way we discover bots to allow easy extending based on the entry point name. (PR#2413 by Kamil Mankowski)
  • intelmq.lib.mixins: Add a new class, StompMixin (defined in a new submodule: stomp),
    which provides certain common STOMP-bot-specific operations, factored out from
    intelmq.bots.collectors.stomp.collector and intelmq.bots.outputs.stomp.output
    (PR#2408 and PR#2414 by Jan Kaliszewski).
  • intelmq.lib.upgrades: Replace deprecated instances of url2fqdn experts by the new url expert in runtime configuration (PR#2432 by Sebastian Wagner).
  • intelmq.lib.bot: Ensure closing log files on reloading (PR#2435 by Kamil Mankowski).
  • AMQP Pipeline: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).
  • Only load the config once when starting intelmqctl (which makes IntelMQ API calls take less time) (PR#2444 by DigitalTrustCenter).

Development

  • Makefile: Add codespell and test commands (PR#2425 by Sebastian Wagner).

Data Format

Bots

Collectors

  • intelmq.bots.collectors.stomp.collector (PR#2408 and PR#2414 by Jan Kaliszewski):
    • Drop support for versions of stomp.py older than 4.1.12.
    • Update the code to support new versions of stomp.py, including the latest (8.1.0);
      fixes #2342.
    • Add support for authentication based on STOMP login and passcode, introducing three
      new configuration parameters (see above: Configuration).
    • Add support for loading the system’s default CA certificates, as an alternative to
      specifying the CA certificate(s) file path explicitly (see above: Configuration).
    • Fix (by carefully targeted monkey patching) certain security problems caused by
      SSL-related weaknesses that some versions of stomp.py suffer from.
    • Fix the reconnection behavior: do not attempt to reconnect after shutdown. Also,
      never attempt to reconnect if the version of stomp.py is older than 4.1.21 (it
      did not work properly anyway).
    • Add coercion of the port config parameter to int.
    • Add implementation of the check hook (verifying, in particular, accessibility
      of necessary file(s)).
    • Remove undocumented and unused attributes of StompCollectorBot instances:
      ssl_ca_cert, ssl_cl_cert, ssl_cl_cert_key.
      • Minor fixes/improvements and some refactoring (see also above: Core...).
  • intelmq.bots.collectors.amqp: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).
  • intelmq.bots.collectors.shadowserver.collector_reports_api:
    • The 'json' option is no longer supported as the 'csv' option provides better performance (PR#2372 by elsif2).
  • intelmq.bots.collectors.alienvault_otx.collector (PR#2449 by qux-bbb):
    • Fix modified_pulses_only is always False.

Parsers

  • intelmq.bots.parsers.netlab_360.parser: Removed as the feed is discontinued. (#2442 by Filip Pokorný)
  • intelmq.bots.parsers.webinspektor.parser: Removed as the feed is discontinued. (#2442 by Filip Pokorný)
  • intelmq.bots.parsers.sucuri.parser: Removed as the feed is discontinued. (#2442 by Filip Pokorný)
  • intelmq.bots.parsers.shadowserver._config:
    • Switch to dynamic configuration to decouple report schema changes from IntelMQ releases by regularly downloading them from the Shadowserver server (PR#2372 by elsif2).
  • intelmq.bots.parsers.cymru: Save current line. (PR by Kamil Mankowski)

Experts

  • intelmq.bots.experts.jinja (PR#2417 by Mikk Margus Möll):
    • Add optional socket_perms and socket_group parameters to change
      file permissions on socket file, if it is in use.
  • intelmq.bots.experts.ripe (PR#2461 by Mikk Margus Möll):
    • Handle "No abuse contact found for" messages for non-ASN resources

Outputs

  • intelmq.bots.outputs.stomp.output (PR#2408 and PR#2414 by Jan Kaliszewski):
    • Drop support for versions of stomp.py older than 4.1.12.
    • Update the code to support new versions of stomp.py, including the latest (8.1.0).
    • Add support for authentication based on STOMP login and passcode, introducing three
      new configuration parameters (see above: Configuration).
    • Add support for loading the system’s default CA certificates, as an alternative to
      specifying the CA certificate(s) file path explicitly (see above: Configuration).
    • Fix (by carefully targeted monkey patching) certain security problems caused by
      SSL-related weaknesses that some versions of stomp.py suffer from.
    • Fix AttributeError caused by attempts to get unset attributes of StompOutputBot
      (ssl_ca_cert et consortes).
    • Add coercion of the port config parameter to int.
    • Add implementation of the check hook (verifying, in particular, accessibility
      of necessary file(s)).
    • Add stomp.py version check (raise MissingDependencyError if not >=4.1.12).
    • Minor fixes/improvements and some refactoring (see also above: Core...).
  • intelmq.bots.outputs.stomp.output (PR#2423 by Kamil Mankowski):
    • Try to reconnect on NotConnectedException.
  • intelmq.bots.outputs.smtp_batch.output (PR #2439 by Edvard Rejthar):
    • Fix ability to send with the default bcc
  • intelmq.bots.outputs.amqp: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).

Documentation

  • Add a readthedocs configuration file to fix the build fail (PR#2403 by Sebastian Wagner).
  • Add a guide of developing extensions packages (PR#2413 by Kamil Mankowski)
  • Update/fix/improve the stuff related to the STOMP bots and integration with the n6's
    Stream API (PR#2408 and PR#2414 by Jan Kaliszewski).
  • Complete documentation overhaul. Change to markdown format. Uses the mkdocs-material (PR#2419 by Filip Pokorný).
  • Adds warning banner if not browsing the latest version of the docs (PR#2445 by Filip Pokorný).
  • Fix logo path in index.md when building the docs (PR#2445 by Filip Pokorný).

Packaging

  • Add pendulum to suggested packages, as it is required for the sieve bot (PR#2424 by Sebastian Wagner).
  • debian/control: in Suggests field, replace python3-stomp.py (>= 4.1.9) with
    python3-stomp (>= 4.1.12), i.e., fix the package name by removing the .py
    suffix and bump the minimum version to 4.1.12 (PR#2414 by Jan Kaliszewski).

Tests

Tools

  • intelmq_psql_initdb:
    • got support for providing custom harmonization file, generating view for storing raw fields separately, and adding IF NOT EXISTS/OR REPLACE clauses (PR#2404 by Kamil Mankowski).
    • got support for generating JSONB fields for PostgreSQL schema (PR#2436 by Kamil Mankowski).

Contrib

Known issues

Assets 2
Loading