Skip to content

3.2.0 Feature release: Running IntelMQ bots as Python Library
Compare
Choose a tag to compare
@sebix sebix released this 18 Jul 20:55
· 307 commits to develop since this release
3.2.0
This tag was signed with the committer’s verified signature. The key has expired.
sebix Sebastian
GPG key ID: 1AF10AF3B3917AB2
Expired
Learn about vigilant mode.
3a32cbe
This commit was signed with the committer’s verified signature. The key has expired.
sebix Sebastian
GPG key ID: 1AF10AF3B3917AB2
Expired
Learn about vigilant mode.

IEP007: Running IntelMQ bots as Python Library is implemented.

Installation: https://intelmq.readthedocs.io/en/develop/user/installation.html
Upgrade: https://intelmq.readthedocs.io/en/develop/user/upgrade.html

The accompanying 3.2.0 release of intelmq-api switches it's backend from the library hug to fastapi.
Deb-packages of intelmq-api 3.2.0 are delayed for some distributions because of necessary changes in packaging.

Core

  • intelmq.lib.utils:
    • resolve_dns: Deprecate dnspython versions pre-2.0.0 and disable search domains (PR#2352)
  • Fixed not resetting destination path statistics in the stats cache after restarting bot (Fixes #2331)
  • Force flushing statistics if bot will sleep longer than flushing delay (Fixes #2336)
  • intelmq.lib.upgrages: Fix a bug in the upgrade function for version 3.1.0 which caused an exception if a generic csv parser instance had no parameter type (PR#2319 by Filip Pokorný).
  • intelmq.lib.datatypes: Adds TimeFormat class to be used for the time_format bot parameter (PR#2329 by Filip Pokorný).
  • intelmq.lib.exceptions: Fixes a bug in InvalidArgument exception (PR#2329 by Filip Pokorný).
  • intelmq.lib.harmonization:
    • Changes signature and names of DateTime conversion functions for consistency, backwards compatible (PR#2329 by Filip Pokorný).
    • Ensure rejecting URLs with leading whitespaces after changes in CPython (fixes #2377)
  • intelmq.lib.bot.Bot: Allow setting the parameters via parameter on bot initialization.

Development

  • CI: pin the Codespell version to omit troubles caused by its new releases (PR #2379).

Bots

Collectors

  • intelmq.bots.collector.rt:
    • restrict python-rt to be below version 3.0 due to introduced breaking changes,
    • added support for Subject NOT LIKE queries,
    • added support for multiple values in ticket subject queries.
  • intelmq.bots.collectors.rsync: Support for optional private key, relative time parsing for the source path, extra rsync parameters and strict host key checking (PR#2241 by Mateo Durante).

Parsers

  • intelmq.bots.parsers.shadowserver._config:
    • Reset detected feedname at shutdown to re-detect the feedname on reloads (PR#2361 by @elsif2, fixes #2360).
  • intelmq.bots.parsers.shadowserver._config:
    • Added 'IPv6-Vulnerable-Exchange' alias and 'Accessible-WS-Discovery-Service' report. (PR#2338)
    • Removed unused p0f_genre and p0f_detail from the 'DNS-Open-Resolvers' report. (PR#2338)
    • Added 'Accessible-SIP' report. (PR#2348)
    • Added 'IPv6-Open-HTTP-Proxy' and 'IPv6-Accessible-HTTP-Proxy' aliases. (PR#2348)
    • Removed duplicate mappings from the 'Spam-URL' report. (PR#2348)
  • intelmq.bots.parsers.generic.parser_csv: Changes time_format parameter to use new TimeFormat class (PR#2329 by Filip Pokorný).
  • intelmq.bots.parsers.html_table.parser: Changes time_format parameter to use new TimeFormat class (PR#2329 by Filip Pokorný).
  • intelmq.bots.parsers.turris.parser.py Updated to the latest data format (issue #2167). (PR#2373 by Filip Pokorný).

Experts

  • intelmq.bots.experts.sieve:
    • Allow empty lists in sieve rule files (PR#2341 by Mikk Margus Möll).
  • intelmq.bots.experts.cymru_whois:
    • Ignore AS names with unexpected unicode characters (PR#2352, fixes #2132)
    • Avoid extraneous search domain-based queries on NXDOMAIN result (PR#2352)
  • intelmq.bots.experts.sieve:
    • Added :before and :after keywords (PR#2374)

Outputs

  • intelmq.bots.outputs.cif3.output: Added (PR#2244 by Michael Davis).
  • intelmq.bots.outputs.sql.output: New parameter fail_on_errors (PR#2362 by Sebastian Wagner).
  • intelmq.bots.outputs.smtp_batch.output: Added a bot to gathering the events and sending them by e-mails at a stroke as CSV files (PR#2253 by Edvard Rejthar)

Documentation

  • API: update API installation to be aligned with the rewritten API, and clarify some missing steps.

Tests

  • New decorator skip_installation and environment variable INTELMQ_TEST_INSTALLATION to skip tests requiring an IntelMQ installation on the test host by default (PR#2370 by Sebastian Wagner, fixes #2369)

Tools

  • intelmqsetup:
    • SECURITY: fixed a low-risk bug causing the tool to change owner of / if run with the INTELMQ_PATHS_NO_OPT environment variable set. This affects only the PIP package as the DEB/RPM packages don't contain this tool. (PR#2355 by Kamil Mańkowski, fixes #2354)
  • contrib.eventdb.separate-raws-table.sql: Added the missing commas to complete the sql syntax. (PR#2386, fixes #2125 by Sebastian Kufner)
  • intelmq_psql_initdb:
    • Added parameter -o to set the output file destination. (by Sebastian Kufner)
  • intelmqctl:
    • Increased the performance through removing unnecessary reads. (by Sebastian Kufner)

Known Issues

This is short list of the most important known issues. The full list can be retrieved from GitHub.

  • intelmq.parsers.html_table may not process invalid URLs in patched Python version due to changes in urllib (#2382).
  • Breaking changes in 'rt' library (#2367).
  • Stomp collector failed (#2342).
  • Type error with SQL output bot's prepare_values returning list instead of tuple (#2255).
  • intelmq_psql_initdb does not work for SQLite (#2202).
  • intelmqsetup: should install a default state file (#2175).
  • Misp Expert - Crash if misp event already exist (#2170).
  • Turris greylist has been updated (#2167).
  • Spamhaus CERT parser uses wrong field (#2165).
  • Custom headers ignored in HTTPCollectorBot (#2150).
  • intelmqctl log: parsing syslog does not work (#2097).
  • Bash completion scripts depend on old JSON-based configuration files (#2094).
  • Bot configuration examples use JSON instead of YAML (#2066).
  • Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952).
  • Corrupt dump files when interrupted during writing (#870).

Contributors

elsif2
Assets 2
Loading
0 Join discussion