Skip to content

Bugfix release
Compare
Choose a tag to compare
released this 26 May 10:58
· 2311 commits to develop since this release
2.1.3
32fb175

Installation documentation:
https://github.com/certtools/intelmq/blob/2.1.3/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.3/docs/UPGRADING.md

Requirements

  • The python library requests is (again) listed as dependency of the core (#1519).

Core

  • intelmq.lib.upgrades:
    • Harmonization upgrade: Also check and update regular expressions.
    • Add function to migrate the deprecated parameter attach_unzip to extract_files for the mail attachment collector.
    • Add function to migrate changed Taichung URL feed.
    • Check for discontinued Abuse.CH Zeus Tracker feed.
  • intelmq.lib.bot:
    • ParserBot.recover_line: Parameter line needs to be optional, fix usage of fallback value self.current_line.
    • start: Handle decoding errors in the pipeline different so that the bot is not stuck in an endless loop (#1494).
    • start: Only acknowledge a message in case of errors, if we actually had a message to dump, which is not the case for collectors.
    • _dump_message: Dump messages with encoding errors base64 encoded, not in JSON format as it's not possible to decode them (#1494).
  • intelmq.lib.test:
    • BotTestCase.run_bot: Add parameters allowed_error_count and allowed_warning_count to allow set the number per run, not per test class.
    • Set source_pipeline_broker and destination_pipeline_broker to pythonlist instead of the old broker, fixes intelmq.tests.lib.test_bot.TestBot.test_pipeline_raising.
    • Fix test for (allowed) errors and warnings.
  • intelmq.lib.exceptions:
    • InvalidKey: Add KeyError as parent class.
    • DecodingError: Added, string representation has all relevant information on the decoding error, including encoding, reason and the affected string (#1494).
  • intelmq.lib.pipeline:
    • Decode messages in Pipeline.receive not in the implementation's _receive so that the internal counter is correct in case of decoding errors (#1494).
  • intelmq.lib.utils:
    • decode: Raise new DecodingError if decoding fails.

Harmonization

  • protocol.transport: Adapt regular expression to allow the value nvp-ii (protocol 11).

Bots

Collectors

  • intelmq.bots.collectors.mail.collector_mail_attach:
    • Fix handling of deprecated parameter name attach_unzip.
    • Fix handling of attachments without filenames (#1538).
  • intelmq.bots.collectors.stomp.collector: Fix compatibility with stomp.py versions > 4.1.20 and catch errors on shutdown.
  • intelmq.bots.collectors.microsoft:
    • Update REQUIREMENTS.txt temporarily fixing deprecated Azure library (#1530, PR#1532).
    • intelmq.bots.collectors.microsoft.collector_interflow: Add method for printing the file list.

Parsers

  • intelmq.bots.parsers.cymru.parser_cap_program: Support for protocol 11 (nvp-ii) and conficker type.
  • intelmq.bots.parsers.taichung.parser: Support more types/classifications:
    • Application Compromise: Apache vulnerability & SQL injections
    • Brute-force: MSSQL & SSH password guess attacks; Office 365, SSH & SIP attacks
    • C2 Sever: Attack controller
    • DDoS
    • DoS: DNS, DoS, Excess connection
    • IDS Alert / known vulnerability exploitation: backdoor
    • Malware: Malware Proxy
    • Warn on new unknown types.
  • intelmq.bots.parsers.bitcash.parser: Removed as feed is discontinued.
  • intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc and intelmq.bots.parsers.fraunhofer.parser_ddosattack_target: Removed as feed is discontinued.
  • intelmq.bots.parsers.malwaredomains.parser: Correctly classify C&C and phishing events.
  • intelmq.bots.parsers.shadowserver.parser: More verbose error message for missing report specification (#1507).
  • intelmq.bots.parsers.n6.parser_n6stomp: Always add n6 field name as malware.name independent of category.
  • intelmq.bots.parsers.anubisnetworks: Update parser with new data format.
  • intelmq.bots.parsers.bambenek: Add new feed URLs with Host faf.bambenekconsulting.com (#1525, PR#1526).
  • intelmq.bots.parsers.abusech.parser_ransomware: Removed, as the feed is discontinued (#1537).
  • intelmq.bots.parsers.nothink.parser: Removed, as the feed is discontinued (#1537).
  • intelmq.bots.parsers.n6.parser: Remove not allowed characters in the name field for malware.name and write original value to event_description.text instead.

Experts

  • intelmq.bots.experts.cymru_whois.lib: Fix parsing of AS names with Unicode characters.

Outputs

  • intelmq.bots.outputs.mongodb:
    • Set default port 27017.
    • Use different authentication mechanisms per MongoDB server version to fix compatibility with server version >= 3.4 (#1439).

Documentation

  • Feeds:
    • Remove unavailable feed Abuse.CH Zeus Tracker.
    • Remove the field status, offline feeds should be removed.
    • Add a new field public to differentiate between private and public feeds.
    • Adding documentation URLs to nearly all feeds.
    • Remove unavailable Bitcash.cz feed.
    • Remove unavailable Fraunhofer DDos Attack feeds.
    • Remove unavailable feed Abuse.CH Ransomware Tracker (#1537).
    • Update information on Bambenek Feeds, many require a license now (#1525).
    • Remove discontinued Nothink Honeypot Feeds (#1537).
  • Developers Guide: Fix the instructions for /opt/intelmq file permissions.

Packaging

  • Patches: fix-logrotate-path.patch: also include path to rotated file in patch.
  • Fix paths from /opt to LSB for setup.py and contrib/logrotate/intelmq in build process (#1500).
  • Add runtime dependency debianutils for the program which, which is required for intelmqctl.

Tests

  • Dropping Travis tests for 3.4 as required libraries dropped 3.4 support.
  • intelmq.tests.bots.experts.cymru_whois:
    • Drop missing ASN test, does not work anymore.
    • IPv6 to IPv4 test: Test for two possible results.
  • intelmq.lib.test: Fix compatibility of logging capture with Python >= 3.7 by reworking the whole process (#1342).
  • intelmq.bots.collectors.tcp.test_collector: Removing custom mocking and bot starting, not necessary anymore.
  • Added tests for intelmq.bin.intelmqctl.IntelMQProcessManager._interpret_commandline.
  • Fix and split tests.bots.experts.ripe.test_expert.test_ripe_stat_error_json.
  • Added tests for invalid encodings in input messages in intelmq.tests.lib.test_bot and intelmq.tests.lib.test_pipeline (#1494).
  • Travis: Explicitly enable RabbitMQ management plugin.
  • intelmq.tests.lib.test_message: Fix usage of the parameter blacklist for Message hash tests (#1539).

Tools

  • intelmqsetup: Copy missing BOTS file to IntelMQ's root directory (#1498).
  • intelmq_gen_docs: Feed documentation generation: Handle missing/empty parameters.
  • intelmqctl:
    • IntelMQProcessManager: For the status of running bots also check the bot ID of the commandline and ignore the path of the executable (#1492).
    • IntelMQController: Fix exit codes of check command for JSON output (now 0 on success and 1 on error, was swapped, #1520).
  • intelmqdump:
    • Handle base64-type messages for show, editor and recovery actions.

Contrib

  • intelmq/bots/experts/asn_lookup/update-asn-data: Use pyasn_util_download.py to download the data instead from RIPE, which cannot be parsed currently (#1517, PR#1518, hadiasghari/pyasn#62).

Known issues

  • HTTP stream collector: retry on regular connection problems? (#1435).
  • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
  • Reverse DNS: Only first record is used (#877).
  • Corrupt dump files when interrupted during writing (#870).
Assets 2
Loading