Releases: certtools/intelmq
3.3.1 Bugfix release
Documentation: https://docs.intelmq.org/latest/
Core
intelmq.lib.utils.drop_privileges
: When IntelMQ is called asroot
and dropping the privileges to userintelmq
, also set the non-primary groups associated with theintelmq
user. Makes the behaviour of running intelmqctl asroot
closer to the behaviour ofsudo -u intelmq ...
(PR#2507 by Mikk Margus Möll).intelmq.lib.utils.unzip
: Ignore directories themselves when extracting data to prevent the extraction of empty data for a directory entries (PR#2512 by Kamil Mankowski).
Bots
Collectors
intelmq.bots.collectors.shadowserver.collector_reports_api.py
:- Added support for the types parameter to be either a string or a list (PR#2495 by elsif2).
- Refactored to utilize the type field returned by the API to match the requested types instead of a sub-string match on the filename.
- Fixed timezone issue for collecting reports (PR#2506 by elsif2).
- Fixed behaviour if parameter
reports
value is empty string, behave the same way as not set, not like no report (PR#2523 by Sebastian Wagner).
intelmq.bots.collectors.shodan.collector_stream
(PR#2492 by Mikk Margus Möll):- Add
alert
parameter to Shodan stream collector to allow fetching streams by configured alert ID
- Add
intelmq.bots.collectors.mail._lib
: Remove deprecated parameterattach_unzip
from default parameters (PR#2511 by Sebastian Wagner).
Parsers
intelmq.bots.parsers.shadowserver._config
:- Fetch schema before first run (PR#2482 by elsif2, fixes #2480).
intelmq.bots.parsers.dataplane.parser
: Use|
as field delimiter, fix parsing of AS names including|
(PR#2488 by DigitalTrustCenter).- all parsers: add
copy_collector_provided_fields
parameter allowing copying additional fields from the report, e.g.extra.file_name
.
(PR#2513 by Kamil Mankowski).
Experts
intelmq.bots.experts.sieve.expert
:- For
:contains
,=~
and!~
, convert the value to string before matching avoiding an exception. If the value is a dict, convert the value to JSON (PR#2500 by Sebastian Wagner). - Add support for variables in Sieve scripts (PR#2514 by Mikk Margus Möll, fixes #2486).
- For
intelmq.bots.experts.filter.expert
:- Treat value
false
for parameterfilter_regex
as false (PR#2499 by Sebastian Wagner).
- Treat value
Outputs
intelmq.bots.outputs.misp.output_feed
: Handle failures if saved current event wasn't saved or is incorrect (PR by Kamil Mankowski).intelmq.bots.outputs.smtp_batch.output
: Documentation on multiple recipients added (PR#2501 by Edvard Rejthar).
Documentation
- Bots: Clarify some section of Mail collectors and the Generic CSV Parser (PR#2510 by Sebastian Wagner).
Known Issues
This is short list of the most important known issues. The full list can be retrieved from GitHub.
intelmq.parsers.html_table
may not process invalid URLs in patched Python version due to changes inurllib
(#2382).- Breaking changes in 'rt' 3.0 library (#2367).
- Type error with SQL output bot's
prepare_values
returning list instead of tuple (#2255). intelmq_psql_initdb
does not work for SQLite (#2202).- intelmqsetup: should install a default state file (#2175).
- Misp Expert - Crash if misp event already exist (#2170).
- Spamhaus CERT parser uses wrong field (#2165).
- Custom headers ignored in HTTPCollectorBot (#2150).
- intelmqctl log: parsing syslog does not work (#2097).
- Bash completion scripts depend on old JSON-based configuration files (#2094).
- Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952).
- Corrupt dump files when interrupted during writing (#870).
3.3.0
3.3.0 (2024-03-01)
Configuration
- Add new optional configuration parameters for
intelmq.bots.collectors.stomp.collector
andintelmq.bots.outputs.stomp.output
(PR#2408 by Jan Kaliszewski):auth_by_ssl_client_certificate
(Boolean, default: true; if false then
ssl_client_certificate
andssl_client_certificate_key
will be ignored);username
(STOMP authentication login, default: "guest"; to be used only
ifauth_by_ssl_client_certificate
is false);password
(STOMP authentication passcode, default: "guest"; to be used only
ifauth_by_ssl_client_certificate
is false).
- Add the possibility to set the
ssl_ca_certificate
configuration parameter for
intelmq.bots.collectors.stomp.collector
and/orintelmq.bots.outputs.stomp.output
to an empty string - which means that the SSL machinery used for STOMP communication
will attempt to load the system’s default CA certificates (PR#2414 by Jan Kaliszewski).
Core
intelmq.lib.message
: For invalid message keys, add a hint on the failure to the exception: not allowed by configuration or not matching regular expression (PR#2398 by Sebastian Wagner).intelmq.lib.exceptions.InvalidKey
: Add optional parameteradditional_text
(PR#2398 by Sebastian Wagner).- Change the way we discover bots to allow easy extending based on the entry point name. (PR#2413 by Kamil Mankowski)
intelmq.lib.mixins
: Add a new class,StompMixin
(defined in a new submodule:stomp
),
which provides certain common STOMP-bot-specific operations, factored out from
intelmq.bots.collectors.stomp.collector
andintelmq.bots.outputs.stomp.output
(PR#2408 and PR#2414 by Jan Kaliszewski).intelmq.lib.upgrades
: Replace deprecated instances ofurl2fqdn
experts by the newurl
expert in runtime configuration (PR#2432 by Sebastian Wagner).intelmq.lib.bot
: Ensure closing log files on reloading (PR#2435 by Kamil Mankowski).- AMQP Pipeline: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).
- Only load the config once when starting intelmqctl (which makes IntelMQ API calls take less time) (PR#2444 by DigitalTrustCenter).
Development
- Makefile: Add codespell and test commands (PR#2425 by Sebastian Wagner).
Data Format
Bots
Collectors
intelmq.bots.collectors.stomp.collector
(PR#2408 and PR#2414 by Jan Kaliszewski):- Drop support for versions of
stomp.py
older than4.1.12
. - Update the code to support new versions of
stomp.py
, including the latest (8.1.0
);
fixes #2342. - Add support for authentication based on STOMP login and passcode, introducing three
new configuration parameters (see above: Configuration). - Add support for loading the system’s default CA certificates, as an alternative to
specifying the CA certificate(s) file path explicitly (see above: Configuration). - Fix (by carefully targeted monkey patching) certain security problems caused by
SSL-related weaknesses that some versions ofstomp.py
suffer from. - Fix the reconnection behavior: do not attempt to reconnect after
shutdown
. Also,
never attempt to reconnect if the version ofstomp.py
is older than4.1.21
(it
did not work properly anyway). - Add coercion of the
port
config parameter toint
. - Add implementation of the
check
hook (verifying, in particular, accessibility
of necessary file(s)). - Remove undocumented and unused attributes of
StompCollectorBot
instances:
ssl_ca_cert
,ssl_cl_cert
,ssl_cl_cert_key
.- Minor fixes/improvements and some refactoring (see also above: Core...).
- Drop support for versions of
intelmq.bots.collectors.amqp
: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).intelmq.bots.collectors.shadowserver.collector_reports_api
:- The 'json' option is no longer supported as the 'csv' option provides better performance (PR#2372 by elsif2).
intelmq.bots.collectors.alienvault_otx.collector
(PR#2449 by qux-bbb):- Fix modified_pulses_only is always False.
Parsers
intelmq.bots.parsers.netlab_360.parser
: Removed as the feed is discontinued. (#2442 by Filip Pokorný)intelmq.bots.parsers.webinspektor.parser
: Removed as the feed is discontinued. (#2442 by Filip Pokorný)intelmq.bots.parsers.sucuri.parser
: Removed as the feed is discontinued. (#2442 by Filip Pokorný)intelmq.bots.parsers.shadowserver._config
:- Switch to dynamic configuration to decouple report schema changes from IntelMQ releases by regularly downloading them from the Shadowserver server (PR#2372 by elsif2).
intelmq.bots.parsers.cymru
: Save current line. (PR by Kamil Mankowski)
Experts
intelmq.bots.experts.jinja
(PR#2417 by Mikk Margus Möll):- Add optional
socket_perms
andsocket_group
parameters to change
file permissions on socket file, if it is in use.
- Add optional
intelmq.bots.experts.ripe
(PR#2461 by Mikk Margus Möll):- Handle "No abuse contact found for" messages for non-ASN resources
Outputs
intelmq.bots.outputs.stomp.output
(PR#2408 and PR#2414 by Jan Kaliszewski):- Drop support for versions of
stomp.py
older than4.1.12
. - Update the code to support new versions of
stomp.py
, including the latest (8.1.0
). - Add support for authentication based on STOMP login and passcode, introducing three
new configuration parameters (see above: Configuration). - Add support for loading the system’s default CA certificates, as an alternative to
specifying the CA certificate(s) file path explicitly (see above: Configuration). - Fix (by carefully targeted monkey patching) certain security problems caused by
SSL-related weaknesses that some versions ofstomp.py
suffer from. - Fix
AttributeError
caused by attempts to get unset attributes ofStompOutputBot
(ssl_ca_cert
et consortes). - Add coercion of the
port
config parameter toint
. - Add implementation of the
check
hook (verifying, in particular, accessibility
of necessary file(s)). - Add
stomp.py
version check (raiseMissingDependencyError
if not>=4.1.12
). - Minor fixes/improvements and some refactoring (see also above: Core...).
- Drop support for versions of
intelmq.bots.outputs.stomp.output
(PR#2423 by Kamil Mankowski):- Try to reconnect on
NotConnectedException
.
- Try to reconnect on
intelmq.bots.outputs.smtp_batch.output
(PR #2439 by Edvard Rejthar):- Fix ability to send with the default
bcc
- Fix ability to send with the default
intelmq.bots.outputs.amqp
: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).
Documentation
- Add a readthedocs configuration file to fix the build fail (PR#2403 by Sebastian Wagner).
- Add a guide of developing extensions packages (PR#2413 by Kamil Mankowski)
- Update/fix/improve the stuff related to the STOMP bots and integration with the n6's
Stream API (PR#2408 and PR#2414 by Jan Kaliszewski). - Complete documentation overhaul. Change to markdown format. Uses the mkdocs-material (PR#2419 by Filip Pokorný).
- Adds warning banner if not browsing the latest version of the docs (PR#2445 by Filip Pokorný).
- Fix logo path in index.md when building the docs (PR#2445 by Filip Pokorný).
Packaging
- Add
pendulum
to suggested packages, as it is required for the sieve bot (PR#2424 by Sebastian Wagner). debian/control
: inSuggests
field, replacepython3-stomp.py (>= 4.1.9)
with
python3-stomp (>= 4.1.12)
, i.e., fix the package name by removing the.py
suffix and bump the minimum version to4.1.12
(PR#2414 by Jan Kaliszewski).
Tests
Tools
intelmq_psql_initdb
:- got support for providing custom harmonization file, generating view for storing
raw
fields separately, and addingIF NOT EXISTS
/OR REPLACE
clauses (PR#2404 by Kamil Mankowski). - got support for generating JSONB fields for PostgreSQL schema (PR#2436 by Kamil Mankowski).
- got support for providing custom harmonization file, generating view for storing
Contrib
Known issues
3.2.1 Important Bugfixes
Installation: https://intelmq.readthedocs.io/en/develop/user/installation.html
Upgrade: https://intelmq.readthedocs.io/en/develop/user/upgrade.html
All Bots
Fixes an issue which prevented bots from stopping gracefully after reloading.
As logrotate reloads all bots regularly, this bug affects most IntelMQ installations.
Reverse DNS Expert
Until IntelMQ version 3.2.0, the bot incorrectly cached and re-used results for /24 networks instead of single IP addresses.
If the bot retrieved the PTR for 192.0.43.7
, it was cached for 192.0.43.0/24
and used for all IP addresses in this range, for example for 192.0.43.8
.
IntelMQ version 3.2.1 fixes this issue.
The bugfix will correctly increase the cache sizes and decrease the performance, as less (incorrect) data is re-used.
3.2.0 Feature release: Running IntelMQ bots as Python Library
IEP007: Running IntelMQ bots as Python Library is implemented.
Installation: https://intelmq.readthedocs.io/en/develop/user/installation.html
Upgrade: https://intelmq.readthedocs.io/en/develop/user/upgrade.html
The accompanying 3.2.0 release of intelmq-api switches it's backend from the library hug to fastapi.
Deb-packages of intelmq-api 3.2.0 are delayed for some distributions because of necessary changes in packaging.
Core
intelmq.lib.utils
:resolve_dns
: Deprecate dnspython versions pre-2.0.0 and disable search domains (PR#2352)
- Fixed not resetting destination path statistics in the stats cache after restarting bot (Fixes #2331)
- Force flushing statistics if bot will sleep longer than flushing delay (Fixes #2336)
intelmq.lib.upgrages
: Fix a bug in the upgrade function for version 3.1.0 which caused an exception if a generic csv parser instance had no parametertype
(PR#2319 by Filip Pokorný).intelmq.lib.datatypes
: AddsTimeFormat
class to be used for thetime_format
bot parameter (PR#2329 by Filip Pokorný).intelmq.lib.exceptions
: Fixes a bug inInvalidArgument
exception (PR#2329 by Filip Pokorný).intelmq.lib.harmonization
:- Changes signature and names of
DateTime
conversion functions for consistency, backwards compatible (PR#2329 by Filip Pokorný). - Ensure rejecting URLs with leading whitespaces after changes in CPython (fixes #2377)
- Changes signature and names of
intelmq.lib.bot.Bot
: Allow setting the parameters via parameter on bot initialization.
Development
- CI: pin the Codespell version to omit troubles caused by its new releases (PR #2379).
Bots
Collectors
intelmq.bots.collector.rt
:- restrict
python-rt
to be below version 3.0 due to introduced breaking changes, - added support for
Subject NOT LIKE
queries, - added support for multiple values in ticket subject queries.
- restrict
intelmq.bots.collectors.rsync
: Support for optional private key, relative time parsing for the source path, extra rsync parameters and strict host key checking (PR#2241 by Mateo Durante).
Parsers
intelmq.bots.parsers.shadowserver._config
:intelmq.bots.parsers.shadowserver._config
:- Added 'IPv6-Vulnerable-Exchange' alias and 'Accessible-WS-Discovery-Service' report. (PR#2338)
- Removed unused
p0f_genre
andp0f_detail
from the 'DNS-Open-Resolvers' report. (PR#2338) - Added 'Accessible-SIP' report. (PR#2348)
- Added 'IPv6-Open-HTTP-Proxy' and 'IPv6-Accessible-HTTP-Proxy' aliases. (PR#2348)
- Removed duplicate mappings from the 'Spam-URL' report. (PR#2348)
intelmq.bots.parsers.generic.parser_csv
: Changestime_format
parameter to use newTimeFormat
class (PR#2329 by Filip Pokorný).intelmq.bots.parsers.html_table.parser
: Changestime_format
parameter to use newTimeFormat
class (PR#2329 by Filip Pokorný).intelmq.bots.parsers.turris.parser.py
Updated to the latest data format (issue #2167). (PR#2373 by Filip Pokorný).
Experts
intelmq.bots.experts.sieve
:- Allow empty lists in sieve rule files (PR#2341 by Mikk Margus Möll).
intelmq.bots.experts.cymru_whois
:- Ignore AS names with unexpected unicode characters (PR#2352, fixes #2132)
- Avoid extraneous search domain-based queries on NXDOMAIN result (PR#2352)
intelmq.bots.experts.sieve
:- Added :before and :after keywords (PR#2374)
Outputs
intelmq.bots.outputs.cif3.output
: Added (PR#2244 by Michael Davis).intelmq.bots.outputs.sql.output
: New parameterfail_on_errors
(PR#2362 by Sebastian Wagner).intelmq.bots.outputs.smtp_batch.output
: Added a bot to gathering the events and sending them by e-mails at a stroke as CSV files (PR#2253 by Edvard Rejthar)
Documentation
- API: update API installation to be aligned with the rewritten API, and clarify some missing steps.
Tests
- New decorator
skip_installation
and environment variableINTELMQ_TEST_INSTALLATION
to skip tests requiring an IntelMQ installation on the test host by default (PR#2370 by Sebastian Wagner, fixes #2369)
Tools
intelmqsetup
:- SECURITY: fixed a low-risk bug causing the tool to change owner of
/
if run with theINTELMQ_PATHS_NO_OPT
environment variable set. This affects only the PIP package as the DEB/RPM packages don't contain this tool. (PR#2355 by Kamil Mańkowski, fixes #2354)
- SECURITY: fixed a low-risk bug causing the tool to change owner of
contrib.eventdb.separate-raws-table.sql
: Added the missing commas to complete the sql syntax. (PR#2386, fixes #2125 by Sebastian Kufner)intelmq_psql_initdb
:- Added parameter
-o
to set the output file destination. (by Sebastian Kufner)
- Added parameter
intelmqctl
:- Increased the performance through removing unnecessary reads. (by Sebastian Kufner)
Known Issues
This is short list of the most important known issues. The full list can be retrieved from GitHub.
intelmq.parsers.html_table
may not process invalid URLs in patched Python version due to changes inurllib
(#2382).- Breaking changes in 'rt' library (#2367).
- Stomp collector failed (#2342).
- Type error with SQL output bot's
prepare_values
returning list instead of tuple (#2255). intelmq_psql_initdb
does not work for SQLite (#2202).- intelmqsetup: should install a default state file (#2175).
- Misp Expert - Crash if misp event already exist (#2170).
- Turris greylist has been updated (#2167).
- Spamhaus CERT parser uses wrong field (#2165).
- Custom headers ignored in HTTPCollectorBot (#2150).
- intelmqctl log: parsing syslog does not work (#2097).
- Bash completion scripts depend on old JSON-based configuration files (#2094).
- Bot configuration examples use JSON instead of YAML (#2066).
- Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952).
- Corrupt dump files when interrupted during writing (#870).
3.1.0 Feature Release
Installation documentation:
https://intelmq.readthedocs.io/en/maintenance/user/installation.html
Upgrade documentation:
https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html
- Upgraded syntax to Python 3.6 (mostly Format-Strings) using pyuprade (PR#2136 by Sebastian Wagner).
Core
intelmq.lib.upgrades
:- Refactor upgrade functions global configuration handling removing the old-style defaults configuration (PR#2058 by Sebastian Wagner).
- Pass version history as parameter to upgrade functions (PR#2058 by Sebastian Wagner).
intelmq.lib.message
:- Fix and pre-compile the regular expression for harmonization key names and also check keys in the
extra.
namespace (PR#2059 by Sebastian Wagner, fixes #1807).
- Fix and pre-compile the regular expression for harmonization key names and also check keys in the
intelmq.lib.bot.SQLBot
was replaced by an SQLMixin inintelmq.lib.mixins.SQLMixin
. The Generic DB Lookup Expert bot and the SQLOutput bot were updated accordingly.- Added support for MSSQL (PR#2171 by Karl-Johan Karlsson).
- Added optional reconnect delay parameter (PR#2171 by Karl-Johan Karlsson).
- Added an ExpertBot class - it should be used by all expert bots as a parent class
- Introduced a module for IntelMQ related datatypes
intelmq.lib.datatypes
which for now only contains an Enum listing the four bot types - Added a
bottype
attribute to CollectorBot, ParserBot, ExpertBot, OutputBot - Introduces a module for IntelMQ processmanagers. The processmanagers were up until now part of the intelmqct script.
They now reside inintelmq.lib.processmanager
which also contains an interface definition the processmanager implementations must adhere to.
Both the processmanagers and theintelmqctl
script were cleaned up a bit.
TheLogLevel
andReturnType
Enums were added tointelmq.lib.datatypes
. intelmq.lib.bot
:- Enhance behaviour if an unconfigured bot is started (PR#2054 by Sebastian Wagner).
- Fix line recovery and message dumping of the
ParserBot
(PR#2192 by Sebastian Wagner).- Previously the dumped message was always the last message of a report if the report contained multiple lines leading to data-loss.
- Fix crashing at start in multithreaded bots (PR#2236 by DigitalTrustCenter).
- Added
default_fields
parameter toParserBot
(PR#2293 by Filip Pokorný)
intelmq.lib.pipeline
:intelmq.lib.utils
:- Added wrapper
resolve_dns
for querying DNS, with the support for recommended methods fromdnspython
package in versions 1 and 2. - Moved line filtering inside
RewindableFileHandle
for easier handling and limiting number of temporary objects.
- Added wrapper
intelmq.lib.harmonization
:- Fixed DateTime handling of naive time strings (previously assumed local timezone, now assumes UTC) (PR#2279 by Filip Pokorný, fixes #2278)
- Removes
tzone
argument fromDateTime.from_timestamp
andDateTime.from_epoch_millis
DateTime.from_timstamp
now also allows string argument
- Removes
pytz
global dependency - Removed support for Python 3.6, including removing conditional dependencies and updating syntax to use features from newest versions. (fixes #2272)
Development
- Removed Python 3.6 from CI.
- Enabled tests against Python 3.11.
Bots
- Set the parent class of all bots to the correct bot class
Collectors
intelmq.bots.collectors.mail._lib
:- Add support for unverified SSL/STARTTLS connections (PR#2055 by Sebastian Wagner).
- Fix exception handling for aborted IMAP connections (PR#2187 by Sebastian Wagner).
intelmq.bots.collectors.blueliv
: Fix Blueliv collector requirements (PR#2161 by Gethvi).intelmq.bots.collectors.github_api._collector_github_api
: Added personal access token support (PR#2145 by Sebastian Waldbauer, fixes #1549).intelmq.bots.collectors.file.collector_file
: Added file lock support, no more race conditions (PR#2147 by Sebastian Waldbauer, fixes #2128)intelmq.bots.collectors.shadowserver.collector_reports_api.py
: Added file_format option to download reports in CSV format for better performance (PR#2246 by elsif2)
Parsers
-
intelmq.bots.parsers.alienvault.parser_otx
: Save CVE data inextra.cve
instead ofextra.CVE
due to the field name restriction on lower-case characters (PR#2059 by Sebastian Wagner). -
intelmq.bots.parsers.anubisnetworks.parser
: Changed field name format fromextra.communication.http.x_forwarded_for_#1
toextra.communication.http.x_forwarded_for_1
due to the field name restriction on alphanumeric characters (PR#2059 by Sebastian Wagner). -
intelmq.bots.parsers.dataplane.parser
:- Add support for additional feeds (PR#2102 by Mikk Margus Möll).
- DNS Recursion Desired
- DNS Recursion Desired ANY
- DNS Version
- Protocol 41
- SMTP Greet
- SMTP Data
- Telnet Login
- VNC/RFB Login
- Fix event object creation (PR#2298 by DigitalTrustCenter).
- Add support for additional feeds (PR#2102 by Mikk Margus Möll).
-
Removed
intelmq.bots.parsers.malc0de
: this bot was marked as deprecated and removed from feed due to offline status (PR#2184 by Tamas Gutsohn, fixes #2178). -
intelmq.bots.parsers.microsoft.parser_ctip
:- New parameter
overwrite
(PR#2112 by Sebastian Wagner, fixes #2022). - Fix handling of field
Payload.domain
if it contains the same IP address asPayload.serverIp
(PR#2144 by Mikk Margus Möll and Sebastian Wagner). - Handle Payload field with non-base64-encoded JSON content and numbered dictionaries (PR#2193 by Sebastian Wagner)
- New parameter
-
intelmq.bots.parsers.shodan.parser
(PR#2117 by Mikk Margus Möll):- Instead of keeping track of
extra.ftp.<something>.parameters
, FTP parameters are collected together intoextra.ftp.features
as a list of said features, reducing field count. - Shodan field
rsync.modules
is collected. - Conversion functions can raise
NoValueException
with a string argument to signify that the conversion would not succeed, such as in the case of a single IP address being given in hostnames, which would then be passed intosource.reverse_dns and
fail to validate as a FQDN. - Variable
_common_keys
is moved out of the class. _dict_dict_to_obj_list
is introduced, for converting a string-to-dict mapping into a list of dicts with the previous key as an attribute of the dict; this can be useful for preventing issues where, when feeding the data into aggregating tools, you'd end up with many more fields than necessary, e.gvulns.CVE-2010-0001.cvss
,CVE-2010-0002.cvss
etc._get_first
to get the first item from a list, withNoValueException
raised on empty lists._get_first_hostname
to handle the first valid FQDN from a list of hostnames for hostnames in the Shodan banner, if there is one, and givesNoValueException
otherwise.ssl.cert.serial
andssl.dhparams.generator
, which may return both integers and strings, are converted to strings.- Changes to method
apply_mapping
, such as reducing needless loop iterations, removing a big try-except, and adding theNoValueException
handling described above. - Stops falsy values (False, 0) besides None from being filtered out.
- Instead of keeping track of
-
intelmq.bots.parsers.shadowserver._config
:- Added support for
Accessible AMQP
,Device Identification Report
(IPv4 and IPv6) (PR#2134 by Mateo Durante). - Added file name mapping for
SSL-POODLE-Vulnerable-Servers IPv6
(file namescan6_ssl_poodle
) (PR#2134 by Mateo Durante). - Added
Malware-URL
,Sandbox-Connection
,Sandbox-DNS
,Accessible-AMQP
,Open-AnonymouIs-MQTT
,Accessible-QUIC
,Accessible-SSH
,SYNful-Knock
, andSpecial
(PR#2227 by elsif2) - Removed legacy reports
Amplification-DDoS-Victim
,CAIDA-IP-Spoofer
,Darknet
,Drone
,Drone-Brute-Force
,IPv6-Sinkhole-HTTP-Drone
,Microsoft-Sinkhole
, andSinkhole-HTTP-Drone
(PR#2227 by elsif2). - Users storing events in a database should be aware that field names and types have been updated (PR#2227 by elsif2).
- Corrected "Accessible-AMQP" message_length type (int) and added "STUN" support (PR#2235 by elsif2).
- Added amplification factor to UDP scan reports (PR#2238 by elsif2).
- Added version and build_date to "Vulnerable-HTTP" report (PR#2238 by elsif2).
- The following field types have been standardized across all Shadowserver reports (PR#2246 by elsif2):
destination.fqdn (validate_fqdn)
destination.url (convert_http_host_and_url)
extra.browser_trusted (convert_bool)
extra.duration (convert_int)
extra.end_time (convert_date_utc)
extra.freak_vulnerable (convert_bool)
extra.ok (convert_bool)
extra.password (validate_to_none)
extra.ssl_poodle (convert_bool)
extra.status (convert_int)
extra.uptime (convert_int)
extra.version (convert_to_none)
source.network (validate_network) - The following report field names have changed to better represent their values:
scan_rsync:extra.password renamed to extra.has_password
scan_elasticsearch:status renamed to http_code - Added
Accessible-HTTP-proxy
andOpen-HTTP-proxy
(PR#2246 by elsif2). - Added http_agent to the
Honeypot-DDoS
report and added theDDoS-Participant
report (PR#2303 by elsif2) - Added
Accessible-SLP
,IPv6 Accesssible-SLP
,IPv6-DNS-Open-Resolvers
, andIPv6-Open-LDAP-TCP
reports (PR#2311 by elsif2) - Standardized response_length to response_size in
Accessible-ICS
andOpen-MSSQL
(PR#2311 by elsif2)
- Added support for
-
intelmq.bots.parsers.cymru.parser_cap_program
: The parser mapped the hostname intosource.fqdn
which is not allowed by the IntelMQ Data Format. Added a check (PR#2215 by Sebastian Waldbauer, fixes #2169) -
intelmq.bots.parsers.generic.parser_csv
:- Use RewindableFileHandle to u...
3.1.0 Release candidate
What's Changed
- BUG: utils/load_configuration: fix loading JSON with tabs in #2010
- CTIP parser: Handle TLP value "unknown" in #2009
- FIX: s/self.parameters/self/ in Shodan collector by @monoidic in #2020
- BUG: provide filename when reading a non-archived file from mail by @kapalex in #2021
- Add new bot: cut string by delimeter, max length by @mariuskarotkis in #1967
- Core fixes by @monoidic in #2027
- DOC: Fix typo by @schacht-certat in #2029
- Make update script less verbose by default by @schacht-certat in #2033
- Remove malwardomains parser by @schacht-certat in #2026
- Add new bot: domain valid by @mariuskarotkis in #1966
- Support Shadowserver Vulnerable SMTP server feed, fix #1984 by @monoidic in #2037
- Shadowserver feed updates by @schacht-certat in #2036
- Remove postgresql output bot by @schacht-certat in #2045
- DOC: remove mention of intelmqctl list bot json output by @schacht-certat in #2030
- Remove workarounds for XMPP bots by @schacht-certat in #2048
- Use the YAML unsafe loader instead of the safe loader by @schacht-certat in #2041
- Make the harmonization attribute of the bot private by @schacht-certat in #2049
- ENH: imap collector: support unverified connections in #2055
- ENH: utils: list_bots: strip descriptions in #2056
- lib/harm: minor change to allowed classification.types list in #2057
- ENH: small refactor of upgrade functions in #2058
- Add Honeypot HTTP Scan feed to Shadowserver parser by @monoidic in #2047
- IDF: Fix and enforce field name restrictions in #2059
- JinjaExpert - modify message fields using jinja2 by @schacht-certat in #2040
- Implement a clear class hierarchy by @schacht-certat in #2042
- DOCS: Docker-compose commands updated by @waldbauer-certat in #2035
- Refactoring intelmqctl by @schacht-certat in #2067
- ENH: lib/bot: fix behavior for unconfigured bots in #2054
- BUG: fix v300_pipeline_file_removal by @schacht-certat in #2083
- Adapt ansible workflow to ansible tests by @schacht-certat in #2081
- BUG: use correct private parameter variable by @schacht-certat in #2087
- DOCS: fix some sphinx warnings by @schacht-certat in #2065
- ENH: make json output indented to make it better readable by humans by @schacht-certat in #2079
- ENH: use the upstream ansible tasks playbook by @schacht-certat in #2089
- Return an INFO string if log file does not yet exist by @schacht-certat in #2100
- MAINT: update the url for the vagrant+ansible test repository by @schacht-certat in #2061
- FIX: statistics_host returns if no host value is given by @waldbauer-certat in #2104
- DOCS: Updated docker installation instructions by @waldbauer-certat in #2069
- BUG: fix logrotate ownership options in #2111
- Add support for new feeds to Dataplane parser by @monoidic in #2102
- Add RPZ file output bot by @mariuskarotkis in #1962
- Add new bot: bro output file by @mariuskarotkis in #1963
- DOC: restructure installation documentation in #2113
- ENH: ctip parser: added parameter overwrite in #2112
- Add new bot: cut string from string by @mariuskarotkis in #1965
- ENH: intelmqsetup: revise installation of manager in #2114
- DOC: enhance misp integration document in #2115
- ENH: Shodan parser: handle invalid hostnames, move _common_keys out of class, adjust apply_mapping by @monoidic in #2117
- Add JSON parser function to Jinja2 templates by @creideiki in #2120
- DOC: feeds.yaml: fix HIBP nginx config by @sebix in #2126
- FIX: Fixes update-database script on the last few days of a month. by @gethvi in #2121
- DOC: add changelog for PR#2121 by @sebix in #2129
- Docs: Remove dead IRC channel by @sebix in #2130
- Added new shadowserver files to _config.py by @mateodurante in #2134
- DOC: Update changelog and docs for shadowserver PR#2134 by @sebix in #2135
- doc: feeds whishlist: add misp warning lists by @sebix in #2141
- docs: feeds whishlist: add avast ioc repo by @sebix in #2139
- ENH: upgrade Python syntax to 3.6+ by @sebix in #2136
- MS CTIP Azure feed invalid hostname filtering by @monoidic in #2144
- TST: also test on python 3.10, switch to pytest by @sebix in #2142
- FIX: CodeQL exponential backtracking on strings by @waldbauer-certat in #2148
- FIX: Added vscode to gitignore & automated docker build with ACT by @waldbauer-certat in #2153
- FIX: threshold expert: configure timeout using Redis cache mixin by @creideiki in #2155
- FIX: Fixes Blueliv collector requirements by @gethvi in #2161
- Add fix Blueliv collector requirements to CHANGELOG.md by @gutsohnCERT in #2173
- FIX: Moved SQLBot to SQLMixin namespace in generic-db-lookup by @waldbauer-certat in #2146
- DOC: installation: remove EOL Fedora versions by @sebix in #2186
- BUG: mail collectors: fix IMAP abort handling by @wagner-intevation in #2187
- bug: fix ctl process manager initialization non-interactive by @wagner-intevation in #2189
- doc: ms azure collector: more precise requirement name by @wagner-intevation in #2190
- intelmqsetup: fix typo in message by @sebix in #2181
- gitignore: add coverage file by @sebix in #2182
- ENH: new parameter autoupdate_cached_database by @sebix in #2180
- tst: fix reverse_dns test by @wagner-intevation in #2208
- pkg: add requests_mock as dev dependency by @sebix in #2210
- intelmqsetup: Fix #2197 by @wagner-intevation in #2198
- Fix reconnection in SQL Mixin by @wagner-intevation in #2201
- ParserBot: Fix line recovery and message dumping by @wagner-intevation in #2192
- FIX: Added check if cymru hostname is an ip address by @waldbauer-certat in #2215
- bug: fix ctip parser handling JSON Payload content by @wagner-intevation in #2193
- docs: workaround for sphinx-doc/sphinx#10701 by @wagner-intevation in #2225
- FIX: Remove AutoShun from IntelMQ as the feed is discontinued by @waldbauer-certat in #2214
- SQL: add MSSQL support and allow storing only a subset of event fields by @creideiki in #2172
- ENH: Added personal access token support to GitHub Collector by @waldbauer-certat in #2145
- FIX: Use redis BLMOVE instead of BRPOPLPUSH (deprecated) by @waldbauer-certat in #2149
- postgresql output: escape all null bytes by @wagner-intevation in #2223
- Remove malc0de feed due to offline by @gutsohnCERT in #2184
- TST: Fixed regex to hold gcc version by @waldbauer-certat in #2216
- Shadowserver 202207 by @elsif2 in https://github.com/certtools/intelmq/p...
3.0.2 Bugfix release
Installation documentation:
https://intelmq.readthedocs.io/en/maintenance/user/installation.html
Upgrade documentation:
https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html
Core
intelmq.lib.bot.CollectorBot
: Fixed an issue with within thenew_report
function, which re-loads the harmonization file after a new incoming dataset, which leads to CPU drain and decreased performance (PR#2106 by Sebastian Waldbauer, fixes #2098).intelmq.lib.bot.Bot
: Make private members__is_multithreadable
and__collector_empty_process
protected members_is_multithreadable
and_collector_empty_process
to make them easily modifiable by Bot classes (PR#2109 by Sebastian Wagner, fixes #2108).
Also affected and adapted bots by this change are:intelmq.bots.collectors.api.collector_api
intelmq.bots.collectors.stomp.collector
intelmq.bots.experts.splunk_saved_search.expert
intelmq.bots.experts.threshold.expert
intelmq.bots.outputs.file.output
intelmq.bots.outputs.misp.output_api
intelmq.bots.outputs.misp.output_feed
intelmq.bots.outputs.tcp.output
intelmq.bots.outputs.udp.output
intelmq.lib.cache
: Do not create the Cache class if the host is null, allows deactivating the bot statistics (PR#2104 by Sebastian Waldbauer, fixes #2103).
Bots
Experts
intelmq.bots.experts.domain_suffix.expert
: Only print skipped database update message if verbose mode is active (PR#2107 by Sebastian Wagner, fixes #2016).
Documentation
- Add configuration upgrade steps for 3.0 to NEWS (PR#2101 by Sebastian Wagner).
Known issues
See open bug reports for a more detailed list.
- ParserBot: erroneous raw line recovery in error handling (#1850).
3.0.1 Bugfix Release
Installation documentation:
https://intelmq.readthedocs.io/en/maintenance/user/installation.html
Upgrade documentation:
https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html
Core
intelmq.lib.bot_debugger
: Fix accessing the bot's destination queues (PR#2027 by Mikk Margus Möll).intelmq.lib.pipeline
: Fix handling ofload_balance
parameter (PR#2027 by Mikk Margus Möll).intelmq.lib.bot
: Fix handling of parameterdestination_queues
if value is an empty dictionary (PR#2051 by Sebastian Wagner, fixes #2034).
Bots
Collectors
intelmq.bots.collectors.shodan.collector_stream
: Fix access to parameters, the bot wrongly usedself.parameters
(PR#2020 by Mikk Margus Möll).intelmq.bots.collectors.mail.collector_mail_attach
: Add attachment file name asextra.file_name
also if the attachment is not compressed (PR#2021 by Alex Kaplan).intelmq.bots.collectors.http.collector_http_stream
: Fix access to parameters, the bot wrongly usedself.parameters
(by Sebastian Wagner).
Parsers
intelmq.bots.parsers.microsoft.parser_ctip
: MapPayload.domain
todestination.fqdn
instead ofextra.payload.domain
as it matches todestination.ip
fromDestinationIp
(PR#2023 by Sebastian Wagner).- Removed
intelmq.bots.parsers.malwaredomains
because the upstream data source (malwaredomains.com) does not exist anymore (PR#2026 by Birger Schacht, fixes #2024). intelmq.bots.parsers.shadowserver.config
:- Add support for feed "Vulnerable SMTP Server" (PR#2037 by Mikk Margus Möll).
- Fix differentiation between feeds "Accessible HTTP" and "Vulnerable HTTP" (PR#2037 by Mikk Margus Möll, fixes #1984).
- Add support for the new feeds Microsoft Sinkhole Events Report, Microsoft Sinkhole HTTP Events Report (PR#2036 by Birger Schacht).
- Complement feed mappings and documentation for feeds with IPv4 and IPv6 variants (PR#2046 by Mikk Margus Möll and Sebastian Wagner).
- Feed names with and without the optional IPv4/IPv6 postfix can be used now consistently.
- Add support for feed "Honeypot HTTP Scan" (PR#2047 by Mikk Margus Möll).
- Update filename mapping for changed filename of feed "Accessible-MSRDPUDP" (PR#2060 by abr4xc).
Experts
intelmq.bots.experts.gethostbyname.expert
: Handle numeric values for thegaierrors_to_ignore
parameter (PR#2073 by Sebastian Wagner, fixes #2072).intelmq.bots.experts.filter.expert
: Fix handling of empty-string parametersnot_after
andnot_before
(PR#2075 by Sebastian Wagner, fixes #2074).
Outputs
intelmq.bots.outputs.mcafee.output_esm_ip
: Fix access to parameters, the bot wrongly usedself.parameters
(by Sebastian Wagner).intelmq.bots.outputs.misp.output_api
: Fix access to parameters, the bot wrongly usedself.parameters
(by Sebastian Wagner).intelmq.bots.outputs.smtp.output
: AddContent-Disposition
-header to the attachment, fixing the display in Mail Clients as actual attachment (PR#2052 by Sebastian Wagner, fixes #2018).
Documentation
- Various formatting fixes (by Sebastian Wagner).
- Removed the malwaredomains feed from the feeds list because the upstream data source (malwaredomains.com) does not exist anymore (PR#2026 by Birger Schacht, fixes #2024).
- Update Docker installation instructions (PR#2035 by Sebastian Waldbauer).
Packaging
- intelmq-update-database crontab: Add missing
recordedfuture_iprisk
update call (by Sebastian Wagner).
Tests
- Replace calls to deprecated/undocumented
logging.warn
withlogging.warning
(by Sebastian Wagner, fixes #2013). intelmq.tests.bots.experts.rdap.test_expert
: Declare cache use, fixes build failures (by Sebastian Wagner, fixes #2014).intelmq.tests.bots.collectors.mail.test_collector_attach
: Test text attachment (by Sebastian Wagner).
Tools
intelmqctl
:- Also honour parameters from environment variables (PR#2068 by Sebastian Wagner, fixes #2063).
- Fix management actions (start/stop/status/reload/restart) for groups (PR#2086 by Sebastian Wagner, fixes #2085).
- Do not use hardcoded logging path in
/opt/intelmq
, use the internal default instead (PR#2092 by Sebastian Wagner, fixes #2091).
Known issues
See open bug reports for a more detailed list.
- ParserBot: erroneous raw line recovery in error handling (#1850).
3.0.0 Major release
Installation documentation:
https://intelmq.readthedocs.io/en/maintenance/user/installation.html
Upgrade documentation:
https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html
Configuration
- The
BOTS
file is no longer used and has been removed (by Sebastian Wagner). - The
defaults.conf
file is no longer used and has been removed (PR#1814 by Birger Schacht). - The
pipeline.conf
file is no longer used and has been removed (PR#1849 by Birger Schacht). - The
runtime.conf
was renamed toruntime.yaml
and is now in YAML format (PR#1812 by Birger Schacht).
Core
intelmq.lib.harmonization
:- New class
ClassificationTaxonomy
with fixed list of taxonomies and sanitiation
- New class
intelmq.lib.bot
:- Handle
InvalidValue
exceptions upon message retrieval by dumping the message instead of repeating endlessly (#1765, PR#1766 by Filip Pokorný). - Rewrite of the parameter loading and handling, getting rid of the
parameters
member (PR#1729 by Birger Schacht). - The pipeline is now initialized before the call of
init
to allow bots accessing data directly on startup/initialization for cleanup or maintenance tasks (PR#1982 by Sebastian Wagner).
- Handle
intelmq.lib.exceptions
:InvalidValue
: Add optional parameterobject
(PR#1766 by Filip Pokorný).
intelmq.lib.utils
:- New function
list_all_bots
to list all available/installed bots as replacement for the BOTS file (#368, #552, #644, #757, #1069, #1750, PR#1751 by Sebastian Waldbauer). - New function
get_bots_settings
to return the effective bot parameters, with global parameters applied. - Removed deprecated function
create_request_session_from_bot
(PR#1997 by Sebastian Wagner, #1404). parse_relative
: Add support for parsing minutes and seconds (PR#1857 by Sebastian Wagner).
- New function
intelmq.lib.bot_debugger
:- Set bot's
logging_level
directly in__init__
before the bot's initialization by changing the default value (by Sebastian Wagner). - Rewrite
load_configuration_patch
by adapting it to the parameter and configuration rewrite (by Sebastian Wagner). - Do not rely on the runtime configuration's
group
setting of bots to determine the required message type of messages given on the command line (PR#1949 by Sebastian Wagner).
- Set bot's
Development
rewrite_config_files.py
: Removed obsolete BOTS-file-related rewriting functionality.- A Github action that checks for reuse compliance of all the license and copyright headers was added.
- PyYAML is no longer a required dependency for development environments, all calls to it have been replaced by ruamel.yaml (by Sebastian Wagner).
Data Format
The IntelMQ Data Harmonization ("DHO") is renamed to IntelMQ Data Format ("IDF"). Internal files remain and work the same as before (PR#1818 by Sebastian Waldbauer, fixes 1810).
Update allowed classification fields to version 1.3 (2021-05-18) (fixes #1409, #1476).
- The taxonomy
abusive content
has been renamed toabusive-content
. - The taxonomy
information content security
has been renamed toinformation-content-security
.- The validation of type
unauthorised-information-access
has been fixed, a bug prevented the use of it. - The validation of type
unauthorised-information-modification
has been fixed, a bug prevented the use of it. - The type
leak
has been renamed todata-leak
. - The type
dropzone
has been removed. Taxonomyother
with typeother
and identifierdropzone
can be used instead. Ongoing discussion in the RSIT WG.
- The validation of type
- The taxonomy
intrusion attempts
has been renamed tointrusion-attempts
. - For the taxonomy intrusions (PR#1993 by Sebastian Wagner, addresses #1409):
- The type
compromised
has been renamed tosystem-compromise
. - The type
unauthorized-command
has been merged intosystem-compromise
. - The type
unauthorized-login
has been merged intosystem-compromise
. - The type
backdoor
has been merged intosystem-compromise
(PR#1995 by Sebastian Wagner, addresses #1409). - The type
defacement
has been merged into taxonomyinformation-content-security
, typeunauthorised-information-modification
(PR#1994 by Sebastian Wagner, addresses #1409).
- The type
- The taxonomy
information gathering
has been rename toinformation-gathering
. - The taxonomy
malicious code
has been renamed tomalicious-code
.- The type
c2server
has been renamed toc2-server
. - The type
malware
has been integrated intoinfected-system
andmalware-distribution
, respectively (PR#1917 by Sebastian Wagner addresses #1409). - The type
ransomware
has been integrated intoinfected-system
. - The type
dga domain
has been moved to the taxonomyother
renameddga-domain
(PR#1992 by Sebastian Wagner fixes #1613).
- The type
- For the taxonomy 'availability', the type
misconfiguration
is new. - For the taxonomy 'other', the type
unknown
has been renamed toundetermined
. - For the taxonomy 'vulnerable':
- The type
vulnerable client
has been renamed tovulnerable-system
. - The type
vulnerable service
has been renamed tovulnerable-system
.
- The type
Bots
- The parameters handling of numerous bots has been refactored (PR#1751, PR#1729, by Birger Schacht, Sebastian Wagner, Sebastian Waldbauer).
Collectors
- Remove
intelmq.bots.collectors.xmpp
: one of the dependencies of the bot was deprecated and according to a short survey on the IntelMQ
users mailinglist, the bot is not used by anyone. (https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html, PR#1761, closes #1614) intelmq.bots.collectors.mail._lib
: Added parametermail_starttls
for STARTTLS in all mail collector bots (PR#1831 by Marius Karotkis, fixes #1128).- Added
intelmq.bots.collectors.fireeye
: A bot that collects indicators from Fireeye MAS appliances (PR#1745 by Christopher Schappelwein). intelmq.bots.collectors.api.collector_api
(PR#1987 by Mikk Margus Möll, fixes #1986):- Added UNIX socket capability.
- Correctly close the IOLoop in the shutdown method to fix reload.
intelmq.bots.collectors.rt.collector_rt
(PR#1997 by Sebastian Wagner, #1404):- compatibility with the deprecated parameter
unzip_attachment
(removed in 2.1.0) was removed.
- compatibility with the deprecated parameter
Parsers
- Added
intelmq.bots.parsers.fireeye
: A bot that parses hashes and URLs from Fireeye MAS indicators (PR#1745 by Christopher Schappelwein). intelmq.bots.parsers.shadowserver._config
:- Improved the feed-mapping and all conversion functions (PR#1971 by Mikk Margus Möll).
intelmq.bots.parsers.generic.parser_csv
:- Fix handling of empty string values for parameter
time_format
(by Sebastian Wagner).
- Fix handling of empty string values for parameter
Experts
intelmq.bots.experts.domain_suffix.expert
:- Added
--update-database
option to update domain suffix database (by Sebastian Wagner). - Fix
check
method: load database with UTF-8 encoding explicitly (by Sebastian Wagner).
- Added
- Added
intelmq.bots.experts.http.expert_status
: A bot that fetches the HTTP Status for a given URI and adds it to the message (PR#1789 by Birger Schacht, fixes #1047 partly). - Added
intelmq.bots.experts.http.expert_content
: A bot that fetches an HTTP resource and checks if it contains a specific string. - Added
intelmq.bots.experts.lookyloo.expert
: A bot that sends requests to a lookyloo instance & addsscreenshot_url
to the event (PR#1844 by Sebastian Waldbauer, fixes #1048). - Added
intelmq.bots.experts.rdap.expert
: A bot that checks the rdap protocol for an abuse contact for a given domain. intelmq.bots.experts.sieve.expert
:- Add operators for comparing lists and sets (PR#1895 by Mikk Margus Möll):
:equals
:overlaps
:supersetof
:subsetof
:equals
- Add support for comparing boolean values (PR#1895 by Mikk Margus Möll).
- Add support for rule negation with
!
(PR#1895, PR#1923 by Mikk Margus Möll). - Add support for values types float, int, bool and string for all lists items (PR#1895 by Mikk Margus Möll).
- Add actions for lists (PR#1895 by Mikk Margus Möll).
append
append!
(forced/overwriting)
- Rewrite the rule-processing and operator-handling code to make it more comprehensible and extensible (PR#1895, PR#1923 by Mikk Margus Möll).
- Nested if statements, plus mixed actions and actions in the same scope (PR #1923 by Mikk Margus Möll).
- The attribute manipulation actions add, add! and update support non-string (bool/int/float) values (PR #1923 by Mikk Margus Möll).
- Drop the
:notcontains
operator, as it made is redundant by generic negation:! foo :contains 'x'
instead offoo :notcontains 'x'
(PR#1957 by Mikk Margus Möll). - Split string and numeric matches into single- and multivalued variants, with the relevant new operators
:in
,:containsany
and:regexin
for string lists, and:in
for numeric value lists (PR#1957 by Mikk Margus Möll).- Removed the
==
operator for lists, with the previous meaning of:in
. Have a look at the NEWS.md for more information.
- Removed the
- Add operators for comparing lists and sets (PR#1895 by Mikk Margus Möll):
- Added
intelmq.bots.experts.uwhoisd
: A bot that fetches the whois entry from a uwhois-instance (PR#1918 by Raphaël Vinot). - Removed deprecated
intelmq.bots.experts.ripencc_abuse_contact.expert
. It was replaced byintelmq.bots.experts.ripe.expert
and marked as deprecated in 2.0.0.beta1 (PR#1997 by Sebastian Wagner, #1404). intelmq.bots.experts.modify.expert
:- Removed compatibility with deprecated configuration format before 1.0.0.dev7 (PR#1997 by Sebastian Wagner, #1404).
- Added
intelmq.bots.experts.aggregate
: A bot that aggregate events based upon given fields & a timespan. (PR#1959 by Sebastian Waldbauer) - Added
intelmq.bots.experts.tuency
: A bot that queries the IntelMQ API of a tuency instance (PR#1857 by Sebastian Wagner, fixes #1856).
Outputs
- Remove
intelmq.bots.outputs.xmpp
: one of the dependencies of the bot was de...
3.0.0 Release candidate 1
This is just an intermediate unstable release towards 3.0.0. Please do not use it in production.