From c8183fec76a30335d34f8f00d8210b6d643bd637 Mon Sep 17 00:00:00 2001 From: Vsevolod Stakhov Date: Fri, 12 Jul 2024 11:46:02 +0100 Subject: [PATCH 1/6] Add migration notes --- doc/tutorials/migration.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/doc/tutorials/migration.md b/doc/tutorials/migration.md index 82e30ff0d..110e4c28f 100644 --- a/doc/tutorials/migration.md +++ b/doc/tutorials/migration.md @@ -33,6 +33,21 @@ Discover a reliable step-by-step process for upgrading your Rspamd cluster while 10. Repeat the entire process starting from `step 1` for future updates. This approach ensures a smooth and controlled upgrade process that minimizes potential downtime and issues in your production environment. +## Migration to Rspamd 3.9.0 + +* `ratelimit` module now works in the non dynamic mode by default. It does not affect any existing buckets, as dynamic rates and dynamic bursts will simply be unused in this mode. If you want old behaviour, please either set `dynamic_rate_limit` option to `true` (globally for all ratelimit rules) or set `ham_factor_rate`/`spam_factor_rate` and/or `ham_factor_burst`/`spam_factor_burst` multipliers for individual rules where desired. + +* Bayes statistics now works with a reduced window size (2 words) that has proven to be faster and more space efficient in our tests. The existing statistics can be used without any modifications nor relearning. To restore the old behaviour, one can set the following in the `local.d/classifier-bayes.conf`: + +~~~ +tokenizer { + name = "osb"; + window = 5; +} +~~~ + +However, it is recommended to use the default settings. + ## Migration to Rspamd 3.7.4 The `exclude_private_ips` setting in RBL module no longer exists in this release (and was broken in previous releases), it can be removed from configuration. This setting is equivalent to `exclude_local`. From 6483138167728cf2d0c25cb00e4a7defe6a6b623 Mon Sep 17 00:00:00 2001 From: Andrew Lewis Date: Fri, 12 Jul 2024 14:02:25 +0200 Subject: [PATCH 2/6] Update index --- index.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/index.md b/index.md index 52d1f139b..669153606 100644 --- a/index.md +++ b/index.md @@ -14,22 +14,22 @@ title: Rspamd spam filtering system
-

Study: Rspamd and GPT integration

- 2024-07-03 -

We have studied the efficiency of the new GPT plugin to access LLM (large language models) intelligence.

+

News: Rspamd 3.9.0 has been released

+ 2024-07-12 +

New stable release is now available.

- Learn more  + Learn more 
-

News: Rspamd 3.8.4 has been released

- 2024-02-22 -

New stable release is now available.

+

Study: Rspamd and GPT integration

+ 2024-07-03 +

We have studied the efficiency of the new GPT plugin to access LLM (large language models) intelligence.

- Learn more  + Learn more 
From 7e24f73b2c308aac3a77a342d018d4412fb09976 Mon Sep 17 00:00:00 2001 From: Andrew Lewis Date: Fri, 12 Jul 2024 14:24:17 +0200 Subject: [PATCH 3/6] Create announce post --- _posts/2024-07-12-rspamd-3.9.0.md | 33 +++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100644 _posts/2024-07-12-rspamd-3.9.0.md diff --git a/_posts/2024-07-12-rspamd-3.9.0.md b/_posts/2024-07-12-rspamd-3.9.0.md new file mode 100644 index 000000000..d0664b837 --- /dev/null +++ b/_posts/2024-07-12-rspamd-3.9.0.md @@ -0,0 +1,33 @@ +--- +layout: post +title: "Rspamd 3.9.0 has been released" +categories: announce +--- + +Today we have released Rspamd 3.9.0 featuring many new features & fixes, the most important of which are highlighted below. Refer to the [migration notes]({{ site.baseurl }}/doc/tutorials/migration.html) for an overview of potentially-breaking changes only. + +## Improvements to Bayes configuration + +Rspamd now uses a reduced window size of 2 words by default; this change does not require statistics to be retrained. In testing this reduced window size has produced the same or better results with better performance & lower storage requirements. The new `rspamadm classifier_test` utility could be used for your own experiments. + +## New GPT module + +This release provides a module Rspamd for using LLMs for text classification, you could read more about it in a [dedicated blog post]({{ site.baseurl }}/misc/2024/07/03/gpt.html). + +## Improvements to `known_senders` & `replies` modules + +These release includes enhancements to the `known_senders` and `replies` modules enabling them to collectively flag verified user contacts, refer to documentation of these modules for more detail. + +## Dynamic multipliers for ratelimits are now disabled by default + +To prevent likely confusion, dynamic ratelimits are now disabled by default and must be configured explicitly, refer to the migration notes for details how. + +## Various bugfixes & new features + +* Reworked `grow_factor` to work in an orderly fashion +* Fixed `SUBJ_ALL_CAPS` for unicase scripts by `@ikedas` +* Fixed relearning of bayes messages by `@aduernberger` +* Fixed retrieving word count in antivirus module by `@PxPert` +* Improvements for rules by `@twesterhever` and `@ishisora` + +Thank you to everyone who contributed. From 9beef0bb29143b818627f84d1c30c7c3913b054c Mon Sep 17 00:00:00 2001 From: moisseev Date: Fri, 12 Jul 2024 16:07:11 +0300 Subject: [PATCH 4/6] Fix grammar and punctuation, improve style --- _posts/2024-07-12-rspamd-3.9.0.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/_posts/2024-07-12-rspamd-3.9.0.md b/_posts/2024-07-12-rspamd-3.9.0.md index d0664b837..938448a38 100644 --- a/_posts/2024-07-12-rspamd-3.9.0.md +++ b/_posts/2024-07-12-rspamd-3.9.0.md @@ -4,30 +4,30 @@ title: "Rspamd 3.9.0 has been released" categories: announce --- -Today we have released Rspamd 3.9.0 featuring many new features & fixes, the most important of which are highlighted below. Refer to the [migration notes]({{ site.baseurl }}/doc/tutorials/migration.html) for an overview of potentially-breaking changes only. +Today we have released Rspamd 3.9.0, featuring many new features and fixes. The most important ones are highlighted below. Refer to the [migration notes]({{ site.baseurl }}/doc/tutorials/migration.html) for an overview of potentially-breaking changes. ## Improvements to Bayes configuration -Rspamd now uses a reduced window size of 2 words by default; this change does not require statistics to be retrained. In testing this reduced window size has produced the same or better results with better performance & lower storage requirements. The new `rspamadm classifier_test` utility could be used for your own experiments. +Rspamd now uses a reduced window size of 2 words by default. This change does not require retraining of statistics. In our tests, this reduced window size has produced the equal or better results with better performance and lower storage requirements. The new `rspamadm classifier_test` utility can be used for your own experiments. ## New GPT module -This release provides a module Rspamd for using LLMs for text classification, you could read more about it in a [dedicated blog post]({{ site.baseurl }}/misc/2024/07/03/gpt.html). +This release provides a module for using LLMs for text classification. You can read more about it in a [dedicated blog post]({{ site.baseurl }}/misc/2024/07/03/gpt.html). -## Improvements to `known_senders` & `replies` modules +## Improvements to `known_senders` and `replies` modules -These release includes enhancements to the `known_senders` and `replies` modules enabling them to collectively flag verified user contacts, refer to documentation of these modules for more detail. +This release includes enhancements to the `known_senders` and `replies` modules, enabling them to collectively flag verified user contacts. Refer to the documentation of these modules for more details. ## Dynamic multipliers for ratelimits are now disabled by default -To prevent likely confusion, dynamic ratelimits are now disabled by default and must be configured explicitly, refer to the migration notes for details how. +To avoid potential confusion, dynamic ratelimits are now disabled by default and must be configured explicitly. Refer to the migration notes for details on how to do this. -## Various bugfixes & new features +## Various bug fixes and new features * Reworked `grow_factor` to work in an orderly fashion * Fixed `SUBJ_ALL_CAPS` for unicase scripts by `@ikedas` -* Fixed relearning of bayes messages by `@aduernberger` -* Fixed retrieving word count in antivirus module by `@PxPert` +* Fixed relearning of Bayes messages by `@aduernberger` +* Fixed retrieving word count in the antivirus module by `@PxPert` * Improvements for rules by `@twesterhever` and `@ishisora` -Thank you to everyone who contributed. +We extend our gratitude to everyone who contributed to this release. From 8f1c0fb71fa8ec70fddf0a0c1224690277bcce64 Mon Sep 17 00:00:00 2001 From: moisseev Date: Fri, 12 Jul 2024 16:22:03 +0300 Subject: [PATCH 5/6] Improve migration notes --- doc/tutorials/migration.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/tutorials/migration.md b/doc/tutorials/migration.md index 110e4c28f..acf91a317 100644 --- a/doc/tutorials/migration.md +++ b/doc/tutorials/migration.md @@ -35,9 +35,9 @@ Discover a reliable step-by-step process for upgrading your Rspamd cluster while ## Migration to Rspamd 3.9.0 -* `ratelimit` module now works in the non dynamic mode by default. It does not affect any existing buckets, as dynamic rates and dynamic bursts will simply be unused in this mode. If you want old behaviour, please either set `dynamic_rate_limit` option to `true` (globally for all ratelimit rules) or set `ham_factor_rate`/`spam_factor_rate` and/or `ham_factor_burst`/`spam_factor_burst` multipliers for individual rules where desired. +* The `ratelimit` module now operates in non-dynamic mode by default. This change does not affect any existing buckets, as dynamic rates and dynamic bursts will simply be unused in this mode. To retain the old behaviour, please either set the `dynamic_rate_limit` option to `true` (globally for all ratelimit rules) or configure the `ham_factor_rate`/`spam_factor_rate` and/or `ham_factor_burst`/`spam_factor_burst` multipliers for individual rules as needed. -* Bayes statistics now works with a reduced window size (2 words) that has proven to be faster and more space efficient in our tests. The existing statistics can be used without any modifications nor relearning. To restore the old behaviour, one can set the following in the `local.d/classifier-bayes.conf`: +* Bayes statistics now use a reduced window size (2 words), which has proven to be faster and more space-efficient in our tests. Existing statistics can be used without any modifications or relearning. To restore the old behaviour, one can set the following to `local.d/classifier-bayes.conf`: ~~~ tokenizer { From dc34bbba92d15d633c49b525bdab4f27863d3321 Mon Sep 17 00:00:00 2001 From: Vsevolod Stakhov Date: Sun, 14 Jul 2024 14:30:02 +0100 Subject: [PATCH 6/6] Add stuff --- _posts/2024-07-12-rspamd-3.9.0.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/_posts/2024-07-12-rspamd-3.9.0.md b/_posts/2024-07-12-rspamd-3.9.0.md index 938448a38..78e598061 100644 --- a/_posts/2024-07-12-rspamd-3.9.0.md +++ b/_posts/2024-07-12-rspamd-3.9.0.md @@ -8,15 +8,15 @@ Today we have released Rspamd 3.9.0, featuring many new features and fixes. The ## Improvements to Bayes configuration -Rspamd now uses a reduced window size of 2 words by default. This change does not require retraining of statistics. In our tests, this reduced window size has produced the equal or better results with better performance and lower storage requirements. The new `rspamadm classifier_test` utility can be used for your own experiments. +Rspamd now uses a reduced window size of 2 words by default. This change does not require retraining of statistics. In our tests, this reduced window size has produced the equal or better results with better performance and lower storage requirements - around 4 times less than with the previous default window size of 5 words. The new `rspamadm classifier_test` utility can be used for your own experiments. ## New GPT module -This release provides a module for using LLMs for text classification. You can read more about it in a [dedicated blog post]({{ site.baseurl }}/misc/2024/07/03/gpt.html). +This release provides a module for using LLMs for text classification and unsupervised learning. You can read more about it in a [dedicated blog post]({{ site.baseurl }}/misc/2024/07/03/gpt.html). ## Improvements to `known_senders` and `replies` modules -This release includes enhancements to the `known_senders` and `replies` modules, enabling them to collectively flag verified user contacts. Refer to the documentation of these modules for more details. +This release introduces enhancements to the `known_senders` and `replies` modules, enabling them to work together to flag verified user contacts. With these improvements, senders to whom a user has previously replied will automatically receive negative scores. For more details, please refer to the documentation of these modules. ## Dynamic multipliers for ratelimits are now disabled by default @@ -24,6 +24,16 @@ To avoid potential confusion, dynamic ratelimits are now disabled by default and ## Various bug fixes and new features +* Rspamd HTTP API now supports IO in messagepack serialization format, `rspamc` client uses it now by default +* Fixed a bug where redis bayes learned cache could grow infinitely (and overflow Redis database) +* Fixed dynamic_symbols in the multimap plugin +* Honor dynamic thresholds for greylisting module +* Fixed a bug with statfiles disabling via settings +* Fixed slow timer so it is can now distinguish slow sync and async rules and act properly in different cases +* Implement fuzzy check retransmits backpressure +* Use libarchive for 7 zip compressed headers +* Serialize control commands to avoid missing/corrupted transfers over the worker <-> main channels +* Improved Lua userdata checks performance * Reworked `grow_factor` to work in an orderly fashion * Fixed `SUBJ_ALL_CAPS` for unicase scripts by `@ikedas` * Fixed relearning of Bayes messages by `@aduernberger`