DPDK rte_flow rules RSS support for more NICs - v2 #12317

adaki4 · 2024-12-21T14:41:26Z

Contribution style:

I have read the contributing guide lines at
https://docs.suricata.io/en/latest/devguide/contributing/contribution-process.html

Our Contribution agreements:

I have signed the Open Information Security Foundation contribution agreement at
https://suricata.io/about/contribution-agreement/ (note: this is only required once)

Changes (if applicable):

I have created a ticket at
https://redmine.openinfosecfoundation.org/projects/suricata/issues

Link to ticket: https://redmine.openinfosecfoundation.org/issues/7337

Previous PR #12315

Describe changes:
v2

expand on individual commit messages
change "ifs" to "else ifs" in DevicePostStartPMDSpecificActions() in source-dpdk.c
remove unused RSS hash key configuration from util-dpdk-ice.c
change RSS hash function to RTE_ETH_HASH_FUNCTION_SYMMETRIC_TOEPLITZ (was RTE_ETH_HASH_FUNCTION_TOEPLITZ) in util-dpdk-ice.c

v1

move the base of rte_flow rules RSS configuration from util-dpdk-i40e.c to separate file
add rte_flow rules RSS configuration for mlx5, ixgbe and ice drivers (only i40e until now)
until now i40e hashed packets by 5 tuple, now (also other mentioned drivers) hash by ip src and ip dst only.

Ticket: #7337

Move and adjust the base of RSS configuration from util-dpdk-i40e.c to a new file that can be later utilized by other cards. RSS configuration can be configured via rte_flow rules. This is useful for possible future features such as specific header offload (vxlan, nvgre) also implemented via rte_flow rules, as rte_flow rules can be chained via groups and priorities. i40e uses multiple different rte_flow rules to setup RSS. At first, function DeviceSetRSSFlowQueues() is used to setup rx queues. This rule matches all types of traffic, so the equivalent to dpdk-testpmd pattern would be "pattern end" This rule can not contain hash types (ipv4, ipv6 etc.) nor hash key. The hash function used here is RTE_ETH_HASH_FUNCTION_DEFAULT. The syntax in dpdk-testpmd for this rule with attributes: port index == 0 used rx queue indices == 0 1 2 3 is as follows: "flow create 0 ingress pattern end actions rss queues 0 1 2 3 end func default / end" The other rules configured by i40eDeviceSetRSSFlowIPv4() and i40eDeviceSetRSSFlowIPv6() match specific type of traffic by l4 protocol (none, TCP, UDP, SCTP). For example, pattern to match l3 ipv4 with l4 tcp traffic in dpdk-testpmd syntax would be equivalent of "pattern eth / ipv4 / tcp / end". These rules can not have rx queues configured, but have hash types (ipv4 src dst / ipv6 src dst) and 52 bytes long symmetric hash key configured. The hash function RTE_ETH_HASH_FUNCTION_TOEPLITZ used in these rules hashes symmetricaly due to the symmetric hash key. The syntax in dpdk-testpmd for rule to match ipv4-tcp traffic with attributes: port index == 0 <hash_key> == 6d5a symmetric hash key is as follows: "flow create 0 ingress pattern eth / ipv4 / tcp / end actions rss types ipv4 end queues end key <hash_key> key_len 52 func toeplitz / end" (queues need to be set to NULL) Ticket: 7337

ixgbe driver requires different configuration of RSS rte_flow rule than i40e, with just one generic rule matching all traffic. The generic rule configured by DeviceCreateRSSFlowGeneric() has pattern equivalent to "pattern eth / end" in dpdk-testpmd syntax. The rule must have rx queues configured. The rule hashes traffic to different queues based on ipv4 and ipv6 hash types (ipv4 src dst / ipv6 src dst). The hash key is 40 bytes long symmetric hash key. ixgbe does not support any other hash function than RTE_ETH_HASH_FUNCTION_DEFAULT. The syntax in dpdk-testpmd for this rule with attributes: port index == 0 used rx queue indices == 0 1 2 3 <hash_key> == 6d5a symmetric hash key is as follows: "flow create 0 ingress pattern eth / end actions rss types ipv4 ipv6 end queues 0 1 2 3 end key <hash_key> key_len 40 func default / end" Ticket: 7337

ice driver requires 2 rte_flow rules for matching and redistributing incoming traffic with RSS. The rules set up by iceDeviceSetRSSFlowIPv4() and iceDeviceSetRSSFlowIPv6() are different only in the pattern ("pattern eth / ipv4 / end" or "pattern eth / ipv6 / end" in dpdk-testpmd syntax) and in the hash type (ipv4 src dst / ipv6 src dst). ice will match all ipv4 or ipv6 traffic independently of following l4 protocol. The rules can not have queues configured, implicitly they will use all queues available. The hash function is set to RTE_ETH_HASH_FUNCTION_SYMMETRIC_TOEPLITZ as the hash key also can not be set explicitly. The syntax in dpdk-testpmd for rule to match all ipv4 traffic with attributes: port index == 0 is as follows: "flow create 0 ingress pattern eth / ipv4 / end actions rss types ipv4 end queues end func symmetric_toeplitz / end" (queues need to be set to NULL) Ticket: 7337

mlx5 driver uses only one generic rte_flow RSS rule to match all traffic The configuration of this rule is the same as for ixgbe driver except the hash function is not RTE_ETH_HASH_FUNCTION_DEFAULT but RTE_ETH_HASH_FUNCTION_TOEPLITZ. The syntax in dpdk-testpmd for this rule with attributes: port index == 0 used rx queue indices == 0 1 2 3 <hash_key> == 6d5a symmetric hash key is as follows: "flow create 0 ingress pattern eth / end actions rss types ipv4 ipv6 end queues 0 1 2 3 end key <hash_key> key_len 40 func toeplitz / end" Ticket: 7337

codecov · 2025-01-02T07:39:51Z

Codecov Report

Attention: Patch coverage is 3.24324% with 179 lines in your changes missing coverage. Please review.

Project coverage is 83.17%. Comparing base (6f937c7) to head (cba979e).

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #12317      +/-   ##
==========================================
- Coverage   83.26%   83.17%   -0.10%     
==========================================
  Files         912      915       +3     
  Lines      257643   257719      +76     
==========================================
- Hits       214521   214352     -169     
- Misses      43122    43367     +245

Flag	Coverage Δ
fuzzcorpus	`61.14% <ø> (+<0.01%)`	⬆️
livemode	`19.39% <3.24%> (-0.01%)`	⬇️
pcap	`44.45% <ø> (+0.02%)`	⬆️
suricata-verify	`62.90% <ø> (+0.03%)`	⬆️
unittests	`59.19% <ø> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

victorjulien

Some minor comments inline.

@lukashino can you review the logic?

victorjulien · 2025-01-04T12:56:11Z

src/util-dpdk-ice.c

+    if (retval != 0) {
+        retval = rte_flow_flush(port_id, &flush_error);
+        if (retval != 0) {
+            SCLogError("Unable to flush rte_flow rules of %s: %s Flush error msg: %s", port_name,


I would like to see all error messages use a pattern of
0000:XXXX: unable to ..., so use the interface identifier as a prefix.

Okay, I will also update SCLogDebugs in src/util-dpdk-rss.c to match this pattern.

victorjulien · 2025-01-04T12:58:23Z

src/util-dpdk-rss.c

+ * \param rss_conf RSS configuration
+ * \return int 0 on success, a negative errno value otherwise
+ */
+int DeviceSetRSSFlowQueues(int port_id, const char *port_name, struct rte_flow_action_rss rss_conf)


these are public functions using the Device prefix. I think we need a Dpdk prefix here, as Device has another meaning in our codebase.

lukashino · 2025-01-06T18:20:07Z

I internally reviewed this before but I'll give it one more pass. Most of my comments were addressed. I'll also individually test it to make sure everything works.

lukashino

[56025 - W#01-af:00.1] 2025-01-12 18:55:33 Perf: dpdk: (W#01-af:00.1) received packets 1200
[56026 - W#02-af:00.1] 2025-01-12 18:55:33 Perf: dpdk: (W#02-af:00.1) received packets 0
[56027 - W#03-af:00.1] 2025-01-12 18:55:33 Perf: dpdk: (W#03-af:00.1) received packets 0
[56028 - W#04-af:00.1] 2025-01-12 18:55:33 Perf: dpdk: (W#04-af:00.1) received packets 0
[56029 - W#01-19:00.1] 2025-01-12 18:55:33 Perf: dpdk: (W#01-19:00.1) received packets 602
[56030 - W#02-19:00.1] 2025-01-12 18:55:33 Perf: dpdk: (W#02-19:00.1) received packets 198
[56031 - W#03-19:00.1] 2025-01-12 18:55:33 Perf: dpdk: (W#03-19:00.1) received packets 198
[56032 - W#04-19:00.1] 2025-01-12 18:55:33 Perf: dpdk: (W#04-19:00.1) received packets 202
[56033 - W#01-3b:00.1] 2025-01-12 18:55:33 Perf: dpdk: (W#01-3b:00.1) received packets 602
[56034 - W#02-3b:00.1] 2025-01-12 18:55:33 Perf: dpdk: (W#02-3b:00.1) received packets 198
[56035 - W#03-3b:00.1] 2025-01-12 18:55:33 Perf: dpdk: (W#03-3b:00.1) received packets 198
[56036 - W#04-3b:00.1] 2025-01-12 18:55:33 Perf: dpdk: (W#04-3b:00.1) received packets 202

The testing shows that the ice card is not distributing the traffic correctly. Tested on DPDK v22.11.1

Baseline results:
[56593 - W#01-af:00.1] 2025-01-12 19:02:27 Perf: dpdk: (W#01-af:00.1) received packets 602 # ice
[56594 - W#02-af:00.1] 2025-01-12 19:02:27 Perf: dpdk: (W#02-af:00.1) received packets 198
[56595 - W#03-af:00.1] 2025-01-12 19:02:27 Perf: dpdk: (W#03-af:00.1) received packets 198
[56596 - W#04-af:00.1] 2025-01-12 19:02:27 Perf: dpdk: (W#04-af:00.1) received packets 202
[56597 - W#01-19:00.1] 2025-01-12 19:02:27 Perf: dpdk: (W#01-19:00.1) received packets 200 # i40e
[56598 - W#02-19:00.1] 2025-01-12 19:02:27 Perf: dpdk: (W#02-19:00.1) received packets 200
[56599 - W#03-19:00.1] 2025-01-12 19:02:27 Perf: dpdk: (W#03-19:00.1) received packets 600
[56600 - W#04-19:00.1] 2025-01-12 19:02:27 Perf: dpdk: (W#04-19:00.1) received packets 200
[56601 - W#01-3b:00.1] 2025-01-12 19:02:27 Perf: dpdk: (W#01-3b:00.1) received packets 602 # mlx5
[56602 - W#02-3b:00.1] 2025-01-12 19:02:27 Perf: dpdk: (W#02-3b:00.1) received packets 198
[56603 - W#03-3b:00.1] 2025-01-12 19:02:27 Perf: dpdk: (W#03-3b:00.1) received packets 198
[56604 - W#04-3b:00.1] 2025-01-12 19:02:27 Perf: dpdk: (W#04-3b:00.1) received packets 202

Edit: finished testing of ixgbe too, and that works ok.

lukashino · 2025-01-12T19:31:15Z

Otherwise, the changes look good to me.

Adam Kiripolsky added 4 commits December 21, 2024 14:29

adaki4 requested a review from victorjulien as a code owner December 21, 2024 14:41

adaki4 mentioned this pull request Dec 21, 2024

DPDK rte_flow rules RSS support for more NICs - v1 #12315

Closed

3 tasks

victorjulien requested changes Jan 4, 2025

View reviewed changes

lukashino requested changes Jan 12, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DPDK rte_flow rules RSS support for more NICs - v2 #12317

DPDK rte_flow rules RSS support for more NICs - v2 #12317

adaki4 commented Dec 21, 2024

codecov bot commented Jan 2, 2025

victorjulien left a comment

victorjulien Jan 4, 2025

adaki4 Jan 6, 2025

victorjulien Jan 4, 2025

lukashino commented Jan 6, 2025

lukashino left a comment •

edited

Loading

lukashino commented Jan 12, 2025

DPDK rte_flow rules RSS support for more NICs - v2 #12317

Are you sure you want to change the base?

DPDK rte_flow rules RSS support for more NICs - v2 #12317

Conversation

adaki4 commented Dec 21, 2024

Contribution style:

Our Contribution agreements:

Changes (if applicable):

codecov bot commented Jan 2, 2025

Codecov Report

victorjulien left a comment

Choose a reason for hiding this comment

victorjulien Jan 4, 2025

Choose a reason for hiding this comment

adaki4 Jan 6, 2025

Choose a reason for hiding this comment

victorjulien Jan 4, 2025

Choose a reason for hiding this comment

lukashino commented Jan 6, 2025

lukashino left a comment • edited Loading

Choose a reason for hiding this comment

lukashino commented Jan 12, 2025

lukashino left a comment •

edited

Loading