From 0e65bf0ff5e7e9530a198e75085ccb64a37d5da5 Mon Sep 17 00:00:00 2001 From: Ian Chen Date: Wed, 20 Sep 2023 18:30:15 +0800 Subject: [PATCH] Update Introduction_of_gtp5g_and_some_kernel_concepts.md --- ...ction_of_gtp5g_and_some_kernel_concepts.md | 91 ++++++++++--------- 1 file changed, 47 insertions(+), 44 deletions(-) diff --git a/docs/blog/Introduction_of_gtp5g_and_some_kernel_concepts.md b/docs/blog/Introduction_of_gtp5g_and_some_kernel_concepts.md index dd74e6a0..bb315bff 100644 --- a/docs/blog/Introduction_of_gtp5g_and_some_kernel_concepts.md +++ b/docs/blog/Introduction_of_gtp5g_and_some_kernel_concepts.md @@ -6,7 +6,7 @@ --- ## Overview -GTP, or the General Packet Radio System Tunneling Protocol, is a group of IP-based communication protocols used to transport General Packet Radio Services (GPRS) within LTE, 5G NR, and other networks. GTP can be broken down into several components: GTP-C, GTP-U, and GTP prime. GTP-C and GTP-U are responsible for the control plane and user plane, respectively, while GTP prime is utilized for transmitting charging data through the Ga interface as defined in the 3GPP GPRS Core Network. +GTP (General Packet Radio System Tunneling Protocol) is a group of IP-based communication protocols used to transport General Packet Radio Services (GPRS) within LTE, 5G NR, and other networks. GTP can be broken down into several components: GTP-C, GTP-U, and GTP prime. GTP-C and GTP-U are responsible for the control plane and user plane, respectively, while GTP prime is utilized for transmitting charging data through the Ga interface as defined in the 3GPP GPRS Core Network. In the context of free5GC, the UPF network function combines the GTP-C control part to correctly instruct the routing path for any packet passing through the core network. GTP-U, on the other hand, is managed by [gtp5g](https://github.com/free5gc/gtp5g), which transports packets using kernel modules generated by gtp5g. This article will introduce how gtp5g assists free5GC in handling packets and some kernel-related concepts. @@ -14,6 +14,7 @@ This article will introduce how gtp5g assists free5GC in handling packets and so Let's start the journey! Additional information: + - Linux kernel version is 5.4.0-159-generic in article. According to other versions, some of content would be different, but the main concept is the same. - Gtp5g version is v0.8.2 - UPF version is v1.2.0 @@ -21,6 +22,7 @@ Additional information: ## Netlink, Generic Netlink and Rtnetlink ### Netlink Before we continue, i need to introduce [Netlink](https://man7.org/linux/man-pages/man7/netlink.7.html) first. What is Netlink? Netlink is an IPC (Inter Process Communicate) protocol which can connect kernel space and user space processes by socket. Traditionally, it used three methods: **Ioctl**, **sysfs**, or **procfs**, to facilitate communication between the kernel and user space. However, it can only be initiated from user space, not from kernel space. Netlink can support not only **initiated from kernel and user space** but also: + - Bidirectional transmission, asynchronous communication. - Standard socket API used in user space. - Specialized API used in kernel space. @@ -75,7 +77,7 @@ The following figure is Generic Netlink structure: - The ***Generic Netlink controller*** which is part of the kernel and is responsible for dynamically allocating Generic Netlink communication channels and other management tasks. The Generic Netlink controller is implemented as a standard Generic Netlink user, however, it listens on a special, pre-allocated Generic Netlink channel. - The kernel socket API. Generic Netlink sockets are created with the PF_NETLINK domain and the NETLINK_GENERIC protocol values. -### Rtnetlink +### RtNetlink The last one is [rtnetlink](https://man7.org/linux/man-pages/man7/rtnetlink.7.html), it also known as Netlink protocol type NETLINK_ROUTE, user space program could read and alter kernel's routing table or create new network device. @@ -114,7 +116,7 @@ type Driver interface { } ``` -UPF use rtnl to create device (interface) name 'upfgtp'. User can observe it while executing **run.sh**. +UPF use rtnl to create device (interface) named `upfgtp`. User can observe it while executing **run.sh**. ```golang func OpenGtp5gLink(mux *nl.Mux, addr string, mtu uint32, log *logrus.Entry) (*Gtp5gLink, error) { g := &Gtp5gLink{ @@ -201,7 +203,7 @@ func OpenGtp5gLink(mux *nl.Mux, addr string, mtu uint32, log *logrus.Entry) (*Gt } ``` -Connect UPF Driver functions and gtp5g_genl_ops. +Connect UPF Driver functions and `gtp5g_genl_ops`. ```golang // internl/forwarder/buffnetlink/server.go func OpenServer(wg *sync.WaitGroup, client *nl.Client, mux *nl.Mux) (*Server, error) { @@ -237,12 +239,12 @@ func OpenServer(wg *sync.WaitGroup, client *nl.Client, mux *nl.Mux) (*Server, er ## GTP5G - Gtp5g utilizes a Linux kernel module to manage packet traffic. A Linux kernel module can be thought of as a small piece of code that is inserted into the Linux kernel, allowing users to customize the program according to the current hardware device - In gtp5g, the primary function is **gtp5g_init** in **gtp5g.c**; it exposes most of the components and techniques provided by gtp5g. This article will choose the following concepts to investigate further: - - Network device -> net_device_ops - - Rtnetlink -> gtp5g_link_ops - - Generic Netlink -> gtp5g_genl_family + - Network device -> `net_device_ops` + - Rtnetlink -> `gtp5g_link_ops` + - Generic Netlink -> `gtp5g_genl_family` - Additionally, the article will present two functions in detail: - - rtnl_link_register() - - genl_register_family() + - `rtnl_link_register()` + - `genl_register_family()` ```cpp // src/gtp5g.c @@ -286,7 +288,7 @@ static int __init gtp5g_init(void) } ``` -### net_device_ops +### `net_device_ops` It is defined in **dev.h** and referenced in **dev.c**. The structure net_device_ops encompasses all operations related to network device, and free5GC inherits some of these operations to implement self-made netdev ops. ```cpp // include/dev.h @@ -307,9 +309,10 @@ const struct net_device_ops gtp5g_netdev_ops = { }; ``` According to [/include/linux/netdevice.h](https://elixir.bootlin.com/linux/latest/source/include/linux/netdevice.h#L1422), you can find the definition of hooks of **net_device_ops**: -- .ndo_init: This function is called once when a network device is registered. The network device can use this for any late stage initialization or semantic validation. It can fail with an error code which will be propagated back to register_netdev. -- .ndo_uninit: This function is called when device is unregistered or when registration fails. It is not called if init fails. -- .ndo_start_xmit: Called when a packet needs to be transmitted. Returns NETDEV_TX_OK. Can return NETDEV_TX_BUSY, but you should stop the queue before that can happen; it's for obsolete devices and weird corner cases, but the stack really does a non-trivial amount of useless work if you return NETDEV_TX_BUSY. Required; cannot be NULL. + +- `.ndo_init`: This function is called once when a network device is registered. The network device can use this for any late stage initialization or semantic validation. It can fail with an error code which will be propagated back to register_netdev. +- `.ndo_uninit`: This function is called when device is unregistered or when registration fails. It is not called if init fails. +- `.ndo_start_xmit`: Called when a packet needs to be transmitted. Returns NETDEV_TX_OK. Can return NETDEV_TX_BUSY, but you should stop the queue before that can happen; it's for obsolete devices and weird corner cases, but the stack really does a non-trivial amount of useless work if you return NETDEV_TX_BUSY. Required; cannot be NULL. ```cpp struct net_device_ops { int (*ndo_init)(struct net_device *dev); @@ -427,7 +430,7 @@ static netdev_tx_t gtp5g_dev_xmit(struct sk_buff *skb, struct net_device *dev) ``` -### gtp5g_link_ops +### `gtp5g_link_ops` Ignore headder file here, structure defines Rtnetlink operations. ```cpp @@ -447,16 +450,16 @@ struct rtnl_link_ops gtp5g_link_ops __read_mostly = { ``` Definition is in [/include/net/rtnetlink.h](https://elixir.bootlin.com/linux/v5.4/source/include/net/rtnetlink.h#L59): -- .kind: Identifier -- .maxtype: Highest device specific netlink attribute number -- .policy: Netlink policy for device specific attribute validation -- .priv_size: sizeof net_device private space -- .setup: net_device setup function -- .validate: Optional validation function for netlink/changelink parameters -- .newlink: Function for configuring and registering a new device -- .dellink: Function to remove a device -- .get_size: Function to calculate required room for dumping device specific netlink attributes -- .fill_info: Function to dump device specific netlink attributes +- `.kind`: Identifier +- `.maxtype`: Highest device specific netlink attribute number +- `.policy`: Netlink policy for device specific attribute validation +- `.priv_size`: sizeof net_device private space +- `.setup`: net_device setup function +- `.validate`: Optional validation function for netlink/changelink parameters +- `.newlink`: Function for configuring and registering a new device +- `.dellink`: Function to remove a device +- `.get_size`: Function to calculate required room for dumping device specific netlink attributes +- `.fill_info`: Function to dump device specific netlink attributes ```cpp struct rtnl_link_ops { @@ -548,7 +551,7 @@ static void gtp5g_link_setup(struct net_device *dev) } ``` -### rtnl_link_register() +### `rtnl_link_register()` - Definition is in [/net/core/rtnetlink.c](https://elixir.bootlin.com/linux/v5.4/source/net/core/rtnetlink.c#L372) - This function should be used by drivers that create devices during module initialization. It must be called before registering the devices - Using the **kind** property of rtnl_link_ops to search for the existence of ops in link_ops. If ops do not exist, then inserted it at the end of link_ops @@ -593,15 +596,15 @@ struct genl_family gtp5g_genl_family __ro_after_init = { }; ``` Definition is in [/include/net/genetlink.h](https://elixir.bootlin.com/linux/v5.4.159/source/include/net/genetlink.h#L46): -- .name: name of family (exclusive) -- .version: protocol version (usually is 1) -- .hdrsize: length of user specific header in bytes -- .maxattr: maximum number of attributes supported -- .netnsok: set to true if the family can handle network namespaces and should be presented in all of them -- .ops: the operations supported by this family -- .n_ops: number of operations supported by this family -- .mcgrps: multicast groups used by this family -- .n_mcgrps: number of multicast groups +- `.name`: name of family (exclusive) +- `.version`: protocol version (usually is 1) +- `.hdrsize`: length of user specific header in bytes +- `.maxattr`: maximum number of attributes supported +- `.netnsok`: set to true if the family can handle network namespaces and should be presented in all of them +- `.ops`: the operations supported by this family +- `.n_ops`: number of operations supported by this family +- `.mcgrps`: multicast groups used by this family +- `.n_mcgrps`: number of multicast groups ```cpp struct genl_family { @@ -629,7 +632,7 @@ struct genl_family { }; ``` -Gtp5g defines gtp5g_genl_ops, all operations are one-to-one match to Driver functions in UPF part: +Gtp5g defines `gtp5g_genl_ops`, all operations are one-to-one match to Driver functions in UPF part: ```cpp // src/genl/genl.c static const struct genl_ops gtp5g_genl_ops[] = { @@ -659,13 +662,13 @@ static const struct genl_ops gtp5g_genl_ops[] = { } ``` -### genl_register_family() +### `genl_register_family()` - Definition is in [/net/netlink/genetlink.c](https://elixir.bootlin.com/linux/v5.4.159/source/net/netlink/genetlink.c#L322) - Registers the specified family after validating it first. Only one family may be registered with the same family name or identifier. The family's ops, multicast groups and module pointer must already be assigned. Return 0 on success or a negative error code - Three functions within this method hold greater significance (all of them in /net/netlink/genetlink.c): - - genl_validate_ops() - - genl_family_find_byname() - - genl_validate_assign_mc_groups() + - `genl_validate_ops()` + - `genl_family_find_byname()` + - `genl_validate_assign_mc_groups()` ```cpp int genl_register_family(struct genl_family *family) @@ -735,7 +738,7 @@ errout_locked: EXPORT_SYMBOL(genl_register_family); ``` -#### genl_validate_ops() +#### `genl_validate_ops()` This function will verify if there is defined function for the operations and will also compare whether any operation is reused by a command. Using gtp5g as an example, command can be considered as action such as add, del, modify PDR rules. ```cpp static int genl_validate_ops(const struct genl_family *family) @@ -762,7 +765,7 @@ static int genl_validate_ops(const struct genl_family *family) } ``` -#### genl_family_find_byname() +#### `genl_family_find_byname()` Function would check every entry in genl_fam_idr whether exists the same family name. ```cpp @@ -779,14 +782,14 @@ static const struct genl_family *genl_family_find_byname(char *name) } ``` -Here is the macro of idr_for_each_entry(): +Here is the macro of `idr_for_each_entry()`: ```cpp #define idr_for_each_entry(idr, entry, id) \ for (id = 0; ((entry) = idr_get_next(idr, &(id))) != NULL; id += 1U) ``` -#### genl_validate_assign_mc_groups() +#### `genl_validate_assign_mc_groups()` This changes the number of multicast groups that are available when **ntensok** is ture: ```cpp @@ -884,5 +887,5 @@ Jimmy Chang - https://www.cnblogs.com/ssyfj/p/16230540.html - https://www.twblogs.net/a/5b81e5852b71772165aedd7a - https://www.cnblogs.com/ssyfj/p/16230540.html -- [IT blog by Lan Chen](https://ithelp.ithome.com.tw/articles/10302887) +- [IT blog by Ian Chen](https://ithelp.ithome.com.tw/articles/10302887) - [IT blog by 0xff07](https://ithelp.ithome.com.tw/articles/10243519)