* Does DPDK supports EPOLL for 10G X550T (ixgbe driver) @ 2023-01-19 19:43 Priya Ramanathan 2023-01-19 22:14 ` Stephen Hemminger 0 siblings, 1 reply; 6+ messages in thread From: Priya Ramanathan @ 2023-01-19 19:43 UTC (permalink / raw) To: users; +Cc: Koteswararao Kondepu, andreaf [-- Attachment #1: Type: text/plain, Size: 513 bytes --] Hello, We use two NIC cards - 10G X550T (ixgbe) for the DPDK-based application. The application uses EPOLL for the UDP communication between the server and the client. With the kernel driver, the EPOLL works properly, but when we enable DPDK support, we cannot see the communication correctly. I would like to know whether anyone has tested DPDK with EPOLL on the 10G X550T model. Also, please suggest which NIC models have been verified using DPDK with EPOLL for UDP communication. Thank you. Regards, Priya. [-- Attachment #2: Type: text/html, Size: 693 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Does DPDK supports EPOLL for 10G X550T (ixgbe driver) 2023-01-19 19:43 Does DPDK supports EPOLL for 10G X550T (ixgbe driver) Priya Ramanathan @ 2023-01-19 22:14 ` Stephen Hemminger 2023-01-19 22:29 ` Priya Ramanathan 0 siblings, 1 reply; 6+ messages in thread From: Stephen Hemminger @ 2023-01-19 22:14 UTC (permalink / raw) To: Priya Ramanathan; +Cc: users, Koteswararao Kondepu, andreaf On Thu, 19 Jan 2023 13:43:19 -0600 Priya Ramanathan <shunmugapriya.ramanathan@gmail.com> wrote: > Hello, > > We use two NIC cards - 10G X550T (ixgbe) for the DPDK-based application. > The application uses EPOLL for the UDP communication between the server and > the client. > With the kernel driver, the EPOLL works properly, but when we enable DPDK > support, we cannot see the communication correctly. > > I would like to know whether anyone has tested DPDK with EPOLL on the 10G > X550T model. > Also, please suggest which NIC models have been verified using DPDK with > EPOLL for UDP communication. Thank you. > > Regards, > Priya. DPDK does not directly support UDP it is designed for raw packet applications. It does not have traditional Linux sockets. DPDK does support device interrupt and rte_epoll model. It requires configuring interrupts and using VFIO. See the l3fwd power for an example. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Does DPDK supports EPOLL for 10G X550T (ixgbe driver) 2023-01-19 22:14 ` Stephen Hemminger @ 2023-01-19 22:29 ` Priya Ramanathan 2023-01-20 4:04 ` Stephen Hemminger 0 siblings, 1 reply; 6+ messages in thread From: Priya Ramanathan @ 2023-01-19 22:29 UTC (permalink / raw) To: stephen; +Cc: users, Koteswararao Kondepu, andreaf [-- Attachment #1: Type: text/plain, Size: 1680 bytes --] Thank you, for the information. I have a UDPDK stack (with DPDK at the physical layer) for UDP communication. I am using VFIO module and binded the 10G X550T NIC to the VFIO module. Then I wrote a simple server-client code for UDP communication using the UDPDK APIs with DPDK support, which works fine. Now, I added the epoll_create, epoll_ctl and epoll_wait commands, and it looks like it is not working as expected. Let me go through the l3fwd example and try to understand it better. If I get any questions, I will get back to you. Thank you. Regards, Priya. On Thu, Jan 19, 2023 at 4:14 PM Stephen Hemminger < stephen@networkplumber.org> wrote: > On Thu, 19 Jan 2023 13:43:19 -0600 > Priya Ramanathan <shunmugapriya.ramanathan@gmail.com> wrote: > > > Hello, > > > > We use two NIC cards - 10G X550T (ixgbe) for the DPDK-based application. > > The application uses EPOLL for the UDP communication between the server > and > > the client. > > With the kernel driver, the EPOLL works properly, but when we enable DPDK > > support, we cannot see the communication correctly. > > > > I would like to know whether anyone has tested DPDK with EPOLL on the 10G > > X550T model. > > Also, please suggest which NIC models have been verified using DPDK with > > EPOLL for UDP communication. Thank you. > > > > Regards, > > Priya. > > DPDK does not directly support UDP it is designed for raw packet > applications. > It does not have traditional Linux sockets. > > DPDK does support device interrupt and rte_epoll model. > It requires configuring interrupts and using VFIO. > See the l3fwd power for an example. > -- winners do not do different things.They do things differently [-- Attachment #2: Type: text/html, Size: 2352 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Does DPDK supports EPOLL for 10G X550T (ixgbe driver) 2023-01-19 22:29 ` Priya Ramanathan @ 2023-01-20 4:04 ` Stephen Hemminger 2023-01-30 17:53 ` Priya Ramanathan 0 siblings, 1 reply; 6+ messages in thread From: Stephen Hemminger @ 2023-01-20 4:04 UTC (permalink / raw) To: Priya Ramanathan; +Cc: users, Koteswararao Kondepu, andreaf On Thu, 19 Jan 2023 16:29:16 -0600 Priya Ramanathan <shunmugapriya.ramanathan@gmail.com> wrote: > Thank you, for the information. > I have a UDPDK stack (with DPDK at the physical layer) for UDP > communication. > > I am using VFIO module and binded the 10G X550T NIC to the VFIO module. > Then I wrote a simple server-client code for UDP communication using the > UDPDK APIs with DPDK support, which works fine. > Now, I added the epoll_create, epoll_ctl and epoll_wait commands, and it > looks like it is not working as expected. > > Let me go through the l3fwd example and try to understand it better. If I > get any questions, I will get back to you. Thank you. > > Regards, > Priya. There is no file descriptor visible to poll on in DPDK. You need to use DPDK interrupt mode which is rte_epoll. Also, if DPDK wakes up in interrupt mode, it might not be a UDP packet, it could be any type of packet that your stack has to digest (ARP, ICMP, UDP, etc) ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Does DPDK supports EPOLL for 10G X550T (ixgbe driver) 2023-01-20 4:04 ` Stephen Hemminger @ 2023-01-30 17:53 ` Priya Ramanathan 2023-01-30 19:21 ` Priya Ramanathan 0 siblings, 1 reply; 6+ messages in thread From: Priya Ramanathan @ 2023-01-30 17:53 UTC (permalink / raw) To: Stephen Hemminger; +Cc: users, Koteswararao Kondepu, andreaf [-- Attachment #1.1: Type: text/plain, Size: 2084 bytes --] Hello, I followed the example from the l3fwd-power to enable the event poll for the ixgbe driver (10G X550T 1563). Default multi-queuing enabled. [image: image.png] As the first step, I bind my NIC to the VFIO module. [image: image.png] The number of rte_eth_devices is shown as 1. Then in my code, I called RTE_INTR_EVENT_ADD with kPort = 0 and kQueue = 0 ->rte_eth_dev_rx_intr_ctl_q(kPort, kQueue, RTE_EPOLL_PER_THREAD, RTE_INTR_EVENT_ADD, nullptr); ->I got the error as "RX Intr vector unset." When I checked inside the DPDK code, it showed intr_handle = dev->intr_handle; + if (!intr_handle->intr_vec) { + RTE_ETHDEV_LOG(ERR, "RX Intr vector unset\n"); + return -EPERM; + } I disabled multi-queueing by setting only one Combined queue but got the same error. Could someone help me understand why and how to overcome this error? Thank you. Regards, Priya. On Thu, Jan 19, 2023 at 10:04 PM Stephen Hemminger < stephen@networkplumber.org> wrote: > On Thu, 19 Jan 2023 16:29:16 -0600 > Priya Ramanathan <shunmugapriya.ramanathan@gmail.com> wrote: > > > Thank you, for the information. > > I have a UDPDK stack (with DPDK at the physical layer) for UDP > > communication. > > > > I am using VFIO module and binded the 10G X550T NIC to the VFIO module. > > Then I wrote a simple server-client code for UDP communication using the > > UDPDK APIs with DPDK support, which works fine. > > Now, I added the epoll_create, epoll_ctl and epoll_wait commands, and it > > looks like it is not working as expected. > > > > Let me go through the l3fwd example and try to understand it better. If I > > get any questions, I will get back to you. Thank you. > > > > Regards, > > Priya. > > > There is no file descriptor visible to poll on in DPDK. > You need to use DPDK interrupt mode which is rte_epoll. > > Also, if DPDK wakes up in interrupt mode, it might not be a UDP > packet, it could be any type of packet that your stack has to digest > (ARP, ICMP, UDP, etc) > -- winners do not do different things.They do things differently [-- Attachment #1.2: Type: text/html, Size: 3917 bytes --] [-- Attachment #2: image.png --] [-- Type: image/png, Size: 91549 bytes --] [-- Attachment #3: image.png --] [-- Type: image/png, Size: 46072 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Does DPDK supports EPOLL for 10G X550T (ixgbe driver) 2023-01-30 17:53 ` Priya Ramanathan @ 2023-01-30 19:21 ` Priya Ramanathan 0 siblings, 0 replies; 6+ messages in thread From: Priya Ramanathan @ 2023-01-30 19:21 UTC (permalink / raw) To: Stephen Hemminger; +Cc: users, Koteswararao Kondepu, andreaf [-- Attachment #1.1: Type: text/plain, Size: 3408 bytes --] //Adding the Interrupt Snapshot details from the kernel Hello, I followed the example from the l3fwd-power to enable the event poll using Interrupt for the ixgbe driver (10G X550T 1563). Default multi-queuing enabled. [image: image.png] The Interrupts associated with the NIC - PCI-MSI [image: image.png] As the first step, I bind my NIC to the VFIO module. [image: image.png] The number of rte_eth_devices is shown as 1. Then in my code, I called RTE_INTR_EVENT_ADD with kPort = 0 and kQueue = 0 ->rte_eth_dev_rx_intr_ctl_q(kPort, kQueue, RTE_EPOLL_PER_THREAD, RTE_INTR_EVENT_ADD, nullptr); ->I got the error as "RX Intr vector unset." When I checked inside the DPDK code, it showed intr_handle = dev->intr_handle; + if (!intr_handle->intr_vec) { + RTE_ETHDEV_LOG(ERR, "RX Intr vector unset\n"); + return -EPERM; + } I disabled multi-queueing by setting only one Combined queue but got the same error. Could someone help me understand why and how to overcome this error? Thank you. Regards, Priya On Mon, Jan 30, 2023 at 11:53 AM Priya Ramanathan < shunmugapriya.ramanathan@gmail.com> wrote: > Hello, > > I followed the example from the l3fwd-power to enable the event poll for > the ixgbe driver (10G X550T 1563). > > Default multi-queuing enabled. > > [image: image.png] > > As the first step, I bind my NIC to the VFIO module. > > [image: image.png] > > > > The number of rte_eth_devices is shown as 1. > > Then in my code, I called RTE_INTR_EVENT_ADD with kPort = 0 and kQueue = 0 > > ->rte_eth_dev_rx_intr_ctl_q(kPort, kQueue, RTE_EPOLL_PER_THREAD, > RTE_INTR_EVENT_ADD, nullptr); > ->I got the error as "RX Intr vector unset." When I checked inside the > DPDK code, it showed > > intr_handle = dev->intr_handle; > + if (!intr_handle->intr_vec) { > + RTE_ETHDEV_LOG(ERR, "RX Intr vector unset\n"); > + return -EPERM; > + } > > > I disabled multi-queueing by setting only one Combined queue but got the same error. Could someone help me understand why and how to overcome this error? Thank you. > > > Regards, > > Priya. > > > > > On Thu, Jan 19, 2023 at 10:04 PM Stephen Hemminger < > stephen@networkplumber.org> wrote: > >> On Thu, 19 Jan 2023 16:29:16 -0600 >> Priya Ramanathan <shunmugapriya.ramanathan@gmail.com> wrote: >> >> > Thank you, for the information. >> > I have a UDPDK stack (with DPDK at the physical layer) for UDP >> > communication. >> > >> > I am using VFIO module and binded the 10G X550T NIC to the VFIO module. >> > Then I wrote a simple server-client code for UDP communication using the >> > UDPDK APIs with DPDK support, which works fine. >> > Now, I added the epoll_create, epoll_ctl and epoll_wait commands, and it >> > looks like it is not working as expected. >> > >> > Let me go through the l3fwd example and try to understand it better. If >> I >> > get any questions, I will get back to you. Thank you. >> > >> > Regards, >> > Priya. >> >> >> There is no file descriptor visible to poll on in DPDK. >> You need to use DPDK interrupt mode which is rte_epoll. >> >> Also, if DPDK wakes up in interrupt mode, it might not be a UDP >> packet, it could be any type of packet that your stack has to digest >> (ARP, ICMP, UDP, etc) >> > > > -- > winners do not do different things.They do things differently > -- winners do not do different things.They do things differently [-- Attachment #1.2: Type: text/html, Size: 6733 bytes --] [-- Attachment #2: image.png --] [-- Type: image/png, Size: 91549 bytes --] [-- Attachment #3: image.png --] [-- Type: image/png, Size: 46072 bytes --] [-- Attachment #4: image.png --] [-- Type: image/png, Size: 232750 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2023-02-02 8:39 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-01-19 19:43 Does DPDK supports EPOLL for 10G X550T (ixgbe driver) Priya Ramanathan 2023-01-19 22:14 ` Stephen Hemminger 2023-01-19 22:29 ` Priya Ramanathan 2023-01-20 4:04 ` Stephen Hemminger 2023-01-30 17:53 ` Priya Ramanathan 2023-01-30 19:21 ` Priya Ramanathan
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).
![[-- Attachment #2: image.png --] [-- Type: image/png, Size: 91549 bytes --]](../../CAMgau3kX1oPZCY1yiWbo4xbfjLxyAD5NTP=Cg4x1MYziB4Z6Qg@mail.gmail.com/2-image.png){kind=link}
![[-- Attachment #3: image.png --] [-- Type: image/png, Size: 46072 bytes --]](../../CAMgau3kX1oPZCY1yiWbo4xbfjLxyAD5NTP=Cg4x1MYziB4Z6Qg@mail.gmail.com/3-image.png){kind=link}
![[-- Attachment #2: image.png --] [-- Type: image/png, Size: 91549 bytes --]](../../CAMgau3naSFWZJbx7pmdnX52fVaa8uXDxQ1zEY=EE_SHdR8hz7w@mail.gmail.com/2-image.png){kind=link}
![[-- Attachment #3: image.png --] [-- Type: image/png, Size: 46072 bytes --]](../../CAMgau3naSFWZJbx7pmdnX52fVaa8uXDxQ1zEY=EE_SHdR8hz7w@mail.gmail.com/3-image.png){kind=link}
![[-- Attachment #4: image.png --] [-- Type: image/png, Size: 232750 bytes --]](../../CAMgau3naSFWZJbx7pmdnX52fVaa8uXDxQ1zEY=EE_SHdR8hz7w@mail.gmail.com/4-image.png){kind=link}