Scheduling of multiple RX/TX queues on a single port

Scheduling of multiple RX/TX queues on a single port

[Fengkai Sun]

[-- Attachment #1: Type: text/plain, Size: 846 bytes --]

Hi list,

I'm curious how DPDK programs the NIC to receive/transmit packets when
there are multiple queues on a single port.

As for RX, the answer might be clear.
The NIC can only receive a packet once at a time, since the cable only
outputs one signal (0 or 1) at a time (correct me if I'm wrong).
Therefore the NIC can receive a packet, check it's information, and finally
put in into the right queue via some policies, e.g. RSS, all sequentially.

However, it confuses me when it comes to TX.
As there are multiple TX queues on the same port, the NIC must decide which
queue to get packets from when it's idle.
This is where scheduling lies. How does the NIC select the queue?
Round-Robin? Does it have to enforce fairness among the queues?

I'm wondering where I can find some documentation on this issue. Thank you!


Best Regards,
Fengkai

[-- Attachment #2: Type: text/html, Size: 1076 bytes --]

Re: Scheduling of multiple RX/TX queues on a single port

[Stephen Hemminger]

On Mon, 29 May 2023 23:02:45 +0800
Fengkai Sun <qcloud1014@gmail.com> wrote:

> Hi list,
> 
> I'm curious how DPDK programs the NIC to receive/transmit packets when
> there are multiple queues on a single port.
> 
> As for RX, the answer might be clear.
> The NIC can only receive a packet once at a time, since the cable only
> outputs one signal (0 or 1) at a time (correct me if I'm wrong).
> Therefore the NIC can receive a packet, check it's information, and finally
> put in into the right queue via some policies, e.g. RSS, all sequentially.
> 
> However, it confuses me when it comes to TX.
> As there are multiple TX queues on the same port, the NIC must decide which
> queue to get packets from when it's idle.
> This is where scheduling lies. How does the NIC select the queue?
> Round-Robin? Does it have to enforce fairness among the queues?
> 
> I'm wondering where I can find some documentation on this issue. Thank you!

Transmit scheduling is up to the hardware (not DPDK).
Generally I assume it is round-robin
but there maybe cases like priority queues (like DCB) or large packets
with segment offload.

Re: Scheduling of multiple RX/TX queues on a single port

[Fengkai Sun]

[-- Attachment #1: Type: text/plain, Size: 1656 bytes --]

Hi Stephen,

Thank you very much for your reply!

After using 'transmit scheduling' as the keyword, much more useful results
now show up.

For others who are also interested in how queues are scheduled, I would
recommend reading this paper (it's open access):
https://www.usenix.org/system/files/nsdi19-stephens.pdf


--
Thanks,
Fengkai

On Wed, May 31, 2023 at 12:24 AM Stephen Hemminger <
stephen@networkplumber.org> wrote:

> On Mon, 29 May 2023 23:02:45 +0800
> Fengkai Sun <qcloud1014@gmail.com> wrote:
>
> > Hi list,
> >
> > I'm curious how DPDK programs the NIC to receive/transmit packets when
> > there are multiple queues on a single port.
> >
> > As for RX, the answer might be clear.
> > The NIC can only receive a packet once at a time, since the cable only
> > outputs one signal (0 or 1) at a time (correct me if I'm wrong).
> > Therefore the NIC can receive a packet, check it's information, and
> finally
> > put in into the right queue via some policies, e.g. RSS, all
> sequentially.
> >
> > However, it confuses me when it comes to TX.
> > As there are multiple TX queues on the same port, the NIC must decide
> which
> > queue to get packets from when it's idle.
> > This is where scheduling lies. How does the NIC select the queue?
> > Round-Robin? Does it have to enforce fairness among the queues?
> >
> > I'm wondering where I can find some documentation on this issue. Thank
> you!
>
> Transmit scheduling is up to the hardware (not DPDK).
> Generally I assume it is round-robin
> but there maybe cases like priority queues (like DCB) or large packets
> with segment offload.
>

[-- Attachment #2: Type: text/html, Size: 2316 bytes --]