From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 318B145502; Thu, 27 Jun 2024 13:54:47 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B7B4A40B99; Thu, 27 Jun 2024 13:54:46 +0200 (CEST) Received: from office2.cesnet.cz (office2.cesnet.cz [78.128.248.237]) by mails.dpdk.org (Postfix) with ESMTP id 7D72D40DDC for ; Thu, 27 Jun 2024 13:42:43 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cesnet.cz; s=office2-2020; t=1719488562; bh=gilqlYcvX+ftShVJTDAhKj7MDYhuRJ62IRK/cEoqjS4=; h=Date:Subject:To:Cc:References:From:In-Reply-To; b=QGTm4p9+orVdy2nXMEGCLi5DB4fzYGudiwPR1oYKR1ydtd7mK2ujAsE4+ZSyFn/2H 0iAiupB96idE6d0Q1GVZxumeCn8zUQQW/oQ2NuzYA98NaMeGp9PVfCr64rcZJQGgib hIAjFkStUHSMTz/bQb1xIr1E6nAxPEa/c5uiQvhL4XrFyuArQufWUSyFz8lqgg8aMm lBWIbYlOstsrIUr+zV+h4cbsH1O43khmXkMHIhs3MSyq1UEZKEAYTGxJnwzfKmghd/ TsUpqwoDJu6LucdoXugMezqiau7wd93D6nZ1aMlT5+/6l8gQZBIi0ukisaF/XtaJxW RVAKrZdQRNOSg== Received: from [IPV6:2a01:c846:1282:6700:358f:431:d8ac:fd03] (unknown [IPv6:2a01:c846:1282:6700:358f:431:d8ac:fd03]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by office2.cesnet.cz (Postfix) with ESMTPSA id D5213118007E; Thu, 27 Jun 2024 13:42:39 +0200 (CEST) Message-ID: <82d8f67c-3b0b-46c2-a94b-8457d0c602c2@cesnet.cz> Date: Thu, 27 Jun 2024 13:42:38 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Hairpin Queues Throughput ConnectX-6 To: Dmitry Kozlyuk Cc: dev@dpdk.org, orika@nvidia.com, bingz@nvidia.com, viktorin@cesnet.cz References: <3d746dbc-330e-403f-b87f-bf495cac3437@cesnet.cz> <20240625032224.45b65339@sovereign> Content-Language: en-US, sk-SK From: =?UTF-8?Q?M=C3=A1rio_Kuka?= In-Reply-To: <20240625032224.45b65339@sovereign> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-256; boundary="------------ms090501010902020902090808" X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This is a cryptographically signed message in MIME format. --------------ms090501010902020902090808 Content-Type: multipart/alternative; boundary="------------VyxGhGcaLD12nM02coTLMGeK" --------------VyxGhGcaLD12nM02coTLMGeK Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 SGkgRG1pdHJ5LA0KDQpUaGFuayB5b3UgZm9yIHlvdXIgaGVscGZ1bCByZXBseS4NCg0KPiBU cnkgZW5hYmxpbmcgIkV4cGxpY2l0IFR4IHJ1bGUiIG1vZGUgaWYgcG9zc2libGUuDQo+IEkg d2FzIGFibGUgdG8gYWNoaWV2ZSAxMzcgTXBwcyBAIDY0QiB3aXRoIHRoZSBmb2xsb3dpbmcg Y29tbWFuZDoNCj4NCj4gZHBkay10ZXN0cG1kIC1hIDIxOjAwLjAgLWEgYzE6MDAuMCAtLWlu LW1lbW9yeSAtLSBcDQo+ICAgICAgLWkgLS1yeHE9MSAtLXR4cT0xIC0taGFpcnBpbnE9OCAt LWhhaXJwaW4tbW9kZT0weDEwDQoNCkJhc2VkIG8gdGhpcyBJIHdhcyBhYmxlIHRvIGFjaGll dmUgMTQyIE1wcHMoOTYuMDggR2JwcykgQCA2NEIgd2l0aCB0aGUgZm9sbG93aW5nIGNvbW1h bmQ6DQoNCnN1ZG8gZHBkay10ZXN0cG1kIC1sIDAtMSAtbiA0IC1hIDAwMDA6YzQ6MDAuMCxo cF9idWZfbG9nX3N6PTEzIFwNCiAgICAgLS1pbi1tZW1vcnkgLS0gLS1yeHE9MSAtLXR4cT0x IC0taGFpcnBpbnE9MTIgLS1oYWlycGluLW1vZGU9MHgxMCAtaQ0KICAgICANCmZsb3cgY3Jl YXRlIDAgaW5ncmVzcyBwYXR0ZXJuIGV0aCBzcmMgaXMgMDA6MTA6OTQ6MDA6MDA6MDIgLyBl bmQgYWN0aW9ucyByc3MgcXVldWVzIDEgMiAzIDQgNSA2IDcgOCA5IDEwIDExIDEyIGVuZCAv IGVuZA0KDQpBbG1vc3QgZnVsbCBzcGVlZCA6KS4NCkFueSBvdGhlciB2YWx1ZSBvZiAiaHBf YnVmX2xvZ19zeiIgb3IgbW9yZSBxdWV1ZXMgZG9lcyBub3QgZ2V0IGJldHRlciByZXN1bHRz LCBidXQgaW5zdGVhZCBtYWtlcyB0aGVtIHdvcnNlLg0KDQo+IFJ4USBwaW5uZWQgaW4gZGV2 aWNlIG1lbW9yeSByZXF1aXJlcyBmaXJtd2FyZSBjb25maWd1cmF0aW9uIFsxXToNCj4NCj4g bWx4Y29uZmlnIC15IC1kICRwY2lfYWRkciBzZXQgTUVNSUNfU0laRV9MSU1JVD0wIEhBSVJQ SU5fREFUQV9CVUZGRVJfTE9DSz0xDQo+IG1seGZ3cmVzZXQgLXkgLWQgJHBjaV9hZGRyIHJl c2V0DQo+DQo+IFsxXTpodHRwczovL2RvYy5kcGRrLm9yZy9ndWlkZXMvcGxhdGZvcm0vbWx4 NS5odG1sP2hpZ2hsaWdodD1oYWlycGluX2RhdGFfYnVmZmVyX2xvY2sNCj4NCj4gSG93ZXZl ciwgcGlubmVkIFJ4USBkaWRuJ3QgaW1wcm92ZSBhbnl0aGluZyBmb3IgbWUuDQoNCkkgdHJp ZWQgaXQsIGJ1dCBpdCBkaWRuJ3QgaW1wcm92ZSBhbnl0aGluZyBmb3IgbWUgZWl0aGVyLg0K DQpNw6FyaW8NCg0KT24gMjUvMDYvMjAyNCAwMjoyMizCoCBLb3pseXVrIHdyb3RlOg0KPiBI aSBNw6FyaW8sDQo+DQo+IDIwMjQtMDYtMTkgMDg6NDUgKFVUQyswMjAwKSwgTcOhcmlvIEt1 a2E6DQo+PiBIZWxsbywNCj4+DQo+PiBJIHdhbnQgdG8gdXNlIGhhaXJwaW4gcXVldWVzIHRv IGZvcndhcmQgaGlnaCBwcmlvcml0eSB0cmFmZmljIChzdWNoIGFzDQo+PiBMQUNQKS4NCj4+ IE15IGdvYWwgaXMgdG8gZW5zdXJlIHRoYXQgdGhpcyB0cmFmZmljIGlzIG5vdCBkcm9wcGVk IGluIGNhc2UgdGhlDQo+PiBzb2Z0d2FyZSBwaXBlbGluZSBpcyBvdmVyd2hlbG1lZC4NCj4+ IEJ1dCBkdXJpbmcgdGVzdGluZyB3aXRoIGRwZGstdGVzdHBtZCBJIGNhbid0IGFjaGlldmUg ZnVsbCB0aHJvdWdocHV0IGZvcg0KPj4gaGFpcnBpbiBxdWV1ZXMuDQo+IEZvciBtYWludGFp bmVyczogSSdkIGxpa2UgdG8gZXhwcmVzcyBpbnRlcmVzdCBpbiB0aGlzIHVzZSBjYXNlIHRv by4NCj4NCj4+IFRoZSBiZXN0IHJlc3VsdCBJIGhhdmUgYmVlbiBhYmxlIHRvIGFjaGlldmUg Zm9yIDY0QiBwYWNrZXRzIGlzIDgzIEdicHMNCj4+IGluIHRoaXMgY29uZmlndXJhdGlvbjoN Cj4+ICQgc3VkbyBkcGRrLXRlc3RwbWQgLWwgMC0xIC1uIDQgLWEgMDAwMDoxNzowMC4wLGhw X2J1Zl9sb2dfc3o9MTkgLS0NCj4+IC0tcnhxPTEgLS10eHE9MSAtLXJ4ZD00MDk2IC0tdHhk PTQwOTYgLS1oYWlycGlucT0yDQo+PiB0ZXN0cG1kPiBmbG93IGNyZWF0ZSAwIGluZ3Jlc3Mg cGF0dGVybiBldGggc3JjIGlzIDAwOjEwOjk0OjAwOjAwOjAzIC8NCj4+IGVuZCBhY3Rpb25z IHJzcyBxdWV1ZXMgMSAyIGVuZCAvIGVuZA0KPiBUcnkgZW5hYmxpbmcgIkV4cGxpY2l0IFR4 IHJ1bGUiIG1vZGUgaWYgcG9zc2libGUuDQo+IEkgd2FzIGFibGUgdG8gYWNoaWV2ZSAxMzcg TXBwcyBAIDY0QiB3aXRoIHRoZSBmb2xsb3dpbmcgY29tbWFuZDoNCj4NCj4gZHBkay10ZXN0 cG1kIC1hIDIxOjAwLjAgLWEgYzE6MDAuMCAtLWluLW1lbW9yeSAtLSBcDQo+ICAgICAgLWkg LS1yeHE9MSAtLXR4cT0xIC0taGFpcnBpbnE9OCAtLWhhaXJwaW4tbW9kZT0weDEwDQo+DQo+ IFlvdSBtaWdodCBnZXQgZXZlbiBiZXR0ZXIgc3BlZWQsIGJlY2F1c2UgbXkgZmxvdyBydWxl cyB3ZXJlIG1vcmUgY29tcGxpY2F0ZWQNCj4gKFJURSBGbG93IGJhc2VkICJyb3V0ZXIgb24t YS1zdGljayIpOg0KPg0KPiBmbG93IGNyZWF0ZSAwIGluZ3Jlc3MgZ3JvdXAgMSBwYXR0ZXJu IGV0aCAvIHZsYW4gdmlkIGlzIDcyMSAvIGVuZCBhY3Rpb25zIG9mX3NldF92bGFuX3ZpZCB2 bGFuX3ZpZCA3MjIgLyByc3MgcXVldWVzIDEgMiAzIDQgNSA2IDcgOCBlbmQgLyBlbmQNCj4g ZmxvdyBjcmVhdGUgMSBpbmdyZXNzIGdyb3VwIDEgcGF0dGVybiBldGggLyB2bGFuIHZpZCBp cyA3MjEgLyBlbmQgYWN0aW9ucyBvZl9zZXRfdmxhbl92aWQgdmxhbl92aWQgNzIyIC8gcnNz IHF1ZXVlcyAxIDIgMyA0IDUgNiA3IDggZW5kIC8gZW5kDQo+IGZsb3cgY3JlYXRlIDAgaW5n cmVzcyBncm91cCAxIHBhdHRlcm4gZXRoIC8gdmxhbiB2aWQgaXMgNzIyIC8gZW5kIGFjdGlv bnMgb2Zfc2V0X3ZsYW5fdmlkIHZsYW5fdmlkIDcyMSAvIHJzcyBxdWV1ZXMgMSAyIDMgNCA1 IDYgNyA4IGVuZCAvIGVuZA0KPiBmbG93IGNyZWF0ZSAxIGluZ3Jlc3MgZ3JvdXAgMSBwYXR0 ZXJuIGV0aCAvIHZsYW4gdmlkIGlzIDcyMiAvIGVuZCBhY3Rpb25zIG9mX3NldF92bGFuX3Zp ZCB2bGFuX3ZpZCA3MjEgLyByc3MgcXVldWVzIDEgMiAzIDQgNSA2IDcgOCBlbmQgLyBlbmQN Cj4gZmxvdyBjcmVhdGUgMCBpbmdyZXNzIGdyb3VwIDAgcGF0dGVybiBlbmQgYWN0aW9ucyBq dW1wIGdyb3VwIDEgLyBlbmQNCj4gZmxvdyBjcmVhdGUgMSBpbmdyZXNzIGdyb3VwIDAgcGF0 dGVybiBlbmQgYWN0aW9ucyBqdW1wIGdyb3VwIDEgLyBlbmQNCj4NCj4+IEZvciBwYWNrZXRz IGluIHRoZSByYW5nZSA2OC04MEIgSSBtZWFzdXJlZCBldmVuIGxvd2VyIHRocm91Z2hwdXQu DQo+PiBGdWxsIHRocm91Z2hwdXQgSSBtZWFzdXJlZCBvbmx5IGZyb20gcGFja2V0cyBsYXJn ZXIgdGhhbiAxMTJCDQo+Pg0KPj4gRm9yIG9ubHkgb25lIHF1ZXVlLCBJIGRpZG4ndCBnZXQg bW9yZSB0aGFuIDU1R2JwczoNCj4+ICQgc3VkbyBkcGRrLXRlc3RwbWQgLWwgMC0xIC1uIDQg LWEgMDAwMDoxNzowMC4wLGhwX2J1Zl9sb2dfc3o9MTkgLS0NCj4+IC0tcnhxPTEgLS10eHE9 MSAtLXJ4ZD00MDk2IC0tdHhkPTQwOTYgLS1oYWlycGlucT0xIC1pDQo+PiB0ZXN0cG1kPiBm bG93IGNyZWF0ZSAwIGluZ3Jlc3MgcGF0dGVybiBldGggc3JjIGlzIDAwOjEwOjk0OjAwOjAw OjAzIC8NCj4+IGVuZCBhY3Rpb25zIHF1ZXVlIGluZGV4IDEgLyBlbmQNCj4+DQo+PiBJIHRy aWVkIHRvIHVzZSBsb2NrZWQgZGV2aWNlIG1lbW9yeSBmb3IgVFggYW5kIFJYIHF1ZXVlcywg YnV0IGl0IHNlZW1zDQo+PiB0aGF0IHRoaXMgaXMgbm90IHN1cHBvcnRlZDoNCj4+ICItLWhh aXJwaW4tbW9kZT0weDAxMTAwMCIgKGJpdCAxNiAtIGhhaXJwaW4gVFggcXVldWVzIHdpbGwg dXNlIGxvY2tlZA0KPj4gZGV2aWNlIG1lbW9yeSwgYml0IDEyIC0gaGFpcnBpbiBSWCBxdWV1 ZXMgd2lsbCB1c2UgbG9ja2VkIGRldmljZSBtZW1vcnkpDQo+IFJ4USBwaW5uZWQgaW4gZGV2 aWNlIG1lbW9yeSByZXF1aXJlcyBmaXJtd2FyZSBjb25maWd1cmF0aW9uIFsxXToNCj4NCj4g bWx4Y29uZmlnIC15IC1kICRwY2lfYWRkciBzZXQgTUVNSUNfU0laRV9MSU1JVD0wIEhBSVJQ SU5fREFUQV9CVUZGRVJfTE9DSz0xDQo+IG1seGZ3cmVzZXQgLXkgLWQgJHBjaV9hZGRyIHJl c2V0DQo+DQo+IFsxXTpodHRwczovL2RvYy5kcGRrLm9yZy9ndWlkZXMvcGxhdGZvcm0vbWx4 NS5odG1sP2hpZ2hsaWdodD1oYWlycGluX2RhdGFfYnVmZmVyX2xvY2sNCj4NCj4gSG93ZXZl ciwgcGlubmVkIFJ4USBkaWRuJ3QgaW1wcm92ZSBhbnl0aGluZyBmb3IgbWUuDQo+DQo+IFR4 USBwaW5uZWQgaW4gZGV2aWNlIG1lbW9yeSBpcyBub3Qgc3VwcG9ydGVkIGJ5IG5ldC9tbHg1 Lg0KPiBUeFEgcGlubmVkIHRvIERQREsgbWVtb3J5IG1hZGUgcGVyZm9ybWFuY2UgYXdmdWwg KHByZWRpY3RhYmx5KS4NCj4NCj4+IEkgd2FzIGV4cGVjdGluZyB0aGF0IGFjaGlldmluZyBm dWxsIHRocm91Z2hwdXQgd2l0aCBoYWlycGluIHF1ZXVlcyB3b3VsZA0KPj4gbm90IGJlIGEg cHJvYmxlbS4NCj4+IElzIG15IGV4cGVjdGF0aW9uIHRvbyBvcHRpbWlzdGljPw0KPj4NCj4+ IFdoYXQgb3RoZXIgcGFyYW1ldGVycyBiZXNpZGVzICdocF9idWZfbG9nX3N6JyBjYW4gSSB1 c2UgdG8gYWNoaWV2ZSBmdWxsDQo+PiB0aHJvdWdocHV0Pw0KPiBJbiBteSBleHBlcmltZW50 cywgZGVmYXVsdCAiaHBfYnVmX2xvZ19zeiIgb2YgMTYgaXMgb3B0aW1hbC4NCj4gVGhlIG1v c3QgaW5mbHVlbnRpYWwgcGFyYW1ldGVyIGFwcGVhcnMgdG8gYmUgdGhlIG51bWJlciBvZiBo YWlycGluIHF1ZXVlcy4NCj4NCj4+IEkgdHJpZWQgY29tYmluaW5nIHRoZSBmb2xsb3dpbmcg cGFyYW1ldGVyczogbXBycV9lbj0sIHJ4cXNfbWluX21wcnE9LA0KPj4gbXBycV9sb2dfc3Ry aWRlX251bT0sIHR4cV9pbmxpbmVfbXB3PSwgcnhxX3BrdF9wYWRfZW49LA0KPj4gYnV0IHdp dGggbm8gcG9zaXRpdmUgaW1wYWN0IG9uIHRocm91Z2hwdXQuDQoNCg== --------------VyxGhGcaLD12nM02coTLMGeK Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Dmitry,

Thank you for your helpful reply.
 
Try enabling "Ex=
plicit Tx rule" mode if possible.
I was able to achieve 137 Mpps @ 64B with the following command:

dpdk-testpmd -a 21:00.0 -a c1:00.0 --in-memory -- \
    -i --rxq=3D1 --txq=3D1 --hairpinq=3D8 --hairpin-mode=3D0x10

Based o this I was able to achieve 142 Mpps(96.08 Gbps) @ 64B with the fo=
llowing command:

sudo dpdk-testpmd -l 0-1 -n 4 -a 0000:c4:00.0,hp_buf_log_sz=3D13 \
    --in-memory -- --rxq=3D1 --txq=3D1 --hairpinq=3D12 --hairpin-mode=3D0=
x10 -i
   =20
flow create 0 ingress pattern eth src is 00:10:94:00:00:02 / end actions =
rss queues 1 2 3 4 5 6 7 8 9 10 11 12 end / end

Almost full speed :).
Any other value of "hp_buf_log_sz" or more queues does not get better res=
ults, but instead makes them worse.


RxQ pinn=
ed in device memory requires firmware configuration [1]:

mlxconfig -y -d $pci_addr set MEMIC_SIZE_LIMIT=3D0 HAIRPIN_DATA_BUFFER_LO=
CK=3D1
mlxfwreset -y -d $pci_addr reset

[1]: https://doc.d=
pdk.org/guides/platform/mlx5.html?highlight=3Dhairpin_data_buffer_lock

However, pinned RxQ didn't improve anything for me.

I tried it, but it didn't improve anything for me either.

M=C3=A1rio



    
On 25/06/2024 02:22,=C2=A0 Kozlyuk wro=
te:

    

    

      
Hi M=C3=A1rio,

2024-06-19 08:45 (UTC+0200), M=C3=A1rio Kuka:


      

        
Hello,

I want to use hairpin queues to forward high priority traffic (such as=20
LACP).
My goal is to ensure that this traffic is not dropped in case the=20
software pipeline is overwhelmed.
But during testing with dpdk-testpmd I can't achieve full throughput for =

hairpin queues.


      

      
For maintainers: I'd like to express interest in this use case too.



      

        
The best result I have been able to achieve for 64B packets is 83 Gbps=20
in this configuration:
$ sudo dpdk-testpmd -l 0-1 -n 4 -a 0000:17:00.0,hp_buf_log_sz=3D19 --=20
--rxq=3D1 --txq=3D1 --rxd=3D4096 --txd=3D4096 --hairpinq=3D2
testpmd> flow create 0 ingress pattern eth src is 00:10:94:00:00:03 / =
 =20
end actions rss queues 1 2 end / end


      

      
Try enabling "Explicit Tx rule" mode if possible.
I was able to achieve 137 Mpps @ 64B with the following command:

dpdk-testpmd -a 21:00.0 -a c1:00.0 --in-memory -- \
    -i --rxq=3D1 --txq=3D1 --hairpinq=3D8 --hairpin-mode=3D0x10

You might get even better speed, because my flow rules were more complica=
ted
(RTE Flow based "router on-a-stick"):

flow create 0 ingress group 1 pattern eth / vlan vid is 721 / end actions=
 of_set_vlan_vid vlan_vid 722 / rss queues 1 2 3 4 5 6 7 8 end / end
flow create 1 ingress group 1 pattern eth / vlan vid is 721 / end actions=
 of_set_vlan_vid vlan_vid 722 / rss queues 1 2 3 4 5 6 7 8 end / end
flow create 0 ingress group 1 pattern eth / vlan vid is 722 / end actions=
 of_set_vlan_vid vlan_vid 721 / rss queues 1 2 3 4 5 6 7 8 end / end
flow create 1 ingress group 1 pattern eth / vlan vid is 722 / end actions=
 of_set_vlan_vid vlan_vid 721 / rss queues 1 2 3 4 5 6 7 8 end / end
flow create 0 ingress group 0 pattern end actions jump group 1 / end
flow create 1 ingress group 0 pattern end actions jump group 1 / end



      

        
For packets in the range 68-80B I measured even lower throughput.
Full throughput I measured only from packets larger than 112B

For only one queue, I didn't get more than 55Gbps:
$ sudo dpdk-testpmd -l 0-1 -n 4 -a 0000:17:00.0,hp_buf_log_sz=3D19 --=20
--rxq=3D1 --txq=3D1 --rxd=3D4096 --txd=3D4096 --hairpinq=3D1 -i
testpmd> flow create 0 ingress pattern eth src is 00:10:94:00:00:03 / =
 =20
end actions queue index 1 / end

I tried to use locked device memory for TX and RX queues, but it seems=20
that this is not supported:
"--hairpin-mode=3D0x011000" (bit 16 - hairpin TX queues will use locked=20
device memory, bit 12 - hairpin RX queues will use locked device memory)


      

      
RxQ pinned in device memory requires firmware configuration [1]:

mlxconfig -y -d $pci_addr set MEMIC_SIZE_LIMIT=3D0 HAIRPIN_DATA_BUFFER_LO=
CK=3D1
mlxfwreset -y -d $pci_addr reset

[1]: https://doc.d=
pdk.org/guides/platform/mlx5.html?highlight=3Dhairpin_data_buffer_lock

However, pinned RxQ didn't improve anything for me.

TxQ pinned in device memory is not supported by net/mlx5.
TxQ pinned to DPDK memory made performance awful (predictably).



      

        
I was expecting that achie=
ving full throughput with hairpin queues would=20
not be a problem.
Is my expectation too optimistic?

What other parameters besides 'hp_buf_log_sz' can I use to achieve full=20
throughput?


      

      
In my experiments, default "hp_buf_log_sz" of 16 is optimal.
The most influential parameter appears to be the number of hairpin queues=
=2E



      

        
I tried combining the foll=
owing parameters: mprq_en=3D, rxqs_min_mprq=3D,=20
mprq_log_stride_num=3D, txq_inline_mpw=3D, rxq_pkt_pad_en=3D,
but with no positive impact on throughput.


      

    

    

  


--------------VyxGhGcaLD12nM02coTLMGeK--

--------------ms090501010902020902090808
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC
DVswggZtMIIEVaADAgECAhEAnbnQ7qfoWj/NNwoRkcscADANBgkqhkiG9w0BAQwFADBGMQsw
CQYDVQQGEwJOTDEZMBcGA1UEChMQR0VBTlQgVmVyZW5pZ2luZzEcMBoGA1UEAxMTR0VBTlQg
UGVyc29uYWwgQ0EgNDAeFw0yNDA0MjIwMDAwMDBaFw0yNjA0MjIyMzU5NTlaMIHXMQswCQYD
VQQGEwJDWjEeMBwGA1UECAwVUHJhaGEsIEhsYXZuw60gbcSbc3RvMTgwNgYDVQQKDC9DRVNO
RVQsIHrDoWptb3bDqSBzZHJ1xb5lbsOtIHByw6F2bmlja8O9Y2ggb3NvYjEZMBcGA1UEYRMQ
TlRSQ1otNjM4IDM5IDE3MjEdMBsGCSqGSIb3DQEJARYOa3VrYUBjZXNuZXQuY3oxDTALBgNV
BAQTBEt1a2ExDzANBgNVBCoMBk3DoXJpbzEUMBIGA1UEAwwLTcOhcmlvIEt1a2EwggEiMA0G
CSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCiHwBI7t3hatQXKJ1X1fANpo1f5rur//PMoRMb
w06yUiWU8lPAQTGshceMnNDTy+PsZ+jvJ6H5xsrISX911zLBH2T2BcF3IVEEiu6k3JBbXMlc
lJvrkKfBcOsV2rnYQUqLHxejKVaeTmvMjTwWGm13HfERf3TRL5JvE9FzwsYHY+XokiFBrHgb
pN18YyjvkvS+LJcMYAPFo+jgEsu7WZY4xjK13fIzKk+qoneD8rrX5bbnhuhAfaHOpnHvowJz
sQR2lLK/8A2pdLAJd4YQi8ITylZeTaIPWK6M13tc2TU2/mCXXyni4LI0q2+n2gzu4oqILCHE
OVam0bRFVAxzZnmFAgMBAAGjggHCMIIBvjAfBgNVHSMEGDAWgBRpAKHHIVj44MUbILAK3adR
vxPZ5DAdBgNVHQ4EFgQUqx/ddGZmQk2PwBpPufqOYZQavXUwDgYDVR0PAQH/BAQDAgWgMAwG
A1UdEwEB/wQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwQGCCsGAQUFBwMCMFAGA1UdIARJMEcw
OgYMKwYBBAGyMQECAQoEMCowKAYIKwYBBQUHAgEWHGh0dHBzOi8vc2VjdGlnby5jb20vU01J
TUVDUFMwCQYHZ4EMAQUDAjBCBgNVHR8EOzA5MDegNaAzhjFodHRwOi8vR0VBTlQuY3JsLnNl
Y3RpZ28uY29tL0dFQU5UUGVyc29uYWxDQTQuY3JsMHgGCCsGAQUFBwEBBGwwajA9BggrBgEF
BQcwAoYxaHR0cDovL0dFQU5ULmNydC5zZWN0aWdvLmNvbS9HRUFOVFBlcnNvbmFsQ0E0LmNy
dDApBggrBgEFBQcwAYYdaHR0cDovL0dFQU5ULm9jc3Auc2VjdGlnby5jb20wLwYDVR0RBCgw
JoEUTWFyaW8uS3VrYUBjZXNuZXQuY3qBDmt1a2FAY2VzbmV0LmN6MA0GCSqGSIb3DQEBDAUA
A4ICAQAlHGBXnHAkPcBj0dtzq+AsVAHCbDjAyrvh6DjaQO6KGfsCdJiI3MKRE7Kmpm0rbdb9
xoSBjtZY5uSJO0ZkAQzz21jcTgiRUJmNgdffTk1r2JaFpXpbQSKrtjOfVG6HfUvOfbgLyVev
A7HH837SaniLCaqj2OzBPH2slV+Q+lYxU2W6pP/S/4tjHvypLruKDsp7GFLwDUYbKm7/vBSK
prBMnjkLq2HQmQ+E3LkPEt8L5In8BpFpKRmAFj+cNm14xv715o+JreioBG1JklsWkFY2X7KE
KYRM6Ov2oJuqK2plhviM4c3r8yQ5bTvale/UWPcm6GXOrGRJXaR0yjuHWk0MZcwrHqXQbfiO
porDmt68zVK05UacMRB0RrEQ+1nEjzzsHTVXzpHIIbCTzKBiEadHsaka3mWMFrLP2fWDh1m4
IRTsKJy9YwYmjzDKYMAdisAXgXE4QHN2K8LLHLMttBGoOI08X91pfSf5WoB4g8UM5+tuemSo
uNpj14gHxFrxoGCC53EImN0f9F/Lh116CpJKDLN+06RYyeVKNjUVWONZfCeEcQLGWNJTDh3p
IDLl13COJGbtUwr2cjHTi1WFG4tYYJUadGgih5+ObDgk9CKqXFFLvkQIXOp7qQnRgRd+Ochq
nCGaOZhLHsd/O7ByimWJ3OB6nDujOahK4PRaHS+G8zCCBuYwggTOoAMCAQICEDECcNQ1vpsk
mvhW0OHihUkwDQYJKoZIhvcNAQEMBQAwgYgxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpOZXcg
SmVyc2V5MRQwEgYDVQQHEwtKZXJzZXkgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBO
ZXR3b3JrMS4wLAYDVQQDEyVVU0VSVHJ1c3QgUlNBIENlcnRpZmljYXRpb24gQXV0aG9yaXR5
MB4XDTIwMDIxODAwMDAwMFoXDTMzMDUwMTIzNTk1OVowRjELMAkGA1UEBhMCTkwxGTAXBgNV
BAoTEEdFQU5UIFZlcmVuaWdpbmcxHDAaBgNVBAMTE0dFQU5UIFBlcnNvbmFsIENBIDQwggIi
MA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCzSuIiXidb6QRbFAQ1MiAUrrTSMUDGzVDA
HqFEyq+eSmF/LZDeYpszai2kQsqWATz/cBA9gGjunvJ45G48ycC4D6gwZFvbBt5JotxlunBe
B8K+crGar3v+RCQ4VfvToX07v+HTJ6EeEONR3IzJPTMyzgAwENGsAWf9va9HePQFJiChCzXq
hKpp0zen53S+8f9itEy06GS8aku7Mvyb4tMBBa9An3Y3ALIqIeymg/iYs8m9WkSkMyekNtFR
B1+1KlnNUpM05G8+sY9EucnQQRUIdHzYsvqjP3XlaSuB4Jj0ia66UGfi5Wx31mm5sKAz8Re9
UGVWIqq6wKFKxkSfuO4iwYiIPJoiGEux3dqabwFLduAroDF1IxE40PqGIdPXzYuZ/wL6BEfF
Ab0xy8bfm5S9G7y/ts9mIlFpPtkLZ/nQ/iVOWdsu9ale/nK/uGF47xsxeW2LIvB4sH5U2+D4
ad8vpNbcCrXIXXKtkBnNHgxumNNZ0R1Isq/Pz7TALCxxDzWdsM7AO32/Jn7R4ldtGRZmKpJy
fACDn8HU1QPhOtiWsjifrMWnanJhQ6K7M/5qz8BmfPrca+MUrr1Y4NHZb9MrgPtWKQQyGDXy
+G5F/iHGdZk7LS+F8NH+Ddolt2wZpz52JqGMTDPIH5Qok4LLO95wbxtn+79Tw+wQxmDTuIg+
LwIDAQABo4IBizCCAYcwHwYDVR0jBBgwFoAUU3m/WqorSs9UgOHYm8Cd8rIDZsswHQYDVR0O
BBYEFGkAocchWPjgxRsgsArdp1G/E9nkMA4GA1UdDwEB/wQEAwIBhjASBgNVHRMBAf8ECDAG
AQH/AgEAMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDBDA4BgNVHSAEMTAvMC0GBFUd
IAAwJTAjBggrBgEFBQcCARYXaHR0cHM6Ly9zZWN0aWdvLmNvbS9DUFMwUAYDVR0fBEkwRzBF
oEOgQYY/aHR0cDovL2NybC51c2VydHJ1c3QuY29tL1VTRVJUcnVzdFJTQUNlcnRpZmljYXRp
b25BdXRob3JpdHkuY3JsMHYGCCsGAQUFBwEBBGowaDA/BggrBgEFBQcwAoYzaHR0cDovL2Ny
dC51c2VydHJ1c3QuY29tL1VTRVJUcnVzdFJTQUFkZFRydXN0Q0EuY3J0MCUGCCsGAQUFBzAB
hhlodHRwOi8vb2NzcC51c2VydHJ1c3QuY29tMA0GCSqGSIb3DQEBDAUAA4ICAQAKBU57DY8f
EzkA/W/sYsbD7e0XquMBzHjcP0eXXXRD4EAEAGCWSs+QRL9XIxmx+52zx9wMa8YTejlR+NKe
jiyKPTF0q63zMxrO/z/hUwo8IDcRRLS0NSgvTW6AN2rCXJe5iLN5fIfYgIBB9cy1L6trPuZ/
vjUJm87nQ7ExQzGqWN5F9U8MlAk0c5iLanG7GCMoNjHiF1n0baj6guUeG7n5qcwOQTyDS19+
NEqfwjUPUGasN1ZH8h1sE6PrzvRpti+rKzWpiU+i2/k3l2b5fFDy+Wu9jv6R9BoBh47es/UM
zwEZ2kSrIVVr4jSukk+FpmR5ZbtwiYNAV6sdb1srMGsILzXlrdasSE2nGHvZklk2zUdgn7b0
1MHq67g0mNozGmT6Dam41Kbhv25WMFs871XqwVIb4gGoT1yRf/VePMm1jwauqijhKJFvrNwe
GnebGPeipaLxIo2iEA4qdRztEg/qyzWGogXK/TFdmivg322fMPQWjQkMhRGMM8SCjlZN22L8
x0ZOYoVA2rHJm5P25IjZe+HPyn7ikJiSJmqlqFmUeowXF3D1dFlCCs/5yC06RYRqI2REFu+2
8t2nswIvY6xCFAR0RtS8Mz2yXNld0js2MmiRUGrc7imWzdUPbPcv9sdUF7SsERGPIzYL8dIi
Hzit+YCoGCSXMg6peF37hHNp1TGCAzswggM3AgEBMFswRjELMAkGA1UEBhMCTkwxGTAXBgNV
BAoTEEdFQU5UIFZlcmVuaWdpbmcxHDAaBgNVBAMTE0dFQU5UIFBlcnNvbmFsIENBIDQCEQCd
udDup+haP803ChGRyxwAMA0GCWCGSAFlAwQCAQUAoIIBsTAYBgkqhkiG9w0BCQMxCwYJKoZI
hvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0yNDA2MjcxMTQyMzhaMC8GCSqGSIb3DQEJBDEiBCDT
7uculp93V3J4K0xnKWgvg+dC4XW04ilv1XKPEKwssDBqBgkrBgEEAYI3EAQxXTBbMEYxCzAJ
BgNVBAYTAk5MMRkwFwYDVQQKExBHRUFOVCBWZXJlbmlnaW5nMRwwGgYDVQQDExNHRUFOVCBQ
ZXJzb25hbCBDQSA0AhEAnbnQ7qfoWj/NNwoRkcscADBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCG
SAFlAwQBKjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqG
SIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMGwGCyqGSIb3DQEJEAILMV2gWzBG
MQswCQYDVQQGEwJOTDEZMBcGA1UEChMQR0VBTlQgVmVyZW5pZ2luZzEcMBoGA1UEAxMTR0VB
TlQgUGVyc29uYWwgQ0EgNAIRAJ250O6n6Fo/zTcKEZHLHAAwDQYJKoZIhvcNAQEBBQAEggEA
FLd7IKKa98xZ1h0bRpkGer8bFUqYMmoBLv2EDITyxUpgW8gm/thTTHtaoxpPq7P7UZOD1dLM
LyQyTTmkD64WRBRltjw3Iba3aalc3HMZ+LSR0PoiAzJFjdyLiYY1D6GzPnjs7wfPLjmv0BW5
Kh3x2GVOp1kufcZZjlfeEIFrFXkD8pxopujVWnNDdGowkg2cY8LwHjBgojw0znVcr8u7o//F
aOSTXCv5mIZTi39tiA/2Y7XYfy/lZxmTwkzMsr/l4OPFnSXKQwxq1zyARWtRC3NUnXL6sI7T
/AnMKju8EpO9pHb+bhHF5t775AEtGJyYeo4jKtE5KIPeSY1kDbAVvwAAAAAAAA==
--------------ms090501010902020902090808--