Hi Bharath,

Apologies for my delay in response. That's strange that it is working on your system. Thank you for testing this, and thanks again for meeting the other day.

I do not see any difference in our bootargs which is likely to cause this cryptodev issue. If the binary dpdk-test that our native build is producing is valid (since it worked on your system), and we are using the same hardware (I think we are both on cn10k), then perhaps the software environment is to blame. Since we just flashed firmware and built the latest SDK, I don't think that is an issue but we can re-do it if you think it makes sense. 

One thing I can do is re-do the setup of the SDK kernel + ubuntu rootfs, on a separate partition, boot to that environment and attempt to use the dpdk-test binary there. What I will use for this process is the "Booting to Ubuntu rootfs using Velox SDK kernel" from doc-base-SDK12.24.11/kernel/ubuntu_rootfs.html. Someone else on our team did this before, and although I believe he did it all according to the docs, it doesn't hurt to run through it again to make sure.

Any other troubleshooting ideas are welcome, if you have any! Also, since Marvell is a member of our lab, as always I can grant VPN access to anyone if they want to poke around this system!  =I am on vacation for the next two weeks for winter holidays, but I will make sure to work on this ASAP when I'm back and report back to you all what happened. 

Thanks and happy holidays.
-Patrick

On Thu, Dec 19, 2024 at 8:35 AM Anoob Joseph <anoobj@marvell.com> wrote:

Hi Vivek,

 

We do not support these cases. The opcodes are optimized for PDCP use case and in the internal branch we have these tests disabled.

 

Thanks,

Anoob

 

From: Vivek Garg <vgarg@marvell.com>
Sent: Thursday, December 19, 2024 6:18 PM
To: JogaRao Nartu <njogarao@marvell.com>; Bharath Rajendra <brajendra@marvell.com>; Patrick Robb <probb@iol.unh.edu>; Hiral Shah <hshah@marvell.com>; Anoob Joseph <anoobj@marvell.com>
Cc: Cody Cheng <ccheng@iol.unh.edu>; Gnanesh Kambalu Palanethra <gpalanethra@marvell.com>; ci@dpdk.org; Akhil Goyal <gakhil@marvell.com>; Anoob Joseph <anoobj@marvell.com>; Jerin Jacob <jerinj@marvell.com>
Subject: RE: [EXTERNAL] CN10K Crypto Test Issue

 

HI Anoob,

Below Jira created for snow 3G Test Failures in cryptodev_cn10k_autotest tests on 106 with native build binary.

[IPBUSW-58120] SNOW 3G Test Suite are failing in cryptodev_cn10k_autotest tests on 106 with native build binary. - Marvell Jira Production System

 

Thanks,

Vivek

 

 

From: JogaRao Nartu <njogarao@marvell.com>
Sent: Thursday, December 19, 2024 5:46 PM
To: Bharath Rajendra <brajendra@marvell.com>; Patrick Robb <probb@iol.unh.edu>; Hiral Shah <hshah@marvell.com>
Cc: Cody Cheng <ccheng@iol.unh.edu>; Gnanesh Kambalu Palanethra <gpalanethra@marvell.com>; ci@dpdk.org; Akhil Goyal <gakhil@marvell.com>; Vivek Garg <vgarg@marvell.com>; Anoob Joseph <anoobj@marvell.com>; Jerin Jacob <jerinj@marvell.com>
Subject: RE: [EXTERNAL] CN10K Crypto Test Issue

 

++

 

From: Bharath Rajendra <brajendra@marvell.com>
Sent: Thursday, December 19, 2024 5:44 PM
To: Patrick Robb <probb@iol.unh.edu>; Hiral Shah <hshah@marvell.com>
Cc: Cody Cheng <ccheng@iol.unh.edu>; Gnanesh Kambalu Palanethra <gpalanethra@marvell.com>; JogaRao Nartu <njogarao@marvell.com>; ci@dpdk.org; Akhil Goyal <gakhil@marvell.com>; Vivek Garg <vgarg@marvell.com>
Subject: RE: [EXTERNAL] CN10K Crypto Test Issue

 

Hi Patrick,

 

We followed the instructions which you used and also the application which you shared over email.

On our setup it is working as expected. Attached the complete logs for the reference.

 

Bootargs:

console=ttyAMA0,115200n8 earlycon=pl011,0x87e028000000 maxcpus=24 rootwait rw root=/dev/nfs nfsroot=10.29.56.91:/srv/tftp/regression-debug-2980-db-cn106xx-crb-b0-cisco-267/dut/rootfs,v3,tcp ip=dhcp coherent_pool=16M default_hugepagesz=512M hugepagesz=512M hugepages=8

 

Enabling CPT VF’s:

# echo 1 > /sys/bus/pci/devices/0002:20:00.0/sriov_numvfs

#/usr/bin/dpdk-devbind.py -b vfio-pci 0002:20:00.1

Setting up huge pages:

#mkdir -p /mnt/huge

#mount -t hugetlbfs nodev /mnt/huge

 

Note: cryptodev_cn10k_asymautotest works without any issues. But with cryptodev_cn10k_autotest observed few test failure, this we will track internally with dev team. But we are not observing “request timeout error”

 

Regards

Bharath Rajendra

 

From: Patrick Robb <probb@iol.unh.edu>
Sent: Thursday, December 19, 2024 12:29 AM
To: Hiral Shah <hshah@marvell.com>
Cc: Cody Cheng <ccheng@iol.unh.edu>; Gnanesh Kambalu Palanethra <gpalanethra@marvell.com>; JogaRao Nartu <njogarao@marvell.com>; Bharath Rajendra <brajendra@marvell.com>; ci@dpdk.org; Akhil Goyal <gakhil@marvell.com>
Subject: Re: [EXTERNAL] CN10K Crypto Test Issue

 

Hi Bharath and others, Here is the download link to the dpdk-test produced with the native compile on the cn10k host: https: //drive. google. com/file/d/127sIjCok3ErxAjrvfncWbv7BOmgL8m6D/view?usp=sharing Thanks, looking forward to hearing back

Hi Bharath and others,

 

Here is the download link to the dpdk-test produced with the native compile on the cn10k host: https://drive.google.com/file/d/127sIjCok3ErxAjrvfncWbv7BOmgL8m6D/view?usp=sharing

 

Thanks, looking forward to hearing back from you about how a native build works on your setup. And, apologies for the late meeting time for you - I did not realize which time zone you were in. If it turns out that a followup meeting is required I will be sure to schedule it earlier in the day.

 

I'm also adding the DPDK crypto tree maintainer (Akhil Goyal) to the CC list for this thread.

 

Best,

Patrick

 

On Wed, Dec 18, 2024 at 12:53 PM Hiral Shah <hshah@marvell.com> wrote:

 

Looks like meeting is over. 

 

Thanks Bharath!

 

Regards,

Hiral

From: Patrick Robb <probb@iol.unh.edu>
Sent: Wednesday, December 18, 2024 9:45 AM
To: Hiral Shah <hshah@marvell.com>
Cc: Cody Cheng <ccheng@iol.unh.edu>; Gnanesh Kambalu Palanethra <gpalanethra@marvell.com>; JogaRao Nartu <njogarao@marvell.com>; Bharath Rajendra <brajendra@marvell.com>; ci@dpdk.org <ci@dpdk.org>
Subject: Re: [EXTERNAL] CN10K Crypto Test Issue

 

No worries, Bharath and the others are demonstrating the driver binding and autotest procedure. On Wed, Dec 18, 2024 at 12: 37 PM Hiral Shah <hshah@ marvell. com> wrote: Hi Cody, Patrick, Give me 10 -15 more meeting. I am on another call. 

No worries, Bharath and the others are demonstrating the driver binding and autotest procedure.

 

On Wed, Dec 18, 2024 at 12:37 PM Hiral Shah <hshah@marvell.com> wrote:

Hi Cody, Patrick,

 

Give me 10 -15 more meeting. I am on another call. 

 

 

Regards,

Hiral

From: Patrick Robb <probb@iol.unh.edu>
Sent: Monday, December 16, 2024 1:57 PM
To: Hiral Shah <hshah@marvell.com>
Cc: Cody Cheng <ccheng@iol.unh.edu>; Gnanesh Kambalu Palanethra <gpalanethra@marvell.com>; JogaRao Nartu <njogarao@marvell.com>; Bharath Rajendra <brajendra@marvell.com>; ci@dpdk.org <ci@dpdk.org>
Subject: Re: [EXTERNAL] CN10K Crypto Test Issue

 

Hi Hiral, Sorry about the last minute planning today - I am scheduling a call for us for Wednesday at 9: 30AM PST. Cody will be in class at that time but I will be available and I believe I know all the details of his issue. -----------------------------------------------------------------------------------------------------------

Hi Hiral,

 

Sorry about the last minute planning today - I am scheduling a call for us for Wednesday at 9:30AM PST. Cody will be in class at that time but I will be available and I believe I know all the details of his issue. 

 

-----------------------------------------------------------------------------------------------------------

 

Patrick Robb is inviting you to a scheduled Zoom meeting.

Topic: Marvell Crypto Sync
Time: Dec 18, 2024 12:30 PM Eastern Time (US and Canada)

Join from PC, Mac, Linux, iOS or Android: https://unh.zoom.us/j/93859170209

Keyboard shortcuts are available to navigate this Zoom meeting or webinar: https://support.zoom.us/hc/en-us/articles/205683899-Hot-Keys-and-Keyboard-for-Zoom
 
Or iPhone one-tap:  13052241968,93859170209# or 13092053325,93859170209#

Or Telephone:
    Dial: +1 305 224 1968 (US Toll)
    Meeting ID: 938 5917 0209
    International numbers available: https://unh.zoom.us/u/acTtiNQKou

Or a H.323/SIP room system:
    H.323: rc.unh.edu or 144.195.19.161 (US West) or 206.247.11.121 (US East)
    Meeting ID: 938 5917 0209
    SIP: 93859170209@zoomcrc.com

TROUBLESHOOTING STEPS:

Audio Echo In A Meeting: https://support.zoom.us/hc/en-us/articles/202050538-Audio-Echo-In-A-Meeting

Want to Join a Test Meeting?: https://zoom.us/test

 

On Mon, Dec 16, 2024 at 3:51 PM Patrick Robb <probb@iol.unh.edu> wrote:

Thanks Hiral for the offer of help, here is a zoom invite for a meeting from 1pm-2pm PST today (in 10 minutes).

 

Sorry for the last notice, I can certainly reschedule if needed.

 

Patrick Robb is inviting you to a scheduled Zoom meeting.

Topic: Marvell DPDK Cryptodev sync
Time: Dec 16, 2024 04:00 PM Eastern Time (US and Canada)

Join from PC, Mac, Linux, iOS or Android: https://unh.zoom.us/j/99749831096

Keyboard shortcuts are available to navigate this Zoom meeting or webinar: https://support.zoom.us/hc/en-us/articles/205683899-Hot-Keys-and-Keyboard-for-Zoom
 
Or iPhone one-tap:  16468769923,99749831096# or 16469313860,99749831096#

Or Telephone:
    Dial: +1 646 876 9923 (US Toll)
    Meeting ID: 997 4983 1096
    International numbers available: https://unh.zoom.us/u/aAZFiJCIr

Or a H.323/SIP room system:
    H.323: rc.unh.edu or 144.195.19.161 (US West) or 206.247.11.121 (US East)
    Meeting ID: 997 4983 1096
    SIP: 99749831096@zoomcrc.com

TROUBLESHOOTING STEPS:

Audio Echo In A Meeting: https://support.zoom.us/hc/en-us/articles/202050538-Audio-Echo-In-A-Meeting

Want to Join a Test Meeting?: https://zoom.us/test

 

On Mon, Dec 16, 2024 at 2:39 PM Patrick Robb <probb@iol.unh.edu> wrote:

Thanks, just confirming is this pacific time?

 

On Mon, Dec 16, 2024 at 2:27 PM Hiral Shah <hshah@marvell.com> wrote:

Sure!

 

We can have sync up today between 1PM -3 PM or Wednesday 8:30AM -1 PM. 

 

 

Regards,

Hiral

From: Patrick Robb <probb@iol.unh.edu>
Sent: Monday, December 16, 2024 10:55 AM
To: Hiral Shah <hshah@marvell.com>
Cc: Cody Cheng <ccheng@iol.unh.edu>; Gnanesh Kambalu Palanethra <gpalanethra@marvell.com>; JogaRao Nartu <njogarao@marvell.com>; Bharath Rajendra <brajendra@marvell.com>; ci@dpdk.org <ci@dpdk.org>
Subject: Re: [EXTERNAL] CN10K Crypto Test Issue

 

Hi Hiral, The cnxk crypto device VF creation and driver binding steps that Cody wrote in his email are actually straight from the SDK document. Specifically he followed the steps from doc-base-SDK12. 24. 11/dpdk/pmd/rte_cryptodev. html#initialization

Hi Hiral,

 

The cnxk crypto device VF creation and driver binding steps that Cody wrote in his email are actually straight from the SDK document. Specifically he followed the steps from doc-base-SDK12.24.11/dpdk/pmd/rte_cryptodev.html#initialization

 

Thanks for offering a sync up - this would be ideal as we are still stuck. We can schedule a Zoom and invite you. Do I remember correctly that you are in the pacific time zone? Is any particular day this week good for you? 

 

Thanks for all the help.

-Patrick

 

On Mon, Dec 16, 2024 at 1:33 PM Hiral Shah <hshah@marvell.com> wrote:

Hi Cody, 

 

Can you please refer our SDK document? It should have clear instructions. We can sync up if you still have any questions. 

 

Regards,

Hiral

From: Cody Cheng <ccheng@iol.unh.edu>
Sent: Friday, December 13, 2024 11:47 AM
To: Gnanesh Kambalu Palanethra <gpalanethra@marvell.com>; Hiral Shah <hshah@marvell.com>
Cc: JogaRao Nartu <njogarao@marvell.com>; Bharath Rajendra <brajendra@marvell.com>; Patrick Robb <probb@iol.unh.edu>; ci@dpdk.org <ci@dpdk.org>
Subject: [EXTERNAL] CN10K Crypto Test Issue

 

Hi Gnanesh & Hiral, My name is Cody Cheng, I'm a tester at the University of New Hampshire DPDK Community Test Lab. We are hosting a CN10K board here which is currently running some ethernet device tests on DPDK, and we would like to extend

Hi Gnanesh & Hiral,
 
My name is Cody Cheng, I'm a tester at the University of New Hampshire
DPDK Community Test Lab. We are hosting a CN10K board here which is
currently running some ethernet device tests on DPDK, and we would
like to extend our testing to include crypto device testing. Gnanesh
has written a patch which adds testcases to the DPDK Test Suite which
should allow us to do so.
 
However, I am running into difficulties with the crypto devices I
create in DPDK from the CN10K board. I would appreciate it if I can
sync with one of you next week so that I can show our current
configuration, and run through the DPDK crypto device setup process
and CN10K autotest (not passing currently). I am guessing there is
some error in our configuration. My coworker Patrick Robb has
mentioned he met with Hiral previously and it was a big help for
understanding how to flash the correct firmware, build the SDK and
tftpboot it, chroot to Ubuntu etc. I hope we can do something similar
to clear up the confusion regarding crypto devs. What timezones are
you in? I would be happy to schedule a Zoom call.
 
Otherwise, I will share some of the system info and the process I have
run through below, which might begin to give you an idea regarding our
status:
 
=======================
Marvell CN10k Boot Stub
=======================
Firmware Version: 2024-12-07 02:04:42
EBF Version: 12.24.11, Branch:
/MarvellSDK/base-SDK12.24.11/cn10ka-release-output/build/marvell-external-fw-SDK12.24.11/firmware/ebf,
Built: Sat, 07 Dec 2024 02:02:27 +0000
 
Board Model:    crb106
Board Revision: r1p1
Board Serial:   <redacted>
 
Chip:  0xb9 Pass A1
SKU:   MV-CN10624-A1-AAP
LLC:   49152 KB
Boot:  SPI1_CS0,EMMC_CS0, using SPI1_CS0
AVS:   Enabled
 
I am setting up 1 crypto virtual function using the commands here:
https://urldefense.proofpoint.com/v2/url?u=https-3A__doc.dpdk.org_guides_cryptodevs_cnxk.html-23initialization&d=DwIBaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=vGy6A_Vxl0vuken84vHnqSHz1sugoMRzgsw2uuNRAQ4&m=dN2bwgTF6BzuBPsPbKfZfmAua-foxlSSnMAoU8EjwCE3XN40flneiz1d7Mwl83iN&s=_d4kcuFSGcAGResAexjf_pFNDJ0Szm68pM2tw5BFzUM&e=
 
So afterwards I am left with 1 VF bound to vfio-pci at 0002:10.00.1
 
At this stage, according to the docs I should be able to launch
dpdk-test and run the cn10k symmetrical crypto autotest, using the
commands below:
 
```
./dpdk-test
RTE>>cryptodev_cn10k_autotest
```
 
However, the auto tests fail and fall into an error loop which I have
attached the logs of in this email.
 
Here is the EAL output from the logs:
 
EAL: Detected CPU lcores: 24
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
EAL: Using IOMMU type 1 (Type 1)
CNXK: RoC Model: cn10ka_a1 (HW_PLATFORM)
CRYPTODEV: Creating cryptodev 0002:20:00.1
CRYPTODEV: Initialisation parameters - name: 0002:20:00.1,socket id:
0, max queue pairs: 0
APP: HPET is not enabled, using TSC as default timer
 
In the EAL output, max queue pairs is 0 even though in the docs, it
says the Maximum queue pairs limit is set to a default of 63. Could
this be related to the issue?
 
Also, here is my kernel cmdline parameters:
 
console=ttyAMA0,115200n8 earlycon=p1011,0x87e028000000 maxcpus=24
rootwait root=/dev/nvme0n1p1 rw coherent_pool=16M
default_hugepagesz=512M hugepagesz=512M hugepages=8
 
Does this look correct?
 
I have also tried setting `iommu.passthrough=1` in the boot arguments
but that resulted in the same dpdk-test failure.
 
Best Regards,
Cody Cheng