Request to map PCI device in a secondary that uses the --block for that device

Request to map PCI device in a secondary that uses the --block for that device

[Antonio Di Bacco]

Is it correct that a primary requests a secondary to map a device that
the secondary explicitly blocks with the --block arg ?

IN my case this requests of mapping creates a crash in the secondary.

Using DPDK 21.11

Best regards,
Antonio.

Re: Request to map PCI device in a secondary that uses the --block for that device

[Dmitry Kozlyuk]

2024-05-22 08:46 (UTC+0200), Antonio Di Bacco:
> Is it correct that a primary requests a secondary to map a device that
> the secondary explicitly blocks with the --block arg ?
> 
> IN my case this requests of mapping creates a crash in the secondary.
> 
> Using DPDK 21.11
> 
> Best regards,
> Antonio.

Hi Antonio,

"Secondary processes which requires access to physical devices in Primary
process, must be passed with the same allow and block options."

https://doc.dpdk.org/guides/prog_guide/multi_proc_support.html

Re: Request to map PCI device in a secondary that uses the --block for that device

[Antonio Di Bacco]

Thank you.
In my case,  I have the same block/allow but the primary does an
explicit probe of a DMA engine on the processor, the secondary is
notified and it crashes.  It should not crash I suppose.
The same software is running on several machines (100) but the problem
is sporadic just on two of them.

Very strange.

Thx,
Antonio.

On Wed, May 22, 2024 at 9:21 AM Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> wrote:
>
> 2024-05-22 08:46 (UTC+0200), Antonio Di Bacco:
> > Is it correct that a primary requests a secondary to map a device that
> > the secondary explicitly blocks with the --block arg ?
> >
> > IN my case this requests of mapping creates a crash in the secondary.
> >
> > Using DPDK 21.11
> >
> > Best regards,
> > Antonio.
>
> Hi Antonio,
>
> "Secondary processes which requires access to physical devices in Primary
> process, must be passed with the same allow and block options."
>
> https://doc.dpdk.org/guides/prog_guide/multi_proc_support.html

Re: Request to map PCI device in a secondary that uses the --block for that device

[Antonio Di Bacco]

I found the exact line where I get the crash (the line saying TAILQ_FOREACH)

    /* if we're in a secondary process, just find our tailq entry */
    TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
        if (rte_pci_addr_cmp(&vfio_res->pci_addr,
                         &dev->addr))
            continue;
        break;
    }
    /* if we haven't found our tailq entry, something's wrong */
    if (vfio_res == NULL) {
        RTE_LOG(ERR, EAL, "%s cannot find TAILQ entry for PCI device!\n",
                pci_addr);
        return -1;
    }

On Wed, May 22, 2024 at 10:06 AM Antonio Di Bacco
<a.dibacco.ks@gmail.com> wrote:
>
> Thank you.
> In my case,  I have the same block/allow but the primary does an
> explicit probe of a DMA engine on the processor, the secondary is
> notified and it crashes.  It should not crash I suppose.
> The same software is running on several machines (100) but the problem
> is sporadic just on two of them.
>
> Very strange.
>
> Thx,
> Antonio.
>
> On Wed, May 22, 2024 at 9:21 AM Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> wrote:
> >
> > 2024-05-22 08:46 (UTC+0200), Antonio Di Bacco:
> > > Is it correct that a primary requests a secondary to map a device that
> > > the secondary explicitly blocks with the --block arg ?
> > >
> > > IN my case this requests of mapping creates a crash in the secondary.
> > >
> > > Using DPDK 21.11
> > >
> > > Best regards,
> > > Antonio.
> >
> > Hi Antonio,
> >
> > "Secondary processes which requires access to physical devices in Primary
> > process, must be passed with the same allow and block options."
> >
> > https://doc.dpdk.org/guides/prog_guide/multi_proc_support.html

Re: Request to map PCI device in a secondary that uses the --block for that device

[Antonio Di Bacco]

For completeness (vfio_res_list is NULL):

(gdb) bt
#0  0x00007f75f074e849 in pci_vfio_map_resource_secondary
(dev=0x7f377c00) at ../drivers/bus/pci/linux/pci_vfio.c:901
#1  0x00007f75f074ea7f in pci_vfio_map_resource (dev=0x7f377c00) at
../drivers/bus/pci/linux/pci_vfio.c:958
#2  0x00007f75f0749eb5 in rte_pci_map_device (dev=0x7f377c00) at
../drivers/bus/pci/linux/pci.c:70
#3  0x00007f75f0747e81 in rte_pci_probe_one_driver (dr=0x7f75f0397100
<ioat_pmd_drv>, dev=0x7f377c00) at ../drivers/bus/pci/pci_common.c:251
#4  0x00007f75f074820a in pci_probe_all_drivers (dev=0x7f377c00) at
../drivers/bus/pci/pci_common.c:353
#5  0x00007f75f07489b8 in pci_plug (dev=0x7f377c10) at
../drivers/bus/pci/pci_common.c:595
#6  0x00007f75f17a0a88 in local_dev_probe (devargs=0x7f75e4000bac
"0000:40:04.3", new_dev=0x7f75ea943868) at
../lib/eal/common/eal_common_dev.c:174
#7  0x00007f75f17c8832 in __handle_primary_request
(param=0x7f75e4000b60) at ../lib/eal/common/hotplug_mp.c:242
#8  0x00007f75f17ce78d in eal_alarm_callback (arg=0x0) at
../lib/eal/linux/eal_alarm.c:110
#9  0x00007f75f17d38c2 in eal_intr_process_interrupts
(events=0x7f75ea943b10, nfds=1) at
../lib/eal/linux/eal_interrupts.c:1025
#10 0x00007f75f17d3ba8 in eal_intr_handle_interrupts (pfd=16,
totalfds=2) at ../lib/eal/linux/eal_interrupts.c:1099
#11 0x00007f75f17d3d8f in eal_intr_thread_main (arg=0x0) at
../lib/eal/linux/eal_interrupts.c:1171
#12 0x00007f75f17b59c6 in ctrl_thread_init (arg=0x7f3640d0) at
../lib/eal/common/eal_common_thread.c:206
#13 0x00007f75f1b5d802 in start_thread () from /lib64/libc.so.6
#14 0x00007f75f1afd450 in clone3 () from /lib64/libc.so.6
(gdb) p vfio_res
$1 = (struct mapped_pci_resource *) 0x0
(gdb) p vfio_res_list
$2 = (struct mapped_pci_res_list *) 0x0
(gdb) p pci_addr
$3 = "0000:40:04.3", '\000' <repeats 4083 times>
(gdb)

On Wed, May 22, 2024 at 7:03 PM Antonio Di Bacco <a.dibacco.ks@gmail.com> wrote:
>
> I found the exact line where I get the crash (the line saying TAILQ_FOREACH)
>
>     /* if we're in a secondary process, just find our tailq entry */
>     TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
>         if (rte_pci_addr_cmp(&vfio_res->pci_addr,
>                          &dev->addr))
>             continue;
>         break;
>     }
>     /* if we haven't found our tailq entry, something's wrong */
>     if (vfio_res == NULL) {
>         RTE_LOG(ERR, EAL, "%s cannot find TAILQ entry for PCI device!\n",
>                 pci_addr);
>         return -1;
>     }
>
> On Wed, May 22, 2024 at 10:06 AM Antonio Di Bacco
> <a.dibacco.ks@gmail.com> wrote:
> >
> > Thank you.
> > In my case,  I have the same block/allow but the primary does an
> > explicit probe of a DMA engine on the processor, the secondary is
> > notified and it crashes.  It should not crash I suppose.
> > The same software is running on several machines (100) but the problem
> > is sporadic just on two of them.
> >
> > Very strange.
> >
> > Thx,
> > Antonio.
> >
> > On Wed, May 22, 2024 at 9:21 AM Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> wrote:
> > >
> > > 2024-05-22 08:46 (UTC+0200), Antonio Di Bacco:
> > > > Is it correct that a primary requests a secondary to map a device that
> > > > the secondary explicitly blocks with the --block arg ?
> > > >
> > > > IN my case this requests of mapping creates a crash in the secondary.
> > > >
> > > > Using DPDK 21.11
> > > >
> > > > Best regards,
> > > > Antonio.
> > >
> > > Hi Antonio,
> > >
> > > "Secondary processes which requires access to physical devices in Primary
> > > process, must be passed with the same allow and block options."
> > >
> > > https://doc.dpdk.org/guides/prog_guide/multi_proc_support.html