[dpdk-dev] SRIOV hot unplug

[dpdk-dev] SRIOV hot unplug

[Eli Britstein]

Hello,

A DPDK application with a DPDK bond device, with 2 slaves: one vnic, and another is a SRIOV VF connected as a pathrough.
The bond device is configured as ACTIVE/BACKUP, and the primary is the VF slave.
Now, I do "virsh detach-device" from the host, and the DPDK process in the VM gets segmentation fault, as it tries to poll an address that is not mmaped anymore.
I wonder if this flow is supposed to be supported by DPDK, or not.
Please advise.

Thanks,
Eli
-------------------------------------------------------------------------------------------------------------------------------------------------
This email and any files transmitted and/or attachments with it are confidential and proprietary information of
Toga Networks Ltd., and intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager. This message contains confidential
information of Toga Networks Ltd., and is intended only for the individual named. If you are not the named
addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately
by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not
the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on
the contents of this information is strictly prohibited.
------------------------------------------------------------------------------------------------------------------------------------------------

Re: [dpdk-dev] SRIOV hot unplug

[Iremonger, Bernard]

Hi Eli,

The DPDK application in the VM should remove the slave device from the bond device, stop, close and detach the device in the VM before doing "virsh detach-device" from the host.

Regards,

Bernard.


> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Eli Britstein
> Sent: Sunday, July 17, 2016 9:58 AM
> To: dev@dpdk.org
> Subject: [dpdk-dev] SRIOV hot unplug
> 
> Hello,
> 
> A DPDK application with a DPDK bond device, with 2 slaves: one vnic, and
> another is a SRIOV VF connected as a pathrough.
> The bond device is configured as ACTIVE/BACKUP, and the primary is the VF
> slave.
> Now, I do "virsh detach-device" from the host, and the DPDK process in the
> VM gets segmentation fault, as it tries to poll an address that is not mmaped
> anymore.
> I wonder if this flow is supposed to be supported by DPDK, or not.
> Please advise.
> 
> Thanks,
> Eli
> ----------------------------------------------------------------------------------------------
> ---------------------------------------------------
> This email and any files transmitted and/or attachments with it are
> confidential and proprietary information of Toga Networks Ltd., and
> intended solely for the use of the individual or entity to whom they are
> addressed.
> If you have received this email in error please notify the system manager.
> This message contains confidential information of Toga Networks Ltd., and is
> intended only for the individual named. If you are not the named addressee
> you should not disseminate, distribute or copy this e-mail. Please notify the
> sender immediately by e-mail if you have received this e-mail by mistake and
> delete this e-mail from your system. If you are not the intended recipient
> you are notified that disclosing, copying, distributing or taking any action in
> reliance on the contents of this information is strictly prohibited.
> ----------------------------------------------------------------------------------------------
> --------------------------------------------------

Re: [dpdk-dev] SRIOV hot unplug

[Eli Britstein]

Hi Bernard,

Thank you for your answer.
However, to do this, I have to have some communication protocol to the VM's application in order for it to do this sequence and acknowledge that it is now safe to proceed with detaching the device.
This implies some kind of integration from the host side, which I would like to avoid.
Do you think might there be any other way for the application to handle such event in a smooth way?

Thanks,
Eli

> -----Original Message-----
> From: Iremonger, Bernard [mailto:bernard.iremonger@intel.com]
> Sent: Sunday, 17 July, 2016 11:53 PM
> To: Eli Britstein; dev@dpdk.org
> Cc: Iremonger, Bernard
> Subject: RE: SRIOV hot unplug
>
> Hi Eli,
>
> The DPDK application in the VM should remove the slave device from the
> bond device, stop, close and detach the device in the VM before doing "virsh
> detach-device" from the host.
>
> Regards,
>
> Bernard.
>
>
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Eli Britstein
> > Sent: Sunday, July 17, 2016 9:58 AM
> > To: dev@dpdk.org
> > Subject: [dpdk-dev] SRIOV hot unplug
> >
> > Hello,
> >
> > A DPDK application with a DPDK bond device, with 2 slaves: one vnic,
> > and another is a SRIOV VF connected as a pathrough.
> > The bond device is configured as ACTIVE/BACKUP, and the primary is the
> > VF slave.
> > Now, I do "virsh detach-device" from the host, and the DPDK process in
> > the VM gets segmentation fault, as it tries to poll an address that is
> > not mmaped anymore.
> > I wonder if this flow is supposed to be supported by DPDK, or not.
> > Please advise.
> >
> > Thanks,
> > Eli
> > ----------------------------------------------------------------------
> > ------------------------
> > ---------------------------------------------------
> > This email and any files transmitted and/or attachments with it are
> > confidential and proprietary information of Toga Networks Ltd., and
> > intended solely for the use of the individual or entity to whom they
> > are addressed.
> > If you have received this email in error please notify the system manager.
> > This message contains confidential information of Toga Networks Ltd.,
> > and is intended only for the individual named. If you are not the
> > named addressee you should not disseminate, distribute or copy this
> > e-mail. Please notify the sender immediately by e-mail if you have
> > received this e-mail by mistake and delete this e-mail from your
> > system. If you are not the intended recipient you are notified that
> > disclosing, copying, distributing or taking any action in reliance on the
> contents of this information is strictly prohibited.
> > ----------------------------------------------------------------------
> > ------------------------
> > --------------------------------------------------

-------------------------------------------------------------------------------------------------------------------------------------------------
This email and any files transmitted and/or attachments with it are confidential and proprietary information of
Toga Networks Ltd., and intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager. This message contains confidential
information of Toga Networks Ltd., and is intended only for the individual named. If you are not the named
addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately
by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not
the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on
the contents of this information is strictly prohibited.
------------------------------------------------------------------------------------------------------------------------------------------------

Re: [dpdk-dev] SRIOV hot unplug

[Tetsuya Mukawa]

Hi Eli,

On 2016/07/18 17:47, Eli Britstein wrote:
> Hi Bernard,
> 
> Thank you for your answer.
> However, to do this, I have to have some communication protocol to the VM's application in order for it to do this sequence and acknowledge that it is now safe to proceed with detaching the device.
> This implies some kind of integration from the host side, which I would like to avoid.

I guess you should have some kind of communication channel to notice the
hotpluging events from host to VM.

> Do you think might there be any other way for the application to handle such event in a smooth way?

So far, I guess having one more virtio-net device will be easiest way.

Thanks,
Tetsuya

> 
> Thanks,
> Eli
> 
>> -----Original Message-----
>> From: Iremonger, Bernard [mailto:bernard.iremonger@intel.com]
>> Sent: Sunday, 17 July, 2016 11:53 PM
>> To: Eli Britstein; dev@dpdk.org
>> Cc: Iremonger, Bernard
>> Subject: RE: SRIOV hot unplug
>>
>> Hi Eli,
>>
>> The DPDK application in the VM should remove the slave device from the
>> bond device, stop, close and detach the device in the VM before doing "virsh
>> detach-device" from the host.
>>
>> Regards,
>>
>> Bernard.
>>
>>
>>> -----Original Message-----
>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Eli Britstein
>>> Sent: Sunday, July 17, 2016 9:58 AM
>>> To: dev@dpdk.org
>>> Subject: [dpdk-dev] SRIOV hot unplug
>>>
>>> Hello,
>>>
>>> A DPDK application with a DPDK bond device, with 2 slaves: one vnic,
>>> and another is a SRIOV VF connected as a pathrough.
>>> The bond device is configured as ACTIVE/BACKUP, and the primary is the
>>> VF slave.
>>> Now, I do "virsh detach-device" from the host, and the DPDK process in
>>> the VM gets segmentation fault, as it tries to poll an address that is
>>> not mmaped anymore.
>>> I wonder if this flow is supposed to be supported by DPDK, or not.
>>> Please advise.
>>>
>>> Thanks,
>>> Eli
>>> ----------------------------------------------------------------------
>>> ------------------------
>>> ---------------------------------------------------
>>> This email and any files transmitted and/or attachments with it are
>>> confidential and proprietary information of Toga Networks Ltd., and
>>> intended solely for the use of the individual or entity to whom they
>>> are addressed.
>>> If you have received this email in error please notify the system manager.
>>> This message contains confidential information of Toga Networks Ltd.,
>>> and is intended only for the individual named. If you are not the
>>> named addressee you should not disseminate, distribute or copy this
>>> e-mail. Please notify the sender immediately by e-mail if you have
>>> received this e-mail by mistake and delete this e-mail from your
>>> system. If you are not the intended recipient you are notified that
>>> disclosing, copying, distributing or taking any action in reliance on the
>> contents of this information is strictly prohibited.
>>> ----------------------------------------------------------------------
>>> ------------------------
>>> --------------------------------------------------
> 
> -------------------------------------------------------------------------------------------------------------------------------------------------
> This email and any files transmitted and/or attachments with it are confidential and proprietary information of
> Toga Networks Ltd., and intended solely for the use of the individual or entity to whom they are addressed.
> If you have received this email in error please notify the system manager. This message contains confidential
> information of Toga Networks Ltd., and is intended only for the individual named. If you are not the named
> addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately
> by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not
> the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on
> the contents of this information is strictly prohibited.
> ------------------------------------------------------------------------------------------------------------------------------------------------
> 

Re: [dpdk-dev] SRIOV hot unplug

[Eli Britstein]

> -----Original Message-----
> From: Tetsuya Mukawa [mailto:mukawa@igel.co.jp]
> Sent: Tuesday, 19 July, 2016 5:49 AM
> To: Eli Britstein; dev@dpdk.org
> Cc: Iremonger, Bernard
> Subject: Re: [dpdk-dev] SRIOV hot unplug
>
> Hi Eli,
>
> On 2016/07/18 17:47, Eli Britstein wrote:
> > Hi Bernard,
> >
> > Thank you for your answer.
> > However, to do this, I have to have some communication protocol to the
> VM's application in order for it to do this sequence and acknowledge that it is
> now safe to proceed with detaching the device.
> > This implies some kind of integration from the host side, which I would like
> to avoid.
>
> I guess you should have some kind of communication channel to notice the
> hotpluging events from host to VM.
[Eli Britstein]
In order just to notice the hotplugging events inside the VM, I can use add some udev action in the VM, in /etc/udev/rules.d/XXX
However, those are asynchronous events. The host proceeds with unplugging without waiting for the VM to acknowledge it.

>
> > Do you think might there be any other way for the application to handle
> such event in a smooth way?
>
> So far, I guess having one more virtio-net device will be easiest way.
[Eli Britstein]
Could you please elaborate your meaning? How do you mean to use this extra virtio-net device?

To clarify: I would like to have my bond device automatically set the vNIC as its primary, and close/remove the VF before it is unplugged.

Thanks,
Eli

>
> Thanks,
> Tetsuya
>
> >
> > Thanks,
> > Eli
> >
> >> -----Original Message-----
> >> From: Iremonger, Bernard [mailto:bernard.iremonger@intel.com]
> >> Sent: Sunday, 17 July, 2016 11:53 PM
> >> To: Eli Britstein; dev@dpdk.org
> >> Cc: Iremonger, Bernard
> >> Subject: RE: SRIOV hot unplug
> >>
> >> Hi Eli,
> >>
> >> The DPDK application in the VM should remove the slave device from
> >> the bond device, stop, close and detach the device in the VM before
> >> doing "virsh detach-device" from the host.
> >>
> >> Regards,
> >>
> >> Bernard.
> >>
> >>
> >>> -----Original Message-----
> >>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Eli Britstein
> >>> Sent: Sunday, July 17, 2016 9:58 AM
> >>> To: dev@dpdk.org
> >>> Subject: [dpdk-dev] SRIOV hot unplug
> >>>
> >>> Hello,
> >>>
> >>> A DPDK application with a DPDK bond device, with 2 slaves: one vnic,
> >>> and another is a SRIOV VF connected as a pathrough.
> >>> The bond device is configured as ACTIVE/BACKUP, and the primary is
> >>> the VF slave.
> >>> Now, I do "virsh detach-device" from the host, and the DPDK process
> >>> in the VM gets segmentation fault, as it tries to poll an address
> >>> that is not mmaped anymore.
> >>> I wonder if this flow is supposed to be supported by DPDK, or not.
> >>> Please advise.
> >>>
> >>> Thanks,
> >>> Eli
> >>> --------------------------------------------------------------------
> >>> --
> >>> ------------------------
> >>> ---------------------------------------------------
> >>> This email and any files transmitted and/or attachments with it are
> >>> confidential and proprietary information of Toga Networks Ltd., and
> >>> intended solely for the use of the individual or entity to whom they
> >>> are addressed.
> >>> If you have received this email in error please notify the system
> manager.
> >>> This message contains confidential information of Toga Networks
> >>> Ltd., and is intended only for the individual named. If you are not
> >>> the named addressee you should not disseminate, distribute or copy
> >>> this e-mail. Please notify the sender immediately by e-mail if you
> >>> have received this e-mail by mistake and delete this e-mail from
> >>> your system. If you are not the intended recipient you are notified
> >>> that disclosing, copying, distributing or taking any action in
> >>> reliance on the
> >> contents of this information is strictly prohibited.
> >>> --------------------------------------------------------------------
> >>> --
> >>> ------------------------
> >>> --------------------------------------------------
> >
> > ----------------------------------------------------------------------
> > ----------------------------------------------------------------------
> > ----- This email and any files transmitted and/or attachments with it
> > are confidential and proprietary information of Toga Networks Ltd.,
> > and intended solely for the use of the individual or entity to whom they are
> addressed.
> > If you have received this email in error please notify the system
> > manager. This message contains confidential information of Toga
> > Networks Ltd., and is intended only for the individual named. If you
> > are not the named addressee you should not disseminate, distribute or
> > copy this e-mail. Please notify the sender immediately by e-mail if
> > you have received this e-mail by mistake and delete this e-mail from your
> system. If you are not the intended recipient you are notified that disclosing,
> copying, distributing or taking any action in reliance on the contents of this
> information is strictly prohibited.
> > ----------------------------------------------------------------------
> > ----------------------------------------------------------------------
> > ----
> >

-------------------------------------------------------------------------------------------------------------------------------------------------
This email and any files transmitted and/or attachments with it are confidential and proprietary information of
Toga Networks Ltd., and intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager. This message contains confidential
information of Toga Networks Ltd., and is intended only for the individual named. If you are not the named
addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately
by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not
the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on
the contents of this information is strictly prohibited.
------------------------------------------------------------------------------------------------------------------------------------------------

Re: [dpdk-dev] SRIOV hot unplug

[Declan Doherty]

On 07/19/2016 12:18 PM, Eli Britstein wrote:
>
>
>> -----Original Message-----
>> From: Tetsuya Mukawa [mailto:mukawa@igel.co.jp]
>> Sent: Tuesday, 19 July, 2016 5:49 AM
>> To: Eli Britstein; dev@dpdk.org
>> Cc: Iremonger, Bernard
>> Subject: Re: [dpdk-dev] SRIOV hot unplug
>>
>> Hi Eli,
>>
>> On 2016/07/18 17:47, Eli Britstein wrote:
>>> Hi Bernard,
>>>
>>> Thank you for your answer.
>>> However, to do this, I have to have some communication protocol to the
>> VM's application in order for it to do this sequence and acknowledge that it is
>> now safe to proceed with detaching the device.
>>> This implies some kind of integration from the host side, which I would like
>> to avoid.
>>
>> I guess you should have some kind of communication channel to notice the
>> hotpluging events from host to VM.
> [Eli Britstein]
> In order just to notice the hotplugging events inside the VM, I can use add some udev action in the VM, in /etc/udev/rules.d/XXX
> However, those are asynchronous events. The host proceeds with unplugging without waiting for the VM to acknowledge it.
>
>>
>>> Do you think might there be any other way for the application to handle
>> such event in a smooth way?
>>
>> So far, I guess having one more virtio-net device will be easiest way.
> [Eli Britstein]
> Could you please elaborate your meaning? How do you mean to use this extra virtio-net device?
>
> To clarify: I would like to have my bond device automatically set the vNIC as its primary, and close/remove the VF before it is unplugged.
>
> Thanks,
> Eli

The only that I can think of which would allow bonding to support this 
is if you could force a link down notification to the VF. The bonding 
driver should automatically then fail over to the vNIC, but again this 
is asynchronous event so I'm not sure how you could confirm this action 
from the host other than by monitoring traffic?

Also I don't know if the DPDK environment has been tested to support the 
removal of dynamic hot-plugging you are suggesting via the udev action. 
There are no hooks that I'm aware of which would detect that the PCI 
device is no longer available and then clean the resources away in DPDK.

>
<snip>
>

Also could you please drop the footer/disclaimer on replies to the 
mailing list (see http://dpdk.org/ml)

Re: [dpdk-dev] SRIOV hot unplug

[Eli Britstein]

> -----Original Message-----
> From: Declan Doherty [mailto:declan.doherty@intel.com]
> Sent: Tuesday, 19 July, 2016 5:15 PM
> To: Eli Britstein; Tetsuya Mukawa; dev@dpdk.org
> Cc: Iremonger, Bernard
> Subject: Re: [dpdk-dev] SRIOV hot unplug
>
> On 07/19/2016 12:18 PM, Eli Britstein wrote:
> >
> >
> >> -----Original Message-----
> >> From: Tetsuya Mukawa [mailto:mukawa@igel.co.jp]
> >> Sent: Tuesday, 19 July, 2016 5:49 AM
> >> To: Eli Britstein; dev@dpdk.org
> >> Cc: Iremonger, Bernard
> >> Subject: Re: [dpdk-dev] SRIOV hot unplug
> >>
> >> Hi Eli,
> >>
> >> On 2016/07/18 17:47, Eli Britstein wrote:
> >>> Hi Bernard,
> >>>
> >>> Thank you for your answer.
> >>> However, to do this, I have to have some communication protocol to
> >>> the
> >> VM's application in order for it to do this sequence and acknowledge
> >> that it is now safe to proceed with detaching the device.
> >>> This implies some kind of integration from the host side, which I
> >>> would like
> >> to avoid.
> >>
> >> I guess you should have some kind of communication channel to notice
> >> the hotpluging events from host to VM.
> > [Eli Britstein]
> > In order just to notice the hotplugging events inside the VM, I can
> > use add some udev action in the VM, in /etc/udev/rules.d/XXX However,
> those are asynchronous events. The host proceeds with unplugging without
> waiting for the VM to acknowledge it.
> >
> >>
> >>> Do you think might there be any other way for the application to
> >>> handle
> >> such event in a smooth way?
> >>
> >> So far, I guess having one more virtio-net device will be easiest way.
> > [Eli Britstein]
> > Could you please elaborate your meaning? How do you mean to use this
> extra virtio-net device?
> >
> > To clarify: I would like to have my bond device automatically set the vNIC as
> its primary, and close/remove the VF before it is unplugged.
> >
> > Thanks,
> > Eli
>
> The only that I can think of which would allow bonding to support this is if you
> could force a link down notification to the VF. The bonding driver should
> automatically then fail over to the vNIC, but again this is asynchronous event
> so I'm not sure how you could confirm this action from the host other than by
> monitoring traffic?
[Eli Britstein]
How can I force link down notification to the VF?

>
> Also I don't know if the DPDK environment has been tested to support the
> removal of dynamic hot-plugging you are suggesting via the udev action.
> There are no hooks that I'm aware of which would detect that the PCI device
> is no longer available and then clean the resources away in DPDK.
>
> >
> <snip>
> >
>
> Also could you please drop the footer/disclaimer on replies to the
> mailing list (see http://dpdk.org/ml)

-------------------------------------------------------------------------------------------------------------------------------------------------
This email and any files transmitted and/or attachments with it are confidential and proprietary information of
Toga Networks Ltd., and intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager. This message contains confidential
information of Toga Networks Ltd., and is intended only for the individual named. If you are not the named
addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately
by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not
the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on
the contents of this information is strictly prohibited.
------------------------------------------------------------------------------------------------------------------------------------------------

Re: [dpdk-dev] SRIOV hot unplug

[Eli Britstein]

Integrating Tetsuya's last comments on top of the latest message and reply.

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Eli Britstein
> Sent: Tuesday, 19 July, 2016 5:34 PM
> To: Declan Doherty; Tetsuya Mukawa; dev@dpdk.org
> Cc: Iremonger, Bernard
> Subject: Re: [dpdk-dev] SRIOV hot unplug
>
>
>
> > -----Original Message-----
> > From: Declan Doherty [mailto:declan.doherty@intel.com]
> > Sent: Tuesday, 19 July, 2016 5:15 PM
> > To: Eli Britstein; Tetsuya Mukawa; dev@dpdk.org
> > Cc: Iremonger, Bernard
> > Subject: Re: [dpdk-dev] SRIOV hot unplug
> >
> > On 07/19/2016 12:18 PM, Eli Britstein wrote:
> > >
> > >
> > >> -----Original Message-----
> > >> From: Tetsuya Mukawa [mailto:mukawa@igel.co.jp]
> > >> Sent: Tuesday, 19 July, 2016 5:49 AM
> > >> To: Eli Britstein; dev@dpdk.org
> > >> Cc: Iremonger, Bernard
> > >> Subject: Re: [dpdk-dev] SRIOV hot unplug
> > >>
> > >> Hi Eli,
> > >>
> > >> On 2016/07/18 17:47, Eli Britstein wrote:
> > >>> Hi Bernard,
> > >>>
> > >>> Thank you for your answer.
> > >>> However, to do this, I have to have some communication protocol to
> > >>> the
> > >> VM's application in order for it to do this sequence and
> > >> acknowledge that it is now safe to proceed with detaching the device.
> > >>> This implies some kind of integration from the host side, which I
> > >>> would like
> > >> to avoid.
> > >>
> > >> I guess you should have some kind of communication channel to
> > >> notice the hotpluging events from host to VM.
> > > [Eli Britstein]
> > > In order just to notice the hotplugging events inside the VM, I can
> > > use add some udev action in the VM, in /etc/udev/rules.d/XXX
> > > However,
> > those are asynchronous events. The host proceeds with unplugging
> > without waiting for the VM to acknowledge it.
> > >
[Tetsuya] Yes, it will be asynchronous.

> > >>
> > >>> Do you think might there be any other way for the application to
> > >>> handle
> > >> such event in a smooth way?
> > >>
> > >> So far, I guess having one more virtio-net device will be easiest way.
> > > [Eli Britstein]
> > > Could you please elaborate your meaning? How do you mean to use this
> > extra virtio-net device?
> > >
[Tetsuya] If you have one more network device, you can send and receive some kind of control messages between host and VM.
It may be okay to use general socket API for this additional network device. So I guess it's not so difficult to additionally implement it.

[Eli Britstein] Again, my goal is to do the hot unplug *without* any change/integration in the host side, just from the guest side.

Thanks, Eli

Thanks,
Tetsuya
> > > To clarify: I would like to have my bond device automatically set
> > > the vNIC as
> > its primary, and close/remove the VF before it is unplugged.
> > >
> > > Thanks,
> > > Eli
> >
> > The only that I can think of which would allow bonding to support this
> > is if you could force a link down notification to the VF. The bonding
> > driver should automatically then fail over to the vNIC, but again this
> > is asynchronous event so I'm not sure how you could confirm this
> > action from the host other than by monitoring traffic?
> [Eli Britstein]
> How can I force link down notification to the VF?
>
> >
> > Also I don't know if the DPDK environment has been tested to support
> > the removal of dynamic hot-plugging you are suggesting via the udev
> action.
> > There are no hooks that I'm aware of which would detect that the PCI
> > device is no longer available and then clean the resources away in DPDK.
> >
> > >
> > <snip>
> > >
> >
> > Also could you please drop the footer/disclaimer on replies to the
> > mailing list (see http://dpdk.org/ml)
>
> ----------------------------------------------------------------------------------------------
> ---------------------------------------------------
> This email and any files transmitted and/or attachments with it are
> confidential and proprietary information of Toga Networks Ltd., and
> intended solely for the use of the individual or entity to whom they are
> addressed.
> If you have received this email in error please notify the system manager.
> This message contains confidential information of Toga Networks Ltd., and is
> intended only for the individual named. If you are not the named addressee
> you should not disseminate, distribute or copy this e-mail. Please notify the
> sender immediately by e-mail if you have received this e-mail by mistake and
> delete this e-mail from your system. If you are not the intended recipient
> you are notified that disclosing, copying, distributing or taking any action in
> reliance on the contents of this information is strictly prohibited.
> ----------------------------------------------------------------------------------------------
> --------------------------------------------------
>
> ----------------------------------------------------------------------------------------------
> ---------------------------------------------------
> This email and any files transmitted and/or attachments with it are
> confidential and proprietary information of Toga Networks Ltd., and
> intended solely for the use of the individual or entity to whom they are
> addressed.
> If you have received this email in error please notify the system manager.
> This message contains confidential information of Toga Networks Ltd., and is
> intended only for the individual named. If you are not the named addressee
> you should not disseminate, distribute or copy this e-mail. Please notify the
> sender immediately by e-mail if you have received this e-mail by mistake and
> delete this e-mail from your system. If you are not the intended recipient
> you are notified that disclosing, copying, distributing or taking any action in
> reliance on the contents of this information is strictly prohibited.
> ----------------------------------------------------------------------------------------------
> --------------------------------------------------

-------------------------------------------------------------------------------------------------------------------------------------------------
This email and any files transmitted and/or attachments with it are confidential and proprietary information of
Toga Networks Ltd., and intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager. This message contains confidential
information of Toga Networks Ltd., and is intended only for the individual named. If you are not the named
addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately
by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not
the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on
the contents of this information is strictly prohibited.
------------------------------------------------------------------------------------------------------------------------------------------------

Re: [dpdk-dev] SRIOV hot unplug

[Tetsuya Mukawa]

Hi Eli,

On 2016/07/19 20:18, Eli Britstein wrote:
> 
> 
>> -----Original Message-----
>> From: Tetsuya Mukawa [mailto:mukawa@igel.co.jp]
>> Sent: Tuesday, 19 July, 2016 5:49 AM
>> To: Eli Britstein; dev@dpdk.org
>> Cc: Iremonger, Bernard
>> Subject: Re: [dpdk-dev] SRIOV hot unplug
>>
>> Hi Eli,
>>
>> On 2016/07/18 17:47, Eli Britstein wrote:
>>> Hi Bernard,
>>>
>>> Thank you for your answer.
>>> However, to do this, I have to have some communication protocol to the
>> VM's application in order for it to do this sequence and acknowledge that it is
>> now safe to proceed with detaching the device.
>>> This implies some kind of integration from the host side, which I would like
>> to avoid.
>>
>> I guess you should have some kind of communication channel to notice the
>> hotpluging events from host to VM.
> [Eli Britstein]
> In order just to notice the hotplugging events inside the VM, I can use add some udev action in the VM, in /etc/udev/rules.d/XXX
> However, those are asynchronous events. The host proceeds with unplugging without waiting for the VM to acknowledge it.

Yes, it will be asynchronous.

> 
>>
>>> Do you think might there be any other way for the application to handle
>> such event in a smooth way?
>>
>> So far, I guess having one more virtio-net device will be easiest way.
> [Eli Britstein]
> Could you please elaborate your meaning? How do you mean to use this extra virtio-net device?

If you have one more network device, you can send and receive some kind
of control messages between host and VM.
It may be okay to use general socket API for this additional network
device. So I guess it's not so difficult to additionally implement it.

Thanks,
Tetsuya

> 
> To clarify: I would like to have my bond device automatically set the vNIC as its primary, and close/remove the VF before it is unplugged.
> 
> Thanks,
> Eli
> 
>>
>> Thanks,
>> Tetsuya
>>
>>>
>>> Thanks,
>>> Eli
>>>
>>>> -----Original Message-----
>>>> From: Iremonger, Bernard [mailto:bernard.iremonger@intel.com]
>>>> Sent: Sunday, 17 July, 2016 11:53 PM
>>>> To: Eli Britstein; dev@dpdk.org
>>>> Cc: Iremonger, Bernard
>>>> Subject: RE: SRIOV hot unplug
>>>>
>>>> Hi Eli,
>>>>
>>>> The DPDK application in the VM should remove the slave device from
>>>> the bond device, stop, close and detach the device in the VM before
>>>> doing "virsh detach-device" from the host.
>>>>
>>>> Regards,
>>>>
>>>> Bernard.
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Eli Britstein
>>>>> Sent: Sunday, July 17, 2016 9:58 AM
>>>>> To: dev@dpdk.org
>>>>> Subject: [dpdk-dev] SRIOV hot unplug
>>>>>
>>>>> Hello,
>>>>>
>>>>> A DPDK application with a DPDK bond device, with 2 slaves: one vnic,
>>>>> and another is a SRIOV VF connected as a pathrough.
>>>>> The bond device is configured as ACTIVE/BACKUP, and the primary is
>>>>> the VF slave.
>>>>> Now, I do "virsh detach-device" from the host, and the DPDK process
>>>>> in the VM gets segmentation fault, as it tries to poll an address
>>>>> that is not mmaped anymore.
>>>>> I wonder if this flow is supposed to be supported by DPDK, or not.
>>>>> Please advise.
>>>>>
>>>>> Thanks,
>>>>> Eli
>>>>> --------------------------------------------------------------------
>>>>> --
>>>>> ------------------------
>>>>> ---------------------------------------------------
>>>>> This email and any files transmitted and/or attachments with it are
>>>>> confidential and proprietary information of Toga Networks Ltd., and
>>>>> intended solely for the use of the individual or entity to whom they
>>>>> are addressed.
>>>>> If you have received this email in error please notify the system
>> manager.
>>>>> This message contains confidential information of Toga Networks
>>>>> Ltd., and is intended only for the individual named. If you are not
>>>>> the named addressee you should not disseminate, distribute or copy
>>>>> this e-mail. Please notify the sender immediately by e-mail if you
>>>>> have received this e-mail by mistake and delete this e-mail from
>>>>> your system. If you are not the intended recipient you are notified
>>>>> that disclosing, copying, distributing or taking any action in
>>>>> reliance on the
>>>> contents of this information is strictly prohibited.
>>>>> --------------------------------------------------------------------
>>>>> --
>>>>> ------------------------
>>>>> --------------------------------------------------
>>>
>>> ----------------------------------------------------------------------
>>> ----------------------------------------------------------------------
>>> ----- This email and any files transmitted and/or attachments with it
>>> are confidential and proprietary information of Toga Networks Ltd.,
>>> and intended solely for the use of the individual or entity to whom they are
>> addressed.
>>> If you have received this email in error please notify the system
>>> manager. This message contains confidential information of Toga
>>> Networks Ltd., and is intended only for the individual named. If you
>>> are not the named addressee you should not disseminate, distribute or
>>> copy this e-mail. Please notify the sender immediately by e-mail if
>>> you have received this e-mail by mistake and delete this e-mail from your
>> system. If you are not the intended recipient you are notified that disclosing,
>> copying, distributing or taking any action in reliance on the contents of this
>> information is strictly prohibited.
>>> ----------------------------------------------------------------------
>>> ----------------------------------------------------------------------
>>> ----
>>>
> 
> -------------------------------------------------------------------------------------------------------------------------------------------------
> This email and any files transmitted and/or attachments with it are confidential and proprietary information of
> Toga Networks Ltd., and intended solely for the use of the individual or entity to whom they are addressed.
> If you have received this email in error please notify the system manager. This message contains confidential
> information of Toga Networks Ltd., and is intended only for the individual named. If you are not the named
> addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately
> by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not
> the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on
> the contents of this information is strictly prohibited.
> ------------------------------------------------------------------------------------------------------------------------------------------------
>