[dpdk-dev] i40e_aq_get_phy_capabilities() fails when using SFP+ with no link

[dpdk-dev] i40e_aq_get_phy_capabilities() fails when using SFP+ with no link

[Christos Ricudis]

Hello, 

Using a X710 based 4-port 4x10Gbit NIC, I have came across the following issue on the i40e PMD: 

When an optical SFP+ (Finisar FTLX8571D3BCL) is used with no active link partner on the other end of the link (or fiber completely disconnected from the SFP+), i40e_aq_get_phy_capabilities() (called by i40e_dev_sync_phy_type() on port initialization), fails with a 0x05 return value (EIO) on the AQ response structure. The struct i40e_aq_get_phy_abilities_resp buffer passed to the Get Phy Abilities command is unmodified upon return. 

This prevents DPDK 16.11 from initializing the port, and ultimately fails with the following error: 

PMD: eth_i40e_dev_init(): Failed to sync phy type: -95

The change introducing this issue was http://dpdk.org/ml/archives/dev/2016-September/047663.html

Reading the X710 datasheet, I notice that no specific mention is given on the meaning of EIO as a response to Get PHY Abilities command (opcode 0x0600), whereas in most other commands, an explicit mention of the meaning of the possible error status responses is given. 

This behaviour is the same across the following NVM releases: 

FW 4.33 API 1.2 NVM 04.04.02 eetrack 800018a6
FW 4.40 API 1.4 NVM 04.05.03 eetrack 80001cd8
FW 5.0 API 1.5 NVM 05.00.04 eetrack 800024da

I will try to get around the issue by falling back to PHY capabilities detection using the device ID in the case i40e_aq_get_phy_capabilities() fails, but conceptually the capabilities of the PHY should not be dependent on whether PHY detects an active link or not. 

I’d be happy to do more testing on this issue per your recommendations. 

Moreover, while trying to debug this issue, I managed to get both 3 NIC adapters on my test system on a state where the PHY has apparently died - no link indication at all on any ports. A reboot solved this, and I am now trying to replicate this behaviour under more controlled conditions. 

Best regards, 
Christos Ricudis.

Re: [dpdk-dev] i40e_aq_get_phy_capabilities() fails when using SFP+ with no link

[Olivier MATZ]

Hi Christos,

+CC i40e maintainers

On Tue, 10 Jan 2017 20:32:26 +0800, Christos Ricudis
<ricudis.christos@gmail.com> wrote:
> Hello, 
> 
> Using a X710 based 4-port 4x10Gbit NIC, I have came across the
> following issue on the i40e PMD: 
> 
> When an optical SFP+ (Finisar FTLX8571D3BCL) is used with no active
> link partner on the other end of the link (or fiber completely
> disconnected from the SFP+), i40e_aq_get_phy_capabilities() (called
> by i40e_dev_sync_phy_type() on port initialization), fails with a
> 0x05 return value (EIO) on the AQ response structure. The struct
> i40e_aq_get_phy_abilities_resp buffer passed to the Get Phy Abilities
> command is unmodified upon return. 
> 
> This prevents DPDK 16.11 from initializing the port, and ultimately
> fails with the following error: 
> 
> PMD: eth_i40e_dev_init(): Failed to sync phy type: -95
> 
> The change introducing this issue was
> http://dpdk.org/ml/archives/dev/2016-September/047663.html
> 
> Reading the X710 datasheet, I notice that no specific mention is
> given on the meaning of EIO as a response to Get PHY Abilities
> command (opcode 0x0600), whereas in most other commands, an explicit
> mention of the meaning of the possible error status responses is
> given. 
> 
> This behaviour is the same across the following NVM releases: 
> 
> FW 4.33 API 1.2 NVM 04.04.02 eetrack 800018a6
> FW 4.40 API 1.4 NVM 04.05.03 eetrack 80001cd8
> FW 5.0 API 1.5 NVM 05.00.04 eetrack 800024da
> 
> I will try to get around the issue by falling back to PHY
> capabilities detection using the device ID in the case
> i40e_aq_get_phy_capabilities() fails, but conceptually the
> capabilities of the PHY should not be dependent on whether PHY
> detects an active link or not. 
> 
> I’d be happy to do more testing on this issue per your
> recommendations. 
> 
> Moreover, while trying to debug this issue, I managed to get both 3
> NIC adapters on my test system on a state where the PHY has
> apparently died - no link indication at all on any ports. A reboot
> solved this, and I am now trying to replicate this behaviour under
> more controlled conditions. 
> 

I'm currently running into a similar issue (I think). I can reproduce
it with testpmd with the following case:

  set link_check off
  port stop 0
  # don't wait between these 2 commands
  port start 0

I added some logs that are displayed after the port start:

  PMD: i40e_set_tx_function(): Vector tx finally be used.
  PMD: i40e_set_rx_function(): Vector rx enabled, please make sure RX
    burst size no less than 4 (port=0).
  PMD: i40e_dev_rx_queue_start():  >>
  PMD: i40e_dev_tx_queue_start():  >>
  PMD: i40e_dev_start(): applying link settings...
  PMD: i40e_apply_link_speed(): abilities = 38, speed = 0
  PMD: i40e_phy_conf_link(): i40e_aq_get_phy_capabilities failed -7
  PMD: i40e_dev_start(): Fail to apply link setting
  PMD: i40e_dev_clear_queues():  >>

The -7 corresponds to I40E_ERR_UNKNOWN_PHY. This happens in
i40e_aq_get_phy_capabilities() in the following code, which makes me
think it's the same problem than yours:

	if (hw->aq.asq_last_status == I40E_AQ_RC_EIO)
		status = I40E_ERR_UNKNOWN_PHY;

A workaround in my usecase is to wait a bit between the stop and the
start.

Any help is welcome.

Regards,
Olivier

Re: [dpdk-dev] i40e_aq_get_phy_capabilities() fails when using SFP+ with no link

[Zhang, Helin]

Hi Aaron

Is the SFP+ (Finisar FTLX8571D3BCL) supported and validated by Intel?
It seems there is some PHY issue in this case.

Regards,
Helin

-----Original Message-----
From: Olivier MATZ [mailto:olivier.matz@6wind.com] 
Sent: Tuesday, January 10, 2017 11:29 PM
To: Christos Ricudis <ricudis.christos@gmail.com>
Cc: dev@dpdk.org; Zhang, Helin <helin.zhang@intel.com>; Wu, Jingjing <jingjing.wu@intel.com>
Subject: Re: [dpdk-dev] i40e_aq_get_phy_capabilities() fails when using SFP+ with no link

Hi Christos,

+CC i40e maintainers

On Tue, 10 Jan 2017 20:32:26 +0800, Christos Ricudis <ricudis.christos@gmail.com> wrote:
> Hello,
> 
> Using a X710 based 4-port 4x10Gbit NIC, I have came across the 
> following issue on the i40e PMD:
> 
> When an optical SFP+ (Finisar FTLX8571D3BCL) is used with no active 
> link partner on the other end of the link (or fiber completely 
> disconnected from the SFP+), i40e_aq_get_phy_capabilities() (called by 
> i40e_dev_sync_phy_type() on port initialization), fails with a
> 0x05 return value (EIO) on the AQ response structure. The struct 
> i40e_aq_get_phy_abilities_resp buffer passed to the Get Phy Abilities 
> command is unmodified upon return.
> 
> This prevents DPDK 16.11 from initializing the port, and ultimately 
> fails with the following error:
> 
> PMD: eth_i40e_dev_init(): Failed to sync phy type: -95
> 
> The change introducing this issue was
> http://dpdk.org/ml/archives/dev/2016-September/047663.html
> 
> Reading the X710 datasheet, I notice that no specific mention is given 
> on the meaning of EIO as a response to Get PHY Abilities command 
> (opcode 0x0600), whereas in most other commands, an explicit mention 
> of the meaning of the possible error status responses is given.
> 
> This behaviour is the same across the following NVM releases: 
> 
> FW 4.33 API 1.2 NVM 04.04.02 eetrack 800018a6 FW 4.40 API 1.4 NVM 
> 04.05.03 eetrack 80001cd8 FW 5.0 API 1.5 NVM 05.00.04 eetrack 800024da
> 
> I will try to get around the issue by falling back to PHY capabilities 
> detection using the device ID in the case
> i40e_aq_get_phy_capabilities() fails, but conceptually the 
> capabilities of the PHY should not be dependent on whether PHY detects 
> an active link or not.
> 
> I’d be happy to do more testing on this issue per your 
> recommendations.
> 
> Moreover, while trying to debug this issue, I managed to get both 3 
> NIC adapters on my test system on a state where the PHY has apparently 
> died - no link indication at all on any ports. A reboot solved this, 
> and I am now trying to replicate this behaviour under more controlled 
> conditions.
> 

I'm currently running into a similar issue (I think). I can reproduce it with testpmd with the following case:

  set link_check off
  port stop 0
  # don't wait between these 2 commands
  port start 0

I added some logs that are displayed after the port start:

  PMD: i40e_set_tx_function(): Vector tx finally be used.
  PMD: i40e_set_rx_function(): Vector rx enabled, please make sure RX
    burst size no less than 4 (port=0).
  PMD: i40e_dev_rx_queue_start():  >>
  PMD: i40e_dev_tx_queue_start():  >>
  PMD: i40e_dev_start(): applying link settings...
  PMD: i40e_apply_link_speed(): abilities = 38, speed = 0
  PMD: i40e_phy_conf_link(): i40e_aq_get_phy_capabilities failed -7
  PMD: i40e_dev_start(): Fail to apply link setting
  PMD: i40e_dev_clear_queues():  >>

The -7 corresponds to I40E_ERR_UNKNOWN_PHY. This happens in
i40e_aq_get_phy_capabilities() in the following code, which makes me think it's the same problem than yours:

	if (hw->aq.asq_last_status == I40E_AQ_RC_EIO)
		status = I40E_ERR_UNKNOWN_PHY;

A workaround in my usecase is to wait a bit between the stop and the start.

Any help is welcome.

Regards,
Olivier

Re: [dpdk-dev] i40e_aq_get_phy_capabilities() fails when using SFP+ with no link

[Christos Ricudis]

Hi Helin, 

> On 11 Jan 2017, at 09:08, Zhang, Helin <helin.zhang@intel.com> wrote:
> 
> Hi Aaron
> 
> Is the SFP+ (Finisar FTLX8571D3BCL) supported and validated by Intel?
> It seems there is some PHY issue in this case.

As the original reporter of this issue, I will test with validated SFP+s and will report on my testing. 

Shouldn’t unsupported SFP+s be blacklisted in the I40E driver? 

Best regards, 
Christos Ricudis

> 
> Regards,
> Helin
> 
> -----Original Message-----
> From: Olivier MATZ [mailto:olivier.matz@6wind.com] 
> Sent: Tuesday, January 10, 2017 11:29 PM
> To: Christos Ricudis <ricudis.christos@gmail.com>
> Cc: dev@dpdk.org; Zhang, Helin <helin.zhang@intel.com>; Wu, Jingjing <jingjing.wu@intel.com>
> Subject: Re: [dpdk-dev] i40e_aq_get_phy_capabilities() fails when using SFP+ with no link
> 
> Hi Christos,
> 
> +CC i40e maintainers
> 
> On Tue, 10 Jan 2017 20:32:26 +0800, Christos Ricudis <ricudis.christos@gmail.com> wrote:
>> Hello,
>> 
>> Using a X710 based 4-port 4x10Gbit NIC, I have came across the 
>> following issue on the i40e PMD:
>> 
>> When an optical SFP+ (Finisar FTLX8571D3BCL) is used with no active 
>> link partner on the other end of the link (or fiber completely 
>> disconnected from the SFP+), i40e_aq_get_phy_capabilities() (called by 
>> i40e_dev_sync_phy_type() on port initialization), fails with a
>> 0x05 return value (EIO) on the AQ response structure. The struct 
>> i40e_aq_get_phy_abilities_resp buffer passed to the Get Phy Abilities 
>> command is unmodified upon return.
>> 
>> This prevents DPDK 16.11 from initializing the port, and ultimately 
>> fails with the following error:
>> 
>> PMD: eth_i40e_dev_init(): Failed to sync phy type: -95
>> 
>> The change introducing this issue was
>> http://dpdk.org/ml/archives/dev/2016-September/047663.html
>> 
>> Reading the X710 datasheet, I notice that no specific mention is given 
>> on the meaning of EIO as a response to Get PHY Abilities command 
>> (opcode 0x0600), whereas in most other commands, an explicit mention 
>> of the meaning of the possible error status responses is given.
>> 
>> This behaviour is the same across the following NVM releases: 
>> 
>> FW 4.33 API 1.2 NVM 04.04.02 eetrack 800018a6 FW 4.40 API 1.4 NVM 
>> 04.05.03 eetrack 80001cd8 FW 5.0 API 1.5 NVM 05.00.04 eetrack 800024da
>> 
>> I will try to get around the issue by falling back to PHY capabilities 
>> detection using the device ID in the case
>> i40e_aq_get_phy_capabilities() fails, but conceptually the 
>> capabilities of the PHY should not be dependent on whether PHY detects 
>> an active link or not.
>> 
>> I’d be happy to do more testing on this issue per your 
>> recommendations.
>> 
>> Moreover, while trying to debug this issue, I managed to get both 3 
>> NIC adapters on my test system on a state where the PHY has apparently 
>> died - no link indication at all on any ports. A reboot solved this, 
>> and I am now trying to replicate this behaviour under more controlled 
>> conditions.
>> 
> 
> I'm currently running into a similar issue (I think). I can reproduce it with testpmd with the following case:
> 
>  set link_check off
>  port stop 0
>  # don't wait between these 2 commands
>  port start 0
> 
> I added some logs that are displayed after the port start:
> 
>  PMD: i40e_set_tx_function(): Vector tx finally be used.
>  PMD: i40e_set_rx_function(): Vector rx enabled, please make sure RX
>    burst size no less than 4 (port=0).
>  PMD: i40e_dev_rx_queue_start():  >>
>  PMD: i40e_dev_tx_queue_start():  >>
>  PMD: i40e_dev_start(): applying link settings...
>  PMD: i40e_apply_link_speed(): abilities = 38, speed = 0
>  PMD: i40e_phy_conf_link(): i40e_aq_get_phy_capabilities failed -7
>  PMD: i40e_dev_start(): Fail to apply link setting
>  PMD: i40e_dev_clear_queues():  >>
> 
> The -7 corresponds to I40E_ERR_UNKNOWN_PHY. This happens in
> i40e_aq_get_phy_capabilities() in the following code, which makes me think it's the same problem than yours:
> 
> 	if (hw->aq.asq_last_status == I40E_AQ_RC_EIO)
> 		status = I40E_ERR_UNKNOWN_PHY;
> 
> A workaround in my usecase is to wait a bit between the stop and the start.
> 
> Any help is welcome.
> 
> Regards,
> Olivier
> 
> 

Re: [dpdk-dev] i40e_aq_get_phy_capabilities() fails when using SFP+ with no link

[Olivier MATZ]

Hi,

On Wed, 11 Jan 2017 20:51:58 +0000, "Rowden, Aaron F"
<aaron.f.rowden@intel.com> wrote:
> Hi Helin,
> 
> I'm checking on this to see why it could be failing but I don’t think
> this is one part of formal validation. Intel modules are always what
> is recommended.
> 
> Aaron
> 
> > Hi Helin, 
> > 
> > > On 11 Jan 2017, at 09:08, Zhang, Helin <helin.zhang@intel.com>
> > > wrote:
> > > 
> > > Hi Aaron
> > > 
> > > Is the SFP+ (Finisar FTLX8571D3BCL) supported and validated by
> > > Intel? It seems there is some PHY issue in this case.  
> > 
> > As the original reporter of this issue, I will test with validated
> > SFP+s and will report on my testing. 
> > 
> > Shouldn’t unsupported SFP+s be blacklisted in the I40E driver? 
> > 

Just to let you know that in my case the SFP are Intel ones.
Maybe it's a different issue.

I see there are some i40e fixes in the net-next repo, I'll give a try
with this version.

Regards,
Olivier

Re: [dpdk-dev] i40e_aq_get_phy_capabilities() fails when using SFP+ with no link

[Olivier Matz]

Hi,

On Thu, 12 Jan 2017 14:55:54 +0100, Olivier MATZ
<olivier.matz@dev.6wind.com> wrote:
> Hi,
> 
> On Wed, 11 Jan 2017 20:51:58 +0000, "Rowden, Aaron F"
> <aaron.f.rowden@intel.com> wrote:
> > Hi Helin,
> > 
> > I'm checking on this to see why it could be failing but I don’t
> > think this is one part of formal validation. Intel modules are
> > always what is recommended.
> > 
> > Aaron
> >   
> > > Hi Helin, 
> > >   
> > > > On 11 Jan 2017, at 09:08, Zhang, Helin <helin.zhang@intel.com>
> > > > wrote:
> > > > 
> > > > Hi Aaron
> > > > 
> > > > Is the SFP+ (Finisar FTLX8571D3BCL) supported and validated by
> > > > Intel? It seems there is some PHY issue in this case.    
> > > 
> > > As the original reporter of this issue, I will test with validated
> > > SFP+s and will report on my testing. 
> > > 
> > > Shouldn’t unsupported SFP+s be blacklisted in the I40E driver? 
> > >   
> 
> Just to let you know that in my case the SFP are Intel ones.
> Maybe it's a different issue.
> 
> I see there are some i40e fixes in the net-next repo, I'll give a try
> with this version.

The issue still exists in net-next.

I did a git bissect, and the commit that introduces the issue is:
f4668a33efe5 ("net/i40e: fix link status change interrupt")  [1]

If I revert it (with some conflicts), the problem I described
in [2] disappear.

Helin, Jinging, do you know what would be the consequences of reverting
this patch?

Christos, I don't know if it also helps for yor issue. If no, sorry for
having squatted your topic, the symptoms looked quite similar at first
glance.

Thanks,
Olivier


[1] http://dpdk.org/browse/dpdk/commit/?id=f4668a33efe5
[2] http://dpdk.org/ml/archives/dev/2017-January/054401.html

Re: [dpdk-dev] i40e_aq_get_phy_capabilities() fails when using SFP+ with no link

[Wu, Jingjing]

Hi, Oliver

Let me 

Christos reports this issue is introduced by the  http://dpdk.org/ml/archives/dev/2016-September/047663.html
Which is the commit

commit edfb226f69bf68d514c9abae4b99b98aeb7c6a32
Author: Qi Zhang <qi.z.zhang@intel.com>
Date:   Tue Sep 27 09:37:21 2016 +0800

    net/i40e: use PHY type to check PHY capability

    Using device ID to check PHY capability is not extensible.

    Now we are using PHY type to detect PHY capability.

    All link speeds supported by the device are encoded into PHY type,
    and PHY type value can be read by admin queue "get_phy_capability"
    command at initialization stage.

While, your issue seems been introduce by
commit f4668a33efe5311301e10bc6e3709a2eccb446ad
Author: Qiming Yang <qiming.yang@intel.com>
Date:   Fri Nov 4 17:10:50 2016 +0800

    net/i40e: fix link status change interrupt

    Previously, link status interrupt in i40e is achieved by checking
    LINK_STAT_CHANGE_MASK in PFINT_ICR0 register which is provided only
    for diagnostic use. Instead, drivers need to get the link status
    change notification by using LSE (Link Status Event).

    This patch enables LSE and calls LSC callback when the event is
    received. This patch also removes the processing on
    LINK_STAT_CHANGE_MASK.


As my understanding, the first commit is try to get the phy type during initialization time,
it may cause error according to different PHYs.
But about the second one, I think the changes are only related to LSC (Link status change interrupt).
And I didn’t get the relationship between your issue and this patch.
Anyway, About the question " the consequences of reverting this patch", the answer is
It may affect the feature Link status change interrupt. Without the patch, there is an issue like:
When lsc is enabled, and link status change happens, interrupt is trigger, but the link status queried
by API rte_eth_link_get is not changed at all immediately. 


Thanks
Jingjing
> -----Original Message-----
> From: Olivier Matz [mailto:olivier.matz@6wind.com]
> Sent: Friday, January 13, 2017 9:25 PM
> To: Olivier MATZ <olivier.matz@dev.6wind.com>
> Cc: Rowden, Aaron F <aaron.f.rowden@intel.com>; Christos Ricudis
> <ricudis.christos@gmail.com>; Zhang, Helin <helin.zhang@intel.com>;
> dev@dpdk.org; Wu, Jingjing <jingjing.wu@intel.com>
> Subject: Re: [dpdk-dev] i40e_aq_get_phy_capabilities() fails when using SFP+
> with no link
> 
> Hi,
> 
> On Thu, 12 Jan 2017 14:55:54 +0100, Olivier MATZ
> <olivier.matz@dev.6wind.com> wrote:
> > Hi,
> >
> > On Wed, 11 Jan 2017 20:51:58 +0000, "Rowden, Aaron F"
> > <aaron.f.rowden@intel.com> wrote:
> > > Hi Helin,
> > >
> > > I'm checking on this to see why it could be failing but I don’t
> > > think this is one part of formal validation. Intel modules are
> > > always what is recommended.
> > >
> > > Aaron
> > >
> > > > Hi Helin,
> > > >
> > > > > On 11 Jan 2017, at 09:08, Zhang, Helin <helin.zhang@intel.com>
> > > > > wrote:
> > > > >
> > > > > Hi Aaron
> > > > >
> > > > > Is the SFP+ (Finisar FTLX8571D3BCL) supported and validated by
> > > > > Intel? It seems there is some PHY issue in this case.
> > > >
> > > > As the original reporter of this issue, I will test with validated
> > > > SFP+s and will report on my testing.
> > > >
> > > > Shouldn’t unsupported SFP+s be blacklisted in the I40E driver?
> > > >
> >
> > Just to let you know that in my case the SFP are Intel ones.
> > Maybe it's a different issue.
> >
> > I see there are some i40e fixes in the net-next repo, I'll give a try
> > with this version.
> 
> The issue still exists in net-next.
> 
> I did a git bissect, and the commit that introduces the issue is:
> f4668a33efe5 ("net/i40e: fix link status change interrupt")  [1]
> 
> If I revert it (with some conflicts), the problem I described in [2] disappear.
> 
> Helin, Jinging, do you know what would be the consequences of reverting this
> patch?
> 
> Christos, I don't know if it also helps for yor issue. If no, sorry for having
> squatted your topic, the symptoms looked quite similar at first glance.
> 
> Thanks,
> Olivier
> 
> 
> [1] http://dpdk.org/browse/dpdk/commit/?id=f4668a33efe5
> [2] http://dpdk.org/ml/archives/dev/2017-January/054401.html

Re: [dpdk-dev] i40e_aq_get_phy_capabilities() fails when using SFP+ with no link

[Christos Ricudis]

Hi, 

> On 12 Jan 2017, at 21:55, Olivier MATZ <olivier.matz@6wind.com> wrote:
> 
> Hi,
> 
> On Wed, 11 Jan 2017 20:51:58 +0000, "Rowden, Aaron F"
> <aaron.f.rowden@intel.com> wrote:
>> Hi Helin,
>> 
>> I'm checking on this to see why it could be failing but I don’t think
>> this is one part of formal validation. Intel modules are always what
>> is recommended.
>> 
>> Aaron
>> 
>>> Hi Helin, 
>>> 
>>>> On 11 Jan 2017, at 09:08, Zhang, Helin <helin.zhang@intel.com>
>>>> wrote:
>>>> 
>>>> Hi Aaron
>>>> 
>>>> Is the SFP+ (Finisar FTLX8571D3BCL) supported and validated by
>>>> Intel? It seems there is some PHY issue in this case.  
>>> 
>>> As the original reporter of this issue, I will test with validated
>>> SFP+s and will report on my testing. 
>>> 
>>> Shouldn’t unsupported SFP+s be blacklisted in the I40E driver? 
>>> 
> 
> Just to let you know that in my case the SFP are Intel ones.
> Maybe it's a different issue.
> 
> I see there are some i40e fixes in the net-next repo, I'll give a try
> with this version.
> 
> Regards,
> Olivier

After further testing, I can confirm that this issue persists with supported Intel SFPs (Intel FTLX8571D3BCV-IT). 

As for the changeset introducing this issue - we had failure reports with previous DPDK versions, probably related to LSE handling, but these weren’t properly investigated. The change in 16.11 which calls get_phy_capability too early in initialization stage might have alleviated the issue making it easier for us to detect and confirm. 

Best regards, 
Christos Ricudis. 

Re: [dpdk-dev] i40e_aq_get_phy_capabilities() fails when using SFP+ with no link

[Ivan Nardi]

Hi guys
any updates on this issue?
We are facing a very similar problem.
We have a server with 4 nics X710 4*10Gbit and the dpdk randomly failed to
start with the error:

PMD: eth_i40e_dev_init(): FW 4.40 API 1.4 NVM 04.05.03 eetrack 80001cd8
PMD: eth_i40e_dev_init(): Failed to sync phy type: -95

It happens randomly (sometimes it works properly, sometimes not), the
"failed" port index is random too and it happens whether the fibers have
been connected or not.

We are using dpdk 16.11.

Any help would be appreciated
Thanks in advance

Ivan

On 18 January 2017 at 11:15, Christos Ricudis <ricudis.christos@gmail.com>
wrote:

> Hi,
>
> > On 12 Jan 2017, at 21:55, Olivier MATZ <olivier.matz@6wind.com> wrote:
> >
> > Hi,
> >
> > On Wed, 11 Jan 2017 20:51:58 +0000, "Rowden, Aaron F"
> > <aaron.f.rowden@intel.com> wrote:
> >> Hi Helin,
> >>
> >> I'm checking on this to see why it could be failing but I don’t think
> >> this is one part of formal validation. Intel modules are always what
> >> is recommended.
> >>
> >> Aaron
> >>
> >>> Hi Helin,
> >>>
> >>>> On 11 Jan 2017, at 09:08, Zhang, Helin <helin.zhang@intel.com>
> >>>> wrote:
> >>>>
> >>>> Hi Aaron
> >>>>
> >>>> Is the SFP+ (Finisar FTLX8571D3BCL) supported and validated by
> >>>> Intel? It seems there is some PHY issue in this case.
> >>>
> >>> As the original reporter of this issue, I will test with validated
> >>> SFP+s and will report on my testing.
> >>>
> >>> Shouldn’t unsupported SFP+s be blacklisted in the I40E driver?
> >>>
> >
> > Just to let you know that in my case the SFP are Intel ones.
> > Maybe it's a different issue.
> >
> > I see there are some i40e fixes in the net-next repo, I'll give a try
> > with this version.
> >
> > Regards,
> > Olivier
>
> After further testing, I can confirm that this issue persists with
> supported Intel SFPs (Intel FTLX8571D3BCV-IT).
>
> As for the changeset introducing this issue - we had failure reports with
> previous DPDK versions, probably related to LSE handling, but these weren’t
> properly investigated. The change in 16.11 which calls get_phy_capability
> too early in initialization stage might have alleviated the issue making it
> easier for us to detect and confirm.
>
> Best regards,
> Christos Ricudis.
>
>

Re: [dpdk-dev] i40e_aq_get_phy_capabilities() fails when using SFP+ with no link

[Ivan Nardi]

HI
same issue with 17.02-rc2
It seems to me the problem I am facing is similar to the ones reported in
these mails; if not, I apologize to have used this thread

Ivan

On 5 February 2017 at 16:30, Ivan Nardi <nardi.ivan@gmail.com> wrote:

> Hi guys
> any updates on this issue?
> We are facing a very similar problem.
> We have a server with 4 nics X710 4*10Gbit and the dpdk randomly failed to
> start with the error:
>
> PMD: eth_i40e_dev_init(): FW 4.40 API 1.4 NVM 04.05.03 eetrack 80001cd8
> PMD: eth_i40e_dev_init(): Failed to sync phy type: -95
>
> It happens randomly (sometimes it works properly, sometimes not), the
> "failed" port index is random too and it happens whether the fibers have
> been connected or not.
>
> We are using dpdk 16.11.
>
> Any help would be appreciated
> Thanks in advance
>
> Ivan
>
> On 18 January 2017 at 11:15, Christos Ricudis <ricudis.christos@gmail.com>
> wrote:
>
>> Hi,
>>
>> > On 12 Jan 2017, at 21:55, Olivier MATZ <olivier.matz@6wind.com> wrote:
>> >
>> > Hi,
>> >
>> > On Wed, 11 Jan 2017 20:51:58 +0000, "Rowden, Aaron F"
>> > <aaron.f.rowden@intel.com> wrote:
>> >> Hi Helin,
>> >>
>> >> I'm checking on this to see why it could be failing but I don’t think
>> >> this is one part of formal validation. Intel modules are always what
>> >> is recommended.
>> >>
>> >> Aaron
>> >>
>> >>> Hi Helin,
>> >>>
>> >>>> On 11 Jan 2017, at 09:08, Zhang, Helin <helin.zhang@intel.com>
>> >>>> wrote:
>> >>>>
>> >>>> Hi Aaron
>> >>>>
>> >>>> Is the SFP+ (Finisar FTLX8571D3BCL) supported and validated by
>> >>>> Intel? It seems there is some PHY issue in this case.
>> >>>
>> >>> As the original reporter of this issue, I will test with validated
>> >>> SFP+s and will report on my testing.
>> >>>
>> >>> Shouldn’t unsupported SFP+s be blacklisted in the I40E driver?
>> >>>
>> >
>> > Just to let you know that in my case the SFP are Intel ones.
>> > Maybe it's a different issue.
>> >
>> > I see there are some i40e fixes in the net-next repo, I'll give a try
>> > with this version.
>> >
>> > Regards,
>> > Olivier
>>
>> After further testing, I can confirm that this issue persists with
>> supported Intel SFPs (Intel FTLX8571D3BCV-IT).
>>
>> As for the changeset introducing this issue - we had failure reports with
>> previous DPDK versions, probably related to LSE handling, but these weren’t
>> properly investigated. The change in 16.11 which calls get_phy_capability
>> too early in initialization stage might have alleviated the issue making it
>> easier for us to detect and confirm.
>>
>> Best regards,
>> Christos Ricudis.
>>
>>
>

Re: [dpdk-dev] i40e_aq_get_phy_capabilities() fails when using SFP+ with no link

[Zhang, Qi Z]

Hi Ivan:

	I'm looking at this issue, but I can't repeat it on my environment both with X710x4 and XL710x1
	Not sure if you could try below things to help narrow down this issue.
	
	1) move i40e_dev_sync_phy_type call after i40e_set_fc call, to see if the problem still exist, since without i40e_dev_sync_phy_type, i40e_set_fc is the first place i40e_aq_get_phy_capabilities get called and we didn't see this issue before 16.11.
	
	2) if above change works, at least we have a work around, if above still fail, please modify the parameter of i40e_aq_get_phy_capabilities in i40e_dev_sync_phy_type as below and check result.
	- status = i40e_aq_get_phy_capabilities(hw, false, true, &phy_ab,
	-				      NULL);
	+ status = i40e_aq_get_phy_capabilities(hw, false, false, &phy_ab,
	+					      NULL);
	
	Thank you!

Regards
Qi

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ivan Nardi
> Sent: Monday, February 6, 2017 4:19 AM
> To: dev@dpdk.org
> Cc: Olivier MATZ <olivier.matz@6wind.com>; Christos Ricudis
> <ricudis.christos@gmail.com>; Rowden, Aaron F
> <aaron.f.rowden@intel.com>; Zhang, Helin <helin.zhang@intel.com>; Wu,
> Jingjing <jingjing.wu@intel.com>
> Subject: Re: [dpdk-dev] i40e_aq_get_phy_capabilities() fails when using SFP+
> with no link
> 
> HI
> same issue with 17.02-rc2
> It seems to me the problem I am facing is similar to the ones reported in
> these mails; if not, I apologize to have used this thread
> 
> Ivan
> 
> On 5 February 2017 at 16:30, Ivan Nardi <nardi.ivan@gmail.com> wrote:
> 
> > Hi guys
> > any updates on this issue?

> > We are facing a very similar problem.
> > We have a server with 4 nics X710 4*10Gbit and the dpdk randomly
> > failed to start with the error:
> >
> > PMD: eth_i40e_dev_init(): FW 4.40 API 1.4 NVM 04.05.03 eetrack
> > 80001cd8
> > PMD: eth_i40e_dev_init(): Failed to sync phy type: -95
> >
> > It happens randomly (sometimes it works properly, sometimes not), the
> > "failed" port index is random too and it happens whether the fibers
> > have been connected or not.
> >
> > We are using dpdk 16.11.
> >
> > Any help would be appreciated
> > Thanks in advance
> >
> > Ivan
> >
> > On 18 January 2017 at 11:15, Christos Ricudis
> > <ricudis.christos@gmail.com>
> > wrote:
> >
> >> Hi,
> >>
> >> > On 12 Jan 2017, at 21:55, Olivier MATZ <olivier.matz@6wind.com>
> wrote:
> >> >
> >> > Hi,
> >> >
> >> > On Wed, 11 Jan 2017 20:51:58 +0000, "Rowden, Aaron F"
> >> > <aaron.f.rowden@intel.com> wrote:
> >> >> Hi Helin,
> >> >>
> >> >> I'm checking on this to see why it could be failing but I don’t
> >> >> think this is one part of formal validation. Intel modules are
> >> >> always what is recommended.
> >> >>
> >> >> Aaron
> >> >>
> >> >>> Hi Helin,
> >> >>>
> >> >>>> On 11 Jan 2017, at 09:08, Zhang, Helin <helin.zhang@intel.com>
> >> >>>> wrote:
> >> >>>>
> >> >>>> Hi Aaron
> >> >>>>
> >> >>>> Is the SFP+ (Finisar FTLX8571D3BCL) supported and validated by
> >> >>>> Intel? It seems there is some PHY issue in this case.
> >> >>>
> >> >>> As the original reporter of this issue, I will test with
> >> >>> validated
> >> >>> SFP+s and will report on my testing.
> >> >>>
> >> >>> Shouldn’t unsupported SFP+s be blacklisted in the I40E driver?
> >> >>>
> >> >
> >> > Just to let you know that in my case the SFP are Intel ones.
> >> > Maybe it's a different issue.
> >> >
> >> > I see there are some i40e fixes in the net-next repo, I'll give a
> >> > try with this version.
> >> >
> >> > Regards,
> >> > Olivier
> >>
> >> After further testing, I can confirm that this issue persists with
> >> supported Intel SFPs (Intel FTLX8571D3BCV-IT).
> >>
> >> As for the changeset introducing this issue - we had failure reports
> >> with previous DPDK versions, probably related to LSE handling, but
> >> these weren’t properly investigated. The change in 16.11 which calls
> >> get_phy_capability too early in initialization stage might have
> >> alleviated the issue making it easier for us to detect and confirm.
> >>
> >> Best regards,
> >> Christos Ricudis.
> >>
> >>
> >

Re: [dpdk-dev] i40e_aq_get_phy_capabilities() fails when using SFP+ with no link

[Zhang, Helin]

Hi guys

There may have firmware dependencies, new DPDK version + old firmware version may not work. Please refer to below link to see what type of firmware has been validated with 16.11 DPDK.
http://dpdk.org/doc/guides/rel_notes/release_16_11.html#tested-nics
Then first, you need to upgrade your firmware. Then let us know if the issue is still there. Thanks!

Regards,
Helin

-----Original Message-----
From: Zhang, Qi Z 
Sent: Monday, February 6, 2017 9:05 AM
To: Ivan Nardi <nardi.ivan@gmail.com>; dev@dpdk.org
Cc: Olivier MATZ <olivier.matz@6wind.com>; Christos Ricudis <ricudis.christos@gmail.com>; Rowden, Aaron F <aaron.f.rowden@intel.com>; Zhang, Helin <helin.zhang@intel.com>; Wu, Jingjing <jingjing.wu@intel.com>
Subject: RE: [dpdk-dev] i40e_aq_get_phy_capabilities() fails when using SFP+ with no link

Hi Ivan:

	I'm looking at this issue, but I can't repeat it on my environment both with X710x4 and XL710x1
	Not sure if you could try below things to help narrow down this issue.
	
	1) move i40e_dev_sync_phy_type call after i40e_set_fc call, to see if the problem still exist, since without i40e_dev_sync_phy_type, i40e_set_fc is the first place i40e_aq_get_phy_capabilities get called and we didn't see this issue before 16.11.
	
	2) if above change works, at least we have a work around, if above still fail, please modify the parameter of i40e_aq_get_phy_capabilities in i40e_dev_sync_phy_type as below and check result.
	- status = i40e_aq_get_phy_capabilities(hw, false, true, &phy_ab,
	-				      NULL);
	+ status = i40e_aq_get_phy_capabilities(hw, false, false, &phy_ab,
	+					      NULL);
	
	Thank you!

Regards
Qi

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ivan Nardi
> Sent: Monday, February 6, 2017 4:19 AM
> To: dev@dpdk.org
> Cc: Olivier MATZ <olivier.matz@6wind.com>; Christos Ricudis 
> <ricudis.christos@gmail.com>; Rowden, Aaron F 
> <aaron.f.rowden@intel.com>; Zhang, Helin <helin.zhang@intel.com>; Wu, 
> Jingjing <jingjing.wu@intel.com>
> Subject: Re: [dpdk-dev] i40e_aq_get_phy_capabilities() fails when 
> using SFP+ with no link
> 
> HI
> same issue with 17.02-rc2
> It seems to me the problem I am facing is similar to the ones reported 
> in these mails; if not, I apologize to have used this thread
> 
> Ivan
> 
> On 5 February 2017 at 16:30, Ivan Nardi <nardi.ivan@gmail.com> wrote:
> 
> > Hi guys
> > any updates on this issue?

> > We are facing a very similar problem.
> > We have a server with 4 nics X710 4*10Gbit and the dpdk randomly 
> > failed to start with the error:
> >
> > PMD: eth_i40e_dev_init(): FW 4.40 API 1.4 NVM 04.05.03 eetrack
> > 80001cd8
> > PMD: eth_i40e_dev_init(): Failed to sync phy type: -95
> >
> > It happens randomly (sometimes it works properly, sometimes not), 
> > the "failed" port index is random too and it happens whether the 
> > fibers have been connected or not.
> >
> > We are using dpdk 16.11.
> >
> > Any help would be appreciated
> > Thanks in advance
> >
> > Ivan
> >
> > On 18 January 2017 at 11:15, Christos Ricudis 
> > <ricudis.christos@gmail.com>
> > wrote:
> >
> >> Hi,
> >>
> >> > On 12 Jan 2017, at 21:55, Olivier MATZ <olivier.matz@6wind.com>
> wrote:
> >> >
> >> > Hi,
> >> >
> >> > On Wed, 11 Jan 2017 20:51:58 +0000, "Rowden, Aaron F"
> >> > <aaron.f.rowden@intel.com> wrote:
> >> >> Hi Helin,
> >> >>
> >> >> I'm checking on this to see why it could be failing but I don’t 
> >> >> think this is one part of formal validation. Intel modules are 
> >> >> always what is recommended.
> >> >>
> >> >> Aaron
> >> >>
> >> >>> Hi Helin,
> >> >>>
> >> >>>> On 11 Jan 2017, at 09:08, Zhang, Helin <helin.zhang@intel.com>
> >> >>>> wrote:
> >> >>>>
> >> >>>> Hi Aaron
> >> >>>>
> >> >>>> Is the SFP+ (Finisar FTLX8571D3BCL) supported and validated by 
> >> >>>> Intel? It seems there is some PHY issue in this case.
> >> >>>
> >> >>> As the original reporter of this issue, I will test with 
> >> >>> validated
> >> >>> SFP+s and will report on my testing.
> >> >>>
> >> >>> Shouldn’t unsupported SFP+s be blacklisted in the I40E driver?
> >> >>>
> >> >
> >> > Just to let you know that in my case the SFP are Intel ones.
> >> > Maybe it's a different issue.
> >> >
> >> > I see there are some i40e fixes in the net-next repo, I'll give a 
> >> > try with this version.
> >> >
> >> > Regards,
> >> > Olivier
> >>
> >> After further testing, I can confirm that this issue persists with 
> >> supported Intel SFPs (Intel FTLX8571D3BCV-IT).
> >>
> >> As for the changeset introducing this issue - we had failure 
> >> reports with previous DPDK versions, probably related to LSE 
> >> handling, but these weren’t properly investigated. The change in 
> >> 16.11 which calls get_phy_capability too early in initialization 
> >> stage might have alleviated the issue making it easier for us to detect and confirm.
> >>
> >> Best regards,
> >> Christos Ricudis.
> >>
> >>
> >

Re: [dpdk-dev] i40e_aq_get_phy_capabilities() fails when using SFP+ with no link

[Olivier MATZ]

Hi Helin,

On Mon, 6 Feb 2017 01:36:45 +0000, "Zhang, Helin"
<helin.zhang@intel.com> wrote:
> Hi guys
> 
> There may have firmware dependencies, new DPDK version + old firmware
> version may not work. Please refer to below link to see what type of
> firmware has been validated with 16.11 DPDK.
> http://dpdk.org/doc/guides/rel_notes/release_16_11.html#tested-nics
> Then first, you need to upgrade your firmware. Then let us know if
> the issue is still there. Thanks!
> 

I updated the firmware, and I confirm it solves my issue described
at: http://dpdk.org/ml/archives/dev/2017-January/054401.html

Thanks,
Olivier

Re: [dpdk-dev] i40e_aq_get_phy_capabilities() fails when using SFP+ with no link

[Zhang, Helin]

Hi Oliver

Very good to know that good news!
Thank you!

Regards,
Helin

-----Original Message-----
From: Olivier MATZ [mailto:olivier.matz@6wind.com] 
Sent: Tuesday, February 7, 2017 10:56 PM
To: Zhang, Helin <helin.zhang@intel.com>
Cc: Zhang, Qi Z <qi.z.zhang@intel.com>; Ivan Nardi <nardi.ivan@gmail.com>; dev@dpdk.org; Olivier MATZ <olivier.matz@6wind.com>; Christos Ricudis <ricudis.christos@gmail.com>; Rowden, Aaron F <aaron.f.rowden@intel.com>; Wu, Jingjing <jingjing.wu@intel.com>
Subject: Re: [dpdk-dev] i40e_aq_get_phy_capabilities() fails when using SFP+ with no link

Hi Helin,

On Mon, 6 Feb 2017 01:36:45 +0000, "Zhang, Helin"
<helin.zhang@intel.com> wrote:
> Hi guys
> 
> There may have firmware dependencies, new DPDK version + old firmware 
> version may not work. Please refer to below link to see what type of 
> firmware has been validated with 16.11 DPDK.
> http://dpdk.org/doc/guides/rel_notes/release_16_11.html#tested-nics
> Then first, you need to upgrade your firmware. Then let us know if the 
> issue is still there. Thanks!
> 

I updated the firmware, and I confirm it solves my issue described
at: http://dpdk.org/ml/archives/dev/2017-January/054401.html

Thanks,
Olivier

Re: [dpdk-dev] i40e_aq_get_phy_capabilities() fails when using SFP+ with no link

[Ivan Nardi]

Hi
I am not able to reproduce the issue anymore with nvm 5.05 + 17.02-rc2
(with LSC disabled)
Lesson learned: always upgrade to latest versions first.
Thanks to everyone for the support
Regards
Ivan

On 7 February 2017 at 15:56, Olivier MATZ <olivier.matz@6wind.com> wrote:
> Hi Helin,
>
> On Mon, 6 Feb 2017 01:36:45 +0000, "Zhang, Helin"
> <helin.zhang@intel.com> wrote:
>> Hi guys
>>
>> There may have firmware dependencies, new DPDK version + old firmware
>> version may not work. Please refer to below link to see what type of
>> firmware has been validated with 16.11 DPDK.
>> http://dpdk.org/doc/guides/rel_notes/release_16_11.html#tested-nics
>> Then first, you need to upgrade your firmware. Then let us know if
>> the issue is still there. Thanks!
>>
>
> I updated the firmware, and I confirm it solves my issue described
> at: http://dpdk.org/ml/archives/dev/2017-January/054401.html
>
> Thanks,
> Olivier

Re: [dpdk-dev] i40e_aq_get_phy_capabilities() fails when using SFP+ with no link

[Ivan Nardi]

Thanks for the suggestions!
I'll try them and I will report back the results in the next days.
Regards
Ivan

On 6 February 2017 at 02:04, Zhang, Qi Z <qi.z.zhang@intel.com> wrote:
> Hi Ivan:
>
>         I'm looking at this issue, but I can't repeat it on my environment both with X710x4 and XL710x1
>         Not sure if you could try below things to help narrow down this issue.
>
>         1) move i40e_dev_sync_phy_type call after i40e_set_fc call, to see if the problem still exist, since without i40e_dev_sync_phy_type, i40e_set_fc is the first place i40e_aq_get_phy_capabilities get called and we didn't see this issue before 16.11.
>
>         2) if above change works, at least we have a work around, if above still fail, please modify the parameter of i40e_aq_get_phy_capabilities in i40e_dev_sync_phy_type as below and check result.
>         - status = i40e_aq_get_phy_capabilities(hw, false, true, &phy_ab,
>         -                                     NULL);
>         + status = i40e_aq_get_phy_capabilities(hw, false, false, &phy_ab,
>         +                                             NULL);
>
>         Thank you!
>
> Regards
> Qi
>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ivan Nardi
>> Sent: Monday, February 6, 2017 4:19 AM
>> To: dev@dpdk.org
>> Cc: Olivier MATZ <olivier.matz@6wind.com>; Christos Ricudis
>> <ricudis.christos@gmail.com>; Rowden, Aaron F
>> <aaron.f.rowden@intel.com>; Zhang, Helin <helin.zhang@intel.com>; Wu,
>> Jingjing <jingjing.wu@intel.com>
>> Subject: Re: [dpdk-dev] i40e_aq_get_phy_capabilities() fails when using SFP+
>> with no link
>>
>> HI
>> same issue with 17.02-rc2
>> It seems to me the problem I am facing is similar to the ones reported in
>> these mails; if not, I apologize to have used this thread
>>
>> Ivan
>>
>> On 5 February 2017 at 16:30, Ivan Nardi <nardi.ivan@gmail.com> wrote:
>>
>> > Hi guys
>> > any updates on this issue?
>
>> > We are facing a very similar problem.
>> > We have a server with 4 nics X710 4*10Gbit and the dpdk randomly
>> > failed to start with the error:
>> >
>> > PMD: eth_i40e_dev_init(): FW 4.40 API 1.4 NVM 04.05.03 eetrack
>> > 80001cd8
>> > PMD: eth_i40e_dev_init(): Failed to sync phy type: -95
>> >
>> > It happens randomly (sometimes it works properly, sometimes not), the
>> > "failed" port index is random too and it happens whether the fibers
>> > have been connected or not.
>> >
>> > We are using dpdk 16.11.
>> >
>> > Any help would be appreciated
>> > Thanks in advance
>> >
>> > Ivan
>> >
>> > On 18 January 2017 at 11:15, Christos Ricudis
>> > <ricudis.christos@gmail.com>
>> > wrote:
>> >
>> >> Hi,
>> >>
>> >> > On 12 Jan 2017, at 21:55, Olivier MATZ <olivier.matz@6wind.com>
>> wrote:
>> >> >
>> >> > Hi,
>> >> >
>> >> > On Wed, 11 Jan 2017 20:51:58 +0000, "Rowden, Aaron F"
>> >> > <aaron.f.rowden@intel.com> wrote:
>> >> >> Hi Helin,
>> >> >>
>> >> >> I'm checking on this to see why it could be failing but I don’t
>> >> >> think this is one part of formal validation. Intel modules are
>> >> >> always what is recommended.
>> >> >>
>> >> >> Aaron
>> >> >>
>> >> >>> Hi Helin,
>> >> >>>
>> >> >>>> On 11 Jan 2017, at 09:08, Zhang, Helin <helin.zhang@intel.com>
>> >> >>>> wrote:
>> >> >>>>
>> >> >>>> Hi Aaron
>> >> >>>>
>> >> >>>> Is the SFP+ (Finisar FTLX8571D3BCL) supported and validated by
>> >> >>>> Intel? It seems there is some PHY issue in this case.
>> >> >>>
>> >> >>> As the original reporter of this issue, I will test with
>> >> >>> validated
>> >> >>> SFP+s and will report on my testing.
>> >> >>>
>> >> >>> Shouldn’t unsupported SFP+s be blacklisted in the I40E driver?
>> >> >>>
>> >> >
>> >> > Just to let you know that in my case the SFP are Intel ones.
>> >> > Maybe it's a different issue.
>> >> >
>> >> > I see there are some i40e fixes in the net-next repo, I'll give a
>> >> > try with this version.
>> >> >
>> >> > Regards,
>> >> > Olivier
>> >>
>> >> After further testing, I can confirm that this issue persists with
>> >> supported Intel SFPs (Intel FTLX8571D3BCV-IT).
>> >>
>> >> As for the changeset introducing this issue - we had failure reports
>> >> with previous DPDK versions, probably related to LSE handling, but
>> >> these weren’t properly investigated. The change in 16.11 which calls
>> >> get_phy_capability too early in initialization stage might have
>> >> alleviated the issue making it easier for us to detect and confirm.
>> >>
>> >> Best regards,
>> >> Christos Ricudis.
>> >>
>> >>
>> >