[PATCH v2 1/7] net/ixgbe: Fix ixgbe_is_sfp() to return valid result for X550EM_a devs

[PATCH v2 1/7] net/ixgbe: Fix ixgbe_is_sfp() to return valid result for X550EM_a devs

[Stephen Douthit]

Currently all X500EM* MAC types fallthrough to the default case and get
reported as non-SFP regardless of media type, which isn't correct.

Fixes: 0790adeb567 ("ixgbe/base: support X550em_a device")
Cc: stable@dpdk.org

Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index fe61dba81d..66f7af95de 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -781,6 +781,20 @@ ixgbe_is_sfp(struct ixgbe_hw *hw)
 	case ixgbe_phy_sfp_passive_unknown:
 		return 1;
 	default:
+		/* x550em devices may be SFP, check media type */
+		switch (hw->mac.type) {
+		case ixgbe_mac_X550EM_x:
+		case ixgbe_mac_X550EM_a:
+			switch (hw->mac.ops.get_media_type(hw)) {
+			case ixgbe_media_type_fiber:
+			case ixgbe_media_type_fiber_qsfp:
+				return 1;
+			default:
+				return 0;
+			}
+		default:
+			return 0;
+		}
 		return 0;
 	}
 }
-- 
2.31.1

RE: [PATCH v2 1/7] net/ixgbe: Fix ixgbe_is_sfp() to return valid result for X550EM_a devs

[Wang, Haiyue]

> -----Original Message-----
> From: Stephen Douthit <stephend@silicom-usa.com>
> Sent: Tuesday, December 7, 2021 06:19
> To: Wang, Haiyue <haiyue.wang@intel.com>; Lu, Wenzhuo <wenzhuo.lu@intel.com>
> Cc: dev@dpdk.org; Wen Wang <wenw@silicom-usa.com>; Stephen Douthit <stephend@silicom-usa.com>;
> stable@dpdk.org
> Subject: [PATCH v2 1/7] net/ixgbe: Fix ixgbe_is_sfp() to return valid result for X550EM_a devs
> 
> Currently all X500EM* MAC types fallthrough to the default case and get
> reported as non-SFP regardless of media type, which isn't correct.
> 
> Fixes: 0790adeb567 ("ixgbe/base: support X550em_a device")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
> ---
>  drivers/net/ixgbe/ixgbe_ethdev.c | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
> 
> diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
> index fe61dba81d..66f7af95de 100644
> --- a/drivers/net/ixgbe/ixgbe_ethdev.c
> +++ b/drivers/net/ixgbe/ixgbe_ethdev.c
> @@ -781,6 +781,20 @@ ixgbe_is_sfp(struct ixgbe_hw *hw)
>  	case ixgbe_phy_sfp_passive_unknown:
>  		return 1;
>  	default:
> +		/* x550em devices may be SFP, check media type */
> +		switch (hw->mac.type) {
> +		case ixgbe_mac_X550EM_x:
> +		case ixgbe_mac_X550EM_a:
> +			switch (hw->mac.ops.get_media_type(hw)) {

Use the API 'ixgbe_get_media_type' to avoid ops null ?

> +			case ixgbe_media_type_fiber:
> +			case ixgbe_media_type_fiber_qsfp:
> +				return 1;
> +			default:
> +				return 0;

Since we care 'return 1' only, then the two defaults just "break;" ?

> +			}
> +		default:
> +			return 0; 

Just 'break;'

> +		}
>  		return 0;

Then this default '0' will be used.

>  	}
>  }
> --
> 2.31.1

Re: [PATCH v2 1/7] net/ixgbe: Fix ixgbe_is_sfp() to return valid result for X550EM_a devs

[Stephen Douthit]

On 12/20/21 02:45, Wang, Haiyue wrote:
>> -----Original Message-----
>> From: Stephen Douthit <stephend@silicom-usa.com>
>> Sent: Tuesday, December 7, 2021 06:19
>> To: Wang, Haiyue <haiyue.wang@intel.com>; Lu, Wenzhuo <wenzhuo.lu@intel.com>
>> Cc: dev@dpdk.org; Wen Wang <wenw@silicom-usa.com>; Stephen Douthit <stephend@silicom-usa.com>;
>> stable@dpdk.org
>> Subject: [PATCH v2 1/7] net/ixgbe: Fix ixgbe_is_sfp() to return valid result for X550EM_a devs
>>
>> Currently all X500EM* MAC types fallthrough to the default case and get
>> reported as non-SFP regardless of media type, which isn't correct.
>>
>> Fixes: 0790adeb567 ("ixgbe/base: support X550em_a device")
>> Cc: stable@dpdk.org
>>
>> Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
>> ---
>>   drivers/net/ixgbe/ixgbe_ethdev.c | 14 ++++++++++++++
>>   1 file changed, 14 insertions(+)
>>
>> diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
>> index fe61dba81d..66f7af95de 100644
>> --- a/drivers/net/ixgbe/ixgbe_ethdev.c
>> +++ b/drivers/net/ixgbe/ixgbe_ethdev.c
>> @@ -781,6 +781,20 @@ ixgbe_is_sfp(struct ixgbe_hw *hw)
>>        case ixgbe_phy_sfp_passive_unknown:
>>                return 1;
>>        default:
>> +             /* x550em devices may be SFP, check media type */
>> +             switch (hw->mac.type) {
>> +             case ixgbe_mac_X550EM_x:
>> +             case ixgbe_mac_X550EM_a:
>> +                     switch (hw->mac.ops.get_media_type(hw)) {
> 
> Use the API 'ixgbe_get_media_type' to avoid ops null ?

Ok, I can change it.

Note that there's already a mix of calls with and without the null check
wrapper in the file already though, so maybe worth future cleanup.

$ grep get_media_type drivers/net/ixgbe/ixgbe_ethdev.c
         if (diag && (hw->mac.ops.get_media_type(hw) == ixgbe_media_type_copper)) {
         if (hw->mac.ops.get_media_type(hw) == ixgbe_media_type_copper) {
         if (hw->mac.ops.get_media_type(hw) == ixgbe_media_type_copper) {
         if (hw->mac.ops.get_media_type(hw) == ixgbe_media_type_copper) {
         if (hw->mac.ops.get_media_type(hw) == ixgbe_media_type_copper) {
         if (ixgbe_get_media_type(hw) == ixgbe_media_type_fiber) {
                 if (ixgbe_get_media_type(hw) == ixgbe_media_type_fiber) {

>> +                     case ixgbe_media_type_fiber:
>> +                     case ixgbe_media_type_fiber_qsfp:
>> +                             return 1;
>> +                     default:
>> +                             return 0;
> 
> Since we care 'return 1' only, then the two defaults just "break;" ?
> 
>> +                     }
>> +             default:
>> +                     return 0;
> 
> Just 'break;'
> 
>> +             }
>>                return 0;
> 
> Then this default '0' will be used.

Ok, will change.

Thanks,
Steve

>>        }
>>   }
>> --
>> 2.31.1
> 

[PATCH v2 2/7] net/ixgbe: Add ixgbe_check_sfp_cage() for testing state of PRSNT# signal

[Stephen Douthit]

Refactor the SFP check code from ixgbe_check_mac_link_generic into its own
function.

Note that the SFP present status was inverted for the X550EM family of
devices, where SDP0 represents the active low PRSNT# signal from the cage.

Call the new function in ixgbe_identify_module_generic() to short circuit
the I2C polling and greatly speed things up for devices we know are
absent.

Fixes: dd3a93cf5a2 ("net/ixgbe/base: bypass checking link for crosstalk")
Cc: stable@dpdk.org

Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
---
 drivers/net/ixgbe/base/ixgbe_common.c | 60 +++++++++++++++++++--------
 drivers/net/ixgbe/base/ixgbe_common.h |  8 ++++
 drivers/net/ixgbe/base/ixgbe_phy.c    |  8 ++++
 3 files changed, 58 insertions(+), 18 deletions(-)

diff --git a/drivers/net/ixgbe/base/ixgbe_common.c b/drivers/net/ixgbe/base/ixgbe_common.c
index aa843bd5c4..2764cf7cf1 100644
--- a/drivers/net/ixgbe/base/ixgbe_common.c
+++ b/drivers/net/ixgbe/base/ixgbe_common.c
@@ -4124,6 +4124,45 @@ static bool ixgbe_need_crosstalk_fix(struct ixgbe_hw *hw)
 	return true;
 }
 
+/**
+ * ixgbe_check_sfp_cage - Find present status of SFP module
+ * @hw: pointer to hardware structure
+ *
+ * Find if a SFP module is present and if this device supports SFPs
+ **/
+enum ixgbe_sfp_cage_status ixgbe_check_sfp_cage(struct ixgbe_hw *hw)
+{
+	enum ixgbe_sfp_cage_status status;
+
+	/* If we're not a fiber/fiber_qsfp, no cage to check */
+	switch (hw->mac.ops.get_media_type(hw)) {
+	case ixgbe_media_type_fiber:
+	case ixgbe_media_type_fiber_qsfp:
+		break;
+	default:
+		return IXGBE_SFP_CAGE_NOCAGE;
+	}
+
+	switch (hw->mac.type) {
+	case ixgbe_mac_82599EB:
+		status = !!(IXGBE_READ_REG(hw, IXGBE_ESDP) &
+			     IXGBE_ESDP_SDP2);
+		break;
+	case ixgbe_mac_X550EM_x:
+	case ixgbe_mac_X550EM_a:
+		/* SDP0 is the active low signal PRSNT#, so invert this */
+		status = !(IXGBE_READ_REG(hw, IXGBE_ESDP) &
+				  IXGBE_ESDP_SDP0);
+		break;
+	default:
+		/* Don't know how to check this device type yet */
+		status = IXGBE_SFP_CAGE_UNKNOWN;
+		break;
+	}
+
+	return status;
+}
+
 /**
  * ixgbe_check_mac_link_generic - Determine link and speed status
  * @hw: pointer to hardware structure
@@ -4145,25 +4184,10 @@ s32 ixgbe_check_mac_link_generic(struct ixgbe_hw *hw, ixgbe_link_speed *speed,
 	 * the SFP+ cage is full.
 	 */
 	if (ixgbe_need_crosstalk_fix(hw)) {
-		u32 sfp_cage_full;
-
-		switch (hw->mac.type) {
-		case ixgbe_mac_82599EB:
-			sfp_cage_full = IXGBE_READ_REG(hw, IXGBE_ESDP) &
-					IXGBE_ESDP_SDP2;
-			break;
-		case ixgbe_mac_X550EM_x:
-		case ixgbe_mac_X550EM_a:
-			sfp_cage_full = IXGBE_READ_REG(hw, IXGBE_ESDP) &
-					IXGBE_ESDP_SDP0;
-			break;
-		default:
-			/* sanity check - No SFP+ devices here */
-			sfp_cage_full = false;
-			break;
-		}
+		enum ixgbe_sfp_cage_status sfp_cage_status;
 
-		if (!sfp_cage_full) {
+		sfp_cage_status = ixgbe_check_sfp_cage(hw);
+		if (sfp_cage_status != IXGBE_SFP_CAGE_FULL) {
 			*link_up = false;
 			*speed = IXGBE_LINK_SPEED_UNKNOWN;
 			return IXGBE_SUCCESS;
diff --git a/drivers/net/ixgbe/base/ixgbe_common.h b/drivers/net/ixgbe/base/ixgbe_common.h
index 5bdb484407..30db9a08c4 100644
--- a/drivers/net/ixgbe/base/ixgbe_common.h
+++ b/drivers/net/ixgbe/base/ixgbe_common.h
@@ -112,6 +112,14 @@ s32 ixgbe_set_vlvf_generic(struct ixgbe_hw *hw, u32 vlan, u32 vind,
 s32 ixgbe_clear_vfta_generic(struct ixgbe_hw *hw);
 s32 ixgbe_find_vlvf_slot(struct ixgbe_hw *hw, u32 vlan, bool vlvf_bypass);
 
+enum ixgbe_sfp_cage_status {
+	IXGBE_SFP_CAGE_EMPTY = 0,
+	IXGBE_SFP_CAGE_FULL,
+	IXGBE_SFP_CAGE_UNKNOWN = -1,
+	IXGBE_SFP_CAGE_NOCAGE = -2,
+};
+enum ixgbe_sfp_cage_status ixgbe_check_sfp_cage(struct ixgbe_hw *hw);
+
 s32 ixgbe_check_mac_link_generic(struct ixgbe_hw *hw,
 			       ixgbe_link_speed *speed,
 			       bool *link_up, bool link_up_wait_to_complete);
diff --git a/drivers/net/ixgbe/base/ixgbe_phy.c b/drivers/net/ixgbe/base/ixgbe_phy.c
index 8d4d9bbfef..d8d51d2c3f 100644
--- a/drivers/net/ixgbe/base/ixgbe_phy.c
+++ b/drivers/net/ixgbe/base/ixgbe_phy.c
@@ -1228,9 +1228,17 @@ s32 ixgbe_reset_phy_nl(struct ixgbe_hw *hw)
 s32 ixgbe_identify_module_generic(struct ixgbe_hw *hw)
 {
 	s32 status = IXGBE_ERR_SFP_NOT_PRESENT;
+	enum ixgbe_sfp_cage_status sfp_cage_status;
 
 	DEBUGFUNC("ixgbe_identify_module_generic");
 
+	sfp_cage_status = ixgbe_check_sfp_cage(hw);
+	if (sfp_cage_status == IXGBE_SFP_CAGE_EMPTY ||
+	    sfp_cage_status == IXGBE_SFP_CAGE_NOCAGE) {
+		hw->phy.sfp_type = ixgbe_sfp_type_not_present;
+		return status;
+	}
+
 	switch (hw->mac.ops.get_media_type(hw)) {
 	case ixgbe_media_type_fiber:
 		status = ixgbe_identify_sfp_module_generic(hw);
-- 
2.31.1

[PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write

[Stephen Douthit]

Make sure an SFP is really a SFF-8472 device that supports the optional
soft rate select feature before just blindly poking those I2C registers.

Skip all I2C traffic if we know there's no SFP.

Fixes: f3430431aba ("ixgbe/base: add SFP+ dual-speed support")
Cc: stable@dpdk.org

Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
---
 drivers/net/ixgbe/base/ixgbe_common.c | 46 +++++++++++++++++++++++++++
 drivers/net/ixgbe/base/ixgbe_phy.h    |  3 ++
 2 files changed, 49 insertions(+)

diff --git a/drivers/net/ixgbe/base/ixgbe_common.c b/drivers/net/ixgbe/base/ixgbe_common.c
index 2764cf7cf1..3be1cc7fa2 100644
--- a/drivers/net/ixgbe/base/ixgbe_common.c
+++ b/drivers/net/ixgbe/base/ixgbe_common.c
@@ -5371,6 +5371,7 @@ s32 ixgbe_setup_mac_link_multispeed_fiber(struct ixgbe_hw *hw,
 void ixgbe_set_soft_rate_select_speed(struct ixgbe_hw *hw,
 					ixgbe_link_speed speed)
 {
+	enum ixgbe_sfp_cage_status sfp_cage_status;
 	s32 status;
 	u8 rs, eeprom_data;
 
@@ -5387,6 +5388,51 @@ void ixgbe_set_soft_rate_select_speed(struct ixgbe_hw *hw,
 		return;
 	}
 
+	/* Can't set rate on missing devices, skip all I2C access */
+	sfp_cage_status = ixgbe_check_sfp_cage(hw);
+	if (sfp_cage_status == IXGBE_SFP_CAGE_EMPTY ||
+	    sfp_cage_status == IXGBE_SFP_CAGE_NOCAGE) {
+		DEBUGOUT("No SFP\n");
+		return;
+	}
+
+	/* This only applies to SFF-8472 devices, so check that this device has
+	 * a non-zero SFF8472 compliance code @ device 0xA0 byte 94
+	 */
+	status = hw->phy.ops.read_i2c_eeprom(hw,
+					     IXGBE_SFF_SFF_8472_COMP,
+					     &eeprom_data);
+	if (status || !eeprom_data) {
+		DEBUGOUT("Not a SFF-8472 device\n");
+		goto out;
+	}
+
+	/* (read|write)_i2c_byte() don't support the address change mechanism
+	 * outlined in section 8.9 "Addressing Modes" of SFF_8472, so if that
+	 * is a requirement give up
+	 */
+	status = hw->phy.ops.read_i2c_eeprom(hw,
+					     IXGBE_SFF_SFF_8472_SWAP,
+					     &eeprom_data);
+	if (status || (eeprom_data & IXGBE_SFF_ADDRESSING_MODE)) {
+		DEBUGOUT("Address change not supported\n");
+		goto out;
+	}
+	/* Digital diagnostic monitoring must be supported for rate select */
+	if (!(eeprom_data & IXGBE_SFF_DDM_IMPLEMENTED)) {
+		DEBUGOUT("DDM not implemented\n");
+		goto out;
+	}
+
+	/* Finally check if the optional rate select feature is implemented */
+	status = hw->phy.ops.read_i2c_eeprom(hw,
+					     IXGBE_SFF_SFF_8472_EOPT,
+					     &eeprom_data);
+	if (status || !(eeprom_data & IXGBE_SFF_HAVE_RS)) {
+		DEBUGOUT("Rate select not supported");
+		goto out;
+	}
+
 	/* Set RS0 */
 	status = hw->phy.ops.read_i2c_byte(hw, IXGBE_SFF_SFF_8472_OSCB,
 					   IXGBE_I2C_EEPROM_DEV_ADDR2,
diff --git a/drivers/net/ixgbe/base/ixgbe_phy.h b/drivers/net/ixgbe/base/ixgbe_phy.h
index ceefbb3e68..cd57ce040f 100644
--- a/drivers/net/ixgbe/base/ixgbe_phy.h
+++ b/drivers/net/ixgbe/base/ixgbe_phy.h
@@ -21,6 +21,7 @@
 #define IXGBE_SFF_CABLE_TECHNOLOGY	0x8
 #define IXGBE_SFF_CABLE_SPEC_COMP	0x3C
 #define IXGBE_SFF_SFF_8472_SWAP		0x5C
+#define IXGBE_SFF_SFF_8472_EOPT		0x5D
 #define IXGBE_SFF_SFF_8472_COMP		0x5E
 #define IXGBE_SFF_SFF_8472_OSCB		0x6E
 #define IXGBE_SFF_SFF_8472_ESCB		0x76
@@ -48,6 +49,8 @@
 #define IXGBE_SFF_SOFT_RS_SELECT_10G	0x8
 #define IXGBE_SFF_SOFT_RS_SELECT_1G	0x0
 #define IXGBE_SFF_ADDRESSING_MODE	0x4
+#define IXGBE_SFF_DDM_IMPLEMENTED	0x40
+#define IXGBE_SFF_HAVE_RS		0x2
 #define IXGBE_SFF_QSFP_DA_ACTIVE_CABLE	0x1
 #define IXGBE_SFF_QSFP_DA_PASSIVE_CABLE	0x8
 #define IXGBE_SFF_QSFP_CONNECTOR_NOT_SEPARABLE	0x23
-- 
2.31.1

RE: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write

[Wang, Haiyue]

> -----Original Message-----
> From: Stephen Douthit <stephend@silicom-usa.com>
> Sent: Tuesday, December 7, 2021 06:19
> To: Wang, Haiyue <haiyue.wang@intel.com>; Lu, Wenzhuo <wenzhuo.lu@intel.com>; Changchun Ouyang
> <changchun.ouyang@intel.com>; Zhang, Helin <helin.zhang@intel.com>
> Cc: dev@dpdk.org; Wen Wang <wenw@silicom-usa.com>; Stephen Douthit <stephend@silicom-usa.com>;
> stable@dpdk.org
> Subject: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write
> 
> Make sure an SFP is really a SFF-8472 device that supports the optional
> soft rate select feature before just blindly poking those I2C registers.
> 
> Skip all I2C traffic if we know there's no SFP.
> 
> Fixes: f3430431aba ("ixgbe/base: add SFP+ dual-speed support")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
> ---


>  	/* Set RS0 */
>  	status = hw->phy.ops.read_i2c_byte(hw, IXGBE_SFF_SFF_8472_OSCB,
>  					   IXGBE_I2C_EEPROM_DEV_ADDR2,
> diff --git a/drivers/net/ixgbe/base/ixgbe_phy.h b/drivers/net/ixgbe/base/ixgbe_phy.h
> index ceefbb3e68..cd57ce040f 100644
> --- a/drivers/net/ixgbe/base/ixgbe_phy.h
> +++ b/drivers/net/ixgbe/base/ixgbe_phy.h
> @@ -21,6 +21,7 @@
>  #define IXGBE_SFF_CABLE_TECHNOLOGY	0x8
>  #define IXGBE_SFF_CABLE_SPEC_COMP	0x3C
>  #define IXGBE_SFF_SFF_8472_SWAP		0x5C
> +#define IXGBE_SFF_SFF_8472_EOPT		0x5D

Looks like this is YOUR platform specific, then this patchset can't be
merged. : - (

>  #define IXGBE_SFF_SFF_8472_COMP		0x5E
>  #define IXGBE_SFF_SFF_8472_OSCB		0x6E
>  #define IXGBE_SFF_SFF_8472_ESCB		0x76
> @@ -48,6 +49,8 @@
>  #define IXGBE_SFF_SOFT_RS_SELECT_10G	0x8
> --
> 2.31.1

Re: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write

[Stephen Douthit]

On 12/20/21 02:53, Wang, Haiyue wrote:
>> -----Original Message-----
>> From: Stephen Douthit <stephend@silicom-usa.com>
>> Sent: Tuesday, December 7, 2021 06:19
>> To: Wang, Haiyue <haiyue.wang@intel.com>; Lu, Wenzhuo <wenzhuo.lu@intel.com>; Changchun Ouyang
>> <changchun.ouyang@intel.com>; Zhang, Helin <helin.zhang@intel.com>
>> Cc: dev@dpdk.org; Wen Wang <wenw@silicom-usa.com>; Stephen Douthit <stephend@silicom-usa.com>;
>> stable@dpdk.org
>> Subject: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write
>>
>> Make sure an SFP is really a SFF-8472 device that supports the optional
>> soft rate select feature before just blindly poking those I2C registers.
>>
>> Skip all I2C traffic if we know there's no SFP.
>>
>> Fixes: f3430431aba ("ixgbe/base: add SFP+ dual-speed support")
>> Cc: stable@dpdk.org
>>
>> Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
>> ---
> 
> 
>>        /* Set RS0 */
>>        status = hw->phy.ops.read_i2c_byte(hw, IXGBE_SFF_SFF_8472_OSCB,
>>                                           IXGBE_I2C_EEPROM_DEV_ADDR2,
>> diff --git a/drivers/net/ixgbe/base/ixgbe_phy.h b/drivers/net/ixgbe/base/ixgbe_phy.h
>> index ceefbb3e68..cd57ce040f 100644
>> --- a/drivers/net/ixgbe/base/ixgbe_phy.h
>> +++ b/drivers/net/ixgbe/base/ixgbe_phy.h
>> @@ -21,6 +21,7 @@
>>   #define IXGBE_SFF_CABLE_TECHNOLOGY   0x8
>>   #define IXGBE_SFF_CABLE_SPEC_COMP    0x3C
>>   #define IXGBE_SFF_SFF_8472_SWAP              0x5C
>> +#define IXGBE_SFF_SFF_8472_EOPT              0x5D
> 
> Looks like this is YOUR platform specific, then this patchset can't be
> merged. : - (

This isn't anything unique to our hardware, these values are coming from
the SFF-8472 SFP+ I2C specification.

The ability to do a soft rate select via I2C is an optional feature, and
modules that support it are supposed to set bit 3 in byte 93 (0x5d), the
"Enhanced Options" register, to advertise the functionality.

Please see section 8.10 and Table 8-6 in the SFF-8472 spec.

Checking the RATE_SELECT bit flag may be overkill since the transceiver
is supposed to ignore writes to rate select control bits if the feature
isn't implemented.  I can drop that check if you like, but the other
checks for a 8472 device (vs 8079) aren't anything different than what
already happens in the driver elsewhere[1].  I'd argue that testing that
a feature is supported in hardware before trying to use it is normal
driver behavior.

If instead you mean that the entire series is somehow applicable only to
our hardware, I'm not sure why.

That hotplug issue isn't seen on the same hardware when using the Linux
driver; so it's a dpdk problem (at least on C3000 ixgbe devs), and not a
hardware problem.  Fixing the hotplug/rateswap issue was my primary
goal, the other patches fix problems I found along the way while
debugging.

I can also reproduce the hotplug/rateswap issue on the PLCC-B, an Intel
reference design for the C3000 family, so again, not unique to this
platform.

Please let me know if that addresses your concerns, or if I've missed
your point.

Thanks,
Steve

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c?h=v5.16-rc6

>>   #define IXGBE_SFF_SFF_8472_COMP              0x5E
>>   #define IXGBE_SFF_SFF_8472_OSCB              0x6E
>>   #define IXGBE_SFF_SFF_8472_ESCB              0x76
>> @@ -48,6 +49,8 @@
>>   #define IXGBE_SFF_SOFT_RS_SELECT_10G 0x8
>> --
>> 2.31.1
> 

RE: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write

[Wang, Haiyue]

> -----Original Message-----
> From: Stephen Douthit <stephend@silicom-usa.com>
> Sent: Tuesday, December 21, 2021 05:33
> To: Wang, Haiyue <haiyue.wang@intel.com>; Lu, Wenzhuo <wenzhuo.lu@intel.com>; Changchun Ouyang
> <changchun.ouyang@intel.com>; Zhang, Helin <helin.zhang@intel.com>
> Cc: dev@dpdk.org; Wang, Wen <wenw@silicom-usa.com>; stable@dpdk.org
> Subject: Re: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write
> 
> On 12/20/21 02:53, Wang, Haiyue wrote:
> >> -----Original Message-----
> >> From: Stephen Douthit <stephend@silicom-usa.com>
> >> Sent: Tuesday, December 7, 2021 06:19
> >> To: Wang, Haiyue <haiyue.wang@intel.com>; Lu, Wenzhuo <wenzhuo.lu@intel.com>; Changchun Ouyang
> >> <changchun.ouyang@intel.com>; Zhang, Helin <helin.zhang@intel.com>
> >> Cc: dev@dpdk.org; Wen Wang <wenw@silicom-usa.com>; Stephen Douthit <stephend@silicom-usa.com>;
> >> stable@dpdk.org
> >> Subject: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write
> >>
> >> Make sure an SFP is really a SFF-8472 device that supports the optional
> >> soft rate select feature before just blindly poking those I2C registers.
> >>
> >> Skip all I2C traffic if we know there's no SFP.
> >>
> >> Fixes: f3430431aba ("ixgbe/base: add SFP+ dual-speed support")
> >> Cc: stable@dpdk.org
> >>
> >> Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
> >> ---
> >
> >
> >>        /* Set RS0 */
> >>        status = hw->phy.ops.read_i2c_byte(hw, IXGBE_SFF_SFF_8472_OSCB,
> >>                                           IXGBE_I2C_EEPROM_DEV_ADDR2,
> >> diff --git a/drivers/net/ixgbe/base/ixgbe_phy.h b/drivers/net/ixgbe/base/ixgbe_phy.h
> >> index ceefbb3e68..cd57ce040f 100644
> >> --- a/drivers/net/ixgbe/base/ixgbe_phy.h
> >> +++ b/drivers/net/ixgbe/base/ixgbe_phy.h
> >> @@ -21,6 +21,7 @@
> >>   #define IXGBE_SFF_CABLE_TECHNOLOGY   0x8
> >>   #define IXGBE_SFF_CABLE_SPEC_COMP    0x3C
> >>   #define IXGBE_SFF_SFF_8472_SWAP              0x5C
> >> +#define IXGBE_SFF_SFF_8472_EOPT              0x5D
> >
> > Looks like this is YOUR platform specific, then this patchset can't be
> > merged. : - (
> 
> This isn't anything unique to our hardware, these values are coming from
> the SFF-8472 SFP+ I2C specification.
> 
> The ability to do a soft rate select via I2C is an optional feature, and
> modules that support it are supposed to set bit 3 in byte 93 (0x5d), the
> "Enhanced Options" register, to advertise the functionality.
> 
> Please see section 8.10 and Table 8-6 in the SFF-8472 spec.
> 
> Checking the RATE_SELECT bit flag may be overkill since the transceiver
> is supposed to ignore writes to rate select control bits if the feature
> isn't implemented.  I can drop that check if you like, but the other
> checks for a 8472 device (vs 8079) aren't anything different than what
> already happens in the driver elsewhere[1].  I'd argue that testing that
> a feature is supported in hardware before trying to use it is normal
> driver behavior.
> 
> If instead you mean that the entire series is somehow applicable only to
> our hardware, I'm not sure why.
> 
> That hotplug issue isn't seen on the same hardware when using the Linux
> driver; so it's a dpdk problem (at least on C3000 ixgbe devs), and not a

I can't find your related fix in two official Linux drivers:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/intel/ixgbe
https://www.intel.com/content/www/us/en/download/14302/14687/intel-network-adapter-driver-for-pcie-intel-10-gigabit-ethernet-network-connections-under-linux.html?

Normally, DPDK keeps sync with this kind of release.

> hardware problem.  Fixing the hotplug/rateswap issue was my primary
> goal, the other patches fix problems I found along the way while
> debugging.
> 
> I can also reproduce the hotplug/rateswap issue on the PLCC-B, an Intel
> reference design for the C3000 family, so again, not unique to this
> platform.

I guess this is just in C3000 reference board SDK ?

I recommend you submit the fix to kernel firstly, you will get more
experts' reviews and fully test:

https://patchwork.ozlabs.org/project/intel-wired-lan/list/
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

> 
> Please let me know if that addresses your concerns, or if I've missed
> your point.
> 





> Thanks,
> Steve
> 
> [1]
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/intel/ixg
> be/ixgbe_ethtool.c?h=v5.16-rc6
> 
> >>   #define IXGBE_SFF_SFF_8472_COMP              0x5E
> >>   #define IXGBE_SFF_SFF_8472_OSCB              0x6E
> >>   #define IXGBE_SFF_SFF_8472_ESCB              0x76
> >> @@ -48,6 +49,8 @@
> >>   #define IXGBE_SFF_SOFT_RS_SELECT_10G 0x8
> >> --
> >> 2.31.1
> >

RE: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write

[Morten Brørup]

> From: Wang, Haiyue [mailto:haiyue.wang@intel.com]
> Sent: Tuesday, 21 December 2021 02.15
> 
> > -----Original Message-----
> > From: Stephen Douthit <stephend@silicom-usa.com>
> > Sent: Tuesday, December 21, 2021 05:33
> >
> > On 12/20/21 02:53, Wang, Haiyue wrote:
> > >> -----Original Message-----
> > >> From: Stephen Douthit <stephend@silicom-usa.com>
> > >> Sent: Tuesday, December 7, 2021 06:19
> > >>
> > >> Make sure an SFP is really a SFF-8472 device that supports the
> optional
> > >> soft rate select feature before just blindly poking those I2C
> registers.
> > >>
> > >> Skip all I2C traffic if we know there's no SFP.
> > >>
> > >> Fixes: f3430431aba ("ixgbe/base: add SFP+ dual-speed support")
> > >> Cc: stable@dpdk.org
> > >>
> > >> Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
> > >> ---
> > >
> > >
> > >>        /* Set RS0 */
> > >>        status = hw->phy.ops.read_i2c_byte(hw,
> IXGBE_SFF_SFF_8472_OSCB,
> > >>
> IXGBE_I2C_EEPROM_DEV_ADDR2,
> > >> diff --git a/drivers/net/ixgbe/base/ixgbe_phy.h
> b/drivers/net/ixgbe/base/ixgbe_phy.h
> > >> index ceefbb3e68..cd57ce040f 100644
> > >> --- a/drivers/net/ixgbe/base/ixgbe_phy.h
> > >> +++ b/drivers/net/ixgbe/base/ixgbe_phy.h
> > >> @@ -21,6 +21,7 @@
> > >>   #define IXGBE_SFF_CABLE_TECHNOLOGY   0x8
> > >>   #define IXGBE_SFF_CABLE_SPEC_COMP    0x3C
> > >>   #define IXGBE_SFF_SFF_8472_SWAP              0x5C
> > >> +#define IXGBE_SFF_SFF_8472_EOPT              0x5D
> > >
> > > Looks like this is YOUR platform specific, then this patchset can't
> be
> > > merged. : - (
> >
> > This isn't anything unique to our hardware, these values are coming
> from
> > the SFF-8472 SFP+ I2C specification.
> >
> > The ability to do a soft rate select via I2C is an optional feature,
> and
> > modules that support it are supposed to set bit 3 in byte 93 (0x5d),
> the
> > "Enhanced Options" register, to advertise the functionality.
> >
> > Please see section 8.10 and Table 8-6 in the SFF-8472 spec.
> >
> > Checking the RATE_SELECT bit flag may be overkill since the
> transceiver
> > is supposed to ignore writes to rate select control bits if the
> feature
> > isn't implemented.  I can drop that check if you like, but the other
> > checks for a 8472 device (vs 8079) aren't anything different than
> what
> > already happens in the driver elsewhere[1].  I'd argue that testing
> that
> > a feature is supported in hardware before trying to use it is normal
> > driver behavior.
> >
> > If instead you mean that the entire series is somehow applicable only
> to
> > our hardware, I'm not sure why.
> >
> > That hotplug issue isn't seen on the same hardware when using the
> Linux
> > driver; so it's a dpdk problem (at least on C3000 ixgbe devs), and
> not a
> 
> I can't find your related fix in two official Linux drivers:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree
> /drivers/net/ethernet/intel/ixgbe
> https://www.intel.com/content/www/us/en/download/14302/14687/intel-
> network-adapter-driver-for-pcie-intel-10-gigabit-ethernet-network-
> connections-under-linux.html?
> 
> Normally, DPDK keeps sync with this kind of release.
> 

Working with the Linux kernel mainline drivers is good advice.

The official Intel Linux drivers seem to be ages behind the Kernel mainline, and they don't fully support the C3000 NICs, so don’t waste any time there! We recently tried using the official Intel Linux drivers for a C3338 based project (using Kernel 3.19 in 32 bit mode with x2APIC disabled), and they didn't work at all. We ended up backporting the necessary changes from the kernel mainline instead.

> > hardware problem.  Fixing the hotplug/rateswap issue was my primary
> > goal, the other patches fix problems I found along the way while
> > debugging.
> >
> > I can also reproduce the hotplug/rateswap issue on the PLCC-B, an
> Intel
> > reference design for the C3000 family, so again, not unique to this
> > platform.
> 
> I guess this is just in C3000 reference board SDK ?
> 
> I recommend you submit the fix to kernel firstly, you will get more
> experts' reviews and fully test:
> 
> https://patchwork.ozlabs.org/project/intel-wired-lan/list/
> https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
> 

RE: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write

[Wang, Haiyue]

> -----Original Message-----
> From: Morten Brørup <mb@smartsharesystems.com>
> Sent: Tuesday, December 21, 2021 16:58
> To: Wang, Haiyue <haiyue.wang@intel.com>; stephend@silicom-usa.com; Lu, Wenzhuo <wenzhuo.lu@intel.com>;
> Changchun Ouyang <changchun.ouyang@intel.com>; Zhang, Helin <helin.zhang@intel.com>
> Cc: dev@dpdk.org; Wang, Wen <wenw@silicom-usa.com>; stable@dpdk.org
> Subject: RE: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write
> 
> > From: Wang, Haiyue [mailto:haiyue.wang@intel.com]
> > Sent: Tuesday, 21 December 2021 02.15
> >
> > > -----Original Message-----
> > > From: Stephen Douthit <stephend@silicom-usa.com>
> > > Sent: Tuesday, December 21, 2021 05:33
> > >
> > > On 12/20/21 02:53, Wang, Haiyue wrote:
> > > >> -----Original Message-----
> > > >> From: Stephen Douthit <stephend@silicom-usa.com>
> > > >> Sent: Tuesday, December 7, 2021 06:19
> > > >>
> > > >> Make sure an SFP is really a SFF-8472 device that supports the
> > optional
> > > >> soft rate select feature before just blindly poking those I2C
> > registers.
> > > >>
> > > >> Skip all I2C traffic if we know there's no SFP.
> > > >>
> > > >> Fixes: f3430431aba ("ixgbe/base: add SFP+ dual-speed support")
> > > >> Cc: stable@dpdk.org
> > > >>
> > > >> Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
> > > >> ---
> > > >


> >
> > Normally, DPDK keeps sync with this kind of release.
> >
> 
> Working with the Linux kernel mainline drivers is good advice.
> 
> The official Intel Linux drivers seem to be ages behind the Kernel mainline, and they don't fully

No, the "ixgbe" drivers is updated on "7/8/2021".

https://www.intel.com/content/www/us/en/download/14302/14687/intel-network-adapter-driver-for-pcie-intel-10-gigabit-ethernet-network-connections-under-linux.html

> support the C3000 NICs, so don’t waste any time there! We recently tried using the official Intel
> Linux drivers for a C3338 based project (using Kernel 3.19 in 32 bit mode with x2APIC disabled), and
> they didn't work at all. We ended up backporting the necessary changes from the kernel mainline
> instead.

From Steve's response: 
     ME: "I guess this is just in C3000 reference board SDK ?"
     Steve: "It's the board covered by Intel Doc # 574437."

I check the doc "Last Updated: 11/07/2018".... It should be some kind of customer release, that's why
they are not in the official *open source* Linux driver, so keep your patch set as private.

RE: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write

[Morten Brørup]

> From: Wang, Haiyue [mailto:haiyue.wang@intel.com]
> Sent: Wednesday, 22 December 2021 02.24
> 
> > -----Original Message-----
> > From: Morten Brørup <mb@smartsharesystems.com>
> > Sent: Tuesday, December 21, 2021 16:58
> >
> > > From: Wang, Haiyue [mailto:haiyue.wang@intel.com]
> > > Sent: Tuesday, 21 December 2021 02.15
> > >
> > > > -----Original Message-----
> > > > From: Stephen Douthit <stephend@silicom-usa.com>
> > > > Sent: Tuesday, December 21, 2021 05:33
> > > >
> > > > On 12/20/21 02:53, Wang, Haiyue wrote:
> > > > >> -----Original Message-----
> > > > >> From: Stephen Douthit <stephend@silicom-usa.com>
> > > > >> Sent: Tuesday, December 7, 2021 06:19
> > > > >>
> > > > >> Make sure an SFP is really a SFF-8472 device that supports the
> > > optional
> > > > >> soft rate select feature before just blindly poking those I2C
> > > registers.
> > > > >>
> > > > >> Skip all I2C traffic if we know there's no SFP.
> > > > >>
> > > > >> Fixes: f3430431aba ("ixgbe/base: add SFP+ dual-speed support")
> > > > >> Cc: stable@dpdk.org
> > > > >>
> > > > >> Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
> > > > >> ---
> > > > >
> 
> 
> > >
> > > Normally, DPDK keeps sync with this kind of release.
> > >
> >
> > Working with the Linux kernel mainline drivers is good advice.
> >
> > The official Intel Linux drivers seem to be ages behind the Kernel
> mainline, and they don't fully
> 
> No, the "ixgbe" drivers is updated on "7/8/2021".
> 
> https://www.intel.com/content/www/us/en/download/14302/14687/intel-
> network-adapter-driver-for-pcie-intel-10-gigabit-ethernet-network-
> connections-under-linux.html

So you can imagine my surprise that they didn't work on the C3338 SoC launched by Intel in Q1'17. The web page says that the drivers supports kernel versions 2.6.18 to 5.12, so we expected them to work with kernel 3.19. Perhaps they haven't been tested with the C3338 SoC. Also, the test section on the web page only mentions 64 bit distributions, so perhaps they haven't been tested with a 32 bit kernel. There is no test report available, so I can only speculate.

I am sorry if I came off as badmouthing the Intel out-of-tree driver. I was only trying to convey to the good folks at Silicom that kernel.org is a better source of inspiration than the Intel out-of-tree driver, which is not as up-to-date as the kernel.org driver, and thus not the optimal source of inspiration for driver development. The out-of-tree drivers serve a different purpose, where they are extremely valuable: In normal production environments where it is not an option to compile and deploy a kernel from scratch.

> 
> > support the C3000 NICs, so don’t waste any time there! We recently
> tried using the official Intel
> > Linux drivers for a C3338 based project (using Kernel 3.19 in 32 bit
> mode with x2APIC disabled), and
> > they didn't work at all. We ended up backporting the necessary
> changes from the kernel mainline
> > instead.
> 
> From Steve's response:
>      ME: "I guess this is just in C3000 reference board SDK ?"
>      Steve: "It's the board covered by Intel Doc # 574437."
> 
> I check the doc "Last Updated: 11/07/2018".... It should be some kind
> of customer release, that's why
> they are not in the official *open source* Linux driver, so keep your
> patch set as private.

I didn't mention it explicitly, but I'm not involved with Silicom, and was not referring to their hardware. The hardware board we had problems with is currently in volume production at a major ODM. But I guess that it is usually being deployed with a 64 bit kernel, as opposed to the 32 bit kernel we were using.


Med venlig hilsen / kind regards

Morten Brørup
CTO


SmartShare Systems A/S
Tonsbakken 16-18
DK-2740 Skovlunde
Denmark

Office      +45 70 20 00 93
Direct      +45 89 93 50 22
Mobile      +45 25 40 82 12

mb@smartsharesystems.com
www.smartsharesystems.com

RE: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write

[Wang, Haiyue]

> -----Original Message-----
> From: Morten Brørup <mb@smartsharesystems.com>
> Sent: Wednesday, December 22, 2021 18:44
> To: Wang, Haiyue <haiyue.wang@intel.com>; stephend@silicom-usa.com; Lu, Wenzhuo <wenzhuo.lu@intel.com>;
> Zhang, Helin <helin.zhang@intel.com>; Zhang, Qi Z <qi.z.zhang@intel.com>
> Cc: dev@dpdk.org; Wang, Wen <wenw@silicom-usa.com>; stable@dpdk.org
> Subject: RE: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write
> 
> > From: Wang, Haiyue [mailto:haiyue.wang@intel.com]
> > Sent: Wednesday, 22 December 2021 02.24
> >
> > > -----Original Message-----
> > > From: Morten Brørup <mb@smartsharesystems.com>
> > > Sent: Tuesday, December 21, 2021 16:58
> > >
> > > > From: Wang, Haiyue [mailto:haiyue.wang@intel.com]
> > > > Sent: Tuesday, 21 December 2021 02.15
> > > >
> > > > > -----Original Message-----
> > > > > From: Stephen Douthit <stephend@silicom-usa.com>
> > > > > Sent: Tuesday, December 21, 2021 05:33
> > > > >
> > > > > On 12/20/21 02:53, Wang, Haiyue wrote:
> > > > > >> -----Original Message-----
> > > > > >> From: Stephen Douthit <stephend@silicom-usa.com>
> > > > > >> Sent: Tuesday, December 7, 2021 06:19
> > > > > >>
> > > > > >> Make sure an SFP is really a SFF-8472 device that supports the
> > > > optional
> > > > > >> soft rate select feature before just blindly poking those I2C
> > > > registers.
> > > > > >>
> > > > > >> Skip all I2C traffic if we know there's no SFP.
> > > > > >>
> > > > > >> Fixes: f3430431aba ("ixgbe/base: add SFP+ dual-speed support")
> > > > > >> Cc: stable@dpdk.org
> > > > > >>
> > > > > >> Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
> > > > > >> ---
> > > > > >
> >
> >
> > > >
> > > > Normally, DPDK keeps sync with this kind of release.
> > > >
> > >
> > > Working with the Linux kernel mainline drivers is good advice.
> > >
> > > The official Intel Linux drivers seem to be ages behind the Kernel
> > mainline, and they don't fully
> >
> > No, the "ixgbe" drivers is updated on "7/8/2021".
> >
> > https://www.intel.com/content/www/us/en/download/14302/14687/intel-
> > network-adapter-driver-for-pcie-intel-10-gigabit-ethernet-network-
> > connections-under-linux.html
> 
> So you can imagine my surprise that they didn't work on the C3338 SoC launched by Intel in Q1'17. The
> web page says that the drivers supports kernel versions 2.6.18 to 5.12, so we expected them to work
> with kernel 3.19. Perhaps they haven't been tested with the C3338 SoC. Also, the test section on the
> web page only mentions 64 bit distributions, so perhaps they haven't been tested with a 32 bit kernel.
> There is no test report available, so I can only speculate.
> 
> I am sorry if I came off as badmouthing the Intel out-of-tree driver. I was only trying to convey to
> the good folks at Silicom that kernel.org is a better source of inspiration than the Intel out-of-tree
> driver, which is not as up-to-date as the kernel.org driver, and thus not the optimal source of
> inspiration for driver development. The out-of-tree drivers serve a different purpose, where they are
> extremely valuable: In normal production environments where it is not an option to compile and deploy
> a kernel from scratch.
>

> >
> > > support the C3000 NICs, so don’t waste any time there! We recently
> > tried using the official Intel
> > > Linux drivers for a C3338 based project (using Kernel 3.19 in 32 bit
> > mode with x2APIC disabled), and
> > > they didn't work at all. We ended up backporting the necessary
> > changes from the kernel mainline
> > > instead.
> >
> > From Steve's response:
> >      ME: "I guess this is just in C3000 reference board SDK ?"
> >      Steve: "It's the board covered by Intel Doc # 574437."
> >
> > I check the doc "Last Updated: 11/07/2018".... It should be some kind
> > of customer release, that's why
> > they are not in the official *open source* Linux driver, so keep your
> > patch set as private.
> 
> I didn't mention it explicitly, but I'm not involved with Silicom, and was not referring to their
> hardware. The hardware board we had problems with is currently in volume production at a major ODM.
> But I guess that it is usually being deployed with a 64 bit kernel, as opposed to the 32 bit kernel we
> were using.

I understood, but we need to follow the open source vs customer release policy,
so not everything is upstream.

The ixgbe (especially in base directory) code is so stable, so in other words,
this patch set can be rebased easily. ;-)

If the patch is about ixgbe ethdev part (vs kernel netdev), it will be welcomed,
since our team mainly work on this (And the base code is mainly developed by the
kernel team, that's why I recommend to send it to
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan).

Hope this will make things clear. ;-)

> 
> 
> Med venlig hilsen / kind regards
> 
> Morten Brørup
> CTO
> 
> 
> SmartShare Systems A/S
> Tonsbakken 16-18
> DK-2740 Skovlunde
> Denmark
> 
> Office      +45 70 20 00 93
> Direct      +45 89 93 50 22
> Mobile      +45 25 40 82 12
> 
> mb@smartsharesystems.com
> www.smartsharesystems.com

RE: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write

[Morten Brørup]

> From: Wang, Haiyue [mailto:haiyue.wang@intel.com]
> Sent: Wednesday, 22 December 2021 17.03
> 
> > -----Original Message-----
> > From: Morten Brørup <mb@smartsharesystems.com>
> > Sent: Wednesday, December 22, 2021 18:44
> >
> > > From: Wang, Haiyue [mailto:haiyue.wang@intel.com]
> > > Sent: Wednesday, 22 December 2021 02.24
> > >
> > > > -----Original Message-----
> > > > From: Morten Brørup <mb@smartsharesystems.com>
> > > > Sent: Tuesday, December 21, 2021 16:58
> > > >
> > > > > From: Wang, Haiyue [mailto:haiyue.wang@intel.com]
> > > > > Sent: Tuesday, 21 December 2021 02.15
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Stephen Douthit <stephend@silicom-usa.com>
> > > > > > Sent: Tuesday, December 21, 2021 05:33
> > > > > >
> > > > > > On 12/20/21 02:53, Wang, Haiyue wrote:
> > > > > > >> -----Original Message-----
> > > > > > >> From: Stephen Douthit <stephend@silicom-usa.com>
> > > > > > >> Sent: Tuesday, December 7, 2021 06:19
> > > > > > >>
> > > > > > >> Make sure an SFP is really a SFF-8472 device that supports
> the
> > > > > optional
> > > > > > >> soft rate select feature before just blindly poking those
> I2C
> > > > > registers.
> > > > > > >>
> > > > > > >> Skip all I2C traffic if we know there's no SFP.
> > > > > > >>
> > > > > > >> Fixes: f3430431aba ("ixgbe/base: add SFP+ dual-speed
> support")
> > > > > > >> Cc: stable@dpdk.org
> > > > > > >>
> > > > > > >> Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
> > > > > > >> ---
> > > > > > >
> > >
> > >
> > > > >
> > > > > Normally, DPDK keeps sync with this kind of release.
> > > > >
> > > >
> > > > Working with the Linux kernel mainline drivers is good advice.
> > > >
> > > > The official Intel Linux drivers seem to be ages behind the
> Kernel
> > > mainline, and they don't fully
> > >
> > > No, the "ixgbe" drivers is updated on "7/8/2021".
> > >
> > > https://www.intel.com/content/www/us/en/download/14302/14687/intel-
> > > network-adapter-driver-for-pcie-intel-10-gigabit-ethernet-network-
> > > connections-under-linux.html
> >
> > So you can imagine my surprise that they didn't work on the C3338 SoC
> launched by Intel in Q1'17. The
> > web page says that the drivers supports kernel versions 2.6.18 to
> 5.12, so we expected them to work
> > with kernel 3.19. Perhaps they haven't been tested with the C3338
> SoC. Also, the test section on the
> > web page only mentions 64 bit distributions, so perhaps they haven't
> been tested with a 32 bit kernel.
> > There is no test report available, so I can only speculate.
> >
> > I am sorry if I came off as badmouthing the Intel out-of-tree driver.
> I was only trying to convey to
> > the good folks at Silicom that kernel.org is a better source of
> inspiration than the Intel out-of-tree
> > driver, which is not as up-to-date as the kernel.org driver, and thus
> not the optimal source of
> > inspiration for driver development. The out-of-tree drivers serve a
> different purpose, where they are
> > extremely valuable: In normal production environments where it is not
> an option to compile and deploy
> > a kernel from scratch.
> >
> 
> > >
> > > > support the C3000 NICs, so don’t waste any time there! We
> recently
> > > tried using the official Intel
> > > > Linux drivers for a C3338 based project (using Kernel 3.19 in 32
> bit
> > > mode with x2APIC disabled), and
> > > > they didn't work at all. We ended up backporting the necessary
> > > changes from the kernel mainline
> > > > instead.
> > >
> > > From Steve's response:
> > >      ME: "I guess this is just in C3000 reference board SDK ?"
> > >      Steve: "It's the board covered by Intel Doc # 574437."
> > >
> > > I check the doc "Last Updated: 11/07/2018".... It should be some
> kind
> > > of customer release, that's why
> > > they are not in the official *open source* Linux driver, so keep
> your
> > > patch set as private.
> >
> > I didn't mention it explicitly, but I'm not involved with Silicom,
> and was not referring to their
> > hardware. The hardware board we had problems with is currently in
> volume production at a major ODM.
> > But I guess that it is usually being deployed with a 64 bit kernel,
> as opposed to the 32 bit kernel we
> > were using.
> 
> I understood, but we need to follow the open source vs customer release
> policy,
> so not everything is upstream.
> 
> The ixgbe (especially in base directory) code is so stable, so in other
> words,
> this patch set can be rebased easily. ;-)
> 
> If the patch is about ixgbe ethdev part (vs kernel netdev), it will be
> welcomed,
> since our team mainly work on this (And the base code is mainly
> developed by the
> kernel team, that's why I recommend to send it to
> https://lists.osuosl.org/mailman/listinfo/intel-wired-lan).
> 
> Hope this will make things clear. ;-)

ACK. :-)

Re: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write

[Stephen Douthit]

<snip>

>>>
>>>  From Steve's response:
>>>       ME: "I guess this is just in C3000 reference board SDK ?"
>>>       Steve: "It's the board covered by Intel Doc # 574437."
>>>
>>> I check the doc "Last Updated: 11/07/2018".... It should be some kind
>>> of customer release, that's why
>>> they are not in the official *open source* Linux driver, so keep your
>>> patch set as private.
>>
>> I didn't mention it explicitly, but I'm not involved with Silicom, and was not referring to their
>> hardware. The hardware board we had problems with is currently in volume production at a major ODM.
>> But I guess that it is usually being deployed with a 64 bit kernel, as opposed to the 32 bit kernel we
>> were using.
> 
> I understood, but we need to follow the open source vs customer release policy,
> so not everything is upstream.

I'm afraid we're still talking past each other here.

I'm using the CRB as a "known good" platform to confirm that the hotplug
issue is _not_ some OEM specific platform quirk.  The CRB is not the
target of the patch set, it's a way for me to make sure I'm not chasing
a bug in our hardware.  If our hardware was the only place the bug was
present I would agree that maintaining this fix outside of mainline
would make sense, but testing on the CRB proves that's not the case.

Implying that testing patches on a CRB somehow makes them ineligible to
be open sourced/upstreamed doesn't really make sense to me.

> The ixgbe (especially in base directory) code is so stable, so in other words,
> this patch set can be rebased easily. ;-)
I found an 82599ES (device ID 0x10fb) this afternoon, and the dpdk ixgbe
driver is "so stable" (I know you meant from a standpoint of few
commits, but couldn't resist) that it has the exact same problem I was
debugging on the C3000 devices. :)

Big log at the end of the email.[4]

Unfortunately my patch series does not fix the issue on the 82599 like
it did for the C3000, so I'll need to look into that now that I have a
card to test on. :(

> If the patch is about ixgbe ethdev part (vs kernel netdev), it will be welcomed,
> since our team mainly work on this (And the base code is mainly developed by the
> kernel team, that's why I recommend to send it to
> https://lists.osuosl.org/mailman/listinfo/intel-wired-lan).

What I believe you're saying is that you want any edits to base/* to
come from the mainline Linux driver, and only fixes to ixgbe_ethdev.c
from this mailing list.

Here's how the hotplug patches -- 1, 2, 4, 5 -- (ignoring 3, 6, & 7 for
now), break down into those two categories:

Patch 1:
All edits are to ixgbe_ethdev.c, and the same code is already in the
mainline.[1]

Seems good to submit here.

Patch 2:
Located in base/, but exports functionality that makes Patch 5 simpler.

I could probably drop this patch and rework Patch 5 to just attempt an
I2C read and see if it's ACKed/NAKed, but I believe that this is
cleaner.  Since the driver bitbangs the I2C bus, I'd prefer to avoid the
overhead and check the SFP present signal via the associated SDP input
and skip the I2C traffic if I know the cage is empty.

This patch seems like it could go either way, add present check to Linux
and backport, or just move the code into ixgbe_ethdev.c since it's not
fixing anything in Linux.

Patch 4:
Located in base/, but addresses a workaround found _only in dpdk_ that
checks the TX laser enable bit (SDP3 on 82599) to qualify the link
status reported by ixgbe_check_mac_link_generic().

The original code smells off to me for a number of reasons.
1) There's nothing like that in mainline.  The only code touching SDP3
there is just controlling the TX laser[2], SDP3's status is never read.

2) The workaround runs on all fiber platforms, even though the commit
message said it was for the 82599eb.  The C3000 doesn't use SDP3 for
TX_DISABLE so poking it on that platform is definitely incorrect, and
may be wrong for others.

3) Open coding this in a single location instead of putting it in a
platform specific mac->ops.check_link() callback implies that the link
status returned at every other check_link() callsite may be bogus if
the workaround is really needed.

I could leave the workaround where it started off in ixgbe_ethdev and
just add a mac type qualifier.  With that edit it seems good to submit
here.

Patch 5:
All edits are to ixgbe_ethdev.c, and addresses an issue not found in
the Linux driver.  The Linux driver uses ixgbe_service_task()[3] to
handle SFP detection and setup, and handles hotplug just fine.  This
patch moves the dpdk driver closer to that working scheme.

Seems good to submit here.

In summary:
1 & 5 seem like clear candidates to upstream directly via dpdk, 2 could
be reworked to avoid touching base/ or dropped and 5 refactored, and 4
can be easily refactored to avoid touching base/.  However my gut says
something's off and commits 1ca05831b9b & ff8162cb957 that made patch 4
necessary in the first place weren't really getting to root cause since
the Linux driver functions without this workaround.

> Hope this will make things clear. ;-)

Same. :)

Either way I'm hoping to be out for the rest of the year, so I'll
revisit this January 3rd.

Happy holidays!

Thanks,
Steve

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c?h=v5.16-rc6#n2887
[2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/intel/ixgbe/ixgbe_82599.c?h=v5.16-rc6#n572
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/intel/ixgbe/ixgbe_82599.c?h=v5.16-rc6#n592
[3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c?h=v5.16-rc6#n7911
[4] Wall of text:

Script started on 2021-12-22 11:19:00-05:00 [TERM="screen.xterm-256color" TTY="/dev/pts/2" COLUMNS="136" LINES="72"]
[root@madrid dpdk]# ./build/app/dpdk-testpmd --log-level *:info -c7 --vdev=0000:20:00.0 --vdev=0000:20:00.1 -- -i --nb-cores=2 --nb-ports=2 --total-num-mbufs=2048
EAL: Detected CPU lcores: 8
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available 1048576 kB hugepages reported
EAL: VFIO support initialized
EAL: Using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: Probe PCI driver: net_ixgbe (8086:10fb) device: 0000:20:00.0 (socket 0)
EAL: Ignore mapping IO port bar(2)
EAL: Probe PCI driver: net_ixgbe (8086:10fb) device: 0000:20:00.1 (socket 0)
Interactive-mode selected
testpmd: create a new mbuf pool <mb_pool_0>: n=2048, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Configuring Port 0 (socket 0)
ixgbe_dev_link_status_print():  Port 0: Link Down
Port 0: 00:E0:ED:97:62:4D
Configuring Port 1 (socket 0)
ixgbe_dev_link_status_print():  Port 1: Link Down
Port 1: 00:E0:ED:97:62:4E
Checking link statuses...
Done
testpmd> # started with empty SFP cages
testpmd> show port info all

********************* Infos for port 0  *********************
MAC address: 00:E0:ED:97:62:4D
Device name: 0000:20:00.0
Driver name: net_ixgbe
Firmware-version: 0x800000cb
Devargs:
Connect to socket: 0
memory allocation on the socket: 0
Link status: down
Link speed: None
Link duplex: half-duplex
Autoneg status: On
MTU: 1500
Promiscuous mode: enabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 128
Maximum number of MAC addresses of hash filtering: 4096
VLAN offload:
   strip off, filter off, extend off, qinq strip off
Hash key size in bytes: 40
Redirection table size: 128
Supported RSS offload flow types:
   ipv4
   ipv4-tcp
   ipv4-udp
   ipv6
   ipv6-tcp
   ipv6-udp
   ipv6-ex
   ipv6-tcp-ex
   ipv6-udp-ex
Minimum size of RX buffer: 1024
Maximum configurable length of RX packet: 15872
Maximum configurable size of LRO aggregated packet: 0
Maximum number of VMDq pools: 64
Current number of RX queues: 1
Max possible RX queues: 128
Max possible number of RXDs per queue: 4096
Min possible number of RXDs per queue: 32
RXDs number alignment: 8
Current number of TX queues: 1
Max possible TX queues: 64
Max possible number of TXDs per queue: 4096
Min possible number of TXDs per queue: 32
TXDs number alignment: 8
Max segment number per packet: 40
Max segment number per MTU/TSO: 40
Device capabilities: 0x0( )

********************* Infos for port 1  *********************
MAC address: 00:E0:ED:97:62:4E
Device name: 0000:20:00.1
Driver name: net_ixgbe
Firmware-version: 0x800000cb
Devargs:
Connect to socket: 0
memory allocation on the socket: 0
Link status: down
Link speed: None
Link duplex: half-duplex
Autoneg status: On
MTU: 1500
Promiscuous mode: enabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 128
Maximum number of MAC addresses of hash filtering: 4096
VLAN offload:
   strip off, filter off, extend off, qinq strip off
Hash key size in bytes: 40
Redirection table size: 128
Supported RSS offload flow types:
   ipv4
   ipv4-tcp
   ipv4-udp
   ipv6
   ipv6-tcp
   ipv6-udp
   ipv6-ex
   ipv6-tcp-ex
   ipv6-udp-ex
Minimum size of RX buffer: 1024
Maximum configurable length of RX packet: 15872
Maximum configurable size of LRO aggregated packet: 0
Maximum number of VMDq pools: 64
Current number of RX queues: 1
Max possible RX queues: 128
Max possible number of RXDs per queue: 4096
Min possible number of RXDs per queue: 32
RXDs number alignment: 8
Current number of TX queues: 1
Max possible TX queues: 64
Max possible number of TXDs per queue: 4096
Min possible number of TXDs per queue: 32
TXDs number alignment: 8
Max segment number per packet: 40
Max segment number per MTU/TSO: 40
Device capabilities: 0x0( )
testpmd> # Cages now filled with 1G SFPs
testpmd> show port info all

********************* Infos for port 0  *********************
MAC address: 00:E0:ED:97:62:4D
Device name: 0000:20:00.0
Driver name: net_ixgbe
Firmware-version: 0x800000cb
Devargs:
Connect to socket: 0
memory allocation on the socket: 0
Link status: down
Link speed: None
Link duplex: half-duplex
Autoneg status: On
MTU: 1500
Promiscuous mode: enabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 128
Maximum number of MAC addresses of hash filtering: 4096
VLAN offload:
   strip off, filter off, extend off, qinq strip off
Hash key size in bytes: 40
Redirection table size: 128
Supported RSS offload flow types:
   ipv4
   ipv4-tcp
   ipv4-udp
   ipv6
   ipv6-tcp
   ipv6-udp
   ipv6-ex
   ipv6-tcp-ex
   ipv6-udp-ex
Minimum size of RX buffer: 1024
Maximum configurable length of RX packet: 15872
Maximum configurable size of LRO aggregated packet: 0
Maximum number of VMDq pools: 64
Current number of RX queues: 1
Max possible RX queues: 128
Max possible number of RXDs per queue: 4096
Min possible number of RXDs per queue: 32
RXDs number alignment: 8
Current number of TX queues: 1
Max possible TX queues: 64
Max possible number of TXDs per queue: 4096
Min possible number of TXDs per queue: 32
TXDs number alignment: 8
Max segment number per packet: 40
Max segment number per MTU/TSO: 40
Device capabilities: 0x0( )

********************* Infos for port 1  *********************
MAC address: 00:E0:ED:97:62:4E
Device name: 0000:20:00.1
Driver name: net_ixgbe
Firmware-version: 0x800000cb
Devargs:
Connect to socket: 0
memory allocation on the socket: 0
Link status: down
Link speed: None
Link duplex: half-duplex
Autoneg status: On
MTU: 1500
Promiscuous mode: enabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 128
Maximum number of MAC addresses of hash filtering: 4096
VLAN offload:
   strip off, filter off, extend off, qinq strip off
Hash key size in bytes: 40
Redirection table size: 128
Supported RSS offload flow types:
   ipv4
   ipv4-tcp
   ipv4-udp
   ipv6
   ipv6-tcp
   ipv6-udp
   ipv6-ex
   ipv6-tcp-ex
   ipv6-udp-ex
Minimum size of RX buffer: 1024
Maximum configurable length of RX packet: 15872
Maximum configurable size of LRO aggregated packet: 0
Maximum number of VMDq pools: 64
Current number of RX queues: 1
Max possible RX queues: 128
Max possible number of RXDs per queue: 4096
Min possible number of RXDs per queue: 32
RXDs number alignment: 8
Current number of TX queues: 1
Max possible TX queues: 64
Max possible number of TXDs per queue: 4096
Min possible number of TXDs per queue: 32
TXDs number alignment: 8
Max segment number per packet: 40
Max segment number per MTU/TSO: 40
Device capabilities: 0x0( )
testpmd> # No link detected if SFPs not present when testpmd launched, bug
testpmd> quit

Stopping port 0...
Stopping ports...
Done

Stopping port 1...
Stopping ports...
Done

Shutting down port 0...
Closing ports...
Port 0 is closed
Done

Shutting down port 1...
Closing ports...
Port 1 is closed
Done

Bye...
[root@madrid dpdk]# # Starting testpmd again with 1G SFPs already in cages
[root@madrid dpdk]# ./build/app/dpdk-testpmd --log-level *:info -c7 --vdev=0000:20:00.0 --vdev=0000:20:00.1 -- -i --nb-cores=2 --nb-ports=2 --total-num-mbufs=2048
EAL: Detected CPU lcores: 8
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available 1048576 kB hugepages reported
EAL: VFIO support initialized
EAL: Using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: Probe PCI driver: net_ixgbe (8086:10fb) device: 0000:20:00.0 (socket 0)
EAL: Ignore mapping IO port bar(2)
EAL: Probe PCI driver: net_ixgbe (8086:10fb) device: 0000:20:00.1 (socket 0)
Interactive-mode selected
testpmd: create a new mbuf pool <mb_pool_0>: n=2048, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Configuring Port 0 (socket 0)
ixgbe_dev_link_status_print():  Port 0: Link Down
Port 0: 00:E0:ED:97:62:4D
Configuring Port 1 (socket 0)
ixgbe_dev_link_status_print():  Port 1: Link Down
Port 1: 00:E0:ED:97:62:4E
Checking link statuses...
ixgbe_dev_link_status_print(): Port 1: Link Up - speed 1000 Mbps - full-duplex
ixgbe_dev_link_status_print(): Port 0: Link Up - speed 1000 Mbps - full-duplex
Done
testpmd> ixgbe_dev_link_status_print(): Port 1: Link Up - speed 1000 Mbps - full-duplex

Port 1: link state change event
ixgbe_dev_link_status_print(): Port 0: Link Up - speed 1000 Mbps - full-duplex

Port 0: link state change event

testpmd> # Link detected correctly for SFPs present on testpmd start
testpmd> show port info all

********************* Infos for port 0  *********************
MAC address: 00:E0:ED:97:62:4D
Device name: 0000:20:00.0
Driver name: net_ixgbe
Firmware-version: 0x800000cb
Devargs:
Connect to socket: 0
memory allocation on the socket: 0
Link status: up
Link speed: 1 Gbps
Link duplex: full-duplex
Autoneg status: On
MTU: 1500
Promiscuous mode: enabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 128
Maximum number of MAC addresses of hash filtering: 4096
VLAN offload:
   strip off, filter off, extend off, qinq strip off
Hash key size in bytes: 40
Redirection table size: 128
Supported RSS offload flow types:
   ipv4
   ipv4-tcp
   ipv4-udp
   ipv6
   ipv6-tcp
   ipv6-udp
   ipv6-ex
   ipv6-tcp-ex
   ipv6-udp-ex
Minimum size of RX buffer: 1024
Maximum configurable length of RX packet: 15872
Maximum configurable size of LRO aggregated packet: 0
Maximum number of VMDq pools: 64
Current number of RX queues: 1
Max possible RX queues: 128
Max possible number of RXDs per queue: 4096
Min possible number of RXDs per queue: 32
RXDs number alignment: 8
Current number of TX queues: 1
Max possible TX queues: 64
Max possible number of TXDs per queue: 4096
Min possible number of TXDs per queue: 32
TXDs number alignment: 8
Max segment number per packet: 40
Max segment number per MTU/TSO: 40
Device capabilities: 0x0( )

********************* Infos for port 1  *********************
MAC address: 00:E0:ED:97:62:4E
Device name: 0000:20:00.1
Driver name: net_ixgbe
Firmware-version: 0x800000cb
Devargs:
Connect to socket: 0
memory allocation on the socket: 0
Link status: up
Link speed: 1 Gbps
Link duplex: full-duplex
Autoneg status: On
MTU: 1500
Promiscuous mode: enabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 128
Maximum number of MAC addresses of hash filtering: 4096
VLAN offload:
   strip off, filter off, extend off, qinq strip off
Hash key size in bytes: 40
Redirection table size: 128
Supported RSS offload flow types:
   ipv4
   ipv4-tcp
   ipv4-udp
   ipv6
   ipv6-tcp
   ipv6-udp
   ipv6-ex
   ipv6-tcp-ex
   ipv6-udp-ex
Minimum size of RX buffer: 1024
Maximum configurable length of RX packet: 15872
Maximum configurable size of LRO aggregated packet: 0
Maximum number of VMDq pools: 64
Current number of RX queues: 1
Max possible RX queues: 128
Max possible number of RXDs per queue: 4096
Min possible number of RXDs per queue: 32
RXDs number alignment: 8
Current number of TX queues: 1
Max possible TX queues: 64
Max possible number of TXDs per queue: 4096
Min possible number of TXDs per queue: 32
TXDs number alignment: 8
Max segment number per packet: 40
Max segment number per MTU/TSO: 40
Device capabilities: 0x0( )
testpmd> # unplug/replug 1G SFPs
testpmd> ixgbe_dev_link_status_print():  Port 0: Link Down
ixgbe_dev_link_status_print():  Port 1: Link Down
ixgbe_dev_link_status_print():  Port 0: Link Down

Port 0: link state change event
ixgbe_dev_link_status_print():  Port 1: Link Down

Port 1: link state change event
ixgbe_dev_link_status_print(): Port 1: Link Up - speed 1000 Mbps - full-duplex
ixgbe_dev_link_status_print(): Port 0: Link Up - speed 1000 Mbps - full-duplex
ixgbe_dev_link_status_print(): Port 1: Link Up - speed 1000 Mbps - full-duplex

Port 1: link state change event
ixgbe_dev_link_status_print(): Port 0: Link Up - speed 1000 Mbps - full-duplex

Port 0: link state change event

testpmd> # Link back up at 1G after unplug/replug
testpmd> # Now swapping 1G SFPs for 10G SFPs
testpmd> ixgbe_dev_link_status_print():  Port 0: Link Down
ixgbe_dev_link_status_print():  Port 1: Link Down
ixgbe_dev_link_status_print():  Port 0: Link Down

Port 0: link state change event
ixgbe_dev_link_status_print():  Port 1: Link Down

Port 1: link state change event
ixgbe_dev_link_status_print(): Port 1: Link Up - speed 1000 Mbps - full-duplex
ixgbe_dev_link_status_print(): Port 0: Link Up - speed 1000 Mbps - full-duplex
ixgbe_dev_link_status_print(): Port 1: Link Up - speed 1000 Mbps - full-duplex

Port 1: link state change event
ixgbe_dev_link_status_print(): Port 0: Link Up - speed 1000 Mbps - full-duplex

Port 0: link state change event

testpmd> # Same bug, testpmd did not correctly ID and link at 10G, still at 1G
testpmd> show port info all

********************* Infos for port 0  *********************
MAC address: 00:E0:ED:97:62:4D
Device name: 0000:20:00.0
Driver name: net_ixgbe
Firmware-version: 0x800000cb
Devargs:
Connect to socket: 0
memory allocation on the socket: 0
Link status: up
Link speed: 1 Gbps
Link duplex: full-duplex
Autoneg status: On
MTU: 1500
Promiscuous mode: enabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 128
Maximum number of MAC addresses of hash filtering: 4096
VLAN offload:
   strip off, filter off, extend off, qinq strip off
Hash key size in bytes: 40
Redirection table size: 128
Supported RSS offload flow types:
   ipv4
   ipv4-tcp
   ipv4-udp
   ipv6
   ipv6-tcp
   ipv6-udp
   ipv6-ex
   ipv6-tcp-ex
   ipv6-udp-ex
Minimum size of RX buffer: 1024
Maximum configurable length of RX packet: 15872
Maximum configurable size of LRO aggregated packet: 0
Maximum number of VMDq pools: 64
Current number of RX queues: 1
Max possible RX queues: 128
Max possible number of RXDs per queue: 4096
Min possible number of RXDs per queue: 32
RXDs number alignment: 8
Current number of TX queues: 1
Max possible TX queues: 64
Max possible number of TXDs per queue: 4096
Min possible number of TXDs per queue: 32
TXDs number alignment: 8
Max segment number per packet: 40
Max segment number per MTU/TSO: 40
Device capabilities: 0x0( )

********************* Infos for port 1  *********************
MAC address: 00:E0:ED:97:62:4E
Device name: 0000:20:00.1
Driver name: net_ixgbe
Firmware-version: 0x800000cb
Devargs:
Connect to socket: 0
memory allocation on the socket: 0
Link status: up
Link speed: 1 Gbps
Link duplex: full-duplex
Autoneg status: On
MTU: 1500
Promiscuous mode: enabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 128
Maximum number of MAC addresses of hash filtering: 4096
VLAN offload:
   strip off, filter off, extend off, qinq strip off
Hash key size in bytes: 40
Redirection table size: 128
Supported RSS offload flow types:
   ipv4
   ipv4-tcp
   ipv4-udp
   ipv6
   ipv6-tcp
   ipv6-udp
   ipv6-ex
   ipv6-tcp-ex
   ipv6-udp-ex
Minimum size of RX buffer: 1024
Maximum configurable length of RX packet: 15872
Maximum configurable size of LRO aggregated packet: 0
Maximum number of VMDq pools: 64
Current number of RX queues: 1
Max possible RX queues: 128
Max possible number of RXDs per queue: 4096
Min possible number of RXDs per queue: 32
RXDs number alignment: 8
Current number of TX queues: 1
Max possible TX queues: 64
Max possible number of TXDs per queue: 4096
Min possible number of TXDs per queue: 32
TXDs number alignment: 8
Max segment number per packet: 40
Max segment number per MTU/TSO: 40
Device capabilities: 0x0( )
testpmd> # quit and restart testpmd with 10G SFPs already installed
testpmd> quit

Stopping port 0...
Stopping ports...
ixgbe_dev_link_status_print():  Port 1: Link Down
Done

Stopping port 1...
Stopping ports...
Done

Shutting down port 0...
Closing ports...
Port 0 is closed
Done

Shutting down port 1...
Closing ports...
Port 1 is closed
Done

Bye...
[root@madrid dpdk]# ./build/app/dpdk-testpmd --log-level *:info -c7 --vdev=0000:20:00.0 --vdev=0000:20:00.1 -- -i --nb-cores=2 --nb-ports=2 --total-num-mbufs=2048
EAL: Detected CPU lcores: 8
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available 1048576 kB hugepages reported
EAL: VFIO support initialized
EAL: Using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: Probe PCI driver: net_ixgbe (8086:10fb) device: 0000:20:00.0 (socket 0)
EAL: Ignore mapping IO port bar(2)
EAL: Probe PCI driver: net_ixgbe (8086:10fb) device: 0000:20:00.1 (socket 0)
Interactive-mode selected
testpmd: create a new mbuf pool <mb_pool_0>: n=2048, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Configuring Port 0 (socket 0)
ixgbe_dev_link_status_print():  Port 0: Link Down
Port 0: 00:E0:ED:97:62:4D
Configuring Port 1 (socket 0)
ixgbe_dev_link_status_print():  Port 1: Link Down
Port 1: 00:E0:ED:97:62:4E
Checking link statuses...
ixgbe_dev_link_status_print(): Port 1: Link Up - speed 10000 Mbps - full-duplex
ixgbe_dev_link_status_print(): Port 0: Link Up - speed 10000 Mbps - full-duplex
Done
testpmd> ixgbe_dev_link_status_print(): Port 1: Link Up - speed 10000 Mbps - full-duplex

Port 1: link state change event
ixgbe_dev_link_status_print(): Port 0: Link Up - speed 10000 Mbps - full-duplex

Port 0: link state change event

testpmd> # now linked correctly @ 10G
testpmd> show port info all

********************* Infos for port 0  *********************
MAC address: 00:E0:ED:97:62:4D
Device name: 0000:20:00.0
Driver name: net_ixgbe
Firmware-version: 0x800000cb
Devargs:
Connect to socket: 0
memory allocation on the socket: 0
Link status: up
Link speed: 10 Gbps
Link duplex: full-duplex
Autoneg status: On
MTU: 1500
Promiscuous mode: enabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 128
Maximum number of MAC addresses of hash filtering: 4096
VLAN offload:
   strip off, filter off, extend off, qinq strip off
Hash key size in bytes: 40
Redirection table size: 128
Supported RSS offload flow types:
   ipv4
   ipv4-tcp
   ipv4-udp
   ipv6
   ipv6-tcp
   ipv6-udp
   ipv6-ex
   ipv6-tcp-ex
   ipv6-udp-ex
Minimum size of RX buffer: 1024
Maximum configurable length of RX packet: 15872
Maximum configurable size of LRO aggregated packet: 0
Maximum number of VMDq pools: 64
Current number of RX queues: 1
Max possible RX queues: 128
Max possible number of RXDs per queue: 4096
Min possible number of RXDs per queue: 32
RXDs number alignment: 8
Current number of TX queues: 1
Max possible TX queues: 64
Max possible number of TXDs per queue: 4096
Min possible number of TXDs per queue: 32
TXDs number alignment: 8
Max segment number per packet: 40
Max segment number per MTU/TSO: 40
Device capabilities: 0x0( )

********************* Infos for port 1  *********************
MAC address: 00:E0:ED:97:62:4E
Device name: 0000:20:00.1
Driver name: net_ixgbe
Firmware-version: 0x800000cb
Devargs:
Connect to socket: 0
memory allocation on the socket: 0
Link status: up
Link speed: 10 Gbps
Link duplex: full-duplex
Autoneg status: On
MTU: 1500
Promiscuous mode: enabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 128
Maximum number of MAC addresses of hash filtering: 4096
VLAN offload:
   strip off, filter off, extend off, qinq strip off
Hash key size in bytes: 40
Redirection table size: 128
Supported RSS offload flow types:
   ipv4
   ipv4-tcp
   ipv4-udp
   ipv6
   ipv6-tcp
   ipv6-udp
   ipv6-ex
   ipv6-tcp-ex
   ipv6-udp-ex
Minimum size of RX buffer: 1024
Maximum configurable length of RX packet: 15872
Maximum configurable size of LRO aggregated packet: 0
Maximum number of VMDq pools: 64
Current number of RX queues: 1
Max possible RX queues: 128
Max possible number of RXDs per queue: 4096
Min possible number of RXDs per queue: 32
RXDs number alignment: 8
Current number of TX queues: 1
Max possible TX queues: 64
Max possible number of TXDs per queue: 4096
Min possible number of TXDs per queue: 32
TXDs number alignment: 8
Max segment number per packet: 40
Max segment number per MTU/TSO: 40
Device capabilities: 0x0( )
testpmd> # swap 10G SFPs for 1G, expect no link since they wont be IDed and setup for 1G, and cant handle 10G
testpmd> ixgbe_dev_link_status_print():  Port 0: Link Down
ixgbe_dev_link_status_print():  Port 1: Link Down
ixgbe_dev_link_status_print():  Port 0: Link Down

Port 0: link state change event
ixgbe_dev_link_status_print():  Port 1: Link Down

Port 1: link state change event

testpmd> # 1G SFPs now installed, and as expected, no link :(
testpmd> show port info all

********************* Infos for port 0  *********************
MAC address: 00:E0:ED:97:62:4D
Device name: 0000:20:00.0
Driver name: net_ixgbe
Firmware-version: 0x800000cb
Devargs:
Connect to socket: 0
memory allocation on the socket: 0
Link status: down
Link speed: None
Link duplex: half-duplex
Autoneg status: On
MTU: 1500
Promiscuous mode: enabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 128
Maximum number of MAC addresses of hash filtering: 4096
VLAN offload:
   strip off, filter off, extend off, qinq strip off
Hash key size in bytes: 40
Redirection table size: 128
Supported RSS offload flow types:
   ipv4
   ipv4-tcp
   ipv4-udp
   ipv6
   ipv6-tcp
   ipv6-udp
   ipv6-ex
   ipv6-tcp-ex
   ipv6-udp-ex
Minimum size of RX buffer: 1024
Maximum configurable length of RX packet: 15872
Maximum configurable size of LRO aggregated packet: 0
Maximum number of VMDq pools: 64
Current number of RX queues: 1
Max possible RX queues: 128
Max possible number of RXDs per queue: 4096
Min possible number of RXDs per queue: 32
RXDs number alignment: 8
Current number of TX queues: 1
Max possible TX queues: 64
Max possible number of TXDs per queue: 4096
Min possible number of TXDs per queue: 32
TXDs number alignment: 8
Max segment number per packet: 40
Max segment number per MTU/TSO: 40
Device capabilities: 0x0( )

********************* Infos for port 1  *********************
MAC address: 00:E0:ED:97:62:4E
Device name: 0000:20:00.1
Driver name: net_ixgbe
Firmware-version: 0x800000cb
Devargs:
Connect to socket: 0
memory allocation on the socket: 0
Link status: down
Link speed: None
Link duplex: half-duplex
Autoneg status: On
MTU: 1500
Promiscuous mode: enabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 128
Maximum number of MAC addresses of hash filtering: 4096
VLAN offload:
   strip off, filter off, extend off, qinq strip off
Hash key size in bytes: 40
Redirection table size: 128
Supported RSS offload flow types:
   ipv4
   ipv4-tcp
   ipv4-udp
   ipv6
   ipv6-tcp
   ipv6-udp
   ipv6-ex
   ipv6-tcp-ex
   ipv6-udp-ex
Minimum size of RX buffer: 1024
Maximum configurable length of RX packet: 15872
Maximum configurable size of LRO aggregated packet: 0
Maximum number of VMDq pools: 64
Current number of RX queues: 1
Max possible RX queues: 128
Max possible number of RXDs per queue: 4096
Min possible number of RXDs per queue: 32
RXDs number alignment: 8
Current number of TX queues: 1
Max possible TX queues: 64
Max possible number of TXDs per queue: 4096
Min possible number of TXDs per queue: 32
TXDs number alignment: 8
Max segment number per packet: 40
Max segment number per MTU/TSO: 40
Device capabilities: 0x0( )
testpmd> quit

Stopping port 0...
Stopping ports...
Done

Stopping port 1...
Stopping ports...
Done

Shutting down port 0...
Closing ports...
Port 0 is closed
Done

Shutting down port 1...
Closing ports...
Port 1 is closed
Done

Bye...
[root@madrid dpdk]# exit
Script done on 2021-12-22 11:32:00-05:00 [COMMAND_EXIT_CODE="0"]

RE: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write

[Wang, Haiyue]

> -----Original Message-----
> From: Stephen Douthit <stephend@silicom-usa.com>
> Sent: Thursday, December 23, 2021 05:44
> To: Wang, Haiyue <haiyue.wang@intel.com>; Morten Brørup <mb@smartsharesystems.com>; Lu, Wenzhuo
> <wenzhuo.lu@intel.com>; Zhang, Helin <helin.zhang@intel.com>; Zhang, Qi Z <qi.z.zhang@intel.com>
> Cc: dev@dpdk.org; Wang, Wen <wenw@silicom-usa.com>; stable@dpdk.org
> Subject: Re: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write
> 
> <snip>
> 


> Same. :)
> 
> Either way I'm hoping to be out for the rest of the year, so I'll
> revisit this January 3rd.
> 
> Happy holidays!

OK, let's see in 2022, happy holidays. ;-)

> 
> Thanks,
> Steve
> 

> 
> Bye...
> [root@madrid dpdk]# exit
> Script done on 2021-12-22 11:32:00-05:00 [COMMAND_EXIT_CODE="0"]

Re: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write

[Stephen Douthit]

On 12/22/21 19:55, Wang, Haiyue wrote:
> OK, let's see in 2022, happy holidays. ;-)

A bit of a slower start to 2022 than I had hoped, but I have a v3 of the
hotplug fix in progress.  It's now working for the 82599 and C3000 ixgbe
devices under Linux, and I'm in the process of getting this built and
tested under FreeBSD.

Assuming the FreeBSD testing comes together smoothly my plan is to split
these patches into three series:

1) The SFP hotplug fix
2) Refactor SDP3 TX_DISABLE 82599 link check stuff
3) Support of additional SFP types under ixgbe

My focus right now is on the SFP hotplug fix, so please let me know if
there's any additional feedback on that portion of the original series.

Right now the feedback I have that impacts the hotplug fix is a cleanup
for patch 1, and the general comment that changes to files under
ixgbe/base are usually backported from Linux, and so I should refactor
my changes to live in ixgbe_ethdev.c

Thanks,
Steve

RE: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write

[Wang, Haiyue]

> -----Original Message-----
> From: Stephen Douthit <stephend@silicom-usa.com>
> Sent: Wednesday, January 19, 2022 05:06
> To: Wang, Haiyue <haiyue.wang@intel.com>; Morten Brørup <mb@smartsharesystems.com>; Lu, Wenzhuo
> <wenzhuo.lu@intel.com>; Zhang, Helin <helin.zhang@intel.com>; Zhang, Qi Z <qi.z.zhang@intel.com>
> Cc: dev@dpdk.org; Wang, Wen <wenw@silicom-usa.com>; stable@dpdk.org
> Subject: Re: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write
> 
> On 12/22/21 19:55, Wang, Haiyue wrote:
> > OK, let's see in 2022, happy holidays. ;-)
> 
> A bit of a slower start to 2022 than I had hoped, but I have a v3 of the
> hotplug fix in progress.  It's now working for the 82599 and C3000 ixgbe
> devices under Linux, and I'm in the process of getting this built and
> tested under FreeBSD.
> 
> Assuming the FreeBSD testing comes together smoothly my plan is to split
> these patches into three series:
> 
> 1) The SFP hotplug fix
> 2) Refactor SDP3 TX_DISABLE 82599 link check stuff
> 3) Support of additional SFP types under ixgbe
> 
> My focus right now is on the SFP hotplug fix, so please let me know if
> there's any additional feedback on that portion of the original series.
> 
> Right now the feedback I have that impacts the hotplug fix is a cleanup
> for patch 1, and the general comment that changes to files under
> ixgbe/base are usually backported from Linux, and so I should refactor
> my changes to live in ixgbe_ethdev.c

Yeah, just put the change in base code to *separate patch*, so that they are
easily to be reviewed, thanks.

> 
> Thanks,
> Steve

Re: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write

[Ferruh Yigit]

On 1/19/2022 12:31 AM, Wang, Haiyue wrote:
>> -----Original Message-----
>> From: Stephen Douthit <stephend@silicom-usa.com>
>> Sent: Wednesday, January 19, 2022 05:06
>> To: Wang, Haiyue <haiyue.wang@intel.com>; Morten Brørup <mb@smartsharesystems.com>; Lu, Wenzhuo
>> <wenzhuo.lu@intel.com>; Zhang, Helin <helin.zhang@intel.com>; Zhang, Qi Z <qi.z.zhang@intel.com>
>> Cc: dev@dpdk.org; Wang, Wen <wenw@silicom-usa.com>; stable@dpdk.org
>> Subject: Re: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write
>>
>> On 12/22/21 19:55, Wang, Haiyue wrote:
>>> OK, let's see in 2022, happy holidays. ;-)
>>
>> A bit of a slower start to 2022 than I had hoped, but I have a v3 of the
>> hotplug fix in progress.  It's now working for the 82599 and C3000 ixgbe
>> devices under Linux, and I'm in the process of getting this built and
>> tested under FreeBSD.
>>
>> Assuming the FreeBSD testing comes together smoothly my plan is to split
>> these patches into three series:
>>
>> 1) The SFP hotplug fix
>> 2) Refactor SDP3 TX_DISABLE 82599 link check stuff
>> 3) Support of additional SFP types under ixgbe
>>
>> My focus right now is on the SFP hotplug fix, so please let me know if
>> there's any additional feedback on that portion of the original series.
>>
>> Right now the feedback I have that impacts the hotplug fix is a cleanup
>> for patch 1, and the general comment that changes to files under
>> ixgbe/base are usually backported from Linux, and so I should refactor
>> my changes to live in ixgbe_ethdev.c
> 
> Yeah, just put the change in base code to *separate patch*, so that they are
> easily to be reviewed, thanks.
> 

Hi Steve, Wen, Haiyue,

Can you please clarify the above change request?

I though it is related to splitting base code updates into their own
patches, but that already seems the case in set (except from a few minor mix).

And what is the status, is there a new version worked on? Or is the
set waiting for more review?

Thanks,
ferruh

RE: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write

[Jeff Daly]

Ferruh,
	Stephen has passed on support of his patches to me, I will be the main point of contact going forward.  I'm still ramping up on the code, expect patch updates to come from me in the future.

-----Original Message-----
From: Ferruh Yigit <ferruh.yigit@intel.com> 
Sent: Monday, February 7, 2022 11:04 AM
To: Wang, Haiyue <haiyue.wang@intel.com>; Stephen Douthit <stephend@silicom-usa.com>; Morten Brørup <mb@smartsharesystems.com>; Lu, Wenzhuo <wenzhuo.lu@intel.com>; Zhang, Helin <helin.zhang@intel.com>; Zhang, Qi Z <qi.z.zhang@intel.com>
Cc: dev@dpdk.org; Wen Wang <wenw@silicom-usa.com>; stable@dpdk.org
Subject: Re: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write

Caution: This is an external email. Please take care when clicking links or opening attachments.


On 1/19/2022 12:31 AM, Wang, Haiyue wrote:
>> -----Original Message-----
>> From: Stephen Douthit <stephend@silicom-usa.com>
>> Sent: Wednesday, January 19, 2022 05:06
>> To: Wang, Haiyue <haiyue.wang@intel.com>; Morten Brørup 
>> <mb@smartsharesystems.com>; Lu, Wenzhuo <wenzhuo.lu@intel.com>; 
>> Zhang, Helin <helin.zhang@intel.com>; Zhang, Qi Z 
>> <qi.z.zhang@intel.com>
>> Cc: dev@dpdk.org; Wang, Wen <wenw@silicom-usa.com>; stable@dpdk.org
>> Subject: Re: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate 
>> select is supported before write
>>
>> On 12/22/21 19:55, Wang, Haiyue wrote:
>>> OK, let's see in 2022, happy holidays. ;-)
>>
>> A bit of a slower start to 2022 than I had hoped, but I have a v3 of 
>> the hotplug fix in progress.  It's now working for the 82599 and 
>> C3000 ixgbe devices under Linux, and I'm in the process of getting 
>> this built and tested under FreeBSD.
>>
>> Assuming the FreeBSD testing comes together smoothly my plan is to 
>> split these patches into three series:
>>
>> 1) The SFP hotplug fix
>> 2) Refactor SDP3 TX_DISABLE 82599 link check stuff
>> 3) Support of additional SFP types under ixgbe
>>
>> My focus right now is on the SFP hotplug fix, so please let me know 
>> if there's any additional feedback on that portion of the original series.
>>
>> Right now the feedback I have that impacts the hotplug fix is a 
>> cleanup for patch 1, and the general comment that changes to files 
>> under ixgbe/base are usually backported from Linux, and so I should 
>> refactor my changes to live in ixgbe_ethdev.c
>
> Yeah, just put the change in base code to *separate patch*, so that 
> they are easily to be reviewed, thanks.
>

Hi Steve, Wen, Haiyue,

Can you please clarify the above change request?

I though it is related to splitting base code updates into their own patches, but that already seems the case in set (except from a few minor mix).

And what is the status, is there a new version worked on? Or is the set waiting for more review?

Thanks,
ferruh

Re: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write

[Ferruh Yigit]

On 2/8/2022 1:50 PM, Jeff Daly wrote:

moved response down, please don't top post.

> -----Original Message-----
> From: Ferruh Yigit <ferruh.yigit@intel.com>
> Sent: Monday, February 7, 2022 11:04 AM
> To: Wang, Haiyue <haiyue.wang@intel.com>; Stephen Douthit <stephend@silicom-usa.com>; Morten Brørup <mb@smartsharesystems.com>; Lu, Wenzhuo <wenzhuo.lu@intel.com>; Zhang, Helin <helin.zhang@intel.com>; Zhang, Qi Z <qi.z.zhang@intel.com>
> Cc: dev@dpdk.org; Wen Wang <wenw@silicom-usa.com>; stable@dpdk.org
> Subject: Re: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write
> 
> Caution: This is an external email. Please take care when clicking links or opening attachments.
> 
> 
> On 1/19/2022 12:31 AM, Wang, Haiyue wrote:
>>> -----Original Message-----
>>> From: Stephen Douthit <stephend@silicom-usa.com>
>>> Sent: Wednesday, January 19, 2022 05:06
>>> To: Wang, Haiyue <haiyue.wang@intel.com>; Morten Brørup
>>> <mb@smartsharesystems.com>; Lu, Wenzhuo <wenzhuo.lu@intel.com>;
>>> Zhang, Helin <helin.zhang@intel.com>; Zhang, Qi Z
>>> <qi.z.zhang@intel.com>
>>> Cc: dev@dpdk.org; Wang, Wen <wenw@silicom-usa.com>; stable@dpdk.org
>>> Subject: Re: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate
>>> select is supported before write
>>>
>>> On 12/22/21 19:55, Wang, Haiyue wrote:
>>>> OK, let's see in 2022, happy holidays. ;-)
>>>
>>> A bit of a slower start to 2022 than I had hoped, but I have a v3 of
>>> the hotplug fix in progress.  It's now working for the 82599 and
>>> C3000 ixgbe devices under Linux, and I'm in the process of getting
>>> this built and tested under FreeBSD.
>>>
>>> Assuming the FreeBSD testing comes together smoothly my plan is to
>>> split these patches into three series:
>>>
>>> 1) The SFP hotplug fix
>>> 2) Refactor SDP3 TX_DISABLE 82599 link check stuff
>>> 3) Support of additional SFP types under ixgbe
>>>
>>> My focus right now is on the SFP hotplug fix, so please let me know
>>> if there's any additional feedback on that portion of the original series.
>>>
>>> Right now the feedback I have that impacts the hotplug fix is a
>>> cleanup for patch 1, and the general comment that changes to files
>>> under ixgbe/base are usually backported from Linux, and so I should
>>> refactor my changes to live in ixgbe_ethdev.c
>>
>> Yeah, just put the change in base code to *separate patch*, so that
>> they are easily to be reviewed, thanks.
>>
> 
> Hi Steve, Wen, Haiyue,
> 
> Can you please clarify the above change request?
> 
> I though it is related to splitting base code updates into their own patches, but that already seems the case in set (except from a few minor mix).
> 
> And what is the status, is there a new version worked on? Or is the set waiting for more review?
> 
> Ferruh,
> 	Stephen has passed on support of his patches to me, I will be the main point of contact going forward.  I'm still ramping up on the code, expect patch updates to come from me in the future.
> 

Hi Jeff,

What is the planned changes in the next version?

I just want to be sure that we are on same page with the change request,
to not waste effort/time.

Haiyue, perhaps can you articulate the request again?

RE: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write

[Wang, Haiyue]

> -----Original Message-----
> From: Yigit, Ferruh <ferruh.yigit@intel.com>
> Sent: Tuesday, February 8, 2022 22:52
> To: Daly, Jeff <jeffd@silicom-usa.com>; Wang, Haiyue <haiyue.wang@intel.com>; Stephen Douthit
> <stephend@silicom-usa.com>; Morten Brørup <mb@smartsharesystems.com>; Lu, Wenzhuo
> <wenzhuo.lu@intel.com>; Zhang, Helin <helin.zhang@intel.com>; Zhang, Qi Z <qi.z.zhang@intel.com>
> Cc: dev@dpdk.org; Wang, Wen <wenw@silicom-usa.com>; stable@dpdk.org
> Subject: Re: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write
> 
> On 2/8/2022 1:50 PM, Jeff Daly wrote:
> 
> moved response down, please don't top post.
> 
> > -----Original Message-----
> > From: Ferruh Yigit <ferruh.yigit@intel.com>
> > Sent: Monday, February 7, 2022 11:04 AM
> > To: Wang, Haiyue <haiyue.wang@intel.com>; Stephen Douthit <stephend@silicom-usa.com>; Morten Brørup
> <mb@smartsharesystems.com>; Lu, Wenzhuo <wenzhuo.lu@intel.com>; Zhang, Helin <helin.zhang@intel.com>;
> Zhang, Qi Z <qi.z.zhang@intel.com>
> > Cc: dev@dpdk.org; Wen Wang <wenw@silicom-usa.com>; stable@dpdk.org
> > Subject: Re: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before
> write
> >
> > Caution: This is an external email. Please take care when clicking links or opening attachments.
> >
> >
> > On 1/19/2022 12:31 AM, Wang, Haiyue wrote:
> >>> -----Original Message-----
> >>> From: Stephen Douthit <stephend@silicom-usa.com>
> >>> Sent: Wednesday, January 19, 2022 05:06
> >>> To: Wang, Haiyue <haiyue.wang@intel.com>; Morten Brørup
> >>> <mb@smartsharesystems.com>; Lu, Wenzhuo <wenzhuo.lu@intel.com>;
> >>> Zhang, Helin <helin.zhang@intel.com>; Zhang, Qi Z
> >>> <qi.z.zhang@intel.com>
> >>> Cc: dev@dpdk.org; Wang, Wen <wenw@silicom-usa.com>; stable@dpdk.org
> >>> Subject: Re: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate
> >>> select is supported before write
> >>>
> >>> On 12/22/21 19:55, Wang, Haiyue wrote:
> >>>> OK, let's see in 2022, happy holidays. ;-)
> >>>
> >>> A bit of a slower start to 2022 than I had hoped, but I have a v3 of
> >>> the hotplug fix in progress.  It's now working for the 82599 and
> >>> C3000 ixgbe devices under Linux, and I'm in the process of getting
> >>> this built and tested under FreeBSD.
> >>>
> >>> Assuming the FreeBSD testing comes together smoothly my plan is to
> >>> split these patches into three series:
> >>>
> >>> 1) The SFP hotplug fix
> >>> 2) Refactor SDP3 TX_DISABLE 82599 link check stuff
> >>> 3) Support of additional SFP types under ixgbe
> >>>
> >>> My focus right now is on the SFP hotplug fix, so please let me know
> >>> if there's any additional feedback on that portion of the original series.
> >>>
> >>> Right now the feedback I have that impacts the hotplug fix is a
> >>> cleanup for patch 1, and the general comment that changes to files
> >>> under ixgbe/base are usually backported from Linux, and so I should
> >>> refactor my changes to live in ixgbe_ethdev.c
> >>
> >> Yeah, just put the change in base code to *separate patch*, so that
> >> they are easily to be reviewed, thanks.
> >>
> >
> > Hi Steve, Wen, Haiyue,
> >
> > Can you please clarify the above change request?
> >
> > I though it is related to splitting base code updates into their own patches, but that already seems
> the case in set (except from a few minor mix).
> >
> > And what is the status, is there a new version worked on? Or is the set waiting for more review?
> >
> > Ferruh,
> > 	Stephen has passed on support of his patches to me, I will be the main point of contact going
> forward.  I'm still ramping up on the code, expect patch updates to come from me in the future.
> >
> 
> Hi Jeff,
> 
> What is the planned changes in the next version?
> 
> I just want to be sure that we are on same page with the change request,
> to not waste effort/time.
> 
> Haiyue, perhaps can you articulate the request again?

Just put the change in base directory into separate patch (es), so that we can ask
different experts to review the patchset easily.

Thanks.

Re: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write

[Ferruh Yigit]

On 2/9/2022 4:00 AM, Wang, Haiyue wrote:
>> -----Original Message-----
>> From: Yigit, Ferruh <ferruh.yigit@intel.com>
>> Sent: Tuesday, February 8, 2022 22:52
>> To: Daly, Jeff <jeffd@silicom-usa.com>; Wang, Haiyue <haiyue.wang@intel.com>; Stephen Douthit
>> <stephend@silicom-usa.com>; Morten Brørup <mb@smartsharesystems.com>; Lu, Wenzhuo
>> <wenzhuo.lu@intel.com>; Zhang, Helin <helin.zhang@intel.com>; Zhang, Qi Z <qi.z.zhang@intel.com>
>> Cc: dev@dpdk.org; Wang, Wen <wenw@silicom-usa.com>; stable@dpdk.org
>> Subject: Re: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write
>>
>> On 2/8/2022 1:50 PM, Jeff Daly wrote:
>>
>> moved response down, please don't top post.
>>
>>> -----Original Message-----
>>> From: Ferruh Yigit <ferruh.yigit@intel.com>
>>> Sent: Monday, February 7, 2022 11:04 AM
>>> To: Wang, Haiyue <haiyue.wang@intel.com>; Stephen Douthit <stephend@silicom-usa.com>; Morten Brørup
>> <mb@smartsharesystems.com>; Lu, Wenzhuo <wenzhuo.lu@intel.com>; Zhang, Helin <helin.zhang@intel.com>;
>> Zhang, Qi Z <qi.z.zhang@intel.com>
>>> Cc: dev@dpdk.org; Wen Wang <wenw@silicom-usa.com>; stable@dpdk.org
>>> Subject: Re: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before
>> write
>>>
>>> Caution: This is an external email. Please take care when clicking links or opening attachments.
>>>
>>>
>>> On 1/19/2022 12:31 AM, Wang, Haiyue wrote:
>>>>> -----Original Message-----
>>>>> From: Stephen Douthit <stephend@silicom-usa.com>
>>>>> Sent: Wednesday, January 19, 2022 05:06
>>>>> To: Wang, Haiyue <haiyue.wang@intel.com>; Morten Brørup
>>>>> <mb@smartsharesystems.com>; Lu, Wenzhuo <wenzhuo.lu@intel.com>;
>>>>> Zhang, Helin <helin.zhang@intel.com>; Zhang, Qi Z
>>>>> <qi.z.zhang@intel.com>
>>>>> Cc: dev@dpdk.org; Wang, Wen <wenw@silicom-usa.com>; stable@dpdk.org
>>>>> Subject: Re: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate
>>>>> select is supported before write
>>>>>
>>>>> On 12/22/21 19:55, Wang, Haiyue wrote:
>>>>>> OK, let's see in 2022, happy holidays. ;-)
>>>>>
>>>>> A bit of a slower start to 2022 than I had hoped, but I have a v3 of
>>>>> the hotplug fix in progress.  It's now working for the 82599 and
>>>>> C3000 ixgbe devices under Linux, and I'm in the process of getting
>>>>> this built and tested under FreeBSD.
>>>>>
>>>>> Assuming the FreeBSD testing comes together smoothly my plan is to
>>>>> split these patches into three series:
>>>>>
>>>>> 1) The SFP hotplug fix
>>>>> 2) Refactor SDP3 TX_DISABLE 82599 link check stuff
>>>>> 3) Support of additional SFP types under ixgbe
>>>>>
>>>>> My focus right now is on the SFP hotplug fix, so please let me know
>>>>> if there's any additional feedback on that portion of the original series.
>>>>>
>>>>> Right now the feedback I have that impacts the hotplug fix is a
>>>>> cleanup for patch 1, and the general comment that changes to files
>>>>> under ixgbe/base are usually backported from Linux, and so I should
>>>>> refactor my changes to live in ixgbe_ethdev.c
>>>>
>>>> Yeah, just put the change in base code to *separate patch*, so that
>>>> they are easily to be reviewed, thanks.
>>>>
>>>
>>> Hi Steve, Wen, Haiyue,
>>>
>>> Can you please clarify the above change request?
>>>
>>> I though it is related to splitting base code updates into their own patches, but that already seems
>> the case in set (except from a few minor mix).
>>>
>>> And what is the status, is there a new version worked on? Or is the set waiting for more review?
>>>
>>> Ferruh,
>>> 	Stephen has passed on support of his patches to me, I will be the main point of contact going
>> forward.  I'm still ramping up on the code, expect patch updates to come from me in the future.
>>>
>>
>> Hi Jeff,
>>
>> What is the planned changes in the next version?
>>
>> I just want to be sure that we are on same page with the change request,
>> to not waste effort/time.
>>
>> Haiyue, perhaps can you articulate the request again?
> 
> Just put the change in base directory into separate patch (es), so that we can ask
> different experts to review the patchset easily.
> 

That seems already the case, only two patch has mix and that is a little
(and related), can you please highlight the patches that requires split?

RE: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write

[Wang, Haiyue]

> -----Original Message-----
> From: Yigit, Ferruh <ferruh.yigit@intel.com>
> Sent: Wednesday, February 9, 2022 21:33
> To: Wang, Haiyue <haiyue.wang@intel.com>; Daly, Jeff <jeffd@silicom-usa.com>; Stephen Douthit
> <stephend@silicom-usa.com>; Morten Brørup <mb@smartsharesystems.com>; Lu, Wenzhuo
> <wenzhuo.lu@intel.com>; Zhang, Helin <helin.zhang@intel.com>; Zhang, Qi Z <qi.z.zhang@intel.com>
> Cc: dev@dpdk.org; Wang, Wen <wenw@silicom-usa.com>; stable@dpdk.org
> Subject: Re: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write
> 
> On 2/9/2022 4:00 AM, Wang, Haiyue wrote:
> >> -----Original Message-----
> >> From: Yigit, Ferruh <ferruh.yigit@intel.com>
> >> Sent: Tuesday, February 8, 2022 22:52
> >> To: Daly, Jeff <jeffd@silicom-usa.com>; Wang, Haiyue <haiyue.wang@intel.com>; Stephen Douthit
> >> <stephend@silicom-usa.com>; Morten Brørup <mb@smartsharesystems.com>; Lu, Wenzhuo
> >> <wenzhuo.lu@intel.com>; Zhang, Helin <helin.zhang@intel.com>; Zhang, Qi Z <qi.z.zhang@intel.com>
> >> Cc: dev@dpdk.org; Wang, Wen <wenw@silicom-usa.com>; stable@dpdk.org
> >> Subject: Re: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before
> write
> >>
> >> On 2/8/2022 1:50 PM, Jeff Daly wrote:
> >>
> >> moved response down, please don't top post.
> >>
> >>> -----Original Message-----
> >>> From: Ferruh Yigit <ferruh.yigit@intel.com>
> >>> Sent: Monday, February 7, 2022 11:04 AM
> >>> To: Wang, Haiyue <haiyue.wang@intel.com>; Stephen Douthit <stephend@silicom-usa.com>; Morten
> Brørup
> >> <mb@smartsharesystems.com>; Lu, Wenzhuo <wenzhuo.lu@intel.com>; Zhang, Helin
> <helin.zhang@intel.com>;
> >> Zhang, Qi Z <qi.z.zhang@intel.com>
> >>> Cc: dev@dpdk.org; Wen Wang <wenw@silicom-usa.com>; stable@dpdk.org
> >>> Subject: Re: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before
> >> write
> >>>
> >>> Caution: This is an external email. Please take care when clicking links or opening attachments.
> >>>
> >>>
> >>> On 1/19/2022 12:31 AM, Wang, Haiyue wrote:
> >>>>> -----Original Message-----
> >>>>> From: Stephen Douthit <stephend@silicom-usa.com>
> >>>>> Sent: Wednesday, January 19, 2022 05:06
> >>>>> To: Wang, Haiyue <haiyue.wang@intel.com>; Morten Brørup
> >>>>> <mb@smartsharesystems.com>; Lu, Wenzhuo <wenzhuo.lu@intel.com>;
> >>>>> Zhang, Helin <helin.zhang@intel.com>; Zhang, Qi Z
> >>>>> <qi.z.zhang@intel.com>
> >>>>> Cc: dev@dpdk.org; Wang, Wen <wenw@silicom-usa.com>; stable@dpdk.org
> >>>>> Subject: Re: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate
> >>>>> select is supported before write
> >>>>>
> >>>>> On 12/22/21 19:55, Wang, Haiyue wrote:
> >>>>>> OK, let's see in 2022, happy holidays. ;-)
> >>>>>
> >>>>> A bit of a slower start to 2022 than I had hoped, but I have a v3 of
> >>>>> the hotplug fix in progress.  It's now working for the 82599 and
> >>>>> C3000 ixgbe devices under Linux, and I'm in the process of getting
> >>>>> this built and tested under FreeBSD.
> >>>>>
> >>>>> Assuming the FreeBSD testing comes together smoothly my plan is to
> >>>>> split these patches into three series:
> >>>>>
> >>>>> 1) The SFP hotplug fix
> >>>>> 2) Refactor SDP3 TX_DISABLE 82599 link check stuff
> >>>>> 3) Support of additional SFP types under ixgbe
> >>>>>
> >>>>> My focus right now is on the SFP hotplug fix, so please let me know
> >>>>> if there's any additional feedback on that portion of the original series.
> >>>>>
> >>>>> Right now the feedback I have that impacts the hotplug fix is a
> >>>>> cleanup for patch 1, and the general comment that changes to files
> >>>>> under ixgbe/base are usually backported from Linux, and so I should
> >>>>> refactor my changes to live in ixgbe_ethdev.c
> >>>>
> >>>> Yeah, just put the change in base code to *separate patch*, so that
> >>>> they are easily to be reviewed, thanks.
> >>>>
> >>>
> >>> Hi Steve, Wen, Haiyue,
> >>>
> >>> Can you please clarify the above change request?
> >>>
> >>> I though it is related to splitting base code updates into their own patches, but that already
> seems
> >> the case in set (except from a few minor mix).
> >>>
> >>> And what is the status, is there a new version worked on? Or is the set waiting for more review?
> >>>
> >>> Ferruh,
> >>> 	Stephen has passed on support of his patches to me, I will be the main point of contact going
> >> forward.  I'm still ramping up on the code, expect patch updates to come from me in the future.
> >>>
> >>
> >> Hi Jeff,
> >>
> >> What is the planned changes in the next version?
> >>
> >> I just want to be sure that we are on same page with the change request,
> >> to not waste effort/time.
> >>
> >> Haiyue, perhaps can you articulate the request again?
> >
> > Just put the change in base directory into separate patch (es), so that we can ask
> > different experts to review the patchset easily.
> >
> 
> That seems already the case, only two patch has mix and that is a little
> (and related), can you please highlight the patches that requires split?

Yes, two patches.

[v2,4/7] net/ixgbe: Run 82599 link status workaround only on affected devices
[v2,5/7] net/ixgbe: Fix SFP detection and linking on hotplug

And adding the "net/ixgbe/base:" prefix explicitly is better.

Re: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write

[Stephen Douthit]

On 12/20/21 20:15, Wang, Haiyue wrote:
>> -----Original Message-----
>> From: Stephen Douthit <stephend@silicom-usa.com>
>> Sent: Tuesday, December 21, 2021 05:33
>> To: Wang, Haiyue <haiyue.wang@intel.com>; Lu, Wenzhuo <wenzhuo.lu@intel.com>; Changchun Ouyang
>> <changchun.ouyang@intel.com>; Zhang, Helin <helin.zhang@intel.com>
>> Cc: dev@dpdk.org; Wang, Wen <wenw@silicom-usa.com>; stable@dpdk.org
>> Subject: Re: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write
>>
>> On 12/20/21 02:53, Wang, Haiyue wrote:
>>>> -----Original Message-----
>>>> From: Stephen Douthit <stephend@silicom-usa.com>
>>>> Sent: Tuesday, December 7, 2021 06:19
>>>> To: Wang, Haiyue <haiyue.wang@intel.com>; Lu, Wenzhuo <wenzhuo.lu@intel.com>; Changchun Ouyang
>>>> <changchun.ouyang@intel.com>; Zhang, Helin <helin.zhang@intel.com>
>>>> Cc: dev@dpdk.org; Wen Wang <wenw@silicom-usa.com>; Stephen Douthit <stephend@silicom-usa.com>;
>>>> stable@dpdk.org
>>>> Subject: [PATCH v2 3/7] net/ixgbe: Check that SFF-8472 soft rate select is supported before write
>>>>
>>>> Make sure an SFP is really a SFF-8472 device that supports the optional
>>>> soft rate select feature before just blindly poking those I2C registers.
>>>>
>>>> Skip all I2C traffic if we know there's no SFP.
>>>>
>>>> Fixes: f3430431aba ("ixgbe/base: add SFP+ dual-speed support")
>>>> Cc: stable@dpdk.org
>>>>
>>>> Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
>>>> ---
>>>
>>>
>>>>         /* Set RS0 */
>>>>         status = hw->phy.ops.read_i2c_byte(hw, IXGBE_SFF_SFF_8472_OSCB,
>>>>                                            IXGBE_I2C_EEPROM_DEV_ADDR2,
>>>> diff --git a/drivers/net/ixgbe/base/ixgbe_phy.h b/drivers/net/ixgbe/base/ixgbe_phy.h
>>>> index ceefbb3e68..cd57ce040f 100644
>>>> --- a/drivers/net/ixgbe/base/ixgbe_phy.h
>>>> +++ b/drivers/net/ixgbe/base/ixgbe_phy.h
>>>> @@ -21,6 +21,7 @@
>>>>    #define IXGBE_SFF_CABLE_TECHNOLOGY   0x8
>>>>    #define IXGBE_SFF_CABLE_SPEC_COMP    0x3C
>>>>    #define IXGBE_SFF_SFF_8472_SWAP              0x5C
>>>> +#define IXGBE_SFF_SFF_8472_EOPT              0x5D
>>>
>>> Looks like this is YOUR platform specific, then this patchset can't be
>>> merged. : - (
>>
>> This isn't anything unique to our hardware, these values are coming from
>> the SFF-8472 SFP+ I2C specification.
>>
>> The ability to do a soft rate select via I2C is an optional feature, and
>> modules that support it are supposed to set bit 3 in byte 93 (0x5d), the
>> "Enhanced Options" register, to advertise the functionality.
>>
>> Please see section 8.10 and Table 8-6 in the SFF-8472 spec.
>>
>> Checking the RATE_SELECT bit flag may be overkill since the transceiver
>> is supposed to ignore writes to rate select control bits if the feature
>> isn't implemented.  I can drop that check if you like, but the other
>> checks for a 8472 device (vs 8079) aren't anything different than what
>> already happens in the driver elsewhere[1].  I'd argue that testing that
>> a feature is supported in hardware before trying to use it is normal
>> driver behavior.
>>
>> If instead you mean that the entire series is somehow applicable only to
>> our hardware, I'm not sure why.
>>
>> That hotplug issue isn't seen on the same hardware when using the Linux
>> driver; so it's a dpdk problem (at least on C3000 ixgbe devs), and not a
> 
> I can't find your related fix in two official Linux drivers:

There's no submission from me on the hotplug issue for the mainline,
because the issue isn't present in Linux, only in DPDK.

> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/intel/ixgbe
> https://www.intel.com/content/www/us/en/download/14302/14687/intel-network-adapter-driver-for-pcie-intel-10-gigabit-ethernet-network-connections-under-linux.html?
> 
> Normally, DPDK keeps sync with this kind of release.
> 
>> hardware problem.  Fixing the hotplug/rateswap issue was my primary
>> goal, the other patches fix problems I found along the way while
>> debugging.
>>
>> I can also reproduce the hotplug/rateswap issue on the PLCC-B, an Intel
>> reference design for the C3000 family, so again, not unique to this
>> platform.
> 
> I guess this is just in C3000 reference board SDK ?

It's the board covered by Intel Doc # 574437.

> I recommend you submit the fix to kernel firstly, you will get more
> experts' reviews and fully test:

Since patch 3 isn't directly related to the hotplug issue should I pull
it from the series for v3 to keep the hotplug fixes moving forward here,
and in parallel submit just that one to Linux?

Thanks,
Steve

> https://patchwork.ozlabs.org/project/intel-wired-lan/list/
> https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
> 
>>
>> Please let me know if that addresses your concerns, or if I've missed
>> your point.
>>
> 
> 
> 
> 
> 
>> Thanks,
>> Steve
>>
>> [1]
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/intel/ixg
>> be/ixgbe_ethtool.c?h=v5.16-rc6
>>
>>>>    #define IXGBE_SFF_SFF_8472_COMP              0x5E
>>>>    #define IXGBE_SFF_SFF_8472_OSCB              0x6E
>>>>    #define IXGBE_SFF_SFF_8472_ESCB              0x76
>>>> @@ -48,6 +49,8 @@
>>>>    #define IXGBE_SFF_SOFT_RS_SELECT_10G 0x8
>>>> --
>>>> 2.31.1
>>>
> 

[PATCH v2 4/7] net/ixgbe: Run 82599 link status workaround only on affected devices

[Stephen Douthit]

1ca05831b9b added a check that SDP3 (used as a TX_DISABLE output to the
SFP cage on these cards) is not asserted to avoid incorrectly reporting
link up when the SFP's laser is turned off.

ff8162cb957 limited this workaround to fiber ports

Refactor this so it's:

* Not open coded in ixgbe_dev_link_update_share()
* Runs only on fiber 82599 devices, not all fiber ixgbe devs (which don't
  all use SDP3 as TX_DISABLE)

Fixes: 1ca05831b9b ("net/ixgbe: fix link status")
Fixes: ff8162cb957 ("net/ixgbe: fix link status")
Cc: stable@dpdk.org

Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
---
 drivers/net/ixgbe/base/ixgbe_82599.c | 41 ++++++++++++++++++++++++++++
 drivers/net/ixgbe/ixgbe_ethdev.c     |  7 -----
 2 files changed, 41 insertions(+), 7 deletions(-)

diff --git a/drivers/net/ixgbe/base/ixgbe_82599.c b/drivers/net/ixgbe/base/ixgbe_82599.c
index 69fd4cd3fb..5786114b0a 100644
--- a/drivers/net/ixgbe/base/ixgbe_82599.c
+++ b/drivers/net/ixgbe/base/ixgbe_82599.c
@@ -28,6 +28,39 @@ STATIC s32 ixgbe_read_i2c_byte_82599(struct ixgbe_hw *hw, u8 byte_offset,
 STATIC s32 ixgbe_write_i2c_byte_82599(struct ixgbe_hw *hw, u8 byte_offset,
 					u8 dev_addr, u8 data);
 
+/**
+ * ixgbe_check_mac_link_82599_fiber - Determine link and speed status
+ *
+ * @hw: pointer to hardware structure
+ * @speed: pointer to link speed
+ * @link_up: true when link is up
+ * @link_up_wait_to_complete: bool used to wait for link up or not
+ *
+ * Call the generic MAC check_link function, but also take into account the
+ * state of SDP3, which is a GPIO configured as an output driving the TX_DISABLE
+ * pin on the SFP cage.  This prevents reporting a false positive link up in the
+ * case where the link partner is transmitting, but we are not.
+ **/
+STATIC s32 ixgbe_check_mac_link_82599_fiber(struct ixgbe_hw *hw,
+					    ixgbe_link_speed *speed,
+					    bool *link_up,
+					    bool link_up_wait_to_complete)
+{
+	u32 esdp_reg;
+	s32 err;
+
+	DEBUGFUNC("ixgbe_check_mac_link_82599_fiber");
+
+	err = ixgbe_check_mac_link_generic(hw, speed, link_up,
+					   link_up_wait_to_complete);
+
+	esdp_reg = IXGBE_READ_REG(hw, IXGBE_ESDP);
+	if ((esdp_reg & IXGBE_ESDP_SDP3))
+		*link_up = 0;
+
+	return err;
+}
+
 void ixgbe_init_mac_link_ops_82599(struct ixgbe_hw *hw)
 {
 	struct ixgbe_mac_info *mac = &hw->mac;
@@ -52,6 +85,14 @@ void ixgbe_init_mac_link_ops_82599(struct ixgbe_hw *hw)
 		mac->ops.flap_tx_laser = NULL;
 	}
 
+	/*
+	 * For 82599 SFP+ fiber, make sure that SDP3 (TX_DISABLE to SFP cage)
+	 * isn't asserted.  Either by mac->ops.disable_tx_laser(), or possibly
+	 * management firmware
+	 */
+	if (mac->ops.get_media_type(hw) == ixgbe_media_type_fiber)
+		mac->ops.check_link = ixgbe_check_mac_link_82599_fiber;
+
 	if (hw->phy.multispeed_fiber) {
 		/* Set up dual speed SFP+ support */
 		mac->ops.setup_link = ixgbe_setup_mac_link_multispeed_fiber;
diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 66f7af95de..34b7cb2d4e 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -4216,7 +4216,6 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 	bool link_up;
 	int diag;
 	int wait = 1;
-	u32 esdp_reg;
 
 	memset(&link, 0, sizeof(link));
 	link.link_status = RTE_ETH_LINK_DOWN;
@@ -4250,12 +4249,6 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 		return rte_eth_linkstatus_set(dev, &link);
 	}
 
-	if (ixgbe_get_media_type(hw) == ixgbe_media_type_fiber) {
-		esdp_reg = IXGBE_READ_REG(hw, IXGBE_ESDP);
-		if ((esdp_reg & IXGBE_ESDP_SDP3))
-			link_up = 0;
-	}
-
 	if (link_up == 0) {
 		if (ixgbe_get_media_type(hw) == ixgbe_media_type_fiber) {
 			ixgbe_dev_wait_setup_link_complete(dev, 0);
-- 
2.31.1

[PATCH v2 5/7] net/ixgbe: Fix SFP detection and linking on hotplug

[Stephen Douthit]

Currently the ixgbe driver does not ID any SFP except for the first one
plugged in. This can lead to no-link, or incorrect speed conditions.

For example:

* If link is initially established with a 1G SFP, and later a 1G/10G
multispeed part is later installed, then the MAC link setup functions are
never called to change from 1000BASE-X to 10GBASE-R mode, and the link
stays running at the slower rate.

* If link is initially established with a 1G SFP, and later a 10G only
module is later installed, no link is established, since we are still
trasnsmitting in 1000BASE-X mode to a 10GBASE-R only partner.

Refactor the SFP ID/setup, and link setup code, to more closely match the
flow of the mainline kernel driver which does not have these issues.  In
that driver a service task runs periodically to handle these operations
based on bit flags that have been set (usually via interrupt or userspace
request), and then get cleared once the requested subtask has been
completed.

Fixes: af75078fece ("first public release")
Cc: stable@dpdk.org

Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
---
 drivers/net/ixgbe/base/ixgbe_type.h |   2 +
 drivers/net/ixgbe/ixgbe_ethdev.c    | 349 +++++++++++++++++-----------
 drivers/net/ixgbe/ixgbe_ethdev.h    |  18 +-
 3 files changed, 224 insertions(+), 145 deletions(-)

diff --git a/drivers/net/ixgbe/base/ixgbe_type.h b/drivers/net/ixgbe/base/ixgbe_type.h
index b7eec45635..c23257aa4c 100644
--- a/drivers/net/ixgbe/base/ixgbe_type.h
+++ b/drivers/net/ixgbe/base/ixgbe_type.h
@@ -45,6 +45,8 @@
 
 #include "ixgbe_osdep.h"
 
+#define BIT(a) (1UL << (a))
+
 /* Override this by setting IOMEM in your ixgbe_osdep.h header */
 
 /* Vendor ID */
diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 34b7cb2d4e..94d5a8f419 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -229,9 +229,6 @@ static int ixgbe_dev_interrupt_get_status(struct rte_eth_dev *dev);
 static int ixgbe_dev_interrupt_action(struct rte_eth_dev *dev);
 static void ixgbe_dev_interrupt_handler(void *param);
 static void ixgbe_dev_interrupt_delayed_handler(void *param);
-static void *ixgbe_dev_setup_link_thread_handler(void *param);
-static int ixgbe_dev_wait_setup_link_complete(struct rte_eth_dev *dev,
-					      uint32_t timeout_ms);
 
 static int ixgbe_add_rar(struct rte_eth_dev *dev,
 			struct rte_ether_addr *mac_addr,
@@ -1032,6 +1029,163 @@ ixgbe_swfw_lock_reset(struct ixgbe_hw *hw)
 	ixgbe_release_swfw_semaphore(hw, mask);
 }
 
+static s32
+ixgbe_sfp_id_and_setup(struct rte_eth_dev *dev)
+{
+	struct ixgbe_hw *hw =
+		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	enum ixgbe_sfp_cage_status sfp_cage_status;
+	s32 err;
+
+	/* Can't ID or setup SFP if it's not plugged in */
+	sfp_cage_status = ixgbe_check_sfp_cage(hw);
+	if (sfp_cage_status == IXGBE_SFP_CAGE_EMPTY ||
+	    sfp_cage_status == IXGBE_SFP_CAGE_NOCAGE)
+		return IXGBE_ERR_SFP_NOT_PRESENT;
+
+	/* Something's in the cage, ID it */
+	hw->phy.ops.identify_sfp(hw);
+
+	/* Unknown module type, give up */
+	if (hw->phy.sfp_type == ixgbe_sfp_type_unknown) {
+		PMD_DRV_LOG(ERR, "unknown SFP type, giving up");
+		return IXGBE_ERR_SFP_NOT_SUPPORTED;
+	}
+
+	/* This should be a redundant check, since we looked at the
+	 * PRSNT# signal from the cage above, but just in case this is
+	 * an SFP that's slow to respond to I2C pokes correctly, try it
+	 * again later
+	 */
+	if (hw->phy.sfp_type == ixgbe_sfp_type_not_present) {
+		PMD_DRV_LOG(ERR, "IDed SFP as absent but cage PRSNT# active!?");
+		return IXGBE_ERR_SFP_NOT_PRESENT;
+	}
+
+	/* SFP is present and identified, try to set it up */
+	err = hw->mac.ops.setup_sfp(hw);
+	if (err)
+		PMD_DRV_LOG(ERR, "setup_sfp() failed %d", err);
+
+	return err;
+}
+
+static void
+ixgbe_sfp_service(struct rte_eth_dev *dev)
+{
+	struct ixgbe_hw *hw =
+		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	struct ixgbe_interrupt *intr =
+		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
+	enum ixgbe_sfp_cage_status sfp_cage_status;
+	s32 err;
+
+	/* No setup requested?  Nothing to do */
+	if (!(intr->flags & IXGBE_FLAG_NEED_SFP_SETUP))
+		return;
+
+	sfp_cage_status = ixgbe_check_sfp_cage(hw);
+	if (sfp_cage_status == IXGBE_SFP_CAGE_EMPTY ||
+	    sfp_cage_status == IXGBE_SFP_CAGE_NOCAGE)
+		return;
+
+	err = ixgbe_sfp_id_and_setup(dev);
+	if (err) {
+		PMD_DRV_LOG(DEBUG, "failed to ID & setup SFP %d", err);
+		return;
+	}
+
+	/* Setup is done, clear the flag, but make sure link config runs for new SFP */
+	intr->flags &= ~IXGBE_FLAG_NEED_SFP_SETUP;
+	intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
+
+	/*
+	 * Since this is a new SFP, clear the old advertised speed mask so we don't
+	 * end up using an old slower rate
+	 */
+	hw->phy.autoneg_advertised = 0;
+}
+
+static void
+ixgbe_link_service(struct rte_eth_dev *dev)
+{
+	struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	struct ixgbe_interrupt *intr =
+		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
+	bool link_up, autoneg = false;
+	u32 speed;
+	s32 err;
+
+	/* Skip if we still need to setup an SFP, or if no link config requested
+	 */
+	if ((intr->flags & IXGBE_FLAG_NEED_SFP_SETUP) ||
+	    !(intr->flags & IXGBE_FLAG_NEED_LINK_CONFIG))
+		return;
+
+	speed = hw->phy.autoneg_advertised;
+	if (!speed)
+		ixgbe_get_link_capabilities(hw, &speed, &autoneg);
+
+	err = ixgbe_setup_link(hw, speed, true);
+	if (err) {
+		PMD_DRV_LOG(ERR, "ixgbe_setup_link failed %d", err);
+		return;
+	}
+
+	/* Update internal link status, waiting for link */
+	err = ixgbe_check_link(hw, &speed, &link_up, 1);
+	if (!err && link_up)
+		intr->flags &= ~IXGBE_FLAG_NEED_LINK_CONFIG;
+
+	if (!ixgbe_dev_link_update(dev, 0)) {
+		ixgbe_dev_link_status_print(dev);
+		rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_LSC, NULL);
+	}
+}
+
+/*
+ * Service task thread to handle periodic tasks
+ */
+static void *
+ixgbe_dev_service_thread_handler(void *param)
+{
+	struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
+	struct ixgbe_hw *hw =
+		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	uint64_t start, ticks, service_ms;
+	uint32_t speed;
+	s32 err;
+	bool link_up;
+
+	while (1) {
+		ixgbe_sfp_service(dev);
+		ixgbe_link_service(dev);
+
+		/* Run the service thread handler more frequently when link is
+		 * down to reduce link up latency (every 200ms vs 1s)
+		 *
+		 * Use a number of smaller sleeps to decrease exit latency when
+		 * ixgbe_dev_stop() wants this thread to join
+		 */
+		err = ixgbe_check_link(hw, &speed, &link_up, 0);
+		if (err == IXGBE_SUCCESS && link_up)
+			service_ms = 2000;
+		else
+			service_ms = 100;
+
+		/* Call nanosleep in a loop with several smaller sleeps to
+		 * provide periodic thread cancellation points
+		 */
+		start = rte_get_timer_cycles();
+		ticks = (uint64_t)service_ms * rte_get_timer_hz() / 1E3;
+		while ((rte_get_timer_cycles() - start) < ticks)
+			msec_delay(100);
+	}
+
+	/* Never return */
+	return NULL;
+}
+
 /*
  * This function is based on code in ixgbe_attach() in base/ixgbe.c.
  * It returns 0 on success.
@@ -1039,7 +1193,6 @@ ixgbe_swfw_lock_reset(struct ixgbe_hw *hw)
 static int
 eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)
 {
-	struct ixgbe_adapter *ad = eth_dev->data->dev_private;
 	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
 	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
 	struct ixgbe_hw *hw =
@@ -1094,7 +1247,6 @@ eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)
 		return 0;
 	}
 
-	rte_atomic32_clear(&ad->link_thread_running);
 	rte_eth_copy_pci_info(eth_dev, pci_dev);
 	eth_dev->data->dev_flags |= RTE_ETH_DEV_AUTOFILL_QUEUE_XSTATS;
 
@@ -1537,7 +1689,6 @@ eth_ixgbevf_dev_init(struct rte_eth_dev *eth_dev)
 {
 	int diag;
 	uint32_t tc, tcs;
-	struct ixgbe_adapter *ad = eth_dev->data->dev_private;
 	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
 	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
 	struct ixgbe_hw *hw =
@@ -1580,7 +1731,6 @@ eth_ixgbevf_dev_init(struct rte_eth_dev *eth_dev)
 		return 0;
 	}
 
-	rte_atomic32_clear(&ad->link_thread_running);
 	ixgbevf_parse_devargs(eth_dev->data->dev_private,
 			      pci_dev->device.devargs);
 
@@ -2382,6 +2532,8 @@ ixgbe_dev_configure(struct rte_eth_dev *dev)
 	struct ixgbe_interrupt *intr =
 		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
 	struct ixgbe_adapter *adapter = dev->data->dev_private;
+	struct ixgbe_hw *hw =
+		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
 	int ret;
 
 	PMD_INIT_FUNC_TRACE();
@@ -2400,6 +2552,10 @@ ixgbe_dev_configure(struct rte_eth_dev *dev)
 	/* set flag to update link status after init */
 	intr->flags |= IXGBE_FLAG_NEED_LINK_UPDATE;
 
+	/* set flag to setup SFP after init */
+	if (ixgbe_is_sfp(hw))
+		intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
+
 	/*
 	 * Initialize to TRUE. If any of Rx queues doesn't meet the bulk
 	 * allocation or vector Rx preconditions we will reset it.
@@ -2419,13 +2575,24 @@ ixgbe_dev_phy_intr_setup(struct rte_eth_dev *dev)
 		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
 	uint32_t gpie;
 
-	/* only set up it on X550EM_X */
+	/* only set up it on X550EM_X (external PHY interrupt)
+	 * or on x550em_a_* for SFP_PRSNT# de-assertion (SFP removal event)
+	 */
 	if (hw->mac.type == ixgbe_mac_X550EM_x) {
 		gpie = IXGBE_READ_REG(hw, IXGBE_GPIE);
 		gpie |= IXGBE_SDP0_GPIEN_X550EM_x;
 		IXGBE_WRITE_REG(hw, IXGBE_GPIE, gpie);
 		if (hw->phy.type == ixgbe_phy_x550em_ext_t)
 			intr->mask |= IXGBE_EICR_GPI_SDP0_X550EM_x;
+	} else if (hw->mac.type == ixgbe_mac_X550EM_a) {
+		gpie = IXGBE_READ_REG(hw, IXGBE_GPIE);
+		gpie |= IXGBE_SDP0_GPIEN_X550EM_a;
+		IXGBE_WRITE_REG(hw, IXGBE_GPIE, gpie);
+		intr->mask |= IXGBE_EICR_GPI_SDP0_X550EM_a;
+	} else {
+		PMD_DRV_LOG(DEBUG,
+			    "No PHY/SFP interrupt for MAC %d, PHY %d\n",
+			    hw->mac.type, hw->phy.type);
 	}
 }
 
@@ -2548,8 +2715,11 @@ ixgbe_flow_ctrl_enable(struct rte_eth_dev *dev, struct ixgbe_hw *hw)
 static int
 ixgbe_dev_start(struct rte_eth_dev *dev)
 {
+	struct ixgbe_adapter *ad = dev->data->dev_private;
 	struct ixgbe_hw *hw =
 		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	struct ixgbe_interrupt *intr =
+		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
 	struct ixgbe_vf_info *vfinfo =
 		*IXGBE_DEV_PRIVATE_TO_P_VFDATA(dev->data->dev_private);
 	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
@@ -2570,9 +2740,6 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
 
 	PMD_INIT_FUNC_TRACE();
 
-	/* Stop the link setup handler before resetting the HW. */
-	ixgbe_dev_wait_setup_link_complete(dev, 0);
-
 	/* disable uio/vfio intr/eventfd mapping */
 	rte_intr_disable(intr_handle);
 
@@ -2688,12 +2855,6 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
 		}
 	}
 
-	if (ixgbe_is_sfp(hw) && hw->phy.multispeed_fiber) {
-		err = hw->mac.ops.setup_sfp(hw);
-		if (err)
-			goto error;
-	}
-
 	if (hw->mac.ops.get_media_type(hw) == ixgbe_media_type_copper) {
 		/* Turn on the copper */
 		ixgbe_set_phy_power(hw, true);
@@ -2805,6 +2966,20 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
 	ixgbe_l2_tunnel_conf(dev);
 	ixgbe_filter_restore(dev);
 
+	/* Spawn service thread */
+	if (ixgbe_is_sfp(hw)) {
+		intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
+		err = rte_ctrl_thread_create(&ad->service_thread_tid,
+					     "ixgbe-service-thread",
+					     NULL,
+					     ixgbe_dev_service_thread_handler,
+					     dev);
+		if (err) {
+			PMD_DRV_LOG(ERR, "service_thread err");
+			goto error;
+		}
+	}
+
 	if (tm_conf->root && !tm_conf->committed)
 		PMD_DRV_LOG(WARNING,
 			    "please call hierarchy_commit() "
@@ -2815,12 +2990,6 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
 	if (err)
 		goto error;
 
-	/*
-	 * Update link status right before return, because it may
-	 * start link configuration process in a separate thread.
-	 */
-	ixgbe_dev_link_update(dev, 0);
-
 	/* setup the macsec setting register */
 	if (macsec_setting->offload_en)
 		ixgbe_dev_macsec_register_enable(dev, macsec_setting);
@@ -2850,13 +3019,21 @@ ixgbe_dev_stop(struct rte_eth_dev *dev)
 	int vf;
 	struct ixgbe_tm_conf *tm_conf =
 		IXGBE_DEV_PRIVATE_TO_TM_CONF(dev->data->dev_private);
+	void *res;
+	s32 err;
 
 	if (hw->adapter_stopped)
 		return 0;
 
 	PMD_INIT_FUNC_TRACE();
 
-	ixgbe_dev_wait_setup_link_complete(dev, 0);
+	/* Cancel the service thread, and wait for it to join */
+	err = pthread_cancel(adapter->service_thread_tid);
+	if (err)
+		PMD_DRV_LOG(ERR, "failed to cancel service thread %d", err);
+	err = pthread_join(adapter->service_thread_tid, &res);
+	if (err)
+		PMD_DRV_LOG(ERR, "failed to join service thread %d", err);
 
 	/* disable interrupts */
 	ixgbe_disable_intr(hw);
@@ -2935,7 +3112,6 @@ ixgbe_dev_set_link_up(struct rte_eth_dev *dev)
 	} else {
 		/* Turn on the laser */
 		ixgbe_enable_tx_laser(hw);
-		ixgbe_dev_link_update(dev, 0);
 	}
 
 	return 0;
@@ -2966,7 +3142,6 @@ ixgbe_dev_set_link_down(struct rte_eth_dev *dev)
 	} else {
 		/* Turn off the laser */
 		ixgbe_disable_tx_laser(hw);
-		ixgbe_dev_link_update(dev, 0);
 	}
 
 	return 0;
@@ -4118,57 +4293,6 @@ ixgbevf_check_link(struct ixgbe_hw *hw, ixgbe_link_speed *speed,
 	return ret_val;
 }
 
-/*
- * If @timeout_ms was 0, it means that it will not return until link complete.
- * It returns 1 on complete, return 0 on timeout.
- */
-static int
-ixgbe_dev_wait_setup_link_complete(struct rte_eth_dev *dev, uint32_t timeout_ms)
-{
-#define WARNING_TIMEOUT    9000 /* 9s  in total */
-	struct ixgbe_adapter *ad = dev->data->dev_private;
-	uint32_t timeout = timeout_ms ? timeout_ms : WARNING_TIMEOUT;
-
-	while (rte_atomic32_read(&ad->link_thread_running)) {
-		msec_delay(1);
-		timeout--;
-
-		if (timeout_ms) {
-			if (!timeout)
-				return 0;
-		} else if (!timeout) {
-			/* It will not return until link complete */
-			timeout = WARNING_TIMEOUT;
-			PMD_DRV_LOG(ERR, "IXGBE link thread not complete too long time!");
-		}
-	}
-
-	return 1;
-}
-
-static void *
-ixgbe_dev_setup_link_thread_handler(void *param)
-{
-	struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
-	struct ixgbe_adapter *ad = dev->data->dev_private;
-	struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
-	struct ixgbe_interrupt *intr =
-		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
-	u32 speed;
-	bool autoneg = false;
-
-	pthread_detach(pthread_self());
-	speed = hw->phy.autoneg_advertised;
-	if (!speed)
-		ixgbe_get_link_capabilities(hw, &speed, &autoneg);
-
-	ixgbe_setup_link(hw, speed, true);
-
-	intr->flags &= ~IXGBE_FLAG_NEED_LINK_CONFIG;
-	rte_atomic32_clear(&ad->link_thread_running);
-	return NULL;
-}
-
 /*
  * In freebsd environment, nic_uio drivers do not support interrupts,
  * rte_intr_callback_register() will fail to register interrupts.
@@ -4208,11 +4332,8 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 			    int wait_to_complete, int vf)
 {
 	struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
-	struct ixgbe_adapter *ad = dev->data->dev_private;
 	struct rte_eth_link link;
 	ixgbe_link_speed link_speed = IXGBE_LINK_SPEED_UNKNOWN;
-	struct ixgbe_interrupt *intr =
-		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
 	bool link_up;
 	int diag;
 	int wait = 1;
@@ -4226,9 +4347,6 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 
 	hw->mac.get_link_status = true;
 
-	if (intr->flags & IXGBE_FLAG_NEED_LINK_CONFIG)
-		return rte_eth_linkstatus_set(dev, &link);
-
 	/* check if it needs to wait to complete, if lsc interrupt is enabled */
 	if (wait_to_complete == 0 || dev->data->dev_conf.intr_conf.lsc != 0)
 		wait = 0;
@@ -4243,38 +4361,12 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 	else
 		diag = ixgbe_check_link(hw, &link_speed, &link_up, wait);
 
-	if (diag != 0) {
+	if (diag != 0 || !link_up) {
 		link.link_speed = RTE_ETH_SPEED_NUM_100M;
 		link.link_duplex = RTE_ETH_LINK_FULL_DUPLEX;
 		return rte_eth_linkstatus_set(dev, &link);
 	}
 
-	if (link_up == 0) {
-		if (ixgbe_get_media_type(hw) == ixgbe_media_type_fiber) {
-			ixgbe_dev_wait_setup_link_complete(dev, 0);
-			if (rte_atomic32_test_and_set(&ad->link_thread_running)) {
-				/* To avoid race condition between threads, set
-				 * the IXGBE_FLAG_NEED_LINK_CONFIG flag only
-				 * when there is no link thread running.
-				 */
-				intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
-				if (rte_ctrl_thread_create(&ad->link_thread_tid,
-					"ixgbe-link-handler",
-					NULL,
-					ixgbe_dev_setup_link_thread_handler,
-					dev) < 0) {
-					PMD_DRV_LOG(ERR,
-						"Create link thread failed!");
-					rte_atomic32_clear(&ad->link_thread_running);
-				}
-			} else {
-				PMD_DRV_LOG(ERR,
-					"Other link thread is running now!");
-			}
-		}
-		return rte_eth_linkstatus_set(dev, &link);
-	}
-
 	link.link_status = RTE_ETH_LINK_UP;
 	link.link_duplex = RTE_ETH_LINK_FULL_DUPLEX;
 
@@ -4480,8 +4572,6 @@ ixgbe_dev_interrupt_get_status(struct rte_eth_dev *dev)
 	eicr = IXGBE_READ_REG(hw, IXGBE_EICR);
 	PMD_DRV_LOG(DEBUG, "eicr %x", eicr);
 
-	intr->flags = 0;
-
 	/* set flag for async link update */
 	if (eicr & IXGBE_EICR_LSC)
 		intr->flags |= IXGBE_FLAG_NEED_LINK_UPDATE;
@@ -4497,6 +4587,11 @@ ixgbe_dev_interrupt_get_status(struct rte_eth_dev *dev)
 	    (eicr & IXGBE_EICR_GPI_SDP0_X550EM_x))
 		intr->flags |= IXGBE_FLAG_PHY_INTERRUPT;
 
+	/* Check for loss of SFP */
+	if (hw->mac.type ==  ixgbe_mac_X550EM_a &&
+	    (eicr & IXGBE_EICR_GPI_SDP0_X550EM_a))
+		intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
+
 	return 0;
 }
 
@@ -4548,11 +4643,13 @@ ixgbe_dev_link_status_print(struct rte_eth_dev *dev)
 static int
 ixgbe_dev_interrupt_action(struct rte_eth_dev *dev)
 {
+	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
 	struct ixgbe_interrupt *intr =
 		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
-	int64_t timeout;
 	struct ixgbe_hw *hw =
 		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	int64_t timeout;
 
 	PMD_DRV_LOG(DEBUG, "intr action type %d", intr->flags);
 
@@ -4587,16 +4684,14 @@ ixgbe_dev_interrupt_action(struct rte_eth_dev *dev)
 		if (rte_eal_alarm_set(timeout * 1000,
 				      ixgbe_dev_interrupt_delayed_handler, (void *)dev) < 0)
 			PMD_DRV_LOG(ERR, "Error setting alarm");
-		else {
-			/* remember original mask */
-			intr->mask_original = intr->mask;
+		else
 			/* only disable lsc interrupt */
 			intr->mask &= ~IXGBE_EIMS_LSC;
-		}
 	}
 
 	PMD_DRV_LOG(DEBUG, "enable intr immediately");
 	ixgbe_enable_intr(dev);
+	rte_intr_ack(intr_handle);
 
 	return 0;
 }
@@ -4619,8 +4714,6 @@ static void
 ixgbe_dev_interrupt_delayed_handler(void *param)
 {
 	struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
-	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
-	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
 	struct ixgbe_interrupt *intr =
 		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
 	struct ixgbe_hw *hw =
@@ -4650,13 +4743,10 @@ ixgbe_dev_interrupt_delayed_handler(void *param)
 		intr->flags &= ~IXGBE_FLAG_MACSEC;
 	}
 
-	/* restore original mask */
-	intr->mask = intr->mask_original;
-	intr->mask_original = 0;
+	if (dev->data->dev_conf.intr_conf.lsc != 0)
+		intr->mask |= IXGBE_EICR_LSC;
 
-	PMD_DRV_LOG(DEBUG, "enable intr in delayed handler S[%08x]", eicr);
 	ixgbe_enable_intr(dev);
-	rte_intr_ack(intr_handle);
 }
 
 /**
@@ -5298,9 +5388,6 @@ ixgbevf_dev_start(struct rte_eth_dev *dev)
 
 	PMD_INIT_FUNC_TRACE();
 
-	/* Stop the link setup handler before resetting the HW. */
-	ixgbe_dev_wait_setup_link_complete(dev, 0);
-
 	err = hw->mac.ops.reset_hw(hw);
 
 	/**
@@ -5380,12 +5467,6 @@ ixgbevf_dev_start(struct rte_eth_dev *dev)
 	/* Re-enable interrupt for VF */
 	ixgbevf_intr_enable(dev);
 
-	/*
-	 * Update link status right before return, because it may
-	 * start link configuration process in a separate thread.
-	 */
-	ixgbevf_dev_link_update(dev, 0);
-
 	hw->adapter_stopped = false;
 
 	return 0;
@@ -5404,8 +5485,6 @@ ixgbevf_dev_stop(struct rte_eth_dev *dev)
 
 	PMD_INIT_FUNC_TRACE();
 
-	ixgbe_dev_wait_setup_link_complete(dev, 0);
-
 	ixgbevf_intr_disable(dev);
 
 	dev->data->dev_started = 0;
diff --git a/drivers/net/ixgbe/ixgbe_ethdev.h b/drivers/net/ixgbe/ixgbe_ethdev.h
index 83e8b5e56a..14658a2a82 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.h
+++ b/drivers/net/ixgbe/ixgbe_ethdev.h
@@ -24,11 +24,12 @@
 #include <rte_tm_driver.h>
 
 /* need update link, bit flag */
-#define IXGBE_FLAG_NEED_LINK_UPDATE (uint32_t)(1 << 0)
-#define IXGBE_FLAG_MAILBOX          (uint32_t)(1 << 1)
-#define IXGBE_FLAG_PHY_INTERRUPT    (uint32_t)(1 << 2)
-#define IXGBE_FLAG_MACSEC           (uint32_t)(1 << 3)
-#define IXGBE_FLAG_NEED_LINK_CONFIG (uint32_t)(1 << 4)
+#define IXGBE_FLAG_NEED_LINK_UPDATE BIT(0)
+#define IXGBE_FLAG_MAILBOX          BIT(1)
+#define IXGBE_FLAG_PHY_INTERRUPT    BIT(2)
+#define IXGBE_FLAG_MACSEC           BIT(3)
+#define IXGBE_FLAG_NEED_LINK_CONFIG BIT(4)
+#define IXGBE_FLAG_NEED_SFP_SETUP   BIT(5)
 
 /*
  * Defines that were not part of ixgbe_type.h as they are not used by the
@@ -41,7 +42,7 @@
 #define IXGBE_RXDADV_ERR_CKSUM_MSK  3
 #define IXGBE_ADVTXD_MACLEN_SHIFT   9          /* Bit shift for l2_len */
 #define IXGBE_NB_STAT_MAPPING_REGS  32
-#define IXGBE_EXTENDED_VLAN	  (uint32_t)(1 << 26) /* EXTENDED VLAN ENABLE */
+#define IXGBE_EXTENDED_VLAN	    BIT(26)    /* EXTENDED VLAN ENABLE */
 #define IXGBE_VFTA_SIZE 128
 #define IXGBE_HKEY_MAX_INDEX 10
 #define IXGBE_MAX_RX_QUEUE_NUM	128
@@ -223,8 +224,6 @@ struct ixgbe_rte_flow_rss_conf {
 struct ixgbe_interrupt {
 	uint32_t flags;
 	uint32_t mask;
-	/*to save original mask during delayed handler */
-	uint32_t mask_original;
 };
 
 struct ixgbe_stat_mapping_registers {
@@ -506,8 +505,7 @@ struct ixgbe_adapter {
 	 */
 	uint8_t pflink_fullchk;
 	uint8_t mac_ctrl_frame_fwd;
-	rte_atomic32_t link_thread_running;
-	pthread_t link_thread_tid;
+	pthread_t service_thread_tid;
 };
 
 struct ixgbe_vf_representor {
-- 
2.31.1

Re: [PATCH v2 5/7] net/ixgbe: Fix SFP detection and linking on hotplug

[Ferruh Yigit]

On 12/6/2021 10:19 PM, Stephen Douthit wrote:
> Currently the ixgbe driver does not ID any SFP except for the first one
> plugged in. This can lead to no-link, or incorrect speed conditions.
> 
> For example:
> 
> * If link is initially established with a 1G SFP, and later a 1G/10G
> multispeed part is later installed, then the MAC link setup functions are
> never called to change from 1000BASE-X to 10GBASE-R mode, and the link
> stays running at the slower rate.
> 
> * If link is initially established with a 1G SFP, and later a 10G only
> module is later installed, no link is established, since we are still
> trasnsmitting in 1000BASE-X mode to a 10GBASE-R only partner.
> 
> Refactor the SFP ID/setup, and link setup code, to more closely match the
> flow of the mainline kernel driver which does not have these issues.  In
> that driver a service task runs periodically to handle these operations
> based on bit flags that have been set (usually via interrupt or userspace
> request), and then get cleared once the requested subtask has been
> completed.
> 
> Fixes: af75078fece ("first public release")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
> ---
>   drivers/net/ixgbe/base/ixgbe_type.h |   2 +
>   drivers/net/ixgbe/ixgbe_ethdev.c    | 349 +++++++++++++++++-----------
>   drivers/net/ixgbe/ixgbe_ethdev.h    |  18 +-
>   3 files changed, 224 insertions(+), 145 deletions(-)
> 
> diff --git a/drivers/net/ixgbe/base/ixgbe_type.h b/drivers/net/ixgbe/base/ixgbe_type.h
> index b7eec45635..c23257aa4c 100644
> --- a/drivers/net/ixgbe/base/ixgbe_type.h
> +++ b/drivers/net/ixgbe/base/ixgbe_type.h
> @@ -45,6 +45,8 @@
>   
>   #include "ixgbe_osdep.h"
>   
> +#define BIT(a) (1UL << (a))
> +

There is already RTE_BIT64/RTE_BIT32 macros in DPDK, they can be used in PMD
instead of defining its own.

[PATCH v3 1/3] net/ixgbe: Fix ixgbe_is_sfp() to return valid result for X550EM_a devs

[Jeff Daly]

From: Stephen Douthit <stephend@silicom-usa.com>

Currently all X500EM* MAC types fallthrough to the default case and get
reported as non-SFP regardless of media type, which isn't correct.

Fixes: 0790adeb567 ("ixgbe/base: support X550em_a device")
Cc: stable@dpdk.org

Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 3d799d2187..68b28b1ce6 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -781,6 +781,20 @@ ixgbe_is_sfp(struct ixgbe_hw *hw)
 	case ixgbe_phy_sfp_passive_unknown:
 		return 1;
 	default:
+		/* x550em devices may be SFP, check media type */
+		switch (hw->mac.type) {
+		case ixgbe_mac_X550EM_x:
+		case ixgbe_mac_X550EM_a:
+			switch (ixgbe_get_media_type(hw)) {
+			case ixgbe_media_type_fiber:
+			case ixgbe_media_type_fiber_qsfp:
+				return 1;
+			default:
+				break;
+			}
+		default:
+			break;
+		}
 		return 0;
 	}
 }
-- 
2.25.1

[PATCH v3 2/3] net/ixgbe: Limit SDP3 check of TX_DISABLE to appropriate devices

[Jeff Daly]

1ca05831b9b added a check that SDP3 (used as a TX_DISABLE output to the
SFP cage on these cards) is not asserted to avoid incorrectly reporting
link up when the SFP's laser is turned off.

ff8162cb957 limited this workaround to fiber ports

This patch:
* Adds additional check for 82599 type, not all fiber ixgbe devs use
  SDP3 as TX_DISABLE

Fixes: 1ca05831b9b ("net/ixgbe: fix link status")
Fixes: ff8162cb957 ("net/ixgbe: fix link status")
Cc: stable@dpdk.org

Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 68b28b1ce6..e8f07cb405 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -4260,7 +4260,8 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 		return rte_eth_linkstatus_set(dev, &link);
 	}
 
-	if (ixgbe_get_media_type(hw) == ixgbe_media_type_fiber) {
+	if (hw->mac.type == ixgbe_mac_82599EB &&
+		ixgbe_get_media_type(hw) == ixgbe_media_type_fiber) {
 		esdp_reg = IXGBE_READ_REG(hw, IXGBE_ESDP);
 		if ((esdp_reg & IXGBE_ESDP_SDP3))
 			link_up = 0;
-- 
2.25.1

[PATCH v3 3/3] net/ixgbe: Fix SFP detection and linking on hotplug

[Jeff Daly]

Currently the ixgbe driver does not ID any SFP except for the first one
plugged in. This can lead to no-link, or incorrect speed conditions.

For example:

* If link is initially established with a 1G SFP, and later a 1G/10G
multispeed part is later installed, then the MAC link setup functions are
never called to change from 1000BASE-X to 10GBASE-R mode, and the link
stays running at the slower rate.

* If link is initially established with a 1G SFP, and later a 10G only
module is later installed, no link is established, since we are still
trasnsmitting in 1000BASE-X mode to a 10GBASE-R only partner.

Refactor the SFP ID/setup, and link setup code, to more closely match the
flow of the mainline kernel driver which does not have these issues.  In
that driver a service task runs periodically to handle these operations
based on bit flags that have been set (usually via interrupt or userspace
request), and then get cleared once the requested subtask has been
completed.

Fixes: af75078fece ("first public release")
Cc: stable@dpdk.org

Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 687 +++++++++++++++++++++----------
 drivers/net/ixgbe/ixgbe_ethdev.h |  14 +-
 2 files changed, 486 insertions(+), 215 deletions(-)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index e8f07cb405..2150b5ed35 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -69,11 +69,11 @@
 
 #define IXGBE_LINK_DOWN_CHECK_TIMEOUT 4000 /* ms */
 #define IXGBE_LINK_UP_CHECK_TIMEOUT   1000 /* ms */
-#define IXGBE_VMDQ_NUM_UC_MAC         4096 /* Maximum nb. of UC MAC addr. */
+#define IXGBE_VMDQ_NUM_UC_MAC	      4096 /* Maximum nb. of UC MAC addr. */
 
-#define IXGBE_MMW_SIZE_DEFAULT        0x4
+#define IXGBE_MMW_SIZE_DEFAULT	      0x4
 #define IXGBE_MMW_SIZE_JUMBO_FRAME    0x14
-#define IXGBE_MAX_RING_DESC           4096 /* replicate define from rxtx */
+#define IXGBE_MAX_RING_DESC	      4096 /* replicate define from rxtx */
 
 /*
  *  Default values for RX/TX configuration
@@ -100,33 +100,33 @@
 #define IXGBE_QUEUE_STAT_COUNTERS (sizeof(hw_stats->qprc) / sizeof(hw_stats->qprc[0]))
 
 /* Additional timesync values. */
-#define NSEC_PER_SEC             1000000000L
-#define IXGBE_INCVAL_10GB        0x66666666
-#define IXGBE_INCVAL_1GB         0x40000000
-#define IXGBE_INCVAL_100         0x50000000
+#define NSEC_PER_SEC		 1000000000L
+#define IXGBE_INCVAL_10GB	 0x66666666
+#define IXGBE_INCVAL_1GB	 0x40000000
+#define IXGBE_INCVAL_100	 0x50000000
 #define IXGBE_INCVAL_SHIFT_10GB  28
-#define IXGBE_INCVAL_SHIFT_1GB   24
-#define IXGBE_INCVAL_SHIFT_100   21
+#define IXGBE_INCVAL_SHIFT_1GB	 24
+#define IXGBE_INCVAL_SHIFT_100	 21
 #define IXGBE_INCVAL_SHIFT_82599 7
 #define IXGBE_INCPER_SHIFT_82599 24
 
 #define IXGBE_CYCLECOUNTER_MASK   0xffffffffffffffffULL
 
-#define IXGBE_VT_CTL_POOLING_MODE_MASK         0x00030000
-#define IXGBE_VT_CTL_POOLING_MODE_ETAG         0x00010000
-#define IXGBE_ETAG_ETYPE                       0x00005084
-#define IXGBE_ETAG_ETYPE_MASK                  0x0000ffff
-#define IXGBE_ETAG_ETYPE_VALID                 0x80000000
-#define IXGBE_RAH_ADTYPE                       0x40000000
-#define IXGBE_RAL_ETAG_FILTER_MASK             0x00003fff
-#define IXGBE_VMVIR_TAGA_MASK                  0x18000000
-#define IXGBE_VMVIR_TAGA_ETAG_INSERT           0x08000000
+#define IXGBE_VT_CTL_POOLING_MODE_MASK	       0x00030000
+#define IXGBE_VT_CTL_POOLING_MODE_ETAG	       0x00010000
+#define IXGBE_ETAG_ETYPE		       0x00005084
+#define IXGBE_ETAG_ETYPE_MASK		       0x0000ffff
+#define IXGBE_ETAG_ETYPE_VALID		       0x80000000
+#define IXGBE_RAH_ADTYPE		       0x40000000
+#define IXGBE_RAL_ETAG_FILTER_MASK	       0x00003fff
+#define IXGBE_VMVIR_TAGA_MASK		       0x18000000
+#define IXGBE_VMVIR_TAGA_ETAG_INSERT	       0x08000000
 #define IXGBE_VMTIR(_i) (0x00017000 + ((_i) * 4)) /* 64 of these (0-63) */
-#define IXGBE_QDE_STRIP_TAG                    0x00000004
-#define IXGBE_VTEICR_MASK                      0x07
+#define IXGBE_QDE_STRIP_TAG		       0x00000004
+#define IXGBE_VTEICR_MASK		       0x07
 
-#define IXGBE_EXVET_VET_EXT_SHIFT              16
-#define IXGBE_DMATXCTL_VT_MASK                 0xFFFF0000
+#define IXGBE_EXVET_VET_EXT_SHIFT	       16
+#define IXGBE_DMATXCTL_VT_MASK		       0xFFFF0000
 
 #define IXGBEVF_DEVARG_PFLINK_FULLCHK		"pflink_fullchk"
 
@@ -229,9 +229,6 @@ static int ixgbe_dev_interrupt_get_status(struct rte_eth_dev *dev);
 static int ixgbe_dev_interrupt_action(struct rte_eth_dev *dev);
 static void ixgbe_dev_interrupt_handler(void *param);
 static void ixgbe_dev_interrupt_delayed_handler(void *param);
-static void *ixgbe_dev_setup_link_thread_handler(void *param);
-static int ixgbe_dev_wait_setup_link_complete(struct rte_eth_dev *dev,
-					      uint32_t timeout_ms);
 
 static int ixgbe_add_rar(struct rte_eth_dev *dev,
 			struct rte_ether_addr *mac_addr,
@@ -352,20 +349,20 @@ static int ixgbe_wait_for_link_up(struct ixgbe_hw *hw);
 /*
  * Define VF Stats MACRO for Non "cleared on read" register
  */
-#define UPDATE_VF_STAT(reg, last, cur)                          \
-{                                                               \
-	uint32_t latest = IXGBE_READ_REG(hw, reg);              \
-	cur += (latest - last) & UINT_MAX;                      \
-	last = latest;                                          \
-}
-
-#define UPDATE_VF_STAT_36BIT(lsb, msb, last, cur)                \
-{                                                                \
-	u64 new_lsb = IXGBE_READ_REG(hw, lsb);                   \
-	u64 new_msb = IXGBE_READ_REG(hw, msb);                   \
-	u64 latest = ((new_msb << 32) | new_lsb);                \
+#define UPDATE_VF_STAT(reg, last, cur)				\
+{								\
+	uint32_t latest = IXGBE_READ_REG(hw, reg);		\
+	cur += (latest - last) & UINT_MAX;			\
+	last = latest;						\
+}
+
+#define UPDATE_VF_STAT_36BIT(lsb, msb, last, cur)		 \
+{								 \
+	u64 new_lsb = IXGBE_READ_REG(hw, lsb);			 \
+	u64 new_msb = IXGBE_READ_REG(hw, msb);			 \
+	u64 latest = ((new_msb << 32) | new_lsb);		 \
 	cur += (0x1000000000LL + latest - last) & 0xFFFFFFFFFLL; \
-	last = latest;                                           \
+	last = latest;						 \
 }
 
 #define IXGBE_SET_HWSTRIP(h, q) do {\
@@ -476,80 +473,80 @@ static const struct rte_eth_desc_lim tx_desc_lim = {
 };
 
 static const struct eth_dev_ops ixgbe_eth_dev_ops = {
-	.dev_configure        = ixgbe_dev_configure,
-	.dev_start            = ixgbe_dev_start,
-	.dev_stop             = ixgbe_dev_stop,
+	.dev_configure	      = ixgbe_dev_configure,
+	.dev_start	      = ixgbe_dev_start,
+	.dev_stop	      = ixgbe_dev_stop,
 	.dev_set_link_up    = ixgbe_dev_set_link_up,
 	.dev_set_link_down  = ixgbe_dev_set_link_down,
-	.dev_close            = ixgbe_dev_close,
+	.dev_close	      = ixgbe_dev_close,
 	.dev_reset	      = ixgbe_dev_reset,
 	.promiscuous_enable   = ixgbe_dev_promiscuous_enable,
 	.promiscuous_disable  = ixgbe_dev_promiscuous_disable,
 	.allmulticast_enable  = ixgbe_dev_allmulticast_enable,
 	.allmulticast_disable = ixgbe_dev_allmulticast_disable,
-	.link_update          = ixgbe_dev_link_update,
-	.stats_get            = ixgbe_dev_stats_get,
-	.xstats_get           = ixgbe_dev_xstats_get,
+	.link_update	      = ixgbe_dev_link_update,
+	.stats_get	      = ixgbe_dev_stats_get,
+	.xstats_get	      = ixgbe_dev_xstats_get,
 	.xstats_get_by_id     = ixgbe_dev_xstats_get_by_id,
-	.stats_reset          = ixgbe_dev_stats_reset,
-	.xstats_reset         = ixgbe_dev_xstats_reset,
+	.stats_reset	      = ixgbe_dev_stats_reset,
+	.xstats_reset	      = ixgbe_dev_xstats_reset,
 	.xstats_get_names     = ixgbe_dev_xstats_get_names,
 	.xstats_get_names_by_id = ixgbe_dev_xstats_get_names_by_id,
 	.queue_stats_mapping_set = ixgbe_dev_queue_stats_mapping_set,
 	.fw_version_get       = ixgbe_fw_version_get,
-	.dev_infos_get        = ixgbe_dev_info_get,
+	.dev_infos_get	      = ixgbe_dev_info_get,
 	.dev_supported_ptypes_get = ixgbe_dev_supported_ptypes_get,
-	.mtu_set              = ixgbe_dev_mtu_set,
+	.mtu_set	      = ixgbe_dev_mtu_set,
 	.vlan_filter_set      = ixgbe_vlan_filter_set,
-	.vlan_tpid_set        = ixgbe_vlan_tpid_set,
+	.vlan_tpid_set	      = ixgbe_vlan_tpid_set,
 	.vlan_offload_set     = ixgbe_vlan_offload_set,
 	.vlan_strip_queue_set = ixgbe_vlan_strip_queue_set,
 	.rx_queue_start	      = ixgbe_dev_rx_queue_start,
-	.rx_queue_stop        = ixgbe_dev_rx_queue_stop,
+	.rx_queue_stop	      = ixgbe_dev_rx_queue_stop,
 	.tx_queue_start	      = ixgbe_dev_tx_queue_start,
-	.tx_queue_stop        = ixgbe_dev_tx_queue_stop,
+	.tx_queue_stop	      = ixgbe_dev_tx_queue_stop,
 	.rx_queue_setup       = ixgbe_dev_rx_queue_setup,
 	.rx_queue_intr_enable = ixgbe_dev_rx_queue_intr_enable,
 	.rx_queue_intr_disable = ixgbe_dev_rx_queue_intr_disable,
 	.rx_queue_release     = ixgbe_dev_rx_queue_release,
 	.tx_queue_setup       = ixgbe_dev_tx_queue_setup,
 	.tx_queue_release     = ixgbe_dev_tx_queue_release,
-	.dev_led_on           = ixgbe_dev_led_on,
-	.dev_led_off          = ixgbe_dev_led_off,
-	.flow_ctrl_get        = ixgbe_flow_ctrl_get,
-	.flow_ctrl_set        = ixgbe_flow_ctrl_set,
+	.dev_led_on	      = ixgbe_dev_led_on,
+	.dev_led_off	      = ixgbe_dev_led_off,
+	.flow_ctrl_get	      = ixgbe_flow_ctrl_get,
+	.flow_ctrl_set	      = ixgbe_flow_ctrl_set,
 	.priority_flow_ctrl_set = ixgbe_priority_flow_ctrl_set,
-	.mac_addr_add         = ixgbe_add_rar,
+	.mac_addr_add	      = ixgbe_add_rar,
 	.mac_addr_remove      = ixgbe_remove_rar,
-	.mac_addr_set         = ixgbe_set_default_mac_addr,
+	.mac_addr_set	      = ixgbe_set_default_mac_addr,
 	.uc_hash_table_set    = ixgbe_uc_hash_table_set,
-	.uc_all_hash_table_set  = ixgbe_uc_all_hash_table_set,
+	.uc_all_hash_table_set	= ixgbe_uc_all_hash_table_set,
 	.set_queue_rate_limit = ixgbe_set_queue_rate_limit,
-	.reta_update          = ixgbe_dev_rss_reta_update,
-	.reta_query           = ixgbe_dev_rss_reta_query,
+	.reta_update	      = ixgbe_dev_rss_reta_update,
+	.reta_query	      = ixgbe_dev_rss_reta_query,
 	.rss_hash_update      = ixgbe_dev_rss_hash_update,
 	.rss_hash_conf_get    = ixgbe_dev_rss_hash_conf_get,
-	.flow_ops_get         = ixgbe_dev_flow_ops_get,
+	.flow_ops_get	      = ixgbe_dev_flow_ops_get,
 	.set_mc_addr_list     = ixgbe_dev_set_mc_addr_list,
-	.rxq_info_get         = ixgbe_rxq_info_get,
-	.txq_info_get         = ixgbe_txq_info_get,
+	.rxq_info_get	      = ixgbe_rxq_info_get,
+	.txq_info_get	      = ixgbe_txq_info_get,
 	.timesync_enable      = ixgbe_timesync_enable,
 	.timesync_disable     = ixgbe_timesync_disable,
 	.timesync_read_rx_timestamp = ixgbe_timesync_read_rx_timestamp,
 	.timesync_read_tx_timestamp = ixgbe_timesync_read_tx_timestamp,
-	.get_reg              = ixgbe_get_regs,
+	.get_reg	      = ixgbe_get_regs,
 	.get_eeprom_length    = ixgbe_get_eeprom_length,
-	.get_eeprom           = ixgbe_get_eeprom,
-	.set_eeprom           = ixgbe_set_eeprom,
+	.get_eeprom	      = ixgbe_get_eeprom,
+	.set_eeprom	      = ixgbe_set_eeprom,
 	.get_module_info      = ixgbe_get_module_info,
 	.get_module_eeprom    = ixgbe_get_module_eeprom,
-	.get_dcb_info         = ixgbe_dev_get_dcb_info,
+	.get_dcb_info	      = ixgbe_dev_get_dcb_info,
 	.timesync_adjust_time = ixgbe_timesync_adjust_time,
 	.timesync_read_time   = ixgbe_timesync_read_time,
 	.timesync_write_time  = ixgbe_timesync_write_time,
 	.udp_tunnel_port_add  = ixgbe_dev_udp_tunnel_port_add,
 	.udp_tunnel_port_del  = ixgbe_dev_udp_tunnel_port_del,
-	.tm_ops_get           = ixgbe_tm_ops_get,
+	.tm_ops_get	      = ixgbe_tm_ops_get,
 	.tx_done_cleanup      = ixgbe_dev_tx_done_cleanup,
 	.get_monitor_addr     = ixgbe_get_monitor_addr,
 };
@@ -559,24 +556,24 @@ static const struct eth_dev_ops ixgbe_eth_dev_ops = {
  * operation have been implemented
  */
 static const struct eth_dev_ops ixgbevf_eth_dev_ops = {
-	.dev_configure        = ixgbevf_dev_configure,
-	.dev_start            = ixgbevf_dev_start,
-	.dev_stop             = ixgbevf_dev_stop,
-	.link_update          = ixgbevf_dev_link_update,
-	.stats_get            = ixgbevf_dev_stats_get,
-	.xstats_get           = ixgbevf_dev_xstats_get,
-	.stats_reset          = ixgbevf_dev_stats_reset,
-	.xstats_reset         = ixgbevf_dev_stats_reset,
+	.dev_configure	      = ixgbevf_dev_configure,
+	.dev_start	      = ixgbevf_dev_start,
+	.dev_stop	      = ixgbevf_dev_stop,
+	.link_update	      = ixgbevf_dev_link_update,
+	.stats_get	      = ixgbevf_dev_stats_get,
+	.xstats_get	      = ixgbevf_dev_xstats_get,
+	.stats_reset	      = ixgbevf_dev_stats_reset,
+	.xstats_reset	      = ixgbevf_dev_stats_reset,
 	.xstats_get_names     = ixgbevf_dev_xstats_get_names,
-	.dev_close            = ixgbevf_dev_close,
+	.dev_close	      = ixgbevf_dev_close,
 	.dev_reset	      = ixgbevf_dev_reset,
 	.promiscuous_enable   = ixgbevf_dev_promiscuous_enable,
 	.promiscuous_disable  = ixgbevf_dev_promiscuous_disable,
 	.allmulticast_enable  = ixgbevf_dev_allmulticast_enable,
 	.allmulticast_disable = ixgbevf_dev_allmulticast_disable,
-	.dev_infos_get        = ixgbevf_dev_info_get,
+	.dev_infos_get	      = ixgbevf_dev_info_get,
 	.dev_supported_ptypes_get = ixgbe_dev_supported_ptypes_get,
-	.mtu_set              = ixgbevf_dev_set_mtu,
+	.mtu_set	      = ixgbevf_dev_set_mtu,
 	.vlan_filter_set      = ixgbevf_vlan_filter_set,
 	.vlan_strip_queue_set = ixgbevf_vlan_strip_queue_set,
 	.vlan_offload_set     = ixgbevf_vlan_offload_set,
@@ -586,15 +583,15 @@ static const struct eth_dev_ops ixgbevf_eth_dev_ops = {
 	.tx_queue_release     = ixgbe_dev_tx_queue_release,
 	.rx_queue_intr_enable = ixgbevf_dev_rx_queue_intr_enable,
 	.rx_queue_intr_disable = ixgbevf_dev_rx_queue_intr_disable,
-	.mac_addr_add         = ixgbevf_add_mac_addr,
+	.mac_addr_add	      = ixgbevf_add_mac_addr,
 	.mac_addr_remove      = ixgbevf_remove_mac_addr,
 	.set_mc_addr_list     = ixgbe_dev_set_mc_addr_list,
-	.rxq_info_get         = ixgbe_rxq_info_get,
-	.txq_info_get         = ixgbe_txq_info_get,
-	.mac_addr_set         = ixgbevf_set_default_mac_addr,
-	.get_reg              = ixgbevf_get_regs,
-	.reta_update          = ixgbe_dev_rss_reta_update,
-	.reta_query           = ixgbe_dev_rss_reta_query,
+	.rxq_info_get	      = ixgbe_rxq_info_get,
+	.txq_info_get	      = ixgbe_txq_info_get,
+	.mac_addr_set	      = ixgbevf_set_default_mac_addr,
+	.get_reg	      = ixgbevf_get_regs,
+	.reta_update	      = ixgbe_dev_rss_reta_update,
+	.reta_query	      = ixgbe_dev_rss_reta_query,
 	.rss_hash_update      = ixgbe_dev_rss_hash_update,
 	.rss_hash_conf_get    = ixgbe_dev_rss_hash_conf_get,
 	.tx_done_cleanup      = ixgbe_dev_tx_done_cleanup,
@@ -766,6 +763,33 @@ static const struct rte_ixgbe_xstats_name_off rte_ixgbevf_stats_strings[] = {
 #define IXGBEVF_NB_XSTATS (sizeof(rte_ixgbevf_stats_strings) /	\
 		sizeof(rte_ixgbevf_stats_strings[0]))
 
+/**
+ * This function is the same as ixgbe_need_crosstalk_fix() in base/ixgbe_common.c
+ *
+ * ixgbe_need_crosstalk_fix - Determine if we need to do cross talk fix
+ * @hw: pointer to hardware structure
+ *
+ * Contains the logic to identify if we need to verify link for the
+ * crosstalk fix
+ **/
+static bool ixgbe_need_crosstalk_fix(struct ixgbe_hw *hw)
+{
+	/* Does FW say we need the fix */
+	if (!hw->need_crosstalk_fix)
+		return false;
+
+	/* Only consider SFP+ PHYs i.e. media type fiber */
+	switch (ixgbe_get_media_type(hw)) {
+	case ixgbe_media_type_fiber:
+	case ixgbe_media_type_fiber_qsfp:
+		break;
+	default:
+		return false;
+	}
+
+	return true;
+}
+
 /*
  * This function is the same as ixgbe_is_sfp() in base/ixgbe.h.
  */
@@ -1032,6 +1056,306 @@ ixgbe_swfw_lock_reset(struct ixgbe_hw *hw)
 	ixgbe_release_swfw_semaphore(hw, mask);
 }
 
+/**
+ * ixgbe_check_sfp_cage - Find present status of SFP module
+ * @hw: pointer to hardware structure
+ *
+ * Find if a SFP module is present and if this device supports SFPs
+ **/
+enum ixgbe_sfp_cage_status ixgbe_check_sfp_cage(struct ixgbe_hw *hw)
+{
+	enum ixgbe_sfp_cage_status sfp_cage_status;
+
+	/* If we're not a fiber/fiber_qsfp, no cage to check */
+	switch (hw->mac.ops.get_media_type(hw)) {
+	case ixgbe_media_type_fiber:
+	case ixgbe_media_type_fiber_qsfp:
+		break;
+	default:
+		return IXGBE_SFP_CAGE_NOCAGE;
+	}
+
+	switch (hw->mac.type) {
+	case ixgbe_mac_82599EB:
+		sfp_cage_status = !!(IXGBE_READ_REG(hw, IXGBE_ESDP) &
+			     IXGBE_ESDP_SDP2);
+		break;
+	case ixgbe_mac_X550EM_x:
+	case ixgbe_mac_X550EM_a:
+		/* SDP0 is the active low signal PRSNT#, so invert this */
+		sfp_cage_status = !(IXGBE_READ_REG(hw, IXGBE_ESDP) &
+				  IXGBE_ESDP_SDP0);
+		break;
+	default:
+		/* Don't know how to check this device type yet */
+		sfp_cage_status = IXGBE_SFP_CAGE_UNKNOWN;
+		DEBUGOUT("IXGBE_SFP_CAGE_UNKNOWN, unknown mac type %d\n",
+			 hw->mac.type);
+		break;
+	}
+
+	DEBUGOUT("sfp status %d for mac type %d\n", sfp_cage_status, hw->mac.type);
+	return sfp_cage_status;
+}
+
+static s32
+ixgbe_sfp_id_and_setup(struct rte_eth_dev *dev)
+{
+	struct ixgbe_hw *hw =
+		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	enum ixgbe_sfp_cage_status sfp_cage_status;
+	s32 err;
+
+	/* Can't ID or setup SFP if it's not plugged in */
+	sfp_cage_status = ixgbe_check_sfp_cage(hw);
+	if (sfp_cage_status == IXGBE_SFP_CAGE_EMPTY ||
+	    sfp_cage_status == IXGBE_SFP_CAGE_NOCAGE)
+		return IXGBE_ERR_SFP_NOT_PRESENT;
+
+	/* Something's in the cage, ID it */
+	hw->phy.ops.identify_sfp(hw);
+
+	/* Unknown module type, give up */
+	if (hw->phy.sfp_type == ixgbe_sfp_type_unknown) {
+		PMD_DRV_LOG(ERR, "unknown SFP type, giving up");
+		return IXGBE_ERR_SFP_NOT_SUPPORTED;
+	}
+
+	/* This should be a redundant check, since we looked at the
+	 * PRSNT# signal from the cage above, but just in case this is
+	 * an SFP that's slow to respond to I2C pokes correctly, try it
+	 * again later
+	 */
+	if (hw->phy.sfp_type == ixgbe_sfp_type_not_present) {
+		PMD_DRV_LOG(ERR, "IDed SFP as absent but cage PRSNT# active!?");
+		return IXGBE_ERR_SFP_NOT_PRESENT;
+	}
+
+	/* SFP is present and identified, try to set it up */
+	err = hw->mac.ops.setup_sfp(hw);
+	if (err)
+		PMD_DRV_LOG(ERR, "setup_sfp() failed %d", err);
+
+	return err;
+}
+
+static void
+ixgbe_sfp_service(struct rte_eth_dev *dev)
+{
+	struct ixgbe_hw *hw =
+		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	struct ixgbe_interrupt *intr =
+		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
+	enum ixgbe_sfp_cage_status sfp_cage_status;
+	s32 err;
+	u8 sff_id;
+	bool have_int = false;
+
+	/* If there's no module cage, then there's nothing to service */
+	sfp_cage_status = ixgbe_check_sfp_cage(hw);
+	if (sfp_cage_status == IXGBE_SFP_CAGE_NOCAGE) {
+		PMD_DRV_LOG(DEBUG, "No SFP to service\n");
+		return;
+	}
+
+	/* TODO - Even for platforms where ixgbe_check_sfp_cage() gives a clear
+	 * status result, if there's no interrupts, or no interrupt for the SFP
+	 * cage present pin, even if other interrupts exist, then we still need
+	 * to poll here to set the flag.
+	 */
+#ifndef RTE_EXEC_ENV_FREEBSD
+	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
+	if (rte_intr_allow_others(intr_handle)) {
+		/* check if lsc interrupt is enabled */
+		if (dev->data->dev_conf.intr_conf.lsc)
+			have_int = true;
+	}
+#endif /* #ifdef RTE_EXEC_ENV_FREEBSD */
+
+	if (!have_int && sfp_cage_status == IXGBE_SFP_CAGE_EMPTY) {
+		intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
+		PMD_DRV_LOG(DEBUG, "No SFP, no LSC, set NEED_SFP_SETUP\n");
+	}
+
+	/* For platforms that don't have a way to read the PRESENT# signal from
+	 * the SFP cage, fallback to doing an I2C read and seeing if it's ACKed
+	 * to determine if a module is present
+	 */
+	if (sfp_cage_status == IXGBE_SFP_CAGE_UNKNOWN) {
+		PMD_DRV_LOG(DEBUG,
+			    "SFP present unknown (int? %d), try I2C read\n",
+			    have_int);
+
+		/* Rather than calling identify_sfp, which will read a lot of I2C
+		 * registers (and in a slow processor intensive fashion due to
+		 * bit-banging, just read the SFF ID register, which is at a
+		 * common address across SFP/SFP+/QSFP modules and see if
+		 * there's a NACK.  This works since we only expect a NACK if no
+		 * module is present
+		 */
+		err = ixgbe_read_i2c_eeprom(hw, IXGBE_SFF_IDENTIFIER, &sff_id);
+		if (err != IXGBE_SUCCESS) {
+			PMD_DRV_LOG(DEBUG, "Received I2C NAK from SFP, set NEED_SFP_SETUP flag\n");
+			intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
+			sfp_cage_status = IXGBE_SFP_CAGE_EMPTY;
+		} else {
+			PMD_DRV_LOG(DEBUG, "SFP ID read ACKed");
+			sfp_cage_status = IXGBE_SFP_CAGE_FULL;
+		}
+	}
+
+	if (sfp_cage_status == IXGBE_SFP_CAGE_EMPTY) {
+		PMD_DRV_LOG(DEBUG, "SFP absent, cage_status %d\n", sfp_cage_status);
+		return;
+	}
+
+	/* No setup requested?	Nothing to do */
+	if (!(intr->flags & IXGBE_FLAG_NEED_SFP_SETUP))
+		return;
+
+	err = ixgbe_sfp_id_and_setup(dev);
+	if (err) {
+		PMD_DRV_LOG(DEBUG, "failed to ID & setup SFP %d", err);
+		return;
+	}
+
+	/* Setup is done, clear the flag, but make sure link config runs for new SFP */
+	intr->flags &= ~IXGBE_FLAG_NEED_SFP_SETUP;
+	intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
+
+	/*
+	 * Since this is a new SFP, clear the old advertised speed mask so we don't
+	 * end up using an old slower rate
+	 */
+	hw->phy.autoneg_advertised = 0;
+}
+
+static void
+ixgbe_link_service(struct rte_eth_dev *dev)
+{
+	struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	struct ixgbe_interrupt *intr =
+		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
+	bool link_up, autoneg = false, have_int = false;
+	u32 speed;
+	s32 err;
+
+	/* Test if we have a LSC interrupt for this platform, if not we need to
+	 * manually check the link register since IXGBE_FLAG_NEED_LINK_CONFIG
+	 * will never be set in the interrupt handler
+	 */
+#ifndef RTE_EXEC_ENV_FREEBSD
+	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
+	if (rte_intr_allow_others(intr_handle)) {
+		/* check if lsc interrupt is enabled */
+		if (dev->data->dev_conf.intr_conf.lsc)
+			have_int = true;
+	}
+#endif /* #ifdef RTE_EXEC_ENV_FREEBSD */
+
+	/* Skip if we still need to setup an SFP, or if no link config requested
+	 */
+	if ((intr->flags & IXGBE_FLAG_NEED_SFP_SETUP) ||
+	    (!(intr->flags & IXGBE_FLAG_NEED_LINK_CONFIG) && have_int))
+		return;
+
+	if (!have_int && !(intr->flags & IXGBE_FLAG_NEED_LINK_CONFIG)) {
+		err = ixgbe_check_link(hw, &speed, &link_up, 0);
+		if (!err && !link_up) {
+			intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
+			PMD_DRV_LOG(DEBUG, "Link down, no LSC, set NEED_LINK_CONFIG\n");
+		} else {
+			return;
+		}
+	}
+
+	speed = hw->phy.autoneg_advertised;
+	if (!speed)
+		ixgbe_get_link_capabilities(hw, &speed, &autoneg);
+
+	err = ixgbe_setup_link(hw, speed, true);
+	if (err) {
+		PMD_DRV_LOG(ERR, "ixgbe_setup_link failed %d", err);
+		return;
+	}
+
+	intr->flags &= ~IXGBE_FLAG_NEED_LINK_CONFIG;
+}
+
+static void
+ixgbe_link_update_service(struct rte_eth_dev *dev)
+{
+	/* Update internal link status, waiting for link */
+	if (!ixgbe_dev_link_update(dev, 0)) {
+		ixgbe_dev_link_status_print(dev);
+		rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_LSC, NULL);
+	}
+}
+
+/*
+ * Service task thread to handle periodic tasks
+ */
+static void *
+ixgbe_dev_service_thread_handler(void *param)
+{
+	struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
+	struct ixgbe_hw *hw =
+		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	uint64_t start, ticks, service_ms;
+	uint32_t speed;
+	s32 err;
+	bool link_up;
+
+	while (1) {
+		ixgbe_sfp_service(dev);
+		ixgbe_link_service(dev);
+		ixgbe_link_update_service(dev);
+
+		/* Run the service thread handler more frequently when link is
+		 * down to reduce link up latency (every 200ms vs 1s)
+		 *
+		 * Use a number of smaller sleeps to decrease exit latency when
+		 * ixgbe_dev_stop() wants this thread to join
+		 */
+		err = ixgbe_check_link(hw, &speed, &link_up, 0);
+		if (err == IXGBE_SUCCESS && link_up)
+			service_ms = 2000;
+		else
+			service_ms = 100;
+
+		/* Call msec_delay in a loop with several smaller sleeps to
+		 * provide periodic thread cancellation points
+		 */
+		start = rte_get_timer_cycles();
+		ticks = (uint64_t)service_ms * rte_get_timer_hz() / 1E3;
+		while ((rte_get_timer_cycles() - start) < ticks)
+			msec_delay(100);
+	}
+
+	/* Never return */
+	return NULL;
+}
+
+static s32
+eth_ixgbe_check_mac_link_generic(struct ixgbe_hw *hw, ixgbe_link_speed *speed,
+				 bool *link_up, bool link_up_wait_to_complete)
+{
+	if (ixgbe_need_crosstalk_fix(hw)) {
+		enum ixgbe_sfp_cage_status sfp_cage_status;
+
+		sfp_cage_status = ixgbe_check_sfp_cage(hw);
+		if (sfp_cage_status != IXGBE_SFP_CAGE_FULL) {
+			*link_up = false;
+			*speed = IXGBE_LINK_SPEED_UNKNOWN;
+			return IXGBE_SUCCESS;
+		}
+	}
+
+	return ixgbe_check_mac_link_generic(hw, speed, link_up, link_up_wait_to_complete);
+}
+
 /*
  * This function is based on code in ixgbe_attach() in base/ixgbe.c.
  * It returns 0 on success.
@@ -1054,6 +1378,7 @@ eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)
 		IXGBE_DEV_PRIVATE_TO_FILTER_INFO(eth_dev->data->dev_private);
 	struct ixgbe_bw_conf *bw_conf =
 		IXGBE_DEV_PRIVATE_TO_BW_CONF(eth_dev->data->dev_private);
+	struct ixgbe_mac_info *mac = &hw->mac;
 	uint32_t ctrl_ext;
 	uint16_t csum;
 	int diag, i, ret;
@@ -1124,6 +1449,17 @@ eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)
 		return -EIO;
 	}
 
+	/* override mac_link_check to check for sfp cage full/empty */
+	switch (hw->mac.type) {
+	case ixgbe_mac_X550EM_x:
+	case ixgbe_mac_X550EM_a:
+	case ixgbe_mac_82599EB:
+		mac->ops.check_link = eth_ixgbe_check_mac_link_generic;
+		break;
+	default:
+		break;
+	}
+
 	/* pick up the PCI bus settings for reporting later */
 	ixgbe_get_bus_info(hw);
 
@@ -2558,8 +2894,11 @@ ixgbe_flow_ctrl_enable(struct rte_eth_dev *dev, struct ixgbe_hw *hw)
 static int
 ixgbe_dev_start(struct rte_eth_dev *dev)
 {
+	struct ixgbe_adapter *ad = dev->data->dev_private;
 	struct ixgbe_hw *hw =
 		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	struct ixgbe_interrupt *intr =
+		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
 	struct ixgbe_vf_info *vfinfo =
 		*IXGBE_DEV_PRIVATE_TO_P_VFDATA(dev->data->dev_private);
 	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
@@ -2580,9 +2919,6 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
 
 	PMD_INIT_FUNC_TRACE();
 
-	/* Stop the link setup handler before resetting the HW. */
-	ixgbe_dev_wait_setup_link_complete(dev, 0);
-
 	/* disable uio/vfio intr/eventfd mapping */
 	rte_intr_disable(intr_handle);
 
@@ -2815,6 +3151,20 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
 	ixgbe_l2_tunnel_conf(dev);
 	ixgbe_filter_restore(dev);
 
+	/* Spawn service thread */
+	if (ixgbe_is_sfp(hw)) {
+		intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
+		err = rte_ctrl_thread_create(&ad->service_thread_tid,
+					     "ixgbe-service-thread",
+					     NULL,
+					     ixgbe_dev_service_thread_handler,
+					     dev);
+		if (err) {
+			PMD_DRV_LOG(ERR, "service_thead err");
+			goto error;
+		}
+	}
+
 	if (tm_conf->root && !tm_conf->committed)
 		PMD_DRV_LOG(WARNING,
 			    "please call hierarchy_commit() "
@@ -2860,13 +3210,21 @@ ixgbe_dev_stop(struct rte_eth_dev *dev)
 	int vf;
 	struct ixgbe_tm_conf *tm_conf =
 		IXGBE_DEV_PRIVATE_TO_TM_CONF(dev->data->dev_private);
+	void *res;
+	s32 err;
 
 	if (hw->adapter_stopped)
 		return 0;
 
 	PMD_INIT_FUNC_TRACE();
 
-	ixgbe_dev_wait_setup_link_complete(dev, 0);
+	/* Cancel the service thread, and wait for it to join */
+	err = pthread_cancel(adapter->service_thread_tid);
+	if (err)
+		PMD_DRV_LOG(ERR, "failed to cancel service thread %d", err);
+	err = pthread_join(adapter->service_thread_tid, &res);
+	if (err)
+		PMD_DRV_LOG(ERR, "failed to join service thread %d", err);
 
 	/* disable interrupts */
 	ixgbe_disable_intr(hw);
@@ -2945,7 +3303,6 @@ ixgbe_dev_set_link_up(struct rte_eth_dev *dev)
 	} else {
 		/* Turn on the laser */
 		ixgbe_enable_tx_laser(hw);
-		ixgbe_dev_link_update(dev, 0);
 	}
 
 	return 0;
@@ -2976,7 +3333,6 @@ ixgbe_dev_set_link_down(struct rte_eth_dev *dev)
 	} else {
 		/* Turn off the laser */
 		ixgbe_disable_tx_laser(hw);
-		ixgbe_dev_link_update(dev, 0);
 	}
 
 	return 0;
@@ -3343,8 +3699,8 @@ ixgbe_dev_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats)
 	}
 
 	/* Rx Errors */
-	stats->imissed  = total_missed_rx;
-	stats->ierrors  = hw_stats->crcerrs +
+	stats->imissed	= total_missed_rx;
+	stats->ierrors	= hw_stats->crcerrs +
 			  hw_stats->mspdc +
 			  hw_stats->rlec +
 			  hw_stats->ruc +
@@ -3363,7 +3719,7 @@ ixgbe_dev_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats)
 		stats->ierrors += hw_stats->xec;
 
 	/* Tx Errors */
-	stats->oerrors  = 0;
+	stats->oerrors	= 0;
 	return 0;
 }
 
@@ -4128,57 +4484,6 @@ ixgbevf_check_link(struct ixgbe_hw *hw, ixgbe_link_speed *speed,
 	return ret_val;
 }
 
-/*
- * If @timeout_ms was 0, it means that it will not return until link complete.
- * It returns 1 on complete, return 0 on timeout.
- */
-static int
-ixgbe_dev_wait_setup_link_complete(struct rte_eth_dev *dev, uint32_t timeout_ms)
-{
-#define WARNING_TIMEOUT    9000 /* 9s  in total */
-	struct ixgbe_adapter *ad = dev->data->dev_private;
-	uint32_t timeout = timeout_ms ? timeout_ms : WARNING_TIMEOUT;
-
-	while (rte_atomic32_read(&ad->link_thread_running)) {
-		msec_delay(1);
-		timeout--;
-
-		if (timeout_ms) {
-			if (!timeout)
-				return 0;
-		} else if (!timeout) {
-			/* It will not return until link complete */
-			timeout = WARNING_TIMEOUT;
-			PMD_DRV_LOG(ERR, "IXGBE link thread not complete too long time!");
-		}
-	}
-
-	return 1;
-}
-
-static void *
-ixgbe_dev_setup_link_thread_handler(void *param)
-{
-	struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
-	struct ixgbe_adapter *ad = dev->data->dev_private;
-	struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
-	struct ixgbe_interrupt *intr =
-		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
-	u32 speed;
-	bool autoneg = false;
-
-	pthread_detach(pthread_self());
-	speed = hw->phy.autoneg_advertised;
-	if (!speed)
-		ixgbe_get_link_capabilities(hw, &speed, &autoneg);
-
-	ixgbe_setup_link(hw, speed, true);
-
-	intr->flags &= ~IXGBE_FLAG_NEED_LINK_CONFIG;
-	rte_atomic32_clear(&ad->link_thread_running);
-	return NULL;
-}
-
 /*
  * In freebsd environment, nic_uio drivers do not support interrupts,
  * rte_intr_callback_register() will fail to register interrupts.
@@ -4218,11 +4523,8 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 			    int wait_to_complete, int vf)
 {
 	struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
-	struct ixgbe_adapter *ad = dev->data->dev_private;
 	struct rte_eth_link link;
 	ixgbe_link_speed link_speed = IXGBE_LINK_SPEED_UNKNOWN;
-	struct ixgbe_interrupt *intr =
-		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
 	bool link_up;
 	int diag;
 	int wait = 1;
@@ -4237,9 +4539,6 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 
 	hw->mac.get_link_status = true;
 
-	if (intr->flags & IXGBE_FLAG_NEED_LINK_CONFIG)
-		return rte_eth_linkstatus_set(dev, &link);
-
 	/* check if it needs to wait to complete, if lsc interrupt is enabled */
 	if (wait_to_complete == 0 || dev->data->dev_conf.intr_conf.lsc != 0)
 		wait = 0;
@@ -4254,7 +4553,7 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 	else
 		diag = ixgbe_check_link(hw, &link_speed, &link_up, wait);
 
-	if (diag != 0) {
+	if (diag != 0 || !link_up) {
 		link.link_speed = RTE_ETH_SPEED_NUM_100M;
 		link.link_duplex = RTE_ETH_LINK_FULL_DUPLEX;
 		return rte_eth_linkstatus_set(dev, &link);
@@ -4267,32 +4566,6 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 			link_up = 0;
 	}
 
-	if (link_up == 0) {
-		if (ixgbe_get_media_type(hw) == ixgbe_media_type_fiber) {
-			ixgbe_dev_wait_setup_link_complete(dev, 0);
-			if (rte_atomic32_test_and_set(&ad->link_thread_running)) {
-				/* To avoid race condition between threads, set
-				 * the IXGBE_FLAG_NEED_LINK_CONFIG flag only
-				 * when there is no link thread running.
-				 */
-				intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
-				if (rte_ctrl_thread_create(&ad->link_thread_tid,
-					"ixgbe-link-handler",
-					NULL,
-					ixgbe_dev_setup_link_thread_handler,
-					dev) < 0) {
-					PMD_DRV_LOG(ERR,
-						"Create link thread failed!");
-					rte_atomic32_clear(&ad->link_thread_running);
-				}
-			} else {
-				PMD_DRV_LOG(ERR,
-					"Other link thread is running now!");
-			}
-		}
-		return rte_eth_linkstatus_set(dev, &link);
-	}
-
 	link.link_status = RTE_ETH_LINK_UP;
 	link.link_duplex = RTE_ETH_LINK_FULL_DUPLEX;
 
@@ -4498,8 +4771,6 @@ ixgbe_dev_interrupt_get_status(struct rte_eth_dev *dev)
 	eicr = IXGBE_READ_REG(hw, IXGBE_EICR);
 	PMD_DRV_LOG(DEBUG, "eicr %x", eicr);
 
-	intr->flags = 0;
-
 	/* set flag for async link update */
 	if (eicr & IXGBE_EICR_LSC)
 		intr->flags |= IXGBE_FLAG_NEED_LINK_UPDATE;
@@ -4515,6 +4786,14 @@ ixgbe_dev_interrupt_get_status(struct rte_eth_dev *dev)
 	    (eicr & IXGBE_EICR_GPI_SDP0_X550EM_x))
 		intr->flags |= IXGBE_FLAG_PHY_INTERRUPT;
 
+	/* Check for loss of SFP */
+	/* TODO - For platforms that don't have this flag, do we need to set
+	 *  NEED_SFP_SETUP on LSC if we're a SFP platform?
+	 */
+	if (hw->mac.type ==  ixgbe_mac_X550EM_a &&
+	    (eicr & IXGBE_EICR_GPI_SDP0_X550EM_a))
+		intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
+
 	return 0;
 }
 
@@ -4566,11 +4845,13 @@ ixgbe_dev_link_status_print(struct rte_eth_dev *dev)
 static int
 ixgbe_dev_interrupt_action(struct rte_eth_dev *dev)
 {
+	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
 	struct ixgbe_interrupt *intr =
 		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
-	int64_t timeout;
 	struct ixgbe_hw *hw =
 		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	int64_t timeout;
 
 	PMD_DRV_LOG(DEBUG, "intr action type %d", intr->flags);
 
@@ -4605,16 +4886,14 @@ ixgbe_dev_interrupt_action(struct rte_eth_dev *dev)
 		if (rte_eal_alarm_set(timeout * 1000,
 				      ixgbe_dev_interrupt_delayed_handler, (void *)dev) < 0)
 			PMD_DRV_LOG(ERR, "Error setting alarm");
-		else {
-			/* remember original mask */
-			intr->mask_original = intr->mask;
+		else
 			/* only disable lsc interrupt */
 			intr->mask &= ~IXGBE_EIMS_LSC;
-		}
 	}
 
 	PMD_DRV_LOG(DEBUG, "enable intr immediately");
 	ixgbe_enable_intr(dev);
+	rte_intr_ack(intr_handle);
 
 	return 0;
 }
@@ -4637,8 +4916,6 @@ static void
 ixgbe_dev_interrupt_delayed_handler(void *param)
 {
 	struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
-	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
-	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
 	struct ixgbe_interrupt *intr =
 		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
 	struct ixgbe_hw *hw =
@@ -4668,13 +4945,10 @@ ixgbe_dev_interrupt_delayed_handler(void *param)
 		intr->flags &= ~IXGBE_FLAG_MACSEC;
 	}
 
-	/* restore original mask */
-	intr->mask = intr->mask_original;
-	intr->mask_original = 0;
+	if (dev->data->dev_conf.intr_conf.lsc != 0)
+		intr->mask |= IXGBE_EICR_LSC;
 
-	PMD_DRV_LOG(DEBUG, "enable intr in delayed handler S[%08x]", eicr);
 	ixgbe_enable_intr(dev);
-	rte_intr_ack(intr_handle);
 }
 
 /**
@@ -4939,7 +5213,7 @@ ixgbe_dcb_pfc_enable_generic(struct ixgbe_hw *hw, uint8_t tc_num)
 		/*
 		 * In order to prevent Tx hangs when the internal Tx
 		 * switch is enabled we must set the high water mark
-		 * to the maximum FCRTH value.  This allows the Tx
+		 * to the maximum FCRTH value.	This allows the Tx
 		 * switch to function even under heavy Rx workloads.
 		 */
 		fcrth = IXGBE_READ_REG(hw, IXGBE_RXPBSIZE(tc_num)) - 32;
@@ -5316,9 +5590,6 @@ ixgbevf_dev_start(struct rte_eth_dev *dev)
 
 	PMD_INIT_FUNC_TRACE();
 
-	/* Stop the link setup handler before resetting the HW. */
-	ixgbe_dev_wait_setup_link_complete(dev, 0);
-
 	err = hw->mac.ops.reset_hw(hw);
 
 	/**
@@ -5398,12 +5669,6 @@ ixgbevf_dev_start(struct rte_eth_dev *dev)
 	/* Re-enable interrupt for VF */
 	ixgbevf_intr_enable(dev);
 
-	/*
-	 * Update link status right before return, because it may
-	 * start link configuration process in a separate thread.
-	 */
-	ixgbevf_dev_link_update(dev, 0);
-
 	hw->adapter_stopped = false;
 
 	return 0;
@@ -5422,8 +5687,6 @@ ixgbevf_dev_stop(struct rte_eth_dev *dev)
 
 	PMD_INIT_FUNC_TRACE();
 
-	ixgbe_dev_wait_setup_link_complete(dev, 0);
-
 	ixgbevf_intr_disable(dev);
 
 	dev->data->dev_started = 0;
@@ -6922,7 +7185,7 @@ ixgbe_timesync_read_rx_timestamp(struct rte_eth_dev *dev,
 	ns = rte_timecounter_update(&adapter->rx_tstamp_tc, rx_tstamp_cycles);
 	*timestamp = rte_ns_to_timespec(ns);
 
-	return  0;
+	return	0;
 }
 
 static int
@@ -7902,7 +8165,7 @@ int ixgbe_disable_sec_tx_path_generic(struct ixgbe_hw *hw)
 	/* For informational purposes only */
 	if (i >= IXGBE_MAX_SECTX_POLL)
 		PMD_DRV_LOG(DEBUG, "Tx unit being enabled before security "
-			 "path fully disabled.  Continuing with init.");
+			 "path fully disabled.	Continuing with init.");
 
 	return IXGBE_SUCCESS;
 }
@@ -7986,8 +8249,8 @@ ixgbe_l2_tn_filter_restore(struct rte_eth_dev *dev)
 
 	TAILQ_FOREACH(node, &l2_tn_info->l2_tn_list, entries) {
 		l2_tn_conf.l2_tunnel_type = node->key.l2_tn_type;
-		l2_tn_conf.tunnel_id      = node->key.tn_id;
-		l2_tn_conf.pool           = node->pool;
+		l2_tn_conf.tunnel_id	  = node->key.tn_id;
+		l2_tn_conf.pool		  = node->pool;
 		(void)ixgbe_dev_l2_tunnel_filter_add(dev, &l2_tn_conf, TRUE);
 	}
 }
@@ -8094,8 +8357,8 @@ ixgbe_clear_all_l2_tn_filter(struct rte_eth_dev *dev)
 
 	while ((l2_tn_filter = TAILQ_FIRST(&l2_tn_info->l2_tn_list))) {
 		l2_tn_conf.l2_tunnel_type = l2_tn_filter->key.l2_tn_type;
-		l2_tn_conf.tunnel_id      = l2_tn_filter->key.tn_id;
-		l2_tn_conf.pool           = l2_tn_filter->pool;
+		l2_tn_conf.tunnel_id	  = l2_tn_filter->key.tn_id;
+		l2_tn_conf.pool		  = l2_tn_filter->pool;
 		ret = ixgbe_dev_l2_tunnel_filter_del(dev, &l2_tn_conf);
 		if (ret < 0)
 			return ret;
diff --git a/drivers/net/ixgbe/ixgbe_ethdev.h b/drivers/net/ixgbe/ixgbe_ethdev.h
index 69e0e82a5b..d243e417e9 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.h
+++ b/drivers/net/ixgbe/ixgbe_ethdev.h
@@ -29,6 +29,7 @@
 #define IXGBE_FLAG_PHY_INTERRUPT    (uint32_t)(1 << 2)
 #define IXGBE_FLAG_MACSEC           (uint32_t)(1 << 3)
 #define IXGBE_FLAG_NEED_LINK_CONFIG (uint32_t)(1 << 4)
+#define IXGBE_FLAG_NEED_SFP_SETUP   ((uint32_t)(1 << 5))
 
 /*
  * Defines that were not part of ixgbe_type.h as they are not used by the
@@ -223,8 +224,6 @@ struct ixgbe_rte_flow_rss_conf {
 struct ixgbe_interrupt {
 	uint32_t flags;
 	uint32_t mask;
-	/*to save original mask during delayed handler */
-	uint32_t mask_original;
 };
 
 struct ixgbe_stat_mapping_registers {
@@ -507,7 +506,7 @@ struct ixgbe_adapter {
 	uint8_t pflink_fullchk;
 	uint8_t mac_ctrl_frame_fwd;
 	rte_atomic32_t link_thread_running;
-	pthread_t link_thread_tid;
+	pthread_t service_thread_tid;
 };
 
 struct ixgbe_vf_representor {
@@ -670,6 +669,15 @@ int ixgbe_syn_filter_set(struct rte_eth_dev *dev,
 			struct rte_eth_syn_filter *filter,
 			bool add);
 
+enum ixgbe_sfp_cage_status {
+	IXGBE_SFP_CAGE_EMPTY = 0,
+	IXGBE_SFP_CAGE_FULL,
+	IXGBE_SFP_CAGE_UNKNOWN = -1,
+	IXGBE_SFP_CAGE_NOCAGE = -2,
+};
+enum ixgbe_sfp_cage_status ixgbe_check_sfp_cage(struct ixgbe_hw *hw);
+
+
 /**
  * l2 tunnel configuration.
  */
-- 
2.25.1

RE: [PATCH v3 3/3] net/ixgbe: Fix SFP detection and linking on hotplug

[Wang, Haiyue]

> -----Original Message-----
> From: Jeff Daly <jeffd@silicom-usa.com>
> Sent: Thursday, February 24, 2022 23:24
> To: dev@dpdk.org
> Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>; Wang, Haiyue <haiyue.wang@intel.com>
> Subject: [PATCH v3 3/3] net/ixgbe: Fix SFP detection and linking on hotplug
> 
> Currently the ixgbe driver does not ID any SFP except for the first one
> plugged in. This can lead to no-link, or incorrect speed conditions.
> 
> For example:
> 
> * If link is initially established with a 1G SFP, and later a 1G/10G
> multispeed part is later installed, then the MAC link setup functions are
> never called to change from 1000BASE-X to 10GBASE-R mode, and the link
> stays running at the slower rate.
> 
> * If link is initially established with a 1G SFP, and later a 10G only
> module is later installed, no link is established, since we are still
> trasnsmitting in 1000BASE-X mode to a 10GBASE-R only partner.
> 
> Refactor the SFP ID/setup, and link setup code, to more closely match the
> flow of the mainline kernel driver which does not have these issues.  In
> that driver a service task runs periodically to handle these operations
> based on bit flags that have been set (usually via interrupt or userspace
> request), and then get cleared once the requested subtask has been
> completed.
> 
> Fixes: af75078fece ("first public release")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
> Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> ---
>  drivers/net/ixgbe/ixgbe_ethdev.c | 687 +++++++++++++++++++++----------
>  drivers/net/ixgbe/ixgbe_ethdev.h |  14 +-
>  2 files changed, 486 insertions(+), 215 deletions(-)
> 
> diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
> index e8f07cb405..2150b5ed35 100644
> --- a/drivers/net/ixgbe/ixgbe_ethdev.c
> +++ b/drivers/net/ixgbe/ixgbe_ethdev.c
> @@ -69,11 +69,11 @@
> 
>  #define IXGBE_LINK_DOWN_CHECK_TIMEOUT 4000 /* ms */
>  #define IXGBE_LINK_UP_CHECK_TIMEOUT   1000 /* ms */
> -#define IXGBE_VMDQ_NUM_UC_MAC         4096 /* Maximum nb. of UC MAC addr. */
> +#define IXGBE_VMDQ_NUM_UC_MAC	      4096 /* Maximum nb. of UC MAC addr. */
> 

Please drop this kind of indentation change, too many.
Then we will be lost in your fix.

>  /**
>   * l2 tunnel configuration.
>   */
> --
> 2.25.1

[PATCH v4 3/3] net/ixgbe: Fix SFP detection and linking on hotplug

[Jeff Daly]

Currently the ixgbe driver does not ID any SFP except for the first one
plugged in. This can lead to no-link, or incorrect speed conditions.

For example:

* If link is initially established with a 1G SFP, and later a 1G/10G
multispeed part is later installed, then the MAC link setup functions are
never called to change from 1000BASE-X to 10GBASE-R mode, and the link
stays running at the slower rate.

* If link is initially established with a 1G SFP, and later a 10G only
module is later installed, no link is established, since we are still
trasnsmitting in 1000BASE-X mode to a 10GBASE-R only partner.

Refactor the SFP ID/setup, and link setup code, to more closely match the
flow of the mainline kernel driver which does not have these issues.  In
that driver a service task runs periodically to handle these operations
based on bit flags that have been set (usually via interrupt or userspace
request), and then get cleared once the requested subtask has been
completed.

Fixes: af75078fece ("first public release")
Cc: stable@dpdk.org

Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 499 +++++++++++++++++++++++--------
 drivers/net/ixgbe/ixgbe_ethdev.h |  14 +-
 2 files changed, 392 insertions(+), 121 deletions(-)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index e8f07cb405..b70985bb1d 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -229,9 +229,6 @@ static int ixgbe_dev_interrupt_get_status(struct rte_eth_dev *dev);
 static int ixgbe_dev_interrupt_action(struct rte_eth_dev *dev);
 static void ixgbe_dev_interrupt_handler(void *param);
 static void ixgbe_dev_interrupt_delayed_handler(void *param);
-static void *ixgbe_dev_setup_link_thread_handler(void *param);
-static int ixgbe_dev_wait_setup_link_complete(struct rte_eth_dev *dev,
-					      uint32_t timeout_ms);
 
 static int ixgbe_add_rar(struct rte_eth_dev *dev,
 			struct rte_ether_addr *mac_addr,
@@ -766,6 +763,33 @@ static const struct rte_ixgbe_xstats_name_off rte_ixgbevf_stats_strings[] = {
 #define IXGBEVF_NB_XSTATS (sizeof(rte_ixgbevf_stats_strings) /	\
 		sizeof(rte_ixgbevf_stats_strings[0]))
 
+/**
+ * This function is the same as ixgbe_need_crosstalk_fix() in base/ixgbe_common.c
+ *
+ * ixgbe_need_crosstalk_fix - Determine if we need to do cross talk fix
+ * @hw: pointer to hardware structure
+ *
+ * Contains the logic to identify if we need to verify link for the
+ * crosstalk fix
+ **/
+static bool ixgbe_need_crosstalk_fix(struct ixgbe_hw *hw)
+{
+	/* Does FW say we need the fix */
+	if (!hw->need_crosstalk_fix)
+		return false;
+
+	/* Only consider SFP+ PHYs i.e. media type fiber */
+	switch (ixgbe_get_media_type(hw)) {
+	case ixgbe_media_type_fiber:
+	case ixgbe_media_type_fiber_qsfp:
+		break;
+	default:
+		return false;
+	}
+
+	return true;
+}
+
 /*
  * This function is the same as ixgbe_is_sfp() in base/ixgbe.h.
  */
@@ -1032,6 +1056,306 @@ ixgbe_swfw_lock_reset(struct ixgbe_hw *hw)
 	ixgbe_release_swfw_semaphore(hw, mask);
 }
 
+/**
+ * ixgbe_check_sfp_cage - Find present status of SFP module
+ * @hw: pointer to hardware structure
+ *
+ * Find if a SFP module is present and if this device supports SFPs
+ **/
+enum ixgbe_sfp_cage_status ixgbe_check_sfp_cage(struct ixgbe_hw *hw)
+{
+	enum ixgbe_sfp_cage_status sfp_cage_status;
+
+	/* If we're not a fiber/fiber_qsfp, no cage to check */
+	switch (hw->mac.ops.get_media_type(hw)) {
+	case ixgbe_media_type_fiber:
+	case ixgbe_media_type_fiber_qsfp:
+		break;
+	default:
+		return IXGBE_SFP_CAGE_NOCAGE;
+	}
+
+	switch (hw->mac.type) {
+	case ixgbe_mac_82599EB:
+		sfp_cage_status = !!(IXGBE_READ_REG(hw, IXGBE_ESDP) &
+			     IXGBE_ESDP_SDP2);
+		break;
+	case ixgbe_mac_X550EM_x:
+	case ixgbe_mac_X550EM_a:
+		/* SDP0 is the active low signal PRSNT#, so invert this */
+		sfp_cage_status = !(IXGBE_READ_REG(hw, IXGBE_ESDP) &
+				  IXGBE_ESDP_SDP0);
+		break;
+	default:
+		/* Don't know how to check this device type yet */
+		sfp_cage_status = IXGBE_SFP_CAGE_UNKNOWN;
+		DEBUGOUT("IXGBE_SFP_CAGE_UNKNOWN, unknown mac type %d\n",
+			 hw->mac.type);
+		break;
+	}
+
+	DEBUGOUT("sfp status %d for mac type %d\n", sfp_cage_status, hw->mac.type);
+	return sfp_cage_status;
+}
+
+static s32
+ixgbe_sfp_id_and_setup(struct rte_eth_dev *dev)
+{
+	struct ixgbe_hw *hw =
+		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	enum ixgbe_sfp_cage_status sfp_cage_status;
+	s32 err;
+
+	/* Can't ID or setup SFP if it's not plugged in */
+	sfp_cage_status = ixgbe_check_sfp_cage(hw);
+	if (sfp_cage_status == IXGBE_SFP_CAGE_EMPTY ||
+	    sfp_cage_status == IXGBE_SFP_CAGE_NOCAGE)
+		return IXGBE_ERR_SFP_NOT_PRESENT;
+
+	/* Something's in the cage, ID it */
+	hw->phy.ops.identify_sfp(hw);
+
+	/* Unknown module type, give up */
+	if (hw->phy.sfp_type == ixgbe_sfp_type_unknown) {
+		PMD_DRV_LOG(ERR, "unknown SFP type, giving up");
+		return IXGBE_ERR_SFP_NOT_SUPPORTED;
+	}
+
+	/* This should be a redundant check, since we looked at the
+	 * PRSNT# signal from the cage above, but just in case this is
+	 * an SFP that's slow to respond to I2C pokes correctly, try it
+	 * again later
+	 */
+	if (hw->phy.sfp_type == ixgbe_sfp_type_not_present) {
+		PMD_DRV_LOG(ERR, "IDed SFP as absent but cage PRSNT# active!?");
+		return IXGBE_ERR_SFP_NOT_PRESENT;
+	}
+
+	/* SFP is present and identified, try to set it up */
+	err = hw->mac.ops.setup_sfp(hw);
+	if (err)
+		PMD_DRV_LOG(ERR, "setup_sfp() failed %d", err);
+
+	return err;
+}
+
+static void
+ixgbe_sfp_service(struct rte_eth_dev *dev)
+{
+	struct ixgbe_hw *hw =
+		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	struct ixgbe_interrupt *intr =
+		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
+	enum ixgbe_sfp_cage_status sfp_cage_status;
+	s32 err;
+	u8 sff_id;
+	bool have_int = false;
+
+	/* If there's no module cage, then there's nothing to service */
+	sfp_cage_status = ixgbe_check_sfp_cage(hw);
+	if (sfp_cage_status == IXGBE_SFP_CAGE_NOCAGE) {
+		PMD_DRV_LOG(DEBUG, "No SFP to service\n");
+		return;
+	}
+
+	/* TODO - Even for platforms where ixgbe_check_sfp_cage() gives a clear
+	 * status result, if there's no interrupts, or no interrupt for the SFP
+	 * cage present pin, even if other interrupts exist, then we still need
+	 * to poll here to set the flag.
+	 */
+#ifndef RTE_EXEC_ENV_FREEBSD
+	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
+	if (rte_intr_allow_others(intr_handle)) {
+		/* check if lsc interrupt is enabled */
+		if (dev->data->dev_conf.intr_conf.lsc)
+			have_int = true;
+	}
+#endif /* #ifdef RTE_EXEC_ENV_FREEBSD */
+
+	if (!have_int && sfp_cage_status == IXGBE_SFP_CAGE_EMPTY) {
+		intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
+		PMD_DRV_LOG(DEBUG, "No SFP, no LSC, set NEED_SFP_SETUP\n");
+	}
+
+	/* For platforms that don't have a way to read the PRESENT# signal from
+	 * the SFP cage, fallback to doing an I2C read and seeing if it's ACKed
+	 * to determine if a module is present
+	 */
+	if (sfp_cage_status == IXGBE_SFP_CAGE_UNKNOWN) {
+		PMD_DRV_LOG(DEBUG,
+			    "SFP present unknown (int? %d), try I2C read\n",
+			    have_int);
+
+		/* Rather than calling identify_sfp, which will read a lot of I2C
+		 * registers (and in a slow processor intensive fashion due to
+		 * bit-banging, just read the SFF ID register, which is at a
+		 * common address across SFP/SFP+/QSFP modules and see if
+		 * there's a NACK.  This works since we only expect a NACK if no
+		 * module is present
+		 */
+		err = ixgbe_read_i2c_eeprom(hw, IXGBE_SFF_IDENTIFIER, &sff_id);
+		if (err != IXGBE_SUCCESS) {
+			PMD_DRV_LOG(DEBUG, "Received I2C NAK from SFP, set NEED_SFP_SETUP flag\n");
+			intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
+			sfp_cage_status = IXGBE_SFP_CAGE_EMPTY;
+		} else {
+			PMD_DRV_LOG(DEBUG, "SFP ID read ACKed");
+			sfp_cage_status = IXGBE_SFP_CAGE_FULL;
+		}
+	}
+
+	if (sfp_cage_status == IXGBE_SFP_CAGE_EMPTY) {
+		PMD_DRV_LOG(DEBUG, "SFP absent, cage_status %d\n", sfp_cage_status);
+		return;
+	}
+
+	/* No setup requested?	Nothing to do */
+	if (!(intr->flags & IXGBE_FLAG_NEED_SFP_SETUP))
+		return;
+
+	err = ixgbe_sfp_id_and_setup(dev);
+	if (err) {
+		PMD_DRV_LOG(DEBUG, "failed to ID & setup SFP %d", err);
+		return;
+	}
+
+	/* Setup is done, clear the flag, but make sure link config runs for new SFP */
+	intr->flags &= ~IXGBE_FLAG_NEED_SFP_SETUP;
+	intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
+
+	/*
+	 * Since this is a new SFP, clear the old advertised speed mask so we don't
+	 * end up using an old slower rate
+	 */
+	hw->phy.autoneg_advertised = 0;
+}
+
+static void
+ixgbe_link_service(struct rte_eth_dev *dev)
+{
+	struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	struct ixgbe_interrupt *intr =
+		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
+	bool link_up, autoneg = false, have_int = false;
+	u32 speed;
+	s32 err;
+
+	/* Test if we have a LSC interrupt for this platform, if not we need to
+	 * manually check the link register since IXGBE_FLAG_NEED_LINK_CONFIG
+	 * will never be set in the interrupt handler
+	 */
+#ifndef RTE_EXEC_ENV_FREEBSD
+	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
+	if (rte_intr_allow_others(intr_handle)) {
+		/* check if lsc interrupt is enabled */
+		if (dev->data->dev_conf.intr_conf.lsc)
+			have_int = true;
+	}
+#endif /* #ifdef RTE_EXEC_ENV_FREEBSD */
+
+	/* Skip if we still need to setup an SFP, or if no link config requested
+	 */
+	if ((intr->flags & IXGBE_FLAG_NEED_SFP_SETUP) ||
+	    (!(intr->flags & IXGBE_FLAG_NEED_LINK_CONFIG) && have_int))
+		return;
+
+	if (!have_int && !(intr->flags & IXGBE_FLAG_NEED_LINK_CONFIG)) {
+		err = ixgbe_check_link(hw, &speed, &link_up, 0);
+		if (!err && !link_up) {
+			intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
+			PMD_DRV_LOG(DEBUG, "Link down, no LSC, set NEED_LINK_CONFIG\n");
+		} else {
+			return;
+		}
+	}
+
+	speed = hw->phy.autoneg_advertised;
+	if (!speed)
+		ixgbe_get_link_capabilities(hw, &speed, &autoneg);
+
+	err = ixgbe_setup_link(hw, speed, true);
+	if (err) {
+		PMD_DRV_LOG(ERR, "ixgbe_setup_link failed %d", err);
+		return;
+	}
+
+	intr->flags &= ~IXGBE_FLAG_NEED_LINK_CONFIG;
+}
+
+static void
+ixgbe_link_update_service(struct rte_eth_dev *dev)
+{
+	/* Update internal link status, waiting for link */
+	if (!ixgbe_dev_link_update(dev, 0)) {
+		ixgbe_dev_link_status_print(dev);
+		rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_LSC, NULL);
+	}
+}
+
+/*
+ * Service task thread to handle periodic tasks
+ */
+static void *
+ixgbe_dev_service_thread_handler(void *param)
+{
+	struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
+	struct ixgbe_hw *hw =
+		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	uint64_t start, ticks, service_ms;
+	uint32_t speed;
+	s32 err;
+	bool link_up;
+
+	while (1) {
+		ixgbe_sfp_service(dev);
+		ixgbe_link_service(dev);
+		ixgbe_link_update_service(dev);
+
+		/* Run the service thread handler more frequently when link is
+		 * down to reduce link up latency (every 200ms vs 1s)
+		 *
+		 * Use a number of smaller sleeps to decrease exit latency when
+		 * ixgbe_dev_stop() wants this thread to join
+		 */
+		err = ixgbe_check_link(hw, &speed, &link_up, 0);
+		if (err == IXGBE_SUCCESS && link_up)
+			service_ms = 2000;
+		else
+			service_ms = 100;
+
+		/* Call msec_delay in a loop with several smaller sleeps to
+		 * provide periodic thread cancellation points
+		 */
+		start = rte_get_timer_cycles();
+		ticks = (uint64_t)service_ms * rte_get_timer_hz() / 1E3;
+		while ((rte_get_timer_cycles() - start) < ticks)
+			msec_delay(100);
+	}
+
+	/* Never return */
+	return NULL;
+}
+
+static s32
+eth_ixgbe_check_mac_link_generic(struct ixgbe_hw *hw, ixgbe_link_speed *speed,
+				 bool *link_up, bool link_up_wait_to_complete)
+{
+	if (ixgbe_need_crosstalk_fix(hw)) {
+		enum ixgbe_sfp_cage_status sfp_cage_status;
+
+		sfp_cage_status = ixgbe_check_sfp_cage(hw);
+		if (sfp_cage_status != IXGBE_SFP_CAGE_FULL) {
+			*link_up = false;
+			*speed = IXGBE_LINK_SPEED_UNKNOWN;
+			return IXGBE_SUCCESS;
+		}
+	}
+
+	return ixgbe_check_mac_link_generic(hw, speed, link_up, link_up_wait_to_complete);
+}
+
 /*
  * This function is based on code in ixgbe_attach() in base/ixgbe.c.
  * It returns 0 on success.
@@ -1054,6 +1378,7 @@ eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)
 		IXGBE_DEV_PRIVATE_TO_FILTER_INFO(eth_dev->data->dev_private);
 	struct ixgbe_bw_conf *bw_conf =
 		IXGBE_DEV_PRIVATE_TO_BW_CONF(eth_dev->data->dev_private);
+	struct ixgbe_mac_info *mac = &hw->mac;
 	uint32_t ctrl_ext;
 	uint16_t csum;
 	int diag, i, ret;
@@ -1124,6 +1449,17 @@ eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)
 		return -EIO;
 	}
 
+	/* override mac_link_check to check for sfp cage full/empty */
+	switch (hw->mac.type) {
+	case ixgbe_mac_X550EM_x:
+	case ixgbe_mac_X550EM_a:
+	case ixgbe_mac_82599EB:
+		mac->ops.check_link = eth_ixgbe_check_mac_link_generic;
+		break;
+	default:
+		break;
+	}
+
 	/* pick up the PCI bus settings for reporting later */
 	ixgbe_get_bus_info(hw);
 
@@ -2558,8 +2894,11 @@ ixgbe_flow_ctrl_enable(struct rte_eth_dev *dev, struct ixgbe_hw *hw)
 static int
 ixgbe_dev_start(struct rte_eth_dev *dev)
 {
+	struct ixgbe_adapter *ad = dev->data->dev_private;
 	struct ixgbe_hw *hw =
 		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	struct ixgbe_interrupt *intr =
+		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
 	struct ixgbe_vf_info *vfinfo =
 		*IXGBE_DEV_PRIVATE_TO_P_VFDATA(dev->data->dev_private);
 	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
@@ -2580,9 +2919,6 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
 
 	PMD_INIT_FUNC_TRACE();
 
-	/* Stop the link setup handler before resetting the HW. */
-	ixgbe_dev_wait_setup_link_complete(dev, 0);
-
 	/* disable uio/vfio intr/eventfd mapping */
 	rte_intr_disable(intr_handle);
 
@@ -2815,6 +3151,20 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
 	ixgbe_l2_tunnel_conf(dev);
 	ixgbe_filter_restore(dev);
 
+	/* Spawn service thread */
+	if (ixgbe_is_sfp(hw)) {
+		intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
+		err = rte_ctrl_thread_create(&ad->service_thread_tid,
+					     "ixgbe-service-thread",
+					     NULL,
+					     ixgbe_dev_service_thread_handler,
+					     dev);
+		if (err) {
+			PMD_DRV_LOG(ERR, "service_thread err");
+			goto error;
+		}
+	}
+
 	if (tm_conf->root && !tm_conf->committed)
 		PMD_DRV_LOG(WARNING,
 			    "please call hierarchy_commit() "
@@ -2860,13 +3210,21 @@ ixgbe_dev_stop(struct rte_eth_dev *dev)
 	int vf;
 	struct ixgbe_tm_conf *tm_conf =
 		IXGBE_DEV_PRIVATE_TO_TM_CONF(dev->data->dev_private);
+	void *res;
+	s32 err;
 
 	if (hw->adapter_stopped)
 		return 0;
 
 	PMD_INIT_FUNC_TRACE();
 
-	ixgbe_dev_wait_setup_link_complete(dev, 0);
+	/* Cancel the service thread, and wait for it to join */
+	err = pthread_cancel(adapter->service_thread_tid);
+	if (err)
+		PMD_DRV_LOG(ERR, "failed to cancel service thread %d", err);
+	err = pthread_join(adapter->service_thread_tid, &res);
+	if (err)
+		PMD_DRV_LOG(ERR, "failed to join service thread %d", err);
 
 	/* disable interrupts */
 	ixgbe_disable_intr(hw);
@@ -2945,7 +3303,6 @@ ixgbe_dev_set_link_up(struct rte_eth_dev *dev)
 	} else {
 		/* Turn on the laser */
 		ixgbe_enable_tx_laser(hw);
-		ixgbe_dev_link_update(dev, 0);
 	}
 
 	return 0;
@@ -2976,7 +3333,6 @@ ixgbe_dev_set_link_down(struct rte_eth_dev *dev)
 	} else {
 		/* Turn off the laser */
 		ixgbe_disable_tx_laser(hw);
-		ixgbe_dev_link_update(dev, 0);
 	}
 
 	return 0;
@@ -4128,57 +4484,6 @@ ixgbevf_check_link(struct ixgbe_hw *hw, ixgbe_link_speed *speed,
 	return ret_val;
 }
 
-/*
- * If @timeout_ms was 0, it means that it will not return until link complete.
- * It returns 1 on complete, return 0 on timeout.
- */
-static int
-ixgbe_dev_wait_setup_link_complete(struct rte_eth_dev *dev, uint32_t timeout_ms)
-{
-#define WARNING_TIMEOUT    9000 /* 9s  in total */
-	struct ixgbe_adapter *ad = dev->data->dev_private;
-	uint32_t timeout = timeout_ms ? timeout_ms : WARNING_TIMEOUT;
-
-	while (rte_atomic32_read(&ad->link_thread_running)) {
-		msec_delay(1);
-		timeout--;
-
-		if (timeout_ms) {
-			if (!timeout)
-				return 0;
-		} else if (!timeout) {
-			/* It will not return until link complete */
-			timeout = WARNING_TIMEOUT;
-			PMD_DRV_LOG(ERR, "IXGBE link thread not complete too long time!");
-		}
-	}
-
-	return 1;
-}
-
-static void *
-ixgbe_dev_setup_link_thread_handler(void *param)
-{
-	struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
-	struct ixgbe_adapter *ad = dev->data->dev_private;
-	struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
-	struct ixgbe_interrupt *intr =
-		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
-	u32 speed;
-	bool autoneg = false;
-
-	pthread_detach(pthread_self());
-	speed = hw->phy.autoneg_advertised;
-	if (!speed)
-		ixgbe_get_link_capabilities(hw, &speed, &autoneg);
-
-	ixgbe_setup_link(hw, speed, true);
-
-	intr->flags &= ~IXGBE_FLAG_NEED_LINK_CONFIG;
-	rte_atomic32_clear(&ad->link_thread_running);
-	return NULL;
-}
-
 /*
  * In freebsd environment, nic_uio drivers do not support interrupts,
  * rte_intr_callback_register() will fail to register interrupts.
@@ -4218,11 +4523,8 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 			    int wait_to_complete, int vf)
 {
 	struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
-	struct ixgbe_adapter *ad = dev->data->dev_private;
 	struct rte_eth_link link;
 	ixgbe_link_speed link_speed = IXGBE_LINK_SPEED_UNKNOWN;
-	struct ixgbe_interrupt *intr =
-		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
 	bool link_up;
 	int diag;
 	int wait = 1;
@@ -4237,9 +4539,6 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 
 	hw->mac.get_link_status = true;
 
-	if (intr->flags & IXGBE_FLAG_NEED_LINK_CONFIG)
-		return rte_eth_linkstatus_set(dev, &link);
-
 	/* check if it needs to wait to complete, if lsc interrupt is enabled */
 	if (wait_to_complete == 0 || dev->data->dev_conf.intr_conf.lsc != 0)
 		wait = 0;
@@ -4254,7 +4553,7 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 	else
 		diag = ixgbe_check_link(hw, &link_speed, &link_up, wait);
 
-	if (diag != 0) {
+	if (diag != 0 || !link_up) {
 		link.link_speed = RTE_ETH_SPEED_NUM_100M;
 		link.link_duplex = RTE_ETH_LINK_FULL_DUPLEX;
 		return rte_eth_linkstatus_set(dev, &link);
@@ -4267,32 +4566,6 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 			link_up = 0;
 	}
 
-	if (link_up == 0) {
-		if (ixgbe_get_media_type(hw) == ixgbe_media_type_fiber) {
-			ixgbe_dev_wait_setup_link_complete(dev, 0);
-			if (rte_atomic32_test_and_set(&ad->link_thread_running)) {
-				/* To avoid race condition between threads, set
-				 * the IXGBE_FLAG_NEED_LINK_CONFIG flag only
-				 * when there is no link thread running.
-				 */
-				intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
-				if (rte_ctrl_thread_create(&ad->link_thread_tid,
-					"ixgbe-link-handler",
-					NULL,
-					ixgbe_dev_setup_link_thread_handler,
-					dev) < 0) {
-					PMD_DRV_LOG(ERR,
-						"Create link thread failed!");
-					rte_atomic32_clear(&ad->link_thread_running);
-				}
-			} else {
-				PMD_DRV_LOG(ERR,
-					"Other link thread is running now!");
-			}
-		}
-		return rte_eth_linkstatus_set(dev, &link);
-	}
-
 	link.link_status = RTE_ETH_LINK_UP;
 	link.link_duplex = RTE_ETH_LINK_FULL_DUPLEX;
 
@@ -4498,8 +4771,6 @@ ixgbe_dev_interrupt_get_status(struct rte_eth_dev *dev)
 	eicr = IXGBE_READ_REG(hw, IXGBE_EICR);
 	PMD_DRV_LOG(DEBUG, "eicr %x", eicr);
 
-	intr->flags = 0;
-
 	/* set flag for async link update */
 	if (eicr & IXGBE_EICR_LSC)
 		intr->flags |= IXGBE_FLAG_NEED_LINK_UPDATE;
@@ -4515,6 +4786,14 @@ ixgbe_dev_interrupt_get_status(struct rte_eth_dev *dev)
 	    (eicr & IXGBE_EICR_GPI_SDP0_X550EM_x))
 		intr->flags |= IXGBE_FLAG_PHY_INTERRUPT;
 
+	/* Check for loss of SFP */
+	/* TODO - For platforms that don't have this flag, do we need to set
+	 *  NEED_SFP_SETUP on LSC if we're a SFP platform?
+	 */
+	if (hw->mac.type ==  ixgbe_mac_X550EM_a &&
+	    (eicr & IXGBE_EICR_GPI_SDP0_X550EM_a))
+		intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
+
 	return 0;
 }
 
@@ -4566,11 +4845,13 @@ ixgbe_dev_link_status_print(struct rte_eth_dev *dev)
 static int
 ixgbe_dev_interrupt_action(struct rte_eth_dev *dev)
 {
+	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
 	struct ixgbe_interrupt *intr =
 		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
-	int64_t timeout;
 	struct ixgbe_hw *hw =
 		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	int64_t timeout;
 
 	PMD_DRV_LOG(DEBUG, "intr action type %d", intr->flags);
 
@@ -4605,16 +4886,14 @@ ixgbe_dev_interrupt_action(struct rte_eth_dev *dev)
 		if (rte_eal_alarm_set(timeout * 1000,
 				      ixgbe_dev_interrupt_delayed_handler, (void *)dev) < 0)
 			PMD_DRV_LOG(ERR, "Error setting alarm");
-		else {
-			/* remember original mask */
-			intr->mask_original = intr->mask;
+		else
 			/* only disable lsc interrupt */
 			intr->mask &= ~IXGBE_EIMS_LSC;
-		}
 	}
 
 	PMD_DRV_LOG(DEBUG, "enable intr immediately");
 	ixgbe_enable_intr(dev);
+	rte_intr_ack(intr_handle);
 
 	return 0;
 }
@@ -4637,8 +4916,6 @@ static void
 ixgbe_dev_interrupt_delayed_handler(void *param)
 {
 	struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
-	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
-	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
 	struct ixgbe_interrupt *intr =
 		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
 	struct ixgbe_hw *hw =
@@ -4668,13 +4945,10 @@ ixgbe_dev_interrupt_delayed_handler(void *param)
 		intr->flags &= ~IXGBE_FLAG_MACSEC;
 	}
 
-	/* restore original mask */
-	intr->mask = intr->mask_original;
-	intr->mask_original = 0;
+	if (dev->data->dev_conf.intr_conf.lsc != 0)
+		intr->mask |= IXGBE_EICR_LSC;
 
-	PMD_DRV_LOG(DEBUG, "enable intr in delayed handler S[%08x]", eicr);
 	ixgbe_enable_intr(dev);
-	rte_intr_ack(intr_handle);
 }
 
 /**
@@ -5316,9 +5590,6 @@ ixgbevf_dev_start(struct rte_eth_dev *dev)
 
 	PMD_INIT_FUNC_TRACE();
 
-	/* Stop the link setup handler before resetting the HW. */
-	ixgbe_dev_wait_setup_link_complete(dev, 0);
-
 	err = hw->mac.ops.reset_hw(hw);
 
 	/**
@@ -5398,12 +5669,6 @@ ixgbevf_dev_start(struct rte_eth_dev *dev)
 	/* Re-enable interrupt for VF */
 	ixgbevf_intr_enable(dev);
 
-	/*
-	 * Update link status right before return, because it may
-	 * start link configuration process in a separate thread.
-	 */
-	ixgbevf_dev_link_update(dev, 0);
-
 	hw->adapter_stopped = false;
 
 	return 0;
@@ -5422,8 +5687,6 @@ ixgbevf_dev_stop(struct rte_eth_dev *dev)
 
 	PMD_INIT_FUNC_TRACE();
 
-	ixgbe_dev_wait_setup_link_complete(dev, 0);
-
 	ixgbevf_intr_disable(dev);
 
 	dev->data->dev_started = 0;
diff --git a/drivers/net/ixgbe/ixgbe_ethdev.h b/drivers/net/ixgbe/ixgbe_ethdev.h
index 69e0e82a5b..d243e417e9 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.h
+++ b/drivers/net/ixgbe/ixgbe_ethdev.h
@@ -29,6 +29,7 @@
 #define IXGBE_FLAG_PHY_INTERRUPT    (uint32_t)(1 << 2)
 #define IXGBE_FLAG_MACSEC           (uint32_t)(1 << 3)
 #define IXGBE_FLAG_NEED_LINK_CONFIG (uint32_t)(1 << 4)
+#define IXGBE_FLAG_NEED_SFP_SETUP   ((uint32_t)(1 << 5))
 
 /*
  * Defines that were not part of ixgbe_type.h as they are not used by the
@@ -223,8 +224,6 @@ struct ixgbe_rte_flow_rss_conf {
 struct ixgbe_interrupt {
 	uint32_t flags;
 	uint32_t mask;
-	/*to save original mask during delayed handler */
-	uint32_t mask_original;
 };
 
 struct ixgbe_stat_mapping_registers {
@@ -507,7 +506,7 @@ struct ixgbe_adapter {
 	uint8_t pflink_fullchk;
 	uint8_t mac_ctrl_frame_fwd;
 	rte_atomic32_t link_thread_running;
-	pthread_t link_thread_tid;
+	pthread_t service_thread_tid;
 };
 
 struct ixgbe_vf_representor {
@@ -670,6 +669,15 @@ int ixgbe_syn_filter_set(struct rte_eth_dev *dev,
 			struct rte_eth_syn_filter *filter,
 			bool add);
 
+enum ixgbe_sfp_cage_status {
+	IXGBE_SFP_CAGE_EMPTY = 0,
+	IXGBE_SFP_CAGE_FULL,
+	IXGBE_SFP_CAGE_UNKNOWN = -1,
+	IXGBE_SFP_CAGE_NOCAGE = -2,
+};
+enum ixgbe_sfp_cage_status ixgbe_check_sfp_cage(struct ixgbe_hw *hw);
+
+
 /**
  * l2 tunnel configuration.
  */
-- 
2.25.1

Re: [PATCH v4 3/3] net/ixgbe: Fix SFP detection and linking on hotplug

[Ferruh Yigit]

On 2/25/2022 8:50 PM, Jeff Daly wrote:
> Currently the ixgbe driver does not ID any SFP except for the first one
> plugged in. This can lead to no-link, or incorrect speed conditions.
> 
> For example:
> 
> * If link is initially established with a 1G SFP, and later a 1G/10G
> multispeed part is later installed, then the MAC link setup functions are
> never called to change from 1000BASE-X to 10GBASE-R mode, and the link
> stays running at the slower rate.
> 
> * If link is initially established with a 1G SFP, and later a 10G only
> module is later installed, no link is established, since we are still
> trasnsmitting in 1000BASE-X mode to a 10GBASE-R only partner.
> 
> Refactor the SFP ID/setup, and link setup code, to more closely match the
> flow of the mainline kernel driver which does not have these issues.  In
> that driver a service task runs periodically to handle these operations
> based on bit flags that have been set (usually via interrupt or userspace
> request), and then get cleared once the requested subtask has been
> completed.
> 
> Fixes: af75078fece ("first public release")
> Cc:stable@dpdk.org
> 
> Signed-off-by: Stephen Douthit<stephend@silicom-usa.com>
> Signed-off-by: Jeff Daly<jeffd@silicom-usa.com>

Hi Jeff,

Can you please send new version of whole set, instead of only
one of the patches?
It is very hard to manage different versions of individual
patches in a set.

[PATCH v4 1/3] net/ixgbe: Fix ixgbe_is_sfp() to return valid result for X550EM_a devs

[Jeff Daly]

From: Stephen Douthit <stephend@silicom-usa.com>

Currently all X500EM* MAC types fallthrough to the default case and get
reported as non-SFP regardless of media type, which isn't correct.

Fixes: 0790adeb567 ("ixgbe/base: support X550em_a device")
Cc: stable@dpdk.org

Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 3d799d2187..68b28b1ce6 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -781,6 +781,20 @@ ixgbe_is_sfp(struct ixgbe_hw *hw)
 	case ixgbe_phy_sfp_passive_unknown:
 		return 1;
 	default:
+		/* x550em devices may be SFP, check media type */
+		switch (hw->mac.type) {
+		case ixgbe_mac_X550EM_x:
+		case ixgbe_mac_X550EM_a:
+			switch (ixgbe_get_media_type(hw)) {
+			case ixgbe_media_type_fiber:
+			case ixgbe_media_type_fiber_qsfp:
+				return 1;
+			default:
+				break;
+			}
+		default:
+			break;
+		}
 		return 0;
 	}
 }
-- 
2.25.1

RE: [PATCH v4 1/3] net/ixgbe: Fix ixgbe_is_sfp() to return valid result for X550EM_a devs

[Wang, Haiyue]

> -----Original Message-----
> From: Jeff Daly <jeffd@silicom-usa.com>
> Sent: Monday, February 28, 2022 23:30
> To: dev@dpdk.org
> Cc: Stephen Douthit <stephend@silicom-usa.com>; stable@dpdk.org; Wang, Haiyue <haiyue.wang@intel.com>;
> Lu, Wenzhuo <wenzhuo.lu@intel.com>
> Subject: [PATCH v4 1/3] net/ixgbe: Fix ixgbe_is_sfp() to return valid result for X550EM_a devs
> 
> From: Stephen Douthit <stephend@silicom-usa.com>
> 
> Currently all X500EM* MAC types fallthrough to the default case and get
> reported as non-SFP regardless of media type, which isn't correct.
> 
> Fixes: 0790adeb567 ("ixgbe/base: support X550em_a device")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
> Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> ---
>  drivers/net/ixgbe/ixgbe_ethdev.c | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
> 

Acked-by: Haiyue Wang <haiyue.wang@intel.com>

> --
> 2.25.1

RE: [PATCH v4 1/3] net/ixgbe: Fix ixgbe_is_sfp() to return valid result for X550EM_a devs

[Zhang, Qi Z]

> -----Original Message-----
> From: Wang, Haiyue <haiyue.wang@intel.com>
> Sent: Tuesday, March 1, 2022 1:57 PM
> To: Daly, Jeff <jeffd@silicom-usa.com>; dev@dpdk.org
> Cc: Stephen Douthit <stephend@silicom-usa.com>; stable@dpdk.org; Lu,
> Wenzhuo <wenzhuo.lu@intel.com>
> Subject: RE: [PATCH v4 1/3] net/ixgbe: Fix ixgbe_is_sfp() to return valid result
> for X550EM_a devs

> 
> > -----Original Message-----
> > From: Jeff Daly <jeffd@silicom-usa.com>
> > Sent: Monday, February 28, 2022 23:30
> > To: dev@dpdk.org
> > Cc: Stephen Douthit <stephend@silicom-usa.com>; stable@dpdk.org;
> Wang,
> > Haiyue <haiyue.wang@intel.com>; Lu, Wenzhuo <wenzhuo.lu@intel.com>
> > Subject: [PATCH v4 1/3] net/ixgbe: Fix ixgbe_is_sfp() to return valid
> > result for X550EM_a devs
> >
> > From: Stephen Douthit <stephend@silicom-usa.com>
> >
> > Currently all X500EM* MAC types fallthrough to the default case and
> > get reported as non-SFP regardless of media type, which isn't correct.
> >
> > Fixes: 0790adeb567 ("ixgbe/base: support X550em_a device")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
> > Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> > ---
> >  drivers/net/ixgbe/ixgbe_ethdev.c | 14 ++++++++++++++
> >  1 file changed, 14 insertions(+)
> >
> 
> Acked-by: Haiyue Wang <haiyue.wang@intel.com>

Applied to dpdk-next-net-intel after renaming the title to "fix FSP check for X550EM devices" to fix check-git-log error.

Thanks
Qi
> 
> > --
> > 2.25.1

Re: [PATCH v4 1/3] net/ixgbe: Fix ixgbe_is_sfp() to return valid result for X550EM_a devs

[Thomas Monjalon]

01/03/2022 12:18, Zhang, Qi Z:
> From: Wang, Haiyue <haiyue.wang@intel.com>
> > From: Jeff Daly <jeffd@silicom-usa.com>
> > > From: Stephen Douthit <stephend@silicom-usa.com>
> > >
> > > Currently all X500EM* MAC types fallthrough to the default case and
> > > get reported as non-SFP regardless of media type, which isn't correct.
> > >
> > > Fixes: 0790adeb567 ("ixgbe/base: support X550em_a device")
> > > Cc: stable@dpdk.org
> > >
> > > Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
> > > Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> > 
> > Acked-by: Haiyue Wang <haiyue.wang@intel.com>
> 
> Applied to dpdk-next-net-intel after renaming the title to "fix FSP check for X550EM devices" to fix check-git-log error.

It seems you have applied only the first patch of the series. Why?
Is there a good reason to split a series without any justification?
What about the two other patches?

RE: [PATCH v4 1/3] net/ixgbe: Fix ixgbe_is_sfp() to return valid result for X550EM_a devs

[Jeff Daly]

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Sunday, March 6, 2022 12:56 PM
> To: Wang, Haiyue <haiyue.wang@intel.com>; Zhang, Qi Z
> <qi.z.zhang@intel.com>; ferruh.yigit@intel.com
> Cc: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org; stable@dpdk.org;
> Stephen Douthit <stephend@silicom-usa.com>; Lu, Wenzhuo
> <wenzhuo.lu@intel.com>
> Subject: Re: [PATCH v4 1/3] net/ixgbe: Fix ixgbe_is_sfp() to return valid result for
> X550EM_a devs
> 
> Caution: This is an external email. Please take care when clicking links or
> opening attachments.
> 
> 
> 01/03/2022 12:18, Zhang, Qi Z:
> > From: Wang, Haiyue <haiyue.wang@intel.com>
> > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > > From: Stephen Douthit <stephend@silicom-usa.com>
> > > >
> > > > Currently all X500EM* MAC types fallthrough to the default case
> > > > and get reported as non-SFP regardless of media type, which isn't correct.
> > > >
> > > > Fixes: 0790adeb567 ("ixgbe/base: support X550em_a device")
> > > > Cc: stable@dpdk.org
> > > >
> > > > Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
> > > > Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> > >
> > > Acked-by: Haiyue Wang <haiyue.wang@intel.com>
> >
> > Applied to dpdk-next-net-intel after renaming the title to "fix FSP check for
> X550EM devices" to fix check-git-log error.
> 
> It seems you have applied only the first patch of the series. Why?
> Is there a good reason to split a series without any justification?
> What about the two other patches?
> 

I should explain the [PATCH v4 x/y] and why it changed from 7 patches to 3 initially.....     In Stephen's initial v1 of the patch series there were 7 patches and during the discussion of the v2 version before Stephen left he intended to break it into 3 separate submissions.  After speaking with Ekinops/Swisscom regarding the best way to move these forward it was decided to move some functionality back to ethdev.c.  So currently:  [PATCH v4 1/3] is basically [PATCH v2 1/7], [PATCH v2 2/7] and [PATCH v2 5/7] were reworked and included in [PATCH v4 3/3] .  [PATCH v2 4/7] became [PATCH v4 2/3].   I submitted 2 separate patches yesterday for what was [PATCH v2 6/7] and [PATCH v2 7/7] since they were small patches to base/ rather than ethdev.   Finally, [PATCH v2 3/7] i'm working out whether it can be reworked into something that only touches ethdev (could be more kludgey) or really kinda just fits into base as originally submitted, like the last 2 patches I submitted for special handling of some SFPs.

I discovered the ixgbe_api functionality the other day while looking into this, and was thinking perhaps it could be somehow done there.  Still under base, but not touching the main driver files.   Is there any documentation that describes the ixgbe_api usage or process to get a wrapper call added to it?

[PATCH v4 2/3] net/ixgbe: Limit SDP3 check of TX_DISABLE to appropriate devices

[Jeff Daly]

1ca05831b9b added a check that SDP3 (used as a TX_DISABLE output to the
SFP cage on these cards) is not asserted to avoid incorrectly reporting
link up when the SFP's laser is turned off.

ff8162cb957 limited this workaround to fiber ports

This patch:
* Adds additional check for 82599 type, not all fiber ixgbe devs use
  SDP3 as TX_DISABLE

Fixes: 1ca05831b9b ("net/ixgbe: fix link status")
Fixes: ff8162cb957 ("net/ixgbe: fix link status")
Cc: stable@dpdk.org

Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 68b28b1ce6..e8f07cb405 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -4260,7 +4260,8 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 		return rte_eth_linkstatus_set(dev, &link);
 	}
 
-	if (ixgbe_get_media_type(hw) == ixgbe_media_type_fiber) {
+	if (hw->mac.type == ixgbe_mac_82599EB &&
+		ixgbe_get_media_type(hw) == ixgbe_media_type_fiber) {
 		esdp_reg = IXGBE_READ_REG(hw, IXGBE_ESDP);
 		if ((esdp_reg & IXGBE_ESDP_SDP3))
 			link_up = 0;
-- 
2.25.1

[PATCH v4 3/3] net/ixgbe: Fix SFP detection and linking on hotplug

[Jeff Daly]

Currently the ixgbe driver does not ID any SFP except for the first one
plugged in. This can lead to no-link, or incorrect speed conditions.

For example:

* If link is initially established with a 1G SFP, and later a 1G/10G
multispeed part is later installed, then the MAC link setup functions are
never called to change from 1000BASE-X to 10GBASE-R mode, and the link
stays running at the slower rate.

* If link is initially established with a 1G SFP, and later a 10G only
module is later installed, no link is established, since we are still
trasnsmitting in 1000BASE-X mode to a 10GBASE-R only partner.

Refactor the SFP ID/setup, and link setup code, to more closely match the
flow of the mainline kernel driver which does not have these issues.  In
that driver a service task runs periodically to handle these operations
based on bit flags that have been set (usually via interrupt or userspace
request), and then get cleared once the requested subtask has been
completed.

Fixes: af75078fece ("first public release")
Cc: stable@dpdk.org

Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 499 +++++++++++++++++++++++--------
 drivers/net/ixgbe/ixgbe_ethdev.h |  14 +-
 2 files changed, 392 insertions(+), 121 deletions(-)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index e8f07cb405..b70985bb1d 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -229,9 +229,6 @@ static int ixgbe_dev_interrupt_get_status(struct rte_eth_dev *dev);
 static int ixgbe_dev_interrupt_action(struct rte_eth_dev *dev);
 static void ixgbe_dev_interrupt_handler(void *param);
 static void ixgbe_dev_interrupt_delayed_handler(void *param);
-static void *ixgbe_dev_setup_link_thread_handler(void *param);
-static int ixgbe_dev_wait_setup_link_complete(struct rte_eth_dev *dev,
-					      uint32_t timeout_ms);
 
 static int ixgbe_add_rar(struct rte_eth_dev *dev,
 			struct rte_ether_addr *mac_addr,
@@ -766,6 +763,33 @@ static const struct rte_ixgbe_xstats_name_off rte_ixgbevf_stats_strings[] = {
 #define IXGBEVF_NB_XSTATS (sizeof(rte_ixgbevf_stats_strings) /	\
 		sizeof(rte_ixgbevf_stats_strings[0]))
 
+/**
+ * This function is the same as ixgbe_need_crosstalk_fix() in base/ixgbe_common.c
+ *
+ * ixgbe_need_crosstalk_fix - Determine if we need to do cross talk fix
+ * @hw: pointer to hardware structure
+ *
+ * Contains the logic to identify if we need to verify link for the
+ * crosstalk fix
+ **/
+static bool ixgbe_need_crosstalk_fix(struct ixgbe_hw *hw)
+{
+	/* Does FW say we need the fix */
+	if (!hw->need_crosstalk_fix)
+		return false;
+
+	/* Only consider SFP+ PHYs i.e. media type fiber */
+	switch (ixgbe_get_media_type(hw)) {
+	case ixgbe_media_type_fiber:
+	case ixgbe_media_type_fiber_qsfp:
+		break;
+	default:
+		return false;
+	}
+
+	return true;
+}
+
 /*
  * This function is the same as ixgbe_is_sfp() in base/ixgbe.h.
  */
@@ -1032,6 +1056,306 @@ ixgbe_swfw_lock_reset(struct ixgbe_hw *hw)
 	ixgbe_release_swfw_semaphore(hw, mask);
 }
 
+/**
+ * ixgbe_check_sfp_cage - Find present status of SFP module
+ * @hw: pointer to hardware structure
+ *
+ * Find if a SFP module is present and if this device supports SFPs
+ **/
+enum ixgbe_sfp_cage_status ixgbe_check_sfp_cage(struct ixgbe_hw *hw)
+{
+	enum ixgbe_sfp_cage_status sfp_cage_status;
+
+	/* If we're not a fiber/fiber_qsfp, no cage to check */
+	switch (hw->mac.ops.get_media_type(hw)) {
+	case ixgbe_media_type_fiber:
+	case ixgbe_media_type_fiber_qsfp:
+		break;
+	default:
+		return IXGBE_SFP_CAGE_NOCAGE;
+	}
+
+	switch (hw->mac.type) {
+	case ixgbe_mac_82599EB:
+		sfp_cage_status = !!(IXGBE_READ_REG(hw, IXGBE_ESDP) &
+			     IXGBE_ESDP_SDP2);
+		break;
+	case ixgbe_mac_X550EM_x:
+	case ixgbe_mac_X550EM_a:
+		/* SDP0 is the active low signal PRSNT#, so invert this */
+		sfp_cage_status = !(IXGBE_READ_REG(hw, IXGBE_ESDP) &
+				  IXGBE_ESDP_SDP0);
+		break;
+	default:
+		/* Don't know how to check this device type yet */
+		sfp_cage_status = IXGBE_SFP_CAGE_UNKNOWN;
+		DEBUGOUT("IXGBE_SFP_CAGE_UNKNOWN, unknown mac type %d\n",
+			 hw->mac.type);
+		break;
+	}
+
+	DEBUGOUT("sfp status %d for mac type %d\n", sfp_cage_status, hw->mac.type);
+	return sfp_cage_status;
+}
+
+static s32
+ixgbe_sfp_id_and_setup(struct rte_eth_dev *dev)
+{
+	struct ixgbe_hw *hw =
+		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	enum ixgbe_sfp_cage_status sfp_cage_status;
+	s32 err;
+
+	/* Can't ID or setup SFP if it's not plugged in */
+	sfp_cage_status = ixgbe_check_sfp_cage(hw);
+	if (sfp_cage_status == IXGBE_SFP_CAGE_EMPTY ||
+	    sfp_cage_status == IXGBE_SFP_CAGE_NOCAGE)
+		return IXGBE_ERR_SFP_NOT_PRESENT;
+
+	/* Something's in the cage, ID it */
+	hw->phy.ops.identify_sfp(hw);
+
+	/* Unknown module type, give up */
+	if (hw->phy.sfp_type == ixgbe_sfp_type_unknown) {
+		PMD_DRV_LOG(ERR, "unknown SFP type, giving up");
+		return IXGBE_ERR_SFP_NOT_SUPPORTED;
+	}
+
+	/* This should be a redundant check, since we looked at the
+	 * PRSNT# signal from the cage above, but just in case this is
+	 * an SFP that's slow to respond to I2C pokes correctly, try it
+	 * again later
+	 */
+	if (hw->phy.sfp_type == ixgbe_sfp_type_not_present) {
+		PMD_DRV_LOG(ERR, "IDed SFP as absent but cage PRSNT# active!?");
+		return IXGBE_ERR_SFP_NOT_PRESENT;
+	}
+
+	/* SFP is present and identified, try to set it up */
+	err = hw->mac.ops.setup_sfp(hw);
+	if (err)
+		PMD_DRV_LOG(ERR, "setup_sfp() failed %d", err);
+
+	return err;
+}
+
+static void
+ixgbe_sfp_service(struct rte_eth_dev *dev)
+{
+	struct ixgbe_hw *hw =
+		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	struct ixgbe_interrupt *intr =
+		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
+	enum ixgbe_sfp_cage_status sfp_cage_status;
+	s32 err;
+	u8 sff_id;
+	bool have_int = false;
+
+	/* If there's no module cage, then there's nothing to service */
+	sfp_cage_status = ixgbe_check_sfp_cage(hw);
+	if (sfp_cage_status == IXGBE_SFP_CAGE_NOCAGE) {
+		PMD_DRV_LOG(DEBUG, "No SFP to service\n");
+		return;
+	}
+
+	/* TODO - Even for platforms where ixgbe_check_sfp_cage() gives a clear
+	 * status result, if there's no interrupts, or no interrupt for the SFP
+	 * cage present pin, even if other interrupts exist, then we still need
+	 * to poll here to set the flag.
+	 */
+#ifndef RTE_EXEC_ENV_FREEBSD
+	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
+	if (rte_intr_allow_others(intr_handle)) {
+		/* check if lsc interrupt is enabled */
+		if (dev->data->dev_conf.intr_conf.lsc)
+			have_int = true;
+	}
+#endif /* #ifdef RTE_EXEC_ENV_FREEBSD */
+
+	if (!have_int && sfp_cage_status == IXGBE_SFP_CAGE_EMPTY) {
+		intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
+		PMD_DRV_LOG(DEBUG, "No SFP, no LSC, set NEED_SFP_SETUP\n");
+	}
+
+	/* For platforms that don't have a way to read the PRESENT# signal from
+	 * the SFP cage, fallback to doing an I2C read and seeing if it's ACKed
+	 * to determine if a module is present
+	 */
+	if (sfp_cage_status == IXGBE_SFP_CAGE_UNKNOWN) {
+		PMD_DRV_LOG(DEBUG,
+			    "SFP present unknown (int? %d), try I2C read\n",
+			    have_int);
+
+		/* Rather than calling identify_sfp, which will read a lot of I2C
+		 * registers (and in a slow processor intensive fashion due to
+		 * bit-banging, just read the SFF ID register, which is at a
+		 * common address across SFP/SFP+/QSFP modules and see if
+		 * there's a NACK.  This works since we only expect a NACK if no
+		 * module is present
+		 */
+		err = ixgbe_read_i2c_eeprom(hw, IXGBE_SFF_IDENTIFIER, &sff_id);
+		if (err != IXGBE_SUCCESS) {
+			PMD_DRV_LOG(DEBUG, "Received I2C NAK from SFP, set NEED_SFP_SETUP flag\n");
+			intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
+			sfp_cage_status = IXGBE_SFP_CAGE_EMPTY;
+		} else {
+			PMD_DRV_LOG(DEBUG, "SFP ID read ACKed");
+			sfp_cage_status = IXGBE_SFP_CAGE_FULL;
+		}
+	}
+
+	if (sfp_cage_status == IXGBE_SFP_CAGE_EMPTY) {
+		PMD_DRV_LOG(DEBUG, "SFP absent, cage_status %d\n", sfp_cage_status);
+		return;
+	}
+
+	/* No setup requested?	Nothing to do */
+	if (!(intr->flags & IXGBE_FLAG_NEED_SFP_SETUP))
+		return;
+
+	err = ixgbe_sfp_id_and_setup(dev);
+	if (err) {
+		PMD_DRV_LOG(DEBUG, "failed to ID & setup SFP %d", err);
+		return;
+	}
+
+	/* Setup is done, clear the flag, but make sure link config runs for new SFP */
+	intr->flags &= ~IXGBE_FLAG_NEED_SFP_SETUP;
+	intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
+
+	/*
+	 * Since this is a new SFP, clear the old advertised speed mask so we don't
+	 * end up using an old slower rate
+	 */
+	hw->phy.autoneg_advertised = 0;
+}
+
+static void
+ixgbe_link_service(struct rte_eth_dev *dev)
+{
+	struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	struct ixgbe_interrupt *intr =
+		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
+	bool link_up, autoneg = false, have_int = false;
+	u32 speed;
+	s32 err;
+
+	/* Test if we have a LSC interrupt for this platform, if not we need to
+	 * manually check the link register since IXGBE_FLAG_NEED_LINK_CONFIG
+	 * will never be set in the interrupt handler
+	 */
+#ifndef RTE_EXEC_ENV_FREEBSD
+	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
+	if (rte_intr_allow_others(intr_handle)) {
+		/* check if lsc interrupt is enabled */
+		if (dev->data->dev_conf.intr_conf.lsc)
+			have_int = true;
+	}
+#endif /* #ifdef RTE_EXEC_ENV_FREEBSD */
+
+	/* Skip if we still need to setup an SFP, or if no link config requested
+	 */
+	if ((intr->flags & IXGBE_FLAG_NEED_SFP_SETUP) ||
+	    (!(intr->flags & IXGBE_FLAG_NEED_LINK_CONFIG) && have_int))
+		return;
+
+	if (!have_int && !(intr->flags & IXGBE_FLAG_NEED_LINK_CONFIG)) {
+		err = ixgbe_check_link(hw, &speed, &link_up, 0);
+		if (!err && !link_up) {
+			intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
+			PMD_DRV_LOG(DEBUG, "Link down, no LSC, set NEED_LINK_CONFIG\n");
+		} else {
+			return;
+		}
+	}
+
+	speed = hw->phy.autoneg_advertised;
+	if (!speed)
+		ixgbe_get_link_capabilities(hw, &speed, &autoneg);
+
+	err = ixgbe_setup_link(hw, speed, true);
+	if (err) {
+		PMD_DRV_LOG(ERR, "ixgbe_setup_link failed %d", err);
+		return;
+	}
+
+	intr->flags &= ~IXGBE_FLAG_NEED_LINK_CONFIG;
+}
+
+static void
+ixgbe_link_update_service(struct rte_eth_dev *dev)
+{
+	/* Update internal link status, waiting for link */
+	if (!ixgbe_dev_link_update(dev, 0)) {
+		ixgbe_dev_link_status_print(dev);
+		rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_LSC, NULL);
+	}
+}
+
+/*
+ * Service task thread to handle periodic tasks
+ */
+static void *
+ixgbe_dev_service_thread_handler(void *param)
+{
+	struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
+	struct ixgbe_hw *hw =
+		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	uint64_t start, ticks, service_ms;
+	uint32_t speed;
+	s32 err;
+	bool link_up;
+
+	while (1) {
+		ixgbe_sfp_service(dev);
+		ixgbe_link_service(dev);
+		ixgbe_link_update_service(dev);
+
+		/* Run the service thread handler more frequently when link is
+		 * down to reduce link up latency (every 200ms vs 1s)
+		 *
+		 * Use a number of smaller sleeps to decrease exit latency when
+		 * ixgbe_dev_stop() wants this thread to join
+		 */
+		err = ixgbe_check_link(hw, &speed, &link_up, 0);
+		if (err == IXGBE_SUCCESS && link_up)
+			service_ms = 2000;
+		else
+			service_ms = 100;
+
+		/* Call msec_delay in a loop with several smaller sleeps to
+		 * provide periodic thread cancellation points
+		 */
+		start = rte_get_timer_cycles();
+		ticks = (uint64_t)service_ms * rte_get_timer_hz() / 1E3;
+		while ((rte_get_timer_cycles() - start) < ticks)
+			msec_delay(100);
+	}
+
+	/* Never return */
+	return NULL;
+}
+
+static s32
+eth_ixgbe_check_mac_link_generic(struct ixgbe_hw *hw, ixgbe_link_speed *speed,
+				 bool *link_up, bool link_up_wait_to_complete)
+{
+	if (ixgbe_need_crosstalk_fix(hw)) {
+		enum ixgbe_sfp_cage_status sfp_cage_status;
+
+		sfp_cage_status = ixgbe_check_sfp_cage(hw);
+		if (sfp_cage_status != IXGBE_SFP_CAGE_FULL) {
+			*link_up = false;
+			*speed = IXGBE_LINK_SPEED_UNKNOWN;
+			return IXGBE_SUCCESS;
+		}
+	}
+
+	return ixgbe_check_mac_link_generic(hw, speed, link_up, link_up_wait_to_complete);
+}
+
 /*
  * This function is based on code in ixgbe_attach() in base/ixgbe.c.
  * It returns 0 on success.
@@ -1054,6 +1378,7 @@ eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)
 		IXGBE_DEV_PRIVATE_TO_FILTER_INFO(eth_dev->data->dev_private);
 	struct ixgbe_bw_conf *bw_conf =
 		IXGBE_DEV_PRIVATE_TO_BW_CONF(eth_dev->data->dev_private);
+	struct ixgbe_mac_info *mac = &hw->mac;
 	uint32_t ctrl_ext;
 	uint16_t csum;
 	int diag, i, ret;
@@ -1124,6 +1449,17 @@ eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)
 		return -EIO;
 	}
 
+	/* override mac_link_check to check for sfp cage full/empty */
+	switch (hw->mac.type) {
+	case ixgbe_mac_X550EM_x:
+	case ixgbe_mac_X550EM_a:
+	case ixgbe_mac_82599EB:
+		mac->ops.check_link = eth_ixgbe_check_mac_link_generic;
+		break;
+	default:
+		break;
+	}
+
 	/* pick up the PCI bus settings for reporting later */
 	ixgbe_get_bus_info(hw);
 
@@ -2558,8 +2894,11 @@ ixgbe_flow_ctrl_enable(struct rte_eth_dev *dev, struct ixgbe_hw *hw)
 static int
 ixgbe_dev_start(struct rte_eth_dev *dev)
 {
+	struct ixgbe_adapter *ad = dev->data->dev_private;
 	struct ixgbe_hw *hw =
 		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	struct ixgbe_interrupt *intr =
+		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
 	struct ixgbe_vf_info *vfinfo =
 		*IXGBE_DEV_PRIVATE_TO_P_VFDATA(dev->data->dev_private);
 	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
@@ -2580,9 +2919,6 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
 
 	PMD_INIT_FUNC_TRACE();
 
-	/* Stop the link setup handler before resetting the HW. */
-	ixgbe_dev_wait_setup_link_complete(dev, 0);
-
 	/* disable uio/vfio intr/eventfd mapping */
 	rte_intr_disable(intr_handle);
 
@@ -2815,6 +3151,20 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
 	ixgbe_l2_tunnel_conf(dev);
 	ixgbe_filter_restore(dev);
 
+	/* Spawn service thread */
+	if (ixgbe_is_sfp(hw)) {
+		intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
+		err = rte_ctrl_thread_create(&ad->service_thread_tid,
+					     "ixgbe-service-thread",
+					     NULL,
+					     ixgbe_dev_service_thread_handler,
+					     dev);
+		if (err) {
+			PMD_DRV_LOG(ERR, "service_thread err");
+			goto error;
+		}
+	}
+
 	if (tm_conf->root && !tm_conf->committed)
 		PMD_DRV_LOG(WARNING,
 			    "please call hierarchy_commit() "
@@ -2860,13 +3210,21 @@ ixgbe_dev_stop(struct rte_eth_dev *dev)
 	int vf;
 	struct ixgbe_tm_conf *tm_conf =
 		IXGBE_DEV_PRIVATE_TO_TM_CONF(dev->data->dev_private);
+	void *res;
+	s32 err;
 
 	if (hw->adapter_stopped)
 		return 0;
 
 	PMD_INIT_FUNC_TRACE();
 
-	ixgbe_dev_wait_setup_link_complete(dev, 0);
+	/* Cancel the service thread, and wait for it to join */
+	err = pthread_cancel(adapter->service_thread_tid);
+	if (err)
+		PMD_DRV_LOG(ERR, "failed to cancel service thread %d", err);
+	err = pthread_join(adapter->service_thread_tid, &res);
+	if (err)
+		PMD_DRV_LOG(ERR, "failed to join service thread %d", err);
 
 	/* disable interrupts */
 	ixgbe_disable_intr(hw);
@@ -2945,7 +3303,6 @@ ixgbe_dev_set_link_up(struct rte_eth_dev *dev)
 	} else {
 		/* Turn on the laser */
 		ixgbe_enable_tx_laser(hw);
-		ixgbe_dev_link_update(dev, 0);
 	}
 
 	return 0;
@@ -2976,7 +3333,6 @@ ixgbe_dev_set_link_down(struct rte_eth_dev *dev)
 	} else {
 		/* Turn off the laser */
 		ixgbe_disable_tx_laser(hw);
-		ixgbe_dev_link_update(dev, 0);
 	}
 
 	return 0;
@@ -4128,57 +4484,6 @@ ixgbevf_check_link(struct ixgbe_hw *hw, ixgbe_link_speed *speed,
 	return ret_val;
 }
 
-/*
- * If @timeout_ms was 0, it means that it will not return until link complete.
- * It returns 1 on complete, return 0 on timeout.
- */
-static int
-ixgbe_dev_wait_setup_link_complete(struct rte_eth_dev *dev, uint32_t timeout_ms)
-{
-#define WARNING_TIMEOUT    9000 /* 9s  in total */
-	struct ixgbe_adapter *ad = dev->data->dev_private;
-	uint32_t timeout = timeout_ms ? timeout_ms : WARNING_TIMEOUT;
-
-	while (rte_atomic32_read(&ad->link_thread_running)) {
-		msec_delay(1);
-		timeout--;
-
-		if (timeout_ms) {
-			if (!timeout)
-				return 0;
-		} else if (!timeout) {
-			/* It will not return until link complete */
-			timeout = WARNING_TIMEOUT;
-			PMD_DRV_LOG(ERR, "IXGBE link thread not complete too long time!");
-		}
-	}
-
-	return 1;
-}
-
-static void *
-ixgbe_dev_setup_link_thread_handler(void *param)
-{
-	struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
-	struct ixgbe_adapter *ad = dev->data->dev_private;
-	struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
-	struct ixgbe_interrupt *intr =
-		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
-	u32 speed;
-	bool autoneg = false;
-
-	pthread_detach(pthread_self());
-	speed = hw->phy.autoneg_advertised;
-	if (!speed)
-		ixgbe_get_link_capabilities(hw, &speed, &autoneg);
-
-	ixgbe_setup_link(hw, speed, true);
-
-	intr->flags &= ~IXGBE_FLAG_NEED_LINK_CONFIG;
-	rte_atomic32_clear(&ad->link_thread_running);
-	return NULL;
-}
-
 /*
  * In freebsd environment, nic_uio drivers do not support interrupts,
  * rte_intr_callback_register() will fail to register interrupts.
@@ -4218,11 +4523,8 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 			    int wait_to_complete, int vf)
 {
 	struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
-	struct ixgbe_adapter *ad = dev->data->dev_private;
 	struct rte_eth_link link;
 	ixgbe_link_speed link_speed = IXGBE_LINK_SPEED_UNKNOWN;
-	struct ixgbe_interrupt *intr =
-		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
 	bool link_up;
 	int diag;
 	int wait = 1;
@@ -4237,9 +4539,6 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 
 	hw->mac.get_link_status = true;
 
-	if (intr->flags & IXGBE_FLAG_NEED_LINK_CONFIG)
-		return rte_eth_linkstatus_set(dev, &link);
-
 	/* check if it needs to wait to complete, if lsc interrupt is enabled */
 	if (wait_to_complete == 0 || dev->data->dev_conf.intr_conf.lsc != 0)
 		wait = 0;
@@ -4254,7 +4553,7 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 	else
 		diag = ixgbe_check_link(hw, &link_speed, &link_up, wait);
 
-	if (diag != 0) {
+	if (diag != 0 || !link_up) {
 		link.link_speed = RTE_ETH_SPEED_NUM_100M;
 		link.link_duplex = RTE_ETH_LINK_FULL_DUPLEX;
 		return rte_eth_linkstatus_set(dev, &link);
@@ -4267,32 +4566,6 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 			link_up = 0;
 	}
 
-	if (link_up == 0) {
-		if (ixgbe_get_media_type(hw) == ixgbe_media_type_fiber) {
-			ixgbe_dev_wait_setup_link_complete(dev, 0);
-			if (rte_atomic32_test_and_set(&ad->link_thread_running)) {
-				/* To avoid race condition between threads, set
-				 * the IXGBE_FLAG_NEED_LINK_CONFIG flag only
-				 * when there is no link thread running.
-				 */
-				intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
-				if (rte_ctrl_thread_create(&ad->link_thread_tid,
-					"ixgbe-link-handler",
-					NULL,
-					ixgbe_dev_setup_link_thread_handler,
-					dev) < 0) {
-					PMD_DRV_LOG(ERR,
-						"Create link thread failed!");
-					rte_atomic32_clear(&ad->link_thread_running);
-				}
-			} else {
-				PMD_DRV_LOG(ERR,
-					"Other link thread is running now!");
-			}
-		}
-		return rte_eth_linkstatus_set(dev, &link);
-	}
-
 	link.link_status = RTE_ETH_LINK_UP;
 	link.link_duplex = RTE_ETH_LINK_FULL_DUPLEX;
 
@@ -4498,8 +4771,6 @@ ixgbe_dev_interrupt_get_status(struct rte_eth_dev *dev)
 	eicr = IXGBE_READ_REG(hw, IXGBE_EICR);
 	PMD_DRV_LOG(DEBUG, "eicr %x", eicr);
 
-	intr->flags = 0;
-
 	/* set flag for async link update */
 	if (eicr & IXGBE_EICR_LSC)
 		intr->flags |= IXGBE_FLAG_NEED_LINK_UPDATE;
@@ -4515,6 +4786,14 @@ ixgbe_dev_interrupt_get_status(struct rte_eth_dev *dev)
 	    (eicr & IXGBE_EICR_GPI_SDP0_X550EM_x))
 		intr->flags |= IXGBE_FLAG_PHY_INTERRUPT;
 
+	/* Check for loss of SFP */
+	/* TODO - For platforms that don't have this flag, do we need to set
+	 *  NEED_SFP_SETUP on LSC if we're a SFP platform?
+	 */
+	if (hw->mac.type ==  ixgbe_mac_X550EM_a &&
+	    (eicr & IXGBE_EICR_GPI_SDP0_X550EM_a))
+		intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
+
 	return 0;
 }
 
@@ -4566,11 +4845,13 @@ ixgbe_dev_link_status_print(struct rte_eth_dev *dev)
 static int
 ixgbe_dev_interrupt_action(struct rte_eth_dev *dev)
 {
+	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
 	struct ixgbe_interrupt *intr =
 		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
-	int64_t timeout;
 	struct ixgbe_hw *hw =
 		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	int64_t timeout;
 
 	PMD_DRV_LOG(DEBUG, "intr action type %d", intr->flags);
 
@@ -4605,16 +4886,14 @@ ixgbe_dev_interrupt_action(struct rte_eth_dev *dev)
 		if (rte_eal_alarm_set(timeout * 1000,
 				      ixgbe_dev_interrupt_delayed_handler, (void *)dev) < 0)
 			PMD_DRV_LOG(ERR, "Error setting alarm");
-		else {
-			/* remember original mask */
-			intr->mask_original = intr->mask;
+		else
 			/* only disable lsc interrupt */
 			intr->mask &= ~IXGBE_EIMS_LSC;
-		}
 	}
 
 	PMD_DRV_LOG(DEBUG, "enable intr immediately");
 	ixgbe_enable_intr(dev);
+	rte_intr_ack(intr_handle);
 
 	return 0;
 }
@@ -4637,8 +4916,6 @@ static void
 ixgbe_dev_interrupt_delayed_handler(void *param)
 {
 	struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
-	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
-	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
 	struct ixgbe_interrupt *intr =
 		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
 	struct ixgbe_hw *hw =
@@ -4668,13 +4945,10 @@ ixgbe_dev_interrupt_delayed_handler(void *param)
 		intr->flags &= ~IXGBE_FLAG_MACSEC;
 	}
 
-	/* restore original mask */
-	intr->mask = intr->mask_original;
-	intr->mask_original = 0;
+	if (dev->data->dev_conf.intr_conf.lsc != 0)
+		intr->mask |= IXGBE_EICR_LSC;
 
-	PMD_DRV_LOG(DEBUG, "enable intr in delayed handler S[%08x]", eicr);
 	ixgbe_enable_intr(dev);
-	rte_intr_ack(intr_handle);
 }
 
 /**
@@ -5316,9 +5590,6 @@ ixgbevf_dev_start(struct rte_eth_dev *dev)
 
 	PMD_INIT_FUNC_TRACE();
 
-	/* Stop the link setup handler before resetting the HW. */
-	ixgbe_dev_wait_setup_link_complete(dev, 0);
-
 	err = hw->mac.ops.reset_hw(hw);
 
 	/**
@@ -5398,12 +5669,6 @@ ixgbevf_dev_start(struct rte_eth_dev *dev)
 	/* Re-enable interrupt for VF */
 	ixgbevf_intr_enable(dev);
 
-	/*
-	 * Update link status right before return, because it may
-	 * start link configuration process in a separate thread.
-	 */
-	ixgbevf_dev_link_update(dev, 0);
-
 	hw->adapter_stopped = false;
 
 	return 0;
@@ -5422,8 +5687,6 @@ ixgbevf_dev_stop(struct rte_eth_dev *dev)
 
 	PMD_INIT_FUNC_TRACE();
 
-	ixgbe_dev_wait_setup_link_complete(dev, 0);
-
 	ixgbevf_intr_disable(dev);
 
 	dev->data->dev_started = 0;
diff --git a/drivers/net/ixgbe/ixgbe_ethdev.h b/drivers/net/ixgbe/ixgbe_ethdev.h
index 69e0e82a5b..d243e417e9 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.h
+++ b/drivers/net/ixgbe/ixgbe_ethdev.h
@@ -29,6 +29,7 @@
 #define IXGBE_FLAG_PHY_INTERRUPT    (uint32_t)(1 << 2)
 #define IXGBE_FLAG_MACSEC           (uint32_t)(1 << 3)
 #define IXGBE_FLAG_NEED_LINK_CONFIG (uint32_t)(1 << 4)
+#define IXGBE_FLAG_NEED_SFP_SETUP   ((uint32_t)(1 << 5))
 
 /*
  * Defines that were not part of ixgbe_type.h as they are not used by the
@@ -223,8 +224,6 @@ struct ixgbe_rte_flow_rss_conf {
 struct ixgbe_interrupt {
 	uint32_t flags;
 	uint32_t mask;
-	/*to save original mask during delayed handler */
-	uint32_t mask_original;
 };
 
 struct ixgbe_stat_mapping_registers {
@@ -507,7 +506,7 @@ struct ixgbe_adapter {
 	uint8_t pflink_fullchk;
 	uint8_t mac_ctrl_frame_fwd;
 	rte_atomic32_t link_thread_running;
-	pthread_t link_thread_tid;
+	pthread_t service_thread_tid;
 };
 
 struct ixgbe_vf_representor {
@@ -670,6 +669,15 @@ int ixgbe_syn_filter_set(struct rte_eth_dev *dev,
 			struct rte_eth_syn_filter *filter,
 			bool add);
 
+enum ixgbe_sfp_cage_status {
+	IXGBE_SFP_CAGE_EMPTY = 0,
+	IXGBE_SFP_CAGE_FULL,
+	IXGBE_SFP_CAGE_UNKNOWN = -1,
+	IXGBE_SFP_CAGE_NOCAGE = -2,
+};
+enum ixgbe_sfp_cage_status ixgbe_check_sfp_cage(struct ixgbe_hw *hw);
+
+
 /**
  * l2 tunnel configuration.
  */
-- 
2.25.1

RE: [PATCH v4 3/3] net/ixgbe: Fix SFP detection and linking on hotplug

[Jeff Daly]

> -----Original Message-----
> From: Jeff Daly <jeffd@silicom-usa.com>
> Sent: Monday, February 28, 2022 10:30 AM
> To: dev@dpdk.org
> Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>; Haiyue
> Wang <haiyue.wang@intel.com>
> Subject: [PATCH v4 3/3] net/ixgbe: Fix SFP detection and linking on hotplug
> 
> Caution: This is an external email. Please take care when clicking links or
> opening attachments.
> 
> 
> Currently the ixgbe driver does not ID any SFP except for the first one plugged in.
> This can lead to no-link, or incorrect speed conditions.
> 
> For example:
> 
> * If link is initially established with a 1G SFP, and later a 1G/10G multispeed part
> is later installed, then the MAC link setup functions are never called to change
> from 1000BASE-X to 10GBASE-R mode, and the link stays running at the slower
> rate.
> 
> * If link is initially established with a 1G SFP, and later a 10G only module is later
> installed, no link is established, since we are still trasnsmitting in 1000BASE-X
> mode to a 10GBASE-R only partner.
> 
> Refactor the SFP ID/setup, and link setup code, to more closely match the flow
> of the mainline kernel driver which does not have these issues.  In that driver a
> service task runs periodically to handle these operations based on bit flags that
> have been set (usually via interrupt or userspace request), and then get cleared
> once the requested subtask has been completed.
> 
> Fixes: af75078fece ("first public release")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
> Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> ---
>  drivers/net/ixgbe/ixgbe_ethdev.c | 499 +++++++++++++++++++++++--------
> drivers/net/ixgbe/ixgbe_ethdev.h |  14 +-
>  2 files changed, 392 insertions(+), 121 deletions(-)
> 
> diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
> index e8f07cb405..b70985bb1d 100644
> --- a/drivers/net/ixgbe/ixgbe_ethdev.c
> +++ b/drivers/net/ixgbe/ixgbe_ethdev.c
> @@ -229,9 +229,6 @@ static int ixgbe_dev_interrupt_get_status(struct
> rte_eth_dev *dev);  static int ixgbe_dev_interrupt_action(struct rte_eth_dev
> *dev);  static void ixgbe_dev_interrupt_handler(void *param);  static void
> ixgbe_dev_interrupt_delayed_handler(void *param); -static void
> *ixgbe_dev_setup_link_thread_handler(void *param); -static int
> ixgbe_dev_wait_setup_link_complete(struct rte_eth_dev *dev,
> -                                             uint32_t timeout_ms);
> 
>  static int ixgbe_add_rar(struct rte_eth_dev *dev,
>                         struct rte_ether_addr *mac_addr, @@ -766,6 +763,33 @@ static
> const struct rte_ixgbe_xstats_name_off rte_ixgbevf_stats_strings[] = {  #define
> IXGBEVF_NB_XSTATS (sizeof(rte_ixgbevf_stats_strings) / \
>                 sizeof(rte_ixgbevf_stats_strings[0]))
> 
> +/**
> + * This function is the same as ixgbe_need_crosstalk_fix() in
> +base/ixgbe_common.c
> + *
> + * ixgbe_need_crosstalk_fix - Determine if we need to do cross talk fix
> + * @hw: pointer to hardware structure
> + *
> + * Contains the logic to identify if we need to verify link for the
> + * crosstalk fix
> + **/
> +static bool ixgbe_need_crosstalk_fix(struct ixgbe_hw *hw) {
> +       /* Does FW say we need the fix */
> +       if (!hw->need_crosstalk_fix)
> +               return false;
> +
> +       /* Only consider SFP+ PHYs i.e. media type fiber */
> +       switch (ixgbe_get_media_type(hw)) {
> +       case ixgbe_media_type_fiber:
> +       case ixgbe_media_type_fiber_qsfp:
> +               break;
> +       default:
> +               return false;
> +       }
> +
> +       return true;
> +}
> +
>  /*
>   * This function is the same as ixgbe_is_sfp() in base/ixgbe.h.
>   */
> @@ -1032,6 +1056,306 @@ ixgbe_swfw_lock_reset(struct ixgbe_hw *hw)
>         ixgbe_release_swfw_semaphore(hw, mask);  }
> 
> +/**
> + * ixgbe_check_sfp_cage - Find present status of SFP module
> + * @hw: pointer to hardware structure
> + *
> + * Find if a SFP module is present and if this device supports SFPs
> +**/ enum ixgbe_sfp_cage_status ixgbe_check_sfp_cage(struct ixgbe_hw
> +*hw) {
> +       enum ixgbe_sfp_cage_status sfp_cage_status;
> +
> +       /* If we're not a fiber/fiber_qsfp, no cage to check */
> +       switch (hw->mac.ops.get_media_type(hw)) {
> +       case ixgbe_media_type_fiber:
> +       case ixgbe_media_type_fiber_qsfp:
> +               break;
> +       default:
> +               return IXGBE_SFP_CAGE_NOCAGE;
> +       }
> +
> +       switch (hw->mac.type) {
> +       case ixgbe_mac_82599EB:
> +               sfp_cage_status = !!(IXGBE_READ_REG(hw, IXGBE_ESDP) &
> +                            IXGBE_ESDP_SDP2);
> +               break;
> +       case ixgbe_mac_X550EM_x:
> +       case ixgbe_mac_X550EM_a:
> +               /* SDP0 is the active low signal PRSNT#, so invert this */
> +               sfp_cage_status = !(IXGBE_READ_REG(hw, IXGBE_ESDP) &
> +                                 IXGBE_ESDP_SDP0);
> +               break;
> +       default:
> +               /* Don't know how to check this device type yet */
> +               sfp_cage_status = IXGBE_SFP_CAGE_UNKNOWN;
> +               DEBUGOUT("IXGBE_SFP_CAGE_UNKNOWN, unknown mac type
> %d\n",
> +                        hw->mac.type);
> +               break;
> +       }
> +
> +       DEBUGOUT("sfp status %d for mac type %d\n", sfp_cage_status, hw-
> >mac.type);
> +       return sfp_cage_status;
> +}
> +
> +static s32
> +ixgbe_sfp_id_and_setup(struct rte_eth_dev *dev) {
> +       struct ixgbe_hw *hw =
> +               IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
> +       enum ixgbe_sfp_cage_status sfp_cage_status;
> +       s32 err;
> +
> +       /* Can't ID or setup SFP if it's not plugged in */
> +       sfp_cage_status = ixgbe_check_sfp_cage(hw);
> +       if (sfp_cage_status == IXGBE_SFP_CAGE_EMPTY ||
> +           sfp_cage_status == IXGBE_SFP_CAGE_NOCAGE)
> +               return IXGBE_ERR_SFP_NOT_PRESENT;
> +
> +       /* Something's in the cage, ID it */
> +       hw->phy.ops.identify_sfp(hw);
> +
> +       /* Unknown module type, give up */
> +       if (hw->phy.sfp_type == ixgbe_sfp_type_unknown) {
> +               PMD_DRV_LOG(ERR, "unknown SFP type, giving up");
> +               return IXGBE_ERR_SFP_NOT_SUPPORTED;
> +       }
> +
> +       /* This should be a redundant check, since we looked at the
> +        * PRSNT# signal from the cage above, but just in case this is
> +        * an SFP that's slow to respond to I2C pokes correctly, try it
> +        * again later
> +        */
> +       if (hw->phy.sfp_type == ixgbe_sfp_type_not_present) {
> +               PMD_DRV_LOG(ERR, "IDed SFP as absent but cage PRSNT# active!?");
> +               return IXGBE_ERR_SFP_NOT_PRESENT;
> +       }
> +
> +       /* SFP is present and identified, try to set it up */
> +       err = hw->mac.ops.setup_sfp(hw);
> +       if (err)
> +               PMD_DRV_LOG(ERR, "setup_sfp() failed %d", err);
> +
> +       return err;
> +}
> +
> +static void
> +ixgbe_sfp_service(struct rte_eth_dev *dev) {
> +       struct ixgbe_hw *hw =
> +               IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
> +       struct ixgbe_interrupt *intr =
> +               IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
> +       enum ixgbe_sfp_cage_status sfp_cage_status;
> +       s32 err;
> +       u8 sff_id;
> +       bool have_int = false;
> +
> +       /* If there's no module cage, then there's nothing to service */
> +       sfp_cage_status = ixgbe_check_sfp_cage(hw);
> +       if (sfp_cage_status == IXGBE_SFP_CAGE_NOCAGE) {
> +               PMD_DRV_LOG(DEBUG, "No SFP to service\n");
> +               return;
> +       }
> +
> +       /* TODO - Even for platforms where ixgbe_check_sfp_cage() gives a clear
> +        * status result, if there's no interrupts, or no interrupt for the SFP
> +        * cage present pin, even if other interrupts exist, then we still need
> +        * to poll here to set the flag.
> +        */
> +#ifndef RTE_EXEC_ENV_FREEBSD
> +       struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
> +       struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
> +       if (rte_intr_allow_others(intr_handle)) {
> +               /* check if lsc interrupt is enabled */
> +               if (dev->data->dev_conf.intr_conf.lsc)
> +                       have_int = true;
> +       }
> +#endif /* #ifdef RTE_EXEC_ENV_FREEBSD */
> +
> +       if (!have_int && sfp_cage_status == IXGBE_SFP_CAGE_EMPTY) {
> +               intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
> +               PMD_DRV_LOG(DEBUG, "No SFP, no LSC, set NEED_SFP_SETUP\n");
> +       }
> +
> +       /* For platforms that don't have a way to read the PRESENT# signal from
> +        * the SFP cage, fallback to doing an I2C read and seeing if it's ACKed
> +        * to determine if a module is present
> +        */
> +       if (sfp_cage_status == IXGBE_SFP_CAGE_UNKNOWN) {
> +               PMD_DRV_LOG(DEBUG,
> +                           "SFP present unknown (int? %d), try I2C read\n",
> +                           have_int);
> +
> +               /* Rather than calling identify_sfp, which will read a lot of I2C
> +                * registers (and in a slow processor intensive fashion due to
> +                * bit-banging, just read the SFF ID register, which is at a
> +                * common address across SFP/SFP+/QSFP modules and see if
> +                * there's a NACK.  This works since we only expect a NACK if no
> +                * module is present
> +                */
> +               err = ixgbe_read_i2c_eeprom(hw, IXGBE_SFF_IDENTIFIER, &sff_id);
> +               if (err != IXGBE_SUCCESS) {
> +                       PMD_DRV_LOG(DEBUG, "Received I2C NAK from SFP, set
> NEED_SFP_SETUP flag\n");
> +                       intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
> +                       sfp_cage_status = IXGBE_SFP_CAGE_EMPTY;
> +               } else {
> +                       PMD_DRV_LOG(DEBUG, "SFP ID read ACKed");
> +                       sfp_cage_status = IXGBE_SFP_CAGE_FULL;
> +               }
> +       }
> +
> +       if (sfp_cage_status == IXGBE_SFP_CAGE_EMPTY) {
> +               PMD_DRV_LOG(DEBUG, "SFP absent, cage_status %d\n",
> sfp_cage_status);
> +               return;
> +       }
> +
> +       /* No setup requested?  Nothing to do */
> +       if (!(intr->flags & IXGBE_FLAG_NEED_SFP_SETUP))
> +               return;
> +
> +       err = ixgbe_sfp_id_and_setup(dev);
> +       if (err) {
> +               PMD_DRV_LOG(DEBUG, "failed to ID & setup SFP %d", err);
> +               return;
> +       }
> +
> +       /* Setup is done, clear the flag, but make sure link config runs for new SFP
> */
> +       intr->flags &= ~IXGBE_FLAG_NEED_SFP_SETUP;
> +       intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
> +
> +       /*
> +        * Since this is a new SFP, clear the old advertised speed mask so we don't
> +        * end up using an old slower rate
> +        */
> +       hw->phy.autoneg_advertised = 0;
> +}
> +
> +static void
> +ixgbe_link_service(struct rte_eth_dev *dev) {
> +       struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data-
> >dev_private);
> +       struct ixgbe_interrupt *intr =
> +               IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
> +       bool link_up, autoneg = false, have_int = false;
> +       u32 speed;
> +       s32 err;
> +
> +       /* Test if we have a LSC interrupt for this platform, if not we need to
> +        * manually check the link register since IXGBE_FLAG_NEED_LINK_CONFIG
> +        * will never be set in the interrupt handler
> +        */
> +#ifndef RTE_EXEC_ENV_FREEBSD
> +       struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
> +       struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
> +       if (rte_intr_allow_others(intr_handle)) {
> +               /* check if lsc interrupt is enabled */
> +               if (dev->data->dev_conf.intr_conf.lsc)
> +                       have_int = true;
> +       }
> +#endif /* #ifdef RTE_EXEC_ENV_FREEBSD */
> +
> +       /* Skip if we still need to setup an SFP, or if no link config requested
> +        */
> +       if ((intr->flags & IXGBE_FLAG_NEED_SFP_SETUP) ||
> +           (!(intr->flags & IXGBE_FLAG_NEED_LINK_CONFIG) && have_int))
> +               return;
> +
> +       if (!have_int && !(intr->flags & IXGBE_FLAG_NEED_LINK_CONFIG)) {
> +               err = ixgbe_check_link(hw, &speed, &link_up, 0);
> +               if (!err && !link_up) {
> +                       intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
> +                       PMD_DRV_LOG(DEBUG, "Link down, no LSC, set
> NEED_LINK_CONFIG\n");
> +               } else {
> +                       return;
> +               }
> +       }
> +
> +       speed = hw->phy.autoneg_advertised;
> +       if (!speed)
> +               ixgbe_get_link_capabilities(hw, &speed, &autoneg);
> +
> +       err = ixgbe_setup_link(hw, speed, true);
> +       if (err) {
> +               PMD_DRV_LOG(ERR, "ixgbe_setup_link failed %d", err);
> +               return;
> +       }
> +
> +       intr->flags &= ~IXGBE_FLAG_NEED_LINK_CONFIG; }
> +
> +static void
> +ixgbe_link_update_service(struct rte_eth_dev *dev) {
> +       /* Update internal link status, waiting for link */
> +       if (!ixgbe_dev_link_update(dev, 0)) {
> +               ixgbe_dev_link_status_print(dev);
> +               rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_LSC,
> NULL);
> +       }
> +}
> +
> +/*
> + * Service task thread to handle periodic tasks  */ static void *
> +ixgbe_dev_service_thread_handler(void *param) {
> +       struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
> +       struct ixgbe_hw *hw =
> +               IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
> +       uint64_t start, ticks, service_ms;
> +       uint32_t speed;
> +       s32 err;
> +       bool link_up;
> +
> +       while (1) {
> +               ixgbe_sfp_service(dev);
> +               ixgbe_link_service(dev);
> +               ixgbe_link_update_service(dev);
> +
> +               /* Run the service thread handler more frequently when link is
> +                * down to reduce link up latency (every 200ms vs 1s)
> +                *
> +                * Use a number of smaller sleeps to decrease exit latency when
> +                * ixgbe_dev_stop() wants this thread to join
> +                */
> +               err = ixgbe_check_link(hw, &speed, &link_up, 0);
> +               if (err == IXGBE_SUCCESS && link_up)
> +                       service_ms = 2000;
> +               else
> +                       service_ms = 100;
> +
> +               /* Call msec_delay in a loop with several smaller sleeps to
> +                * provide periodic thread cancellation points
> +                */
> +               start = rte_get_timer_cycles();
> +               ticks = (uint64_t)service_ms * rte_get_timer_hz() / 1E3;
> +               while ((rte_get_timer_cycles() - start) < ticks)
> +                       msec_delay(100);
> +       }
> +
> +       /* Never return */
> +       return NULL;
> +}
> +
> +static s32
> +eth_ixgbe_check_mac_link_generic(struct ixgbe_hw *hw, ixgbe_link_speed
> *speed,
> +                                bool *link_up, bool
> +link_up_wait_to_complete) {
> +       if (ixgbe_need_crosstalk_fix(hw)) {
> +               enum ixgbe_sfp_cage_status sfp_cage_status;
> +
> +               sfp_cage_status = ixgbe_check_sfp_cage(hw);
> +               if (sfp_cage_status != IXGBE_SFP_CAGE_FULL) {
> +                       *link_up = false;
> +                       *speed = IXGBE_LINK_SPEED_UNKNOWN;
> +                       return IXGBE_SUCCESS;
> +               }
> +       }
> +
> +       return ixgbe_check_mac_link_generic(hw, speed, link_up,
> +link_up_wait_to_complete); }
> +
>  /*
>   * This function is based on code in ixgbe_attach() in base/ixgbe.c.
>   * It returns 0 on success.
> @@ -1054,6 +1378,7 @@ eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev, void
> *init_params __rte_unused)
>                 IXGBE_DEV_PRIVATE_TO_FILTER_INFO(eth_dev->data->dev_private);
>         struct ixgbe_bw_conf *bw_conf =
>                 IXGBE_DEV_PRIVATE_TO_BW_CONF(eth_dev->data->dev_private);
> +       struct ixgbe_mac_info *mac = &hw->mac;
>         uint32_t ctrl_ext;
>         uint16_t csum;
>         int diag, i, ret;
> @@ -1124,6 +1449,17 @@ eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev,
> void *init_params __rte_unused)
>                 return -EIO;
>         }
> 
> +       /* override mac_link_check to check for sfp cage full/empty */
> +       switch (hw->mac.type) {
> +       case ixgbe_mac_X550EM_x:
> +       case ixgbe_mac_X550EM_a:
> +       case ixgbe_mac_82599EB:
> +               mac->ops.check_link = eth_ixgbe_check_mac_link_generic;
> +               break;
> +       default:
> +               break;
> +       }
> +
>         /* pick up the PCI bus settings for reporting later */
>         ixgbe_get_bus_info(hw);
> 
> @@ -2558,8 +2894,11 @@ ixgbe_flow_ctrl_enable(struct rte_eth_dev *dev,
> struct ixgbe_hw *hw)  static int  ixgbe_dev_start(struct rte_eth_dev *dev)  {
> +       struct ixgbe_adapter *ad = dev->data->dev_private;
>         struct ixgbe_hw *hw =
>                 IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
> +       struct ixgbe_interrupt *intr =
> +               IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
>         struct ixgbe_vf_info *vfinfo =
>                 *IXGBE_DEV_PRIVATE_TO_P_VFDATA(dev->data->dev_private);
>         struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev); @@ -2580,9
> +2919,6 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
> 
>         PMD_INIT_FUNC_TRACE();
> 
> -       /* Stop the link setup handler before resetting the HW. */
> -       ixgbe_dev_wait_setup_link_complete(dev, 0);
> -
>         /* disable uio/vfio intr/eventfd mapping */
>         rte_intr_disable(intr_handle);
> 
> @@ -2815,6 +3151,20 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
>         ixgbe_l2_tunnel_conf(dev);
>         ixgbe_filter_restore(dev);
> 
> +       /* Spawn service thread */
> +       if (ixgbe_is_sfp(hw)) {
> +               intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
> +               err = rte_ctrl_thread_create(&ad->service_thread_tid,
> +                                            "ixgbe-service-thread",
> +                                            NULL,
> +                                            ixgbe_dev_service_thread_handler,
> +                                            dev);
> +               if (err) {
> +                       PMD_DRV_LOG(ERR, "service_thread err");
> +                       goto error;
> +               }
> +       }
> +
>         if (tm_conf->root && !tm_conf->committed)
>                 PMD_DRV_LOG(WARNING,
>                             "please call hierarchy_commit() "
> @@ -2860,13 +3210,21 @@ ixgbe_dev_stop(struct rte_eth_dev *dev)
>         int vf;
>         struct ixgbe_tm_conf *tm_conf =
>                 IXGBE_DEV_PRIVATE_TO_TM_CONF(dev->data->dev_private);
> +       void *res;
> +       s32 err;
> 
>         if (hw->adapter_stopped)
>                 return 0;
> 
>         PMD_INIT_FUNC_TRACE();
> 
> -       ixgbe_dev_wait_setup_link_complete(dev, 0);
> +       /* Cancel the service thread, and wait for it to join */
> +       err = pthread_cancel(adapter->service_thread_tid);
> +       if (err)
> +               PMD_DRV_LOG(ERR, "failed to cancel service thread %d", err);
> +       err = pthread_join(adapter->service_thread_tid, &res);
> +       if (err)
> +               PMD_DRV_LOG(ERR, "failed to join service thread %d",
> + err);
> 
>         /* disable interrupts */
>         ixgbe_disable_intr(hw);
> @@ -2945,7 +3303,6 @@ ixgbe_dev_set_link_up(struct rte_eth_dev *dev)
>         } else {
>                 /* Turn on the laser */
>                 ixgbe_enable_tx_laser(hw);
> -               ixgbe_dev_link_update(dev, 0);
>         }
> 
>         return 0;
> @@ -2976,7 +3333,6 @@ ixgbe_dev_set_link_down(struct rte_eth_dev *dev)
>         } else {
>                 /* Turn off the laser */
>                 ixgbe_disable_tx_laser(hw);
> -               ixgbe_dev_link_update(dev, 0);
>         }
> 
>         return 0;
> @@ -4128,57 +4484,6 @@ ixgbevf_check_link(struct ixgbe_hw *hw,
> ixgbe_link_speed *speed,
>         return ret_val;
>  }
> 
> -/*
> - * If @timeout_ms was 0, it means that it will not return until link complete.
> - * It returns 1 on complete, return 0 on timeout.
> - */
> -static int
> -ixgbe_dev_wait_setup_link_complete(struct rte_eth_dev *dev, uint32_t
> timeout_ms) -{
> -#define WARNING_TIMEOUT    9000 /* 9s  in total */
> -       struct ixgbe_adapter *ad = dev->data->dev_private;
> -       uint32_t timeout = timeout_ms ? timeout_ms : WARNING_TIMEOUT;
> -
> -       while (rte_atomic32_read(&ad->link_thread_running)) {
> -               msec_delay(1);
> -               timeout--;
> -
> -               if (timeout_ms) {
> -                       if (!timeout)
> -                               return 0;
> -               } else if (!timeout) {
> -                       /* It will not return until link complete */
> -                       timeout = WARNING_TIMEOUT;
> -                       PMD_DRV_LOG(ERR, "IXGBE link thread not complete too long
> time!");
> -               }
> -       }
> -
> -       return 1;
> -}
> -
> -static void *
> -ixgbe_dev_setup_link_thread_handler(void *param) -{
> -       struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
> -       struct ixgbe_adapter *ad = dev->data->dev_private;
> -       struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data-
> >dev_private);
> -       struct ixgbe_interrupt *intr =
> -               IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
> -       u32 speed;
> -       bool autoneg = false;
> -
> -       pthread_detach(pthread_self());
> -       speed = hw->phy.autoneg_advertised;
> -       if (!speed)
> -               ixgbe_get_link_capabilities(hw, &speed, &autoneg);
> -
> -       ixgbe_setup_link(hw, speed, true);
> -
> -       intr->flags &= ~IXGBE_FLAG_NEED_LINK_CONFIG;
> -       rte_atomic32_clear(&ad->link_thread_running);
> -       return NULL;
> -}
> -
>  /*
>   * In freebsd environment, nic_uio drivers do not support interrupts,
>   * rte_intr_callback_register() will fail to register interrupts.
> @@ -4218,11 +4523,8 @@ ixgbe_dev_link_update_share(struct rte_eth_dev
> *dev,
>                             int wait_to_complete, int vf)  {
>         struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data-
> >dev_private);
> -       struct ixgbe_adapter *ad = dev->data->dev_private;
>         struct rte_eth_link link;
>         ixgbe_link_speed link_speed = IXGBE_LINK_SPEED_UNKNOWN;
> -       struct ixgbe_interrupt *intr =
> -               IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
>         bool link_up;
>         int diag;
>         int wait = 1;
> @@ -4237,9 +4539,6 @@ ixgbe_dev_link_update_share(struct rte_eth_dev
> *dev,
> 
>         hw->mac.get_link_status = true;
> 
> -       if (intr->flags & IXGBE_FLAG_NEED_LINK_CONFIG)
> -               return rte_eth_linkstatus_set(dev, &link);
> -
>         /* check if it needs to wait to complete, if lsc interrupt is enabled */
>         if (wait_to_complete == 0 || dev->data->dev_conf.intr_conf.lsc != 0)
>                 wait = 0;
> @@ -4254,7 +4553,7 @@ ixgbe_dev_link_update_share(struct rte_eth_dev
> *dev,
>         else
>                 diag = ixgbe_check_link(hw, &link_speed, &link_up, wait);
> 
> -       if (diag != 0) {
> +       if (diag != 0 || !link_up) {
>                 link.link_speed = RTE_ETH_SPEED_NUM_100M;
>                 link.link_duplex = RTE_ETH_LINK_FULL_DUPLEX;
>                 return rte_eth_linkstatus_set(dev, &link); @@ -4267,32 +4566,6 @@
> ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
>                         link_up = 0;
>         }
> 
> -       if (link_up == 0) {
> -               if (ixgbe_get_media_type(hw) == ixgbe_media_type_fiber) {
> -                       ixgbe_dev_wait_setup_link_complete(dev, 0);
> -                       if (rte_atomic32_test_and_set(&ad->link_thread_running)) {
> -                               /* To avoid race condition between threads, set
> -                                * the IXGBE_FLAG_NEED_LINK_CONFIG flag only
> -                                * when there is no link thread running.
> -                                */
> -                               intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
> -                               if (rte_ctrl_thread_create(&ad->link_thread_tid,
> -                                       "ixgbe-link-handler",
> -                                       NULL,
> -                                       ixgbe_dev_setup_link_thread_handler,
> -                                       dev) < 0) {
> -                                       PMD_DRV_LOG(ERR,
> -                                               "Create link thread failed!");
> -                                       rte_atomic32_clear(&ad->link_thread_running);
> -                               }
> -                       } else {
> -                               PMD_DRV_LOG(ERR,
> -                                       "Other link thread is running now!");
> -                       }
> -               }
> -               return rte_eth_linkstatus_set(dev, &link);
> -       }
> -
>         link.link_status = RTE_ETH_LINK_UP;
>         link.link_duplex = RTE_ETH_LINK_FULL_DUPLEX;
> 
> @@ -4498,8 +4771,6 @@ ixgbe_dev_interrupt_get_status(struct rte_eth_dev
> *dev)
>         eicr = IXGBE_READ_REG(hw, IXGBE_EICR);
>         PMD_DRV_LOG(DEBUG, "eicr %x", eicr);
> 
> -       intr->flags = 0;
> -
>         /* set flag for async link update */
>         if (eicr & IXGBE_EICR_LSC)
>                 intr->flags |= IXGBE_FLAG_NEED_LINK_UPDATE; @@ -4515,6
> +4786,14 @@ ixgbe_dev_interrupt_get_status(struct rte_eth_dev *dev)
>             (eicr & IXGBE_EICR_GPI_SDP0_X550EM_x))
>                 intr->flags |= IXGBE_FLAG_PHY_INTERRUPT;
> 
> +       /* Check for loss of SFP */
> +       /* TODO - For platforms that don't have this flag, do we need to set
> +        *  NEED_SFP_SETUP on LSC if we're a SFP platform?
> +        */
> +       if (hw->mac.type ==  ixgbe_mac_X550EM_a &&
> +           (eicr & IXGBE_EICR_GPI_SDP0_X550EM_a))
> +               intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
> +
>         return 0;
>  }
> 
> @@ -4566,11 +4845,13 @@ ixgbe_dev_link_status_print(struct rte_eth_dev
> *dev)  static int  ixgbe_dev_interrupt_action(struct rte_eth_dev *dev)  {
> +       struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
> +       struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
>         struct ixgbe_interrupt *intr =
>                 IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
> -       int64_t timeout;
>         struct ixgbe_hw *hw =
>                 IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
> +       int64_t timeout;
> 
>         PMD_DRV_LOG(DEBUG, "intr action type %d", intr->flags);
> 
> @@ -4605,16 +4886,14 @@ ixgbe_dev_interrupt_action(struct rte_eth_dev
> *dev)
>                 if (rte_eal_alarm_set(timeout * 1000,
>                                       ixgbe_dev_interrupt_delayed_handler, (void *)dev) < 0)
>                         PMD_DRV_LOG(ERR, "Error setting alarm");
> -               else {
> -                       /* remember original mask */
> -                       intr->mask_original = intr->mask;
> +               else
>                         /* only disable lsc interrupt */
>                         intr->mask &= ~IXGBE_EIMS_LSC;
> -               }
>         }
> 
>         PMD_DRV_LOG(DEBUG, "enable intr immediately");
>         ixgbe_enable_intr(dev);
> +       rte_intr_ack(intr_handle);
> 
>         return 0;
>  }
> @@ -4637,8 +4916,6 @@ static void
>  ixgbe_dev_interrupt_delayed_handler(void *param)  {
>         struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
> -       struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
> -       struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
>         struct ixgbe_interrupt *intr =
>                 IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
>         struct ixgbe_hw *hw =
> @@ -4668,13 +4945,10 @@ ixgbe_dev_interrupt_delayed_handler(void
> *param)
>                 intr->flags &= ~IXGBE_FLAG_MACSEC;
>         }
> 
> -       /* restore original mask */
> -       intr->mask = intr->mask_original;
> -       intr->mask_original = 0;
> +       if (dev->data->dev_conf.intr_conf.lsc != 0)
> +               intr->mask |= IXGBE_EICR_LSC;
> 
> -       PMD_DRV_LOG(DEBUG, "enable intr in delayed handler S[%08x]", eicr);
>         ixgbe_enable_intr(dev);
> -       rte_intr_ack(intr_handle);
>  }
> 
>  /**
> @@ -5316,9 +5590,6 @@ ixgbevf_dev_start(struct rte_eth_dev *dev)
> 
>         PMD_INIT_FUNC_TRACE();
> 
> -       /* Stop the link setup handler before resetting the HW. */
> -       ixgbe_dev_wait_setup_link_complete(dev, 0);
> -
>         err = hw->mac.ops.reset_hw(hw);
> 
>         /**
> @@ -5398,12 +5669,6 @@ ixgbevf_dev_start(struct rte_eth_dev *dev)
>         /* Re-enable interrupt for VF */
>         ixgbevf_intr_enable(dev);
> 
> -       /*
> -        * Update link status right before return, because it may
> -        * start link configuration process in a separate thread.
> -        */
> -       ixgbevf_dev_link_update(dev, 0);
> -
>         hw->adapter_stopped = false;
> 
>         return 0;
> @@ -5422,8 +5687,6 @@ ixgbevf_dev_stop(struct rte_eth_dev *dev)
> 
>         PMD_INIT_FUNC_TRACE();
> 
> -       ixgbe_dev_wait_setup_link_complete(dev, 0);
> -
>         ixgbevf_intr_disable(dev);
> 
>         dev->data->dev_started = 0;
> diff --git a/drivers/net/ixgbe/ixgbe_ethdev.h
> b/drivers/net/ixgbe/ixgbe_ethdev.h
> index 69e0e82a5b..d243e417e9 100644
> --- a/drivers/net/ixgbe/ixgbe_ethdev.h
> +++ b/drivers/net/ixgbe/ixgbe_ethdev.h
> @@ -29,6 +29,7 @@
>  #define IXGBE_FLAG_PHY_INTERRUPT    (uint32_t)(1 << 2)
>  #define IXGBE_FLAG_MACSEC           (uint32_t)(1 << 3)
>  #define IXGBE_FLAG_NEED_LINK_CONFIG (uint32_t)(1 << 4)
> +#define IXGBE_FLAG_NEED_SFP_SETUP   ((uint32_t)(1 << 5))
> 
>  /*
>   * Defines that were not part of ixgbe_type.h as they are not used by the @@ -
> 223,8 +224,6 @@ struct ixgbe_rte_flow_rss_conf {  struct ixgbe_interrupt {
>         uint32_t flags;
>         uint32_t mask;
> -       /*to save original mask during delayed handler */
> -       uint32_t mask_original;
>  };
> 
>  struct ixgbe_stat_mapping_registers {
> @@ -507,7 +506,7 @@ struct ixgbe_adapter {
>         uint8_t pflink_fullchk;
>         uint8_t mac_ctrl_frame_fwd;
>         rte_atomic32_t link_thread_running;
> -       pthread_t link_thread_tid;
> +       pthread_t service_thread_tid;
>  };
> 
>  struct ixgbe_vf_representor {
> @@ -670,6 +669,15 @@ int ixgbe_syn_filter_set(struct rte_eth_dev *dev,
>                         struct rte_eth_syn_filter *filter,
>                         bool add);
> 
> +enum ixgbe_sfp_cage_status {
> +       IXGBE_SFP_CAGE_EMPTY = 0,
> +       IXGBE_SFP_CAGE_FULL,
> +       IXGBE_SFP_CAGE_UNKNOWN = -1,
> +       IXGBE_SFP_CAGE_NOCAGE = -2,
> +};
> +enum ixgbe_sfp_cage_status ixgbe_check_sfp_cage(struct ixgbe_hw *hw);
> +
> +
>  /**
>   * l2 tunnel configuration.
>   */
> --
> 2.25.1

Hm, testing on a different platform there appears to be an issue when plugging in a module with a different speed from a prior module does not correctly update the link speed.  Currently investigating, could be a platform-specific issue.

[PATCH v5 1/2] net/ixgbe: Limit SDP3 check of TX_DISABLE to appropriate devices

[Jeff Daly]

1ca05831b9b added a check that SDP3 (used as a TX_DISABLE output to the
SFP cage on these cards) is not asserted to avoid incorrectly reporting
link up when the SFP's laser is turned off.

ff8162cb957 limited this workaround to fiber ports

This patch:
* Adds devarg 'fiber_sdp3_no_tx_disable' not all fiber ixgbe devs use
  SDP3 as TX_DISABLE

Fixes: 1ca05831b9b ("net/ixgbe: fix link status")
Fixes: ff8162cb957 ("net/ixgbe: fix link status")
Cc: stable@dpdk.org

Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 39 +++++++++++++++++++++++++++++++-
 drivers/net/ixgbe/ixgbe_ethdev.h |  3 +++
 2 files changed, 41 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 2da3f67bbc..596ea769ae 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -128,6 +128,13 @@
 #define IXGBE_EXVET_VET_EXT_SHIFT              16
 #define IXGBE_DMATXCTL_VT_MASK                 0xFFFF0000
 
+#define IXGBE_DEVARG_FIBER_SDP3_NOT_TX_DISABLE	"fiber_sdp3_no_tx_disable"
+
+static const char * const ixgbe_valid_arguments[] = {
+	IXGBE_DEVARG_FIBER_SDP3_NOT_TX_DISABLE,
+	NULL
+};
+
 #define IXGBEVF_DEVARG_PFLINK_FULLCHK		"pflink_fullchk"
 
 static const char * const ixgbevf_valid_arguments[] = {
@@ -348,6 +355,8 @@ static int ixgbe_dev_udp_tunnel_port_del(struct rte_eth_dev *dev,
 static int ixgbe_filter_restore(struct rte_eth_dev *dev);
 static void ixgbe_l2_tunnel_conf(struct rte_eth_dev *dev);
 static int ixgbe_wait_for_link_up(struct ixgbe_hw *hw);
+static int devarg_handle_int(__rte_unused const char *key, const char *value,
+                  void *extra_args);
 
 /*
  * Define VF Stats MACRO for Non "cleared on read" register
@@ -1032,6 +1041,29 @@ ixgbe_swfw_lock_reset(struct ixgbe_hw *hw)
 	ixgbe_release_swfw_semaphore(hw, mask);
 }
 
+static void
+ixgbe_parse_devargs(struct ixgbe_adapter *adapter,
+		      struct rte_devargs *devargs)
+{
+	struct rte_kvargs *kvlist;
+	uint16_t sdp3_no_tx_disable;
+
+	if (devargs == NULL)
+		return;
+
+	kvlist = rte_kvargs_parse(devargs->args, ixgbe_valid_arguments);
+	if (kvlist == NULL)
+		return;
+
+	if (rte_kvargs_count(kvlist, IXGBE_DEVARG_FIBER_SDP3_NOT_TX_DISABLE) == 1 &&
+	    rte_kvargs_process(kvlist, IXGBE_DEVARG_FIBER_SDP3_NOT_TX_DISABLE,
+			       devarg_handle_int, &sdp3_no_tx_disable) == 0 &&
+	    sdp3_no_tx_disable == 1)
+		adapter->sdp3_no_tx_disable = 1;
+
+	rte_kvargs_free(kvlist);
+}
+
 /*
  * This function is based on code in ixgbe_attach() in base/ixgbe.c.
  * It returns 0 on success.
@@ -1095,6 +1127,8 @@ eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)
 	}
 
 	rte_atomic32_clear(&ad->link_thread_running);
+	ixgbe_parse_devargs(eth_dev->data->dev_private,
+			    pci_dev->device.devargs);
 	rte_eth_copy_pci_info(eth_dev, pci_dev);
 	eth_dev->data->dev_flags |= RTE_ETH_DEV_AUTOFILL_QUEUE_XSTATS;
 
@@ -4261,7 +4295,8 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 		return rte_eth_linkstatus_set(dev, &link);
 	}
 
-	if (ixgbe_get_media_type(hw) == ixgbe_media_type_fiber) {
+	if (ixgbe_get_media_type(hw) == ixgbe_media_type_fiber &&
+	    !ad->sdp3_no_tx_disable) {
 		esdp_reg = IXGBE_READ_REG(hw, IXGBE_ESDP);
 		if ((esdp_reg & IXGBE_ESDP_SDP3))
 			link_up = 0;
@@ -8250,6 +8285,8 @@ ixgbe_dev_macsec_register_disable(struct rte_eth_dev *dev)
 RTE_PMD_REGISTER_PCI(net_ixgbe, rte_ixgbe_pmd);
 RTE_PMD_REGISTER_PCI_TABLE(net_ixgbe, pci_id_ixgbe_map);
 RTE_PMD_REGISTER_KMOD_DEP(net_ixgbe, "* igb_uio | uio_pci_generic | vfio-pci");
+RTE_PMD_REGISTER_PARAM_STRING(net_ixgbe,
+			      IXGBE_DEVARG_FIBER_SDP3_NOT_TX_DISABLE "=<0|1>");
 RTE_PMD_REGISTER_PCI(net_ixgbe_vf, rte_ixgbevf_pmd);
 RTE_PMD_REGISTER_PCI_TABLE(net_ixgbe_vf, pci_id_ixgbevf_map);
 RTE_PMD_REGISTER_KMOD_DEP(net_ixgbe_vf, "* igb_uio | vfio-pci");
diff --git a/drivers/net/ixgbe/ixgbe_ethdev.h b/drivers/net/ixgbe/ixgbe_ethdev.h
index 69e0e82a5b..cc6049a66a 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.h
+++ b/drivers/net/ixgbe/ixgbe_ethdev.h
@@ -501,6 +501,9 @@ struct ixgbe_adapter {
 	/* For RSS reta table update */
 	uint8_t rss_reta_updated;
 
+	/* Used for limiting SDP3 TX_DISABLE checks */
+	uint8_t sdp3_no_tx_disable;
+
 	/* Used for VF link sync with PF's physical and logical (by checking
 	 * mailbox status) link status.
 	 */
-- 
2.25.1

[PATCH v5 2/2] net/ixgbe: Fix SFP detection and linking on hotplug

[Jeff Daly]

Currently the ixgbe driver does not ID any SFP except for the first one
plugged in. This can lead to no-link, or incorrect speed conditions.

For example:

* If link is initially established with a 1G SFP, and later a 1G/10G
multispeed part is later installed, then the MAC link setup functions are
never called to change from 1000BASE-X to 10GBASE-R mode, and the link
stays running at the slower rate.

* If link is initially established with a 1G SFP, and later a 10G only
module is later installed, no link is established, since we are still
trasnsmitting in 1000BASE-X mode to a 10GBASE-R only partner.

Refactor the SFP ID/setup, and link setup code, to more closely match the
flow of the mainline kernel driver which does not have these issues.  In
that driver a service task runs periodically to handle these operations
based on bit flags that have been set (usually via interrupt or userspace
request), and then get cleared once the requested subtask has been
completed.

Fixes: af75078fece ("first public release")
Cc: stable@dpdk.org

Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 533 +++++++++++++++++++++++--------
 drivers/net/ixgbe/ixgbe_ethdev.h |  14 +-
 2 files changed, 410 insertions(+), 137 deletions(-)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 596ea769ae..4adb6dd5ec 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -236,9 +236,6 @@ static int ixgbe_dev_interrupt_get_status(struct rte_eth_dev *dev);
 static int ixgbe_dev_interrupt_action(struct rte_eth_dev *dev);
 static void ixgbe_dev_interrupt_handler(void *param);
 static void ixgbe_dev_interrupt_delayed_handler(void *param);
-static void *ixgbe_dev_setup_link_thread_handler(void *param);
-static int ixgbe_dev_wait_setup_link_complete(struct rte_eth_dev *dev,
-					      uint32_t timeout_ms);
 
 static int ixgbe_add_rar(struct rte_eth_dev *dev,
 			struct rte_ether_addr *mac_addr,
@@ -775,6 +772,33 @@ static const struct rte_ixgbe_xstats_name_off rte_ixgbevf_stats_strings[] = {
 #define IXGBEVF_NB_XSTATS (sizeof(rte_ixgbevf_stats_strings) /	\
 		sizeof(rte_ixgbevf_stats_strings[0]))
 
+/**
+ * This function is the same as ixgbe_need_crosstalk_fix() in base/ixgbe_common.c
+ *
+ * ixgbe_need_crosstalk_fix - Determine if we need to do cross talk fix
+ * @hw: pointer to hardware structure
+ *
+ * Contains the logic to identify if we need to verify link for the
+ * crosstalk fix
+ **/
+static bool ixgbe_need_crosstalk_fix(struct ixgbe_hw *hw)
+{
+	/* Does FW say we need the fix */
+	if (!hw->need_crosstalk_fix)
+		return false;
+
+	/* Only consider SFP+ PHYs i.e. media type fiber */
+	switch (ixgbe_get_media_type(hw)) {
+	case ixgbe_media_type_fiber:
+	case ixgbe_media_type_fiber_qsfp:
+		break;
+	default:
+		return false;
+	}
+
+	return true;
+}
+
 /*
  * This function is the same as ixgbe_is_sfp() in base/ixgbe.h.
  */
@@ -1064,6 +1088,306 @@ ixgbe_parse_devargs(struct ixgbe_adapter *adapter,
 	rte_kvargs_free(kvlist);
 }
 
+/**
+ * ixgbe_check_sfp_cage - Find present status of SFP module
+ * @hw: pointer to hardware structure
+ *
+ * Find if a SFP module is present and if this device supports SFPs
+ **/
+enum ixgbe_sfp_cage_status ixgbe_check_sfp_cage(struct ixgbe_hw *hw)
+{
+	enum ixgbe_sfp_cage_status sfp_cage_status;
+
+	/* If we're not a fiber/fiber_qsfp, no cage to check */
+	switch (hw->mac.ops.get_media_type(hw)) {
+	case ixgbe_media_type_fiber:
+	case ixgbe_media_type_fiber_qsfp:
+		break;
+	default:
+		return IXGBE_SFP_CAGE_NOCAGE;
+	}
+
+	switch (hw->mac.type) {
+	case ixgbe_mac_82599EB:
+		sfp_cage_status = !!(IXGBE_READ_REG(hw, IXGBE_ESDP) &
+			     IXGBE_ESDP_SDP2);
+		break;
+	case ixgbe_mac_X550EM_x:
+	case ixgbe_mac_X550EM_a:
+		/* SDP0 is the active low signal PRSNT#, so invert this */
+		sfp_cage_status = !(IXGBE_READ_REG(hw, IXGBE_ESDP) &
+				  IXGBE_ESDP_SDP0);
+		break;
+	default:
+		/* Don't know how to check this device type yet */
+		sfp_cage_status = IXGBE_SFP_CAGE_UNKNOWN;
+		DEBUGOUT("IXGBE_SFP_CAGE_UNKNOWN, unknown mac type %d\n",
+			 hw->mac.type);
+		break;
+	}
+
+	DEBUGOUT("sfp status %d for mac type %d\n", sfp_cage_status, hw->mac.type);
+	return sfp_cage_status;
+}
+
+static s32
+ixgbe_sfp_id_and_setup(struct rte_eth_dev *dev)
+{
+	struct ixgbe_hw *hw =
+		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	enum ixgbe_sfp_cage_status sfp_cage_status;
+	s32 err;
+
+	/* Can't ID or setup SFP if it's not plugged in */
+	sfp_cage_status = ixgbe_check_sfp_cage(hw);
+	if (sfp_cage_status == IXGBE_SFP_CAGE_EMPTY ||
+	    sfp_cage_status == IXGBE_SFP_CAGE_NOCAGE)
+		return IXGBE_ERR_SFP_NOT_PRESENT;
+
+	/* Something's in the cage, ID it */
+	hw->phy.ops.identify_sfp(hw);
+
+	/* Unknown module type, give up */
+	if (hw->phy.sfp_type == ixgbe_sfp_type_unknown) {
+		PMD_DRV_LOG(ERR, "unknown SFP type, giving up");
+		return IXGBE_ERR_SFP_NOT_SUPPORTED;
+	}
+
+	/* This should be a redundant check, since we looked at the
+	 * PRSNT# signal from the cage above, but just in case this is
+	 * an SFP that's slow to respond to I2C pokes correctly, try it
+	 * again later
+	 */
+	if (hw->phy.sfp_type == ixgbe_sfp_type_not_present) {
+		PMD_DRV_LOG(ERR, "IDed SFP as absent but cage PRSNT# active!?");
+		return IXGBE_ERR_SFP_NOT_PRESENT;
+	}
+
+	/* SFP is present and identified, try to set it up */
+	err = hw->mac.ops.setup_sfp(hw);
+	if (err)
+		PMD_DRV_LOG(ERR, "setup_sfp() failed %d", err);
+
+	return err;
+}
+
+static void
+ixgbe_sfp_service(struct rte_eth_dev *dev)
+{
+	struct ixgbe_hw *hw =
+		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	struct ixgbe_interrupt *intr =
+		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
+	enum ixgbe_sfp_cage_status sfp_cage_status;
+	s32 err;
+	u8 sff_id;
+	bool have_int = false;
+
+	/* If there's no module cage, then there's nothing to service */
+	sfp_cage_status = ixgbe_check_sfp_cage(hw);
+	if (sfp_cage_status == IXGBE_SFP_CAGE_NOCAGE) {
+		PMD_DRV_LOG(DEBUG, "No SFP to service\n");
+		return;
+	}
+
+	/* TODO - Even for platforms where ixgbe_check_sfp_cage() gives a clear
+	 * status result, if there's no interrupts, or no interrupt for the SFP
+	 * cage present pin, even if other interrupts exist, then we still need
+	 * to poll here to set the flag.
+	 */
+#ifndef RTE_EXEC_ENV_FREEBSD
+	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
+	if (rte_intr_allow_others(intr_handle)) {
+		/* check if lsc interrupt is enabled */
+		if (dev->data->dev_conf.intr_conf.lsc)
+			have_int = true;
+	}
+#endif /* #ifdef RTE_EXEC_ENV_FREEBSD */
+
+	if (!have_int && sfp_cage_status == IXGBE_SFP_CAGE_EMPTY) {
+		intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
+		PMD_DRV_LOG(DEBUG, "No SFP, no LSC, set NEED_SFP_SETUP\n");
+	}
+
+	/* For platforms that don't have a way to read the PRESENT# signal from
+	 * the SFP cage, fallback to doing an I2C read and seeing if it's ACKed
+	 * to determine if a module is present
+	 */
+	if (sfp_cage_status == IXGBE_SFP_CAGE_UNKNOWN) {
+		PMD_DRV_LOG(DEBUG,
+			    "SFP present unknown (int? %d), try I2C read\n",
+			    have_int);
+
+		/* Rather than calling identify_sfp, which will read a lot of I2C
+		 * registers (and in a slow processor intensive fashion due to
+		 * bit-banging, just read the SFF ID register, which is at a
+		 * common address across SFP/SFP+/QSFP modules and see if
+		 * there's a NACK.  This works since we only expect a NACK if no
+		 * module is present
+		 */
+		err = ixgbe_read_i2c_eeprom(hw, IXGBE_SFF_IDENTIFIER, &sff_id);
+		if (err != IXGBE_SUCCESS) {
+			PMD_DRV_LOG(DEBUG, "Received I2C NAK from SFP, set NEED_SFP_SETUP flag\n");
+			intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
+			sfp_cage_status = IXGBE_SFP_CAGE_EMPTY;
+		} else {
+			PMD_DRV_LOG(DEBUG, "SFP ID read ACKed");
+			sfp_cage_status = IXGBE_SFP_CAGE_FULL;
+		}
+	}
+
+	if (sfp_cage_status == IXGBE_SFP_CAGE_EMPTY) {
+		PMD_DRV_LOG(DEBUG, "SFP absent, cage_status %d\n", sfp_cage_status);
+		return;
+	}
+
+	/* No setup requested?	Nothing to do */
+	if (!(intr->flags & IXGBE_FLAG_NEED_SFP_SETUP))
+		return;
+
+	err = ixgbe_sfp_id_and_setup(dev);
+	if (err) {
+		PMD_DRV_LOG(DEBUG, "failed to ID & setup SFP %d", err);
+		return;
+	}
+
+	/* Setup is done, clear the flag, but make sure link config runs for new SFP */
+	intr->flags &= ~IXGBE_FLAG_NEED_SFP_SETUP;
+	intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
+
+	/*
+	 * Since this is a new SFP, clear the old advertised speed mask so we don't
+	 * end up using an old slower rate
+	 */
+	hw->phy.autoneg_advertised = 0;
+}
+
+static void
+ixgbe_link_service(struct rte_eth_dev *dev)
+{
+	struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	struct ixgbe_interrupt *intr =
+		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
+	bool link_up, autoneg = false, have_int = false;
+	u32 speed;
+	s32 err;
+
+	/* Test if we have a LSC interrupt for this platform, if not we need to
+	 * manually check the link register since IXGBE_FLAG_NEED_LINK_CONFIG
+	 * will never be set in the interrupt handler
+	 */
+#ifndef RTE_EXEC_ENV_FREEBSD
+	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
+	if (rte_intr_allow_others(intr_handle)) {
+		/* check if lsc interrupt is enabled */
+		if (dev->data->dev_conf.intr_conf.lsc)
+			have_int = true;
+	}
+#endif /* #ifdef RTE_EXEC_ENV_FREEBSD */
+
+	/* Skip if we still need to setup an SFP, or if no link config requested
+	 */
+	if ((intr->flags & IXGBE_FLAG_NEED_SFP_SETUP) ||
+	    (!(intr->flags & IXGBE_FLAG_NEED_LINK_CONFIG) && have_int))
+		return;
+
+	if (!have_int && !(intr->flags & IXGBE_FLAG_NEED_LINK_CONFIG)) {
+		err = ixgbe_check_link(hw, &speed, &link_up, 0);
+		if (!err && !link_up) {
+			intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
+			PMD_DRV_LOG(DEBUG, "Link down, no LSC, set NEED_LINK_CONFIG\n");
+		} else {
+			return;
+		}
+	}
+
+	speed = hw->phy.autoneg_advertised;
+	if (!speed)
+		ixgbe_get_link_capabilities(hw, &speed, &autoneg);
+
+	err = ixgbe_setup_link(hw, speed, true);
+	if (err) {
+		PMD_DRV_LOG(ERR, "ixgbe_setup_link failed %d", err);
+		return;
+	}
+
+	intr->flags &= ~IXGBE_FLAG_NEED_LINK_CONFIG;
+}
+
+static void
+ixgbe_link_update_service(struct rte_eth_dev *dev)
+{
+	/* Update internal link status, waiting for link */
+	if (!ixgbe_dev_link_update(dev, 0)) {
+		ixgbe_dev_link_status_print(dev);
+		rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_LSC, NULL);
+	}
+}
+
+/*
+ * Service task thread to handle periodic tasks
+ */
+static void *
+ixgbe_dev_service_thread_handler(void *param)
+{
+	struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
+	struct ixgbe_hw *hw =
+		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	uint64_t start, ticks, service_ms;
+	uint32_t speed;
+	s32 err;
+	bool link_up;
+
+	while (1) {
+		ixgbe_sfp_service(dev);
+		ixgbe_link_service(dev);
+		ixgbe_link_update_service(dev);
+
+		/* Run the service thread handler more frequently when link is
+		 * down to reduce link up latency (every 200ms vs 1s)
+		 *
+		 * Use a number of smaller sleeps to decrease exit latency when
+		 * ixgbe_dev_stop() wants this thread to join
+		 */
+		err = ixgbe_check_link(hw, &speed, &link_up, 0);
+		if (err == IXGBE_SUCCESS && link_up)
+			service_ms = 2000;
+		else
+			service_ms = 100;
+
+		/* Call msec_delay in a loop with several smaller sleeps to
+		 * provide periodic thread cancellation points
+		 */
+		start = rte_get_timer_cycles();
+		ticks = (uint64_t)service_ms * rte_get_timer_hz() / 1E3;
+		while ((rte_get_timer_cycles() - start) < ticks)
+			msec_delay(100);
+	}
+
+	/* Never return */
+	return NULL;
+}
+
+static s32
+eth_ixgbe_check_mac_link_generic(struct ixgbe_hw *hw, ixgbe_link_speed *speed,
+				 bool *link_up, bool link_up_wait_to_complete)
+{
+	if (ixgbe_need_crosstalk_fix(hw)) {
+		enum ixgbe_sfp_cage_status sfp_cage_status;
+
+		sfp_cage_status = ixgbe_check_sfp_cage(hw);
+		if (sfp_cage_status != IXGBE_SFP_CAGE_FULL) {
+			*link_up = false;
+			*speed = IXGBE_LINK_SPEED_UNKNOWN;
+			return IXGBE_SUCCESS;
+		}
+	}
+
+	return ixgbe_check_mac_link_generic(hw, speed, link_up, link_up_wait_to_complete);
+}
+
 /*
  * This function is based on code in ixgbe_attach() in base/ixgbe.c.
  * It returns 0 on success.
@@ -1071,7 +1395,6 @@ ixgbe_parse_devargs(struct ixgbe_adapter *adapter,
 static int
 eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)
 {
-	struct ixgbe_adapter *ad = eth_dev->data->dev_private;
 	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
 	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
 	struct ixgbe_hw *hw =
@@ -1086,6 +1409,7 @@ eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)
 		IXGBE_DEV_PRIVATE_TO_FILTER_INFO(eth_dev->data->dev_private);
 	struct ixgbe_bw_conf *bw_conf =
 		IXGBE_DEV_PRIVATE_TO_BW_CONF(eth_dev->data->dev_private);
+	struct ixgbe_mac_info *mac = &hw->mac;
 	uint32_t ctrl_ext;
 	uint16_t csum;
 	int diag, i, ret;
@@ -1126,7 +1450,6 @@ eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)
 		return 0;
 	}
 
-	rte_atomic32_clear(&ad->link_thread_running);
 	ixgbe_parse_devargs(eth_dev->data->dev_private,
 			    pci_dev->device.devargs);
 	rte_eth_copy_pci_info(eth_dev, pci_dev);
@@ -1158,6 +1481,17 @@ eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)
 		return -EIO;
 	}
 
+	/* override mac_link_check to check for sfp cage full/empty */
+	switch (hw->mac.type) {
+	case ixgbe_mac_X550EM_x:
+	case ixgbe_mac_X550EM_a:
+	case ixgbe_mac_82599EB:
+		mac->ops.check_link = eth_ixgbe_check_mac_link_generic;
+		break;
+	default:
+		break;
+	}
+
 	/* pick up the PCI bus settings for reporting later */
 	ixgbe_get_bus_info(hw);
 
@@ -1581,7 +1915,6 @@ eth_ixgbevf_dev_init(struct rte_eth_dev *eth_dev)
 {
 	int diag;
 	uint32_t tc, tcs;
-	struct ixgbe_adapter *ad = eth_dev->data->dev_private;
 	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
 	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
 	struct ixgbe_hw *hw =
@@ -1624,7 +1957,6 @@ eth_ixgbevf_dev_init(struct rte_eth_dev *eth_dev)
 		return 0;
 	}
 
-	rte_atomic32_clear(&ad->link_thread_running);
 	ixgbevf_parse_devargs(eth_dev->data->dev_private,
 			      pci_dev->device.devargs);
 
@@ -2426,6 +2758,8 @@ ixgbe_dev_configure(struct rte_eth_dev *dev)
 	struct ixgbe_interrupt *intr =
 		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
 	struct ixgbe_adapter *adapter = dev->data->dev_private;
+	struct ixgbe_hw *hw =
+		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
 	int ret;
 
 	PMD_INIT_FUNC_TRACE();
@@ -2444,6 +2778,10 @@ ixgbe_dev_configure(struct rte_eth_dev *dev)
 	/* set flag to update link status after init */
 	intr->flags |= IXGBE_FLAG_NEED_LINK_UPDATE;
 
+	/* set flag to setup SFP after init */
+	if (ixgbe_is_sfp(hw))
+		intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
+
 	/*
 	 * Initialize to TRUE. If any of Rx queues doesn't meet the bulk
 	 * allocation or vector Rx preconditions we will reset it.
@@ -2463,13 +2801,24 @@ ixgbe_dev_phy_intr_setup(struct rte_eth_dev *dev)
 		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
 	uint32_t gpie;
 
-	/* only set up it on X550EM_X */
+	/* only set up it on X550EM_X (external PHY interrupt)
+	 * or on X550EM_a_* for SFP_PRSNT# de-assertion (SFP removal)
+	 */
 	if (hw->mac.type == ixgbe_mac_X550EM_x) {
 		gpie = IXGBE_READ_REG(hw, IXGBE_GPIE);
 		gpie |= IXGBE_SDP0_GPIEN_X550EM_x;
 		IXGBE_WRITE_REG(hw, IXGBE_GPIE, gpie);
 		if (hw->phy.type == ixgbe_phy_x550em_ext_t)
 			intr->mask |= IXGBE_EICR_GPI_SDP0_X550EM_x;
+	} else if (hw->mac.type == ixgbe_mac_X550EM_a) {
+		gpie = IXGBE_READ_REG(hw, IXGBE_GPIE);
+		gpie |= IXGBE_SDP0_GPIEN_X550EM_a;
+		IXGBE_WRITE_REG(hw, IXGBE_GPIE, gpie);
+		intr->mask |= IXGBE_EICR_GPI_SDP0_X550EM_a;
+	} else {
+		PMD_DRV_LOG(DEBUG,
+			    "No PHY/SFP interrupt for MAC %d, PHY %d\n",
+			    hw->mac.type, hw->phy.type);
 	}
 }
 
@@ -2592,8 +2941,11 @@ ixgbe_flow_ctrl_enable(struct rte_eth_dev *dev, struct ixgbe_hw *hw)
 static int
 ixgbe_dev_start(struct rte_eth_dev *dev)
 {
+	struct ixgbe_adapter *ad = dev->data->dev_private;
 	struct ixgbe_hw *hw =
 		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	struct ixgbe_interrupt *intr =
+		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
 	struct ixgbe_vf_info *vfinfo =
 		*IXGBE_DEV_PRIVATE_TO_P_VFDATA(dev->data->dev_private);
 	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
@@ -2614,9 +2966,6 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
 
 	PMD_INIT_FUNC_TRACE();
 
-	/* Stop the link setup handler before resetting the HW. */
-	ixgbe_dev_wait_setup_link_complete(dev, 0);
-
 	/* disable uio/vfio intr/eventfd mapping */
 	rte_intr_disable(intr_handle);
 
@@ -2732,12 +3081,6 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
 		}
 	}
 
-	if (ixgbe_is_sfp(hw) && hw->phy.multispeed_fiber) {
-		err = hw->mac.ops.setup_sfp(hw);
-		if (err)
-			goto error;
-	}
-
 	if (hw->mac.ops.get_media_type(hw) == ixgbe_media_type_copper) {
 		/* Turn on the copper */
 		ixgbe_set_phy_power(hw, true);
@@ -2849,6 +3192,20 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
 	ixgbe_l2_tunnel_conf(dev);
 	ixgbe_filter_restore(dev);
 
+	/* Spawn service thread */
+	if (ixgbe_is_sfp(hw)) {
+		intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
+		err = rte_ctrl_thread_create(&ad->service_thread_tid,
+					     "ixgbe-service-thread",
+					     NULL,
+					     ixgbe_dev_service_thread_handler,
+					     dev);
+		if (err) {
+			PMD_DRV_LOG(ERR, "service_thread err");
+			goto error;
+		}
+	}
+
 	if (tm_conf->root && !tm_conf->committed)
 		PMD_DRV_LOG(WARNING,
 			    "please call hierarchy_commit() "
@@ -2859,12 +3216,6 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
 	if (err)
 		goto error;
 
-	/*
-	 * Update link status right before return, because it may
-	 * start link configuration process in a separate thread.
-	 */
-	ixgbe_dev_link_update(dev, 0);
-
 	/* setup the macsec setting register */
 	if (macsec_setting->offload_en)
 		ixgbe_dev_macsec_register_enable(dev, macsec_setting);
@@ -2894,13 +3245,21 @@ ixgbe_dev_stop(struct rte_eth_dev *dev)
 	int vf;
 	struct ixgbe_tm_conf *tm_conf =
 		IXGBE_DEV_PRIVATE_TO_TM_CONF(dev->data->dev_private);
+	void *res;
+	s32 err;
 
 	if (hw->adapter_stopped)
 		return 0;
 
 	PMD_INIT_FUNC_TRACE();
 
-	ixgbe_dev_wait_setup_link_complete(dev, 0);
+	/* Cancel the service thread, and wait for it to join */
+	err = pthread_cancel(adapter->service_thread_tid);
+	if (err)
+		PMD_DRV_LOG(ERR, "failed to cancel service thread %d", err);
+	err = pthread_join(adapter->service_thread_tid, &res);
+	if (err)
+		PMD_DRV_LOG(ERR, "failed to join service thread %d", err);
 
 	/* disable interrupts */
 	ixgbe_disable_intr(hw);
@@ -2979,7 +3338,6 @@ ixgbe_dev_set_link_up(struct rte_eth_dev *dev)
 	} else {
 		/* Turn on the laser */
 		ixgbe_enable_tx_laser(hw);
-		ixgbe_dev_link_update(dev, 0);
 	}
 
 	return 0;
@@ -3010,7 +3368,6 @@ ixgbe_dev_set_link_down(struct rte_eth_dev *dev)
 	} else {
 		/* Turn off the laser */
 		ixgbe_disable_tx_laser(hw);
-		ixgbe_dev_link_update(dev, 0);
 	}
 
 	return 0;
@@ -4163,57 +4520,6 @@ ixgbevf_check_link(struct ixgbe_hw *hw, ixgbe_link_speed *speed,
 	return ret_val;
 }
 
-/*
- * If @timeout_ms was 0, it means that it will not return until link complete.
- * It returns 1 on complete, return 0 on timeout.
- */
-static int
-ixgbe_dev_wait_setup_link_complete(struct rte_eth_dev *dev, uint32_t timeout_ms)
-{
-#define WARNING_TIMEOUT    9000 /* 9s  in total */
-	struct ixgbe_adapter *ad = dev->data->dev_private;
-	uint32_t timeout = timeout_ms ? timeout_ms : WARNING_TIMEOUT;
-
-	while (rte_atomic32_read(&ad->link_thread_running)) {
-		msec_delay(1);
-		timeout--;
-
-		if (timeout_ms) {
-			if (!timeout)
-				return 0;
-		} else if (!timeout) {
-			/* It will not return until link complete */
-			timeout = WARNING_TIMEOUT;
-			PMD_DRV_LOG(ERR, "IXGBE link thread not complete too long time!");
-		}
-	}
-
-	return 1;
-}
-
-static void *
-ixgbe_dev_setup_link_thread_handler(void *param)
-{
-	struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
-	struct ixgbe_adapter *ad = dev->data->dev_private;
-	struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
-	struct ixgbe_interrupt *intr =
-		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
-	u32 speed;
-	bool autoneg = false;
-
-	pthread_detach(pthread_self());
-	speed = hw->phy.autoneg_advertised;
-	if (!speed)
-		ixgbe_get_link_capabilities(hw, &speed, &autoneg);
-
-	ixgbe_setup_link(hw, speed, true);
-
-	intr->flags &= ~IXGBE_FLAG_NEED_LINK_CONFIG;
-	rte_atomic32_clear(&ad->link_thread_running);
-	return NULL;
-}
-
 /*
  * In freebsd environment, nic_uio drivers do not support interrupts,
  * rte_intr_callback_register() will fail to register interrupts.
@@ -4256,8 +4562,6 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 	struct ixgbe_adapter *ad = dev->data->dev_private;
 	struct rte_eth_link link;
 	ixgbe_link_speed link_speed = IXGBE_LINK_SPEED_UNKNOWN;
-	struct ixgbe_interrupt *intr =
-		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
 	bool link_up;
 	int diag;
 	int wait = 1;
@@ -4272,9 +4576,6 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 
 	hw->mac.get_link_status = true;
 
-	if (intr->flags & IXGBE_FLAG_NEED_LINK_CONFIG)
-		return rte_eth_linkstatus_set(dev, &link);
-
 	/* check if it needs to wait to complete, if lsc interrupt is enabled */
 	if (wait_to_complete == 0 || dev->data->dev_conf.intr_conf.lsc != 0)
 		wait = 0;
@@ -4289,7 +4590,7 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 	else
 		diag = ixgbe_check_link(hw, &link_speed, &link_up, wait);
 
-	if (diag != 0) {
+	if (diag != 0 || !link_up) {
 		link.link_speed = RTE_ETH_SPEED_NUM_100M;
 		link.link_duplex = RTE_ETH_LINK_FULL_DUPLEX;
 		return rte_eth_linkstatus_set(dev, &link);
@@ -4302,32 +4603,6 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 			link_up = 0;
 	}
 
-	if (link_up == 0) {
-		if (ixgbe_get_media_type(hw) == ixgbe_media_type_fiber) {
-			ixgbe_dev_wait_setup_link_complete(dev, 0);
-			if (rte_atomic32_test_and_set(&ad->link_thread_running)) {
-				/* To avoid race condition between threads, set
-				 * the IXGBE_FLAG_NEED_LINK_CONFIG flag only
-				 * when there is no link thread running.
-				 */
-				intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
-				if (rte_ctrl_thread_create(&ad->link_thread_tid,
-					"ixgbe-link-handler",
-					NULL,
-					ixgbe_dev_setup_link_thread_handler,
-					dev) < 0) {
-					PMD_DRV_LOG(ERR,
-						"Create link thread failed!");
-					rte_atomic32_clear(&ad->link_thread_running);
-				}
-			} else {
-				PMD_DRV_LOG(ERR,
-					"Other link thread is running now!");
-			}
-		}
-		return rte_eth_linkstatus_set(dev, &link);
-	}
-
 	link.link_status = RTE_ETH_LINK_UP;
 	link.link_duplex = RTE_ETH_LINK_FULL_DUPLEX;
 
@@ -4533,8 +4808,6 @@ ixgbe_dev_interrupt_get_status(struct rte_eth_dev *dev)
 	eicr = IXGBE_READ_REG(hw, IXGBE_EICR);
 	PMD_DRV_LOG(DEBUG, "eicr %x", eicr);
 
-	intr->flags = 0;
-
 	/* set flag for async link update */
 	if (eicr & IXGBE_EICR_LSC)
 		intr->flags |= IXGBE_FLAG_NEED_LINK_UPDATE;
@@ -4550,6 +4823,14 @@ ixgbe_dev_interrupt_get_status(struct rte_eth_dev *dev)
 	    (eicr & IXGBE_EICR_GPI_SDP0_X550EM_x))
 		intr->flags |= IXGBE_FLAG_PHY_INTERRUPT;
 
+	/* Check for loss of SFP */
+	/* TODO - For platforms that don't have this flag, do we need to set
+	 *  NEED_SFP_SETUP on LSC if we're a SFP platform?
+	 */
+	if (hw->mac.type ==  ixgbe_mac_X550EM_a &&
+	    (eicr & IXGBE_EICR_GPI_SDP0_X550EM_a))
+		intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
+
 	return 0;
 }
 
@@ -4601,11 +4882,13 @@ ixgbe_dev_link_status_print(struct rte_eth_dev *dev)
 static int
 ixgbe_dev_interrupt_action(struct rte_eth_dev *dev)
 {
+	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
 	struct ixgbe_interrupt *intr =
 		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
-	int64_t timeout;
 	struct ixgbe_hw *hw =
 		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	int64_t timeout;
 
 	PMD_DRV_LOG(DEBUG, "intr action type %d", intr->flags);
 
@@ -4640,16 +4923,14 @@ ixgbe_dev_interrupt_action(struct rte_eth_dev *dev)
 		if (rte_eal_alarm_set(timeout * 1000,
 				      ixgbe_dev_interrupt_delayed_handler, (void *)dev) < 0)
 			PMD_DRV_LOG(ERR, "Error setting alarm");
-		else {
-			/* remember original mask */
-			intr->mask_original = intr->mask;
+		else
 			/* only disable lsc interrupt */
 			intr->mask &= ~IXGBE_EIMS_LSC;
-		}
 	}
 
 	PMD_DRV_LOG(DEBUG, "enable intr immediately");
 	ixgbe_enable_intr(dev);
+	rte_intr_ack(intr_handle);
 
 	return 0;
 }
@@ -4672,8 +4953,6 @@ static void
 ixgbe_dev_interrupt_delayed_handler(void *param)
 {
 	struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
-	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
-	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
 	struct ixgbe_interrupt *intr =
 		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
 	struct ixgbe_hw *hw =
@@ -4703,13 +4982,10 @@ ixgbe_dev_interrupt_delayed_handler(void *param)
 		intr->flags &= ~IXGBE_FLAG_MACSEC;
 	}
 
-	/* restore original mask */
-	intr->mask = intr->mask_original;
-	intr->mask_original = 0;
+	if (dev->data->dev_conf.intr_conf.lsc != 0)
+		intr->mask |= IXGBE_EICR_LSC;
 
-	PMD_DRV_LOG(DEBUG, "enable intr in delayed handler S[%08x]", eicr);
 	ixgbe_enable_intr(dev);
-	rte_intr_ack(intr_handle);
 }
 
 /**
@@ -5351,9 +5627,6 @@ ixgbevf_dev_start(struct rte_eth_dev *dev)
 
 	PMD_INIT_FUNC_TRACE();
 
-	/* Stop the link setup handler before resetting the HW. */
-	ixgbe_dev_wait_setup_link_complete(dev, 0);
-
 	err = hw->mac.ops.reset_hw(hw);
 
 	/**
@@ -5433,12 +5706,6 @@ ixgbevf_dev_start(struct rte_eth_dev *dev)
 	/* Re-enable interrupt for VF */
 	ixgbevf_intr_enable(dev);
 
-	/*
-	 * Update link status right before return, because it may
-	 * start link configuration process in a separate thread.
-	 */
-	ixgbevf_dev_link_update(dev, 0);
-
 	hw->adapter_stopped = false;
 
 	return 0;
@@ -5457,8 +5724,6 @@ ixgbevf_dev_stop(struct rte_eth_dev *dev)
 
 	PMD_INIT_FUNC_TRACE();
 
-	ixgbe_dev_wait_setup_link_complete(dev, 0);
-
 	ixgbevf_intr_disable(dev);
 
 	dev->data->dev_started = 0;
diff --git a/drivers/net/ixgbe/ixgbe_ethdev.h b/drivers/net/ixgbe/ixgbe_ethdev.h
index cc6049a66a..d8d0bebd04 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.h
+++ b/drivers/net/ixgbe/ixgbe_ethdev.h
@@ -29,6 +29,7 @@
 #define IXGBE_FLAG_PHY_INTERRUPT    (uint32_t)(1 << 2)
 #define IXGBE_FLAG_MACSEC           (uint32_t)(1 << 3)
 #define IXGBE_FLAG_NEED_LINK_CONFIG (uint32_t)(1 << 4)
+#define IXGBE_FLAG_NEED_SFP_SETUP   ((uint32_t)(1 << 5))
 
 /*
  * Defines that were not part of ixgbe_type.h as they are not used by the
@@ -223,8 +224,6 @@ struct ixgbe_rte_flow_rss_conf {
 struct ixgbe_interrupt {
 	uint32_t flags;
 	uint32_t mask;
-	/*to save original mask during delayed handler */
-	uint32_t mask_original;
 };
 
 struct ixgbe_stat_mapping_registers {
@@ -510,7 +509,7 @@ struct ixgbe_adapter {
 	uint8_t pflink_fullchk;
 	uint8_t mac_ctrl_frame_fwd;
 	rte_atomic32_t link_thread_running;
-	pthread_t link_thread_tid;
+	pthread_t service_thread_tid;
 };
 
 struct ixgbe_vf_representor {
@@ -673,6 +672,15 @@ int ixgbe_syn_filter_set(struct rte_eth_dev *dev,
 			struct rte_eth_syn_filter *filter,
 			bool add);
 
+enum ixgbe_sfp_cage_status {
+	IXGBE_SFP_CAGE_EMPTY = 0,
+	IXGBE_SFP_CAGE_FULL,
+	IXGBE_SFP_CAGE_UNKNOWN = -1,
+	IXGBE_SFP_CAGE_NOCAGE = -2,
+};
+enum ixgbe_sfp_cage_status ixgbe_check_sfp_cage(struct ixgbe_hw *hw);
+
+
 /**
  * l2 tunnel configuration.
  */
-- 
2.25.1

[PATCH v6 1/2] net/ixgbe: Limit SDP3 check of TX_DISABLE to appropriate devices

[Jeff Daly]

1ca05831b9b added a check that SDP3 (used as a TX_DISABLE output to the
SFP cage on these cards) is not asserted to avoid incorrectly reporting
link up when the SFP's laser is turned off.

ff8162cb957 limited this workaround to fiber ports

This patch:
* Adds devarg 'fiber_sdp3_no_tx_disable' not all fiber ixgbe devs use
  SDP3 as TX_DISABLE

Fixes: 1ca05831b9b ("net/ixgbe: fix link status")
Fixes: ff8162cb957 ("net/ixgbe: fix link status")
Cc: stable@dpdk.org

Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 39 +++++++++++++++++++++++++++++++-
 drivers/net/ixgbe/ixgbe_ethdev.h |  3 +++
 2 files changed, 41 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 2da3f67bbc..f31bbb7895 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -128,6 +128,13 @@
 #define IXGBE_EXVET_VET_EXT_SHIFT              16
 #define IXGBE_DMATXCTL_VT_MASK                 0xFFFF0000
 
+#define IXGBE_DEVARG_FIBER_SDP3_NOT_TX_DISABLE	"fiber_sdp3_no_tx_disable"
+
+static const char * const ixgbe_valid_arguments[] = {
+	IXGBE_DEVARG_FIBER_SDP3_NOT_TX_DISABLE,
+	NULL
+};
+
 #define IXGBEVF_DEVARG_PFLINK_FULLCHK		"pflink_fullchk"
 
 static const char * const ixgbevf_valid_arguments[] = {
@@ -348,6 +355,8 @@ static int ixgbe_dev_udp_tunnel_port_del(struct rte_eth_dev *dev,
 static int ixgbe_filter_restore(struct rte_eth_dev *dev);
 static void ixgbe_l2_tunnel_conf(struct rte_eth_dev *dev);
 static int ixgbe_wait_for_link_up(struct ixgbe_hw *hw);
+static int devarg_handle_int(__rte_unused const char *key, const char *value,
+			     void *extra_args);
 
 /*
  * Define VF Stats MACRO for Non "cleared on read" register
@@ -1032,6 +1041,29 @@ ixgbe_swfw_lock_reset(struct ixgbe_hw *hw)
 	ixgbe_release_swfw_semaphore(hw, mask);
 }
 
+static void
+ixgbe_parse_devargs(struct ixgbe_adapter *adapter,
+		      struct rte_devargs *devargs)
+{
+	struct rte_kvargs *kvlist;
+	uint16_t sdp3_no_tx_disable;
+
+	if (devargs == NULL)
+		return;
+
+	kvlist = rte_kvargs_parse(devargs->args, ixgbe_valid_arguments);
+	if (kvlist == NULL)
+		return;
+
+	if (rte_kvargs_count(kvlist, IXGBE_DEVARG_FIBER_SDP3_NOT_TX_DISABLE) == 1 &&
+	    rte_kvargs_process(kvlist, IXGBE_DEVARG_FIBER_SDP3_NOT_TX_DISABLE,
+			       devarg_handle_int, &sdp3_no_tx_disable) == 0 &&
+	    sdp3_no_tx_disable == 1)
+		adapter->sdp3_no_tx_disable = 1;
+
+	rte_kvargs_free(kvlist);
+}
+
 /*
  * This function is based on code in ixgbe_attach() in base/ixgbe.c.
  * It returns 0 on success.
@@ -1095,6 +1127,8 @@ eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)
 	}
 
 	rte_atomic32_clear(&ad->link_thread_running);
+	ixgbe_parse_devargs(eth_dev->data->dev_private,
+			    pci_dev->device.devargs);
 	rte_eth_copy_pci_info(eth_dev, pci_dev);
 	eth_dev->data->dev_flags |= RTE_ETH_DEV_AUTOFILL_QUEUE_XSTATS;
 
@@ -4261,7 +4295,8 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 		return rte_eth_linkstatus_set(dev, &link);
 	}
 
-	if (ixgbe_get_media_type(hw) == ixgbe_media_type_fiber) {
+	if (ixgbe_get_media_type(hw) == ixgbe_media_type_fiber &&
+	    !ad->sdp3_no_tx_disable) {
 		esdp_reg = IXGBE_READ_REG(hw, IXGBE_ESDP);
 		if ((esdp_reg & IXGBE_ESDP_SDP3))
 			link_up = 0;
@@ -8250,6 +8285,8 @@ ixgbe_dev_macsec_register_disable(struct rte_eth_dev *dev)
 RTE_PMD_REGISTER_PCI(net_ixgbe, rte_ixgbe_pmd);
 RTE_PMD_REGISTER_PCI_TABLE(net_ixgbe, pci_id_ixgbe_map);
 RTE_PMD_REGISTER_KMOD_DEP(net_ixgbe, "* igb_uio | uio_pci_generic | vfio-pci");
+RTE_PMD_REGISTER_PARAM_STRING(net_ixgbe,
+			      IXGBE_DEVARG_FIBER_SDP3_NOT_TX_DISABLE "=<0|1>");
 RTE_PMD_REGISTER_PCI(net_ixgbe_vf, rte_ixgbevf_pmd);
 RTE_PMD_REGISTER_PCI_TABLE(net_ixgbe_vf, pci_id_ixgbevf_map);
 RTE_PMD_REGISTER_KMOD_DEP(net_ixgbe_vf, "* igb_uio | vfio-pci");
diff --git a/drivers/net/ixgbe/ixgbe_ethdev.h b/drivers/net/ixgbe/ixgbe_ethdev.h
index 69e0e82a5b..cc6049a66a 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.h
+++ b/drivers/net/ixgbe/ixgbe_ethdev.h
@@ -501,6 +501,9 @@ struct ixgbe_adapter {
 	/* For RSS reta table update */
 	uint8_t rss_reta_updated;
 
+	/* Used for limiting SDP3 TX_DISABLE checks */
+	uint8_t sdp3_no_tx_disable;
+
 	/* Used for VF link sync with PF's physical and logical (by checking
 	 * mailbox status) link status.
 	 */
-- 
2.25.1

RE: [PATCH v6 1/2] net/ixgbe: Limit SDP3 check of TX_DISABLE to appropriate devices

[Wang, Haiyue]

> -----Original Message-----
> From: Jeff Daly <jeffd@silicom-usa.com>
> Sent: Wednesday, April 13, 2022 01:42
> To: dev@dpdk.org
> Cc: stable@dpdk.org; Wang, Haiyue <haiyue.wang@intel.com>; Xiaolong Ye <xiaolong.ye@intel.com>; Xiao
> Zhang <xiao.zhang@intel.com>; Zhao1, Wei <wei.zhao1@intel.com>; Lunyuan Cui <lunyuanx.cui@intel.com>
> Subject: [PATCH v6 1/2] net/ixgbe: Limit SDP3 check of TX_DISABLE to appropriate devices
> 
> 1ca05831b9b added a check that SDP3 (used as a TX_DISABLE output to the
> SFP cage on these cards) is not asserted to avoid incorrectly reporting
> link up when the SFP's laser is turned off.
> 
> ff8162cb957 limited this workaround to fiber ports
> 
> This patch:
> * Adds devarg 'fiber_sdp3_no_tx_disable' not all fiber ixgbe devs use
>   SDP3 as TX_DISABLE
> 
> Fixes: 1ca05831b9b ("net/ixgbe: fix link status")
> Fixes: ff8162cb957 ("net/ixgbe: fix link status")
> Cc: stable@dpdk.org

This is new for soc for BIG change, not cc to stable.

> 
> Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> ---
>  drivers/net/ixgbe/ixgbe_ethdev.c | 39 +++++++++++++++++++++++++++++++-
>  drivers/net/ixgbe/ixgbe_ethdev.h |  3 +++
>  2 files changed, 41 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
> index 2da3f67bbc..f31bbb7895 100644
> --- a/drivers/net/ixgbe/ixgbe_ethdev.c
> +++ b/drivers/net/ixgbe/ixgbe_ethdev.c
> @@ -128,6 +128,13 @@
>  #define IXGBE_EXVET_VET_EXT_SHIFT              16
>  #define IXGBE_DMATXCTL_VT_MASK                 0xFFFF0000
> 
> +#define IXGBE_DEVARG_FIBER_SDP3_NOT_TX_DISABLE	"fiber_sdp3_no_tx_disable"
> +

'platform' may be a good arg for the soc related change.

dpdk-testpmd -a af:10.0,platform=xxx - -i

enum ixgbe_platform_type {
	ixgbe_platform_unknown = 0,
	ixgbe_platform_soc_atom, ??? You can specify it.



enum ixgbe_media_type ixgbe_get_platform_type(xxx)
{
	return xxx;
}


> +static const char * const ixgbe_valid_arguments[] = {
> +	IXGBE_DEVARG_FIBER_SDP3_NOT_TX_DISABLE,
> +	NULL
> +};
> +
>  #define IXGBEVF_DEVARG_PFLINK_FULLCHK		"pflink_fullchk"
> 
>  static const char * const ixgbevf_valid_arguments[] = {
> @@ -348,6 +355,8 @@ static int ixgbe_dev_udp_tunnel_port_del(struct rte_eth_dev *dev,
>  static int ixgbe_filter_restore(struct rte_eth_dev *dev);
>  static void ixgbe_l2_tunnel_conf(struct rte_eth_dev *dev);
>  static int ixgbe_wait_for_link_up(struct ixgbe_hw *hw);
> +static int devarg_handle_int(__rte_unused const char *key, const char *value,
> +			     void *extra_args);
> 
>  /*
>   * Define VF Stats MACRO for Non "cleared on read" register
> @@ -1032,6 +1041,29 @@ ixgbe_swfw_lock_reset(struct ixgbe_hw *hw)
>  	ixgbe_release_swfw_semaphore(hw, mask);
>  }
> 
> +static void
> +ixgbe_parse_devargs(struct ixgbe_adapter *adapter,
> +		      struct rte_devargs *devargs)
> +{
> +	struct rte_kvargs *kvlist;
> +	uint16_t sdp3_no_tx_disable;
> +
> +	if (devargs == NULL)
> +		return;
> +
> +	kvlist = rte_kvargs_parse(devargs->args, ixgbe_valid_arguments);
> +	if (kvlist == NULL)
> +		return;
> +
> +	if (rte_kvargs_count(kvlist, IXGBE_DEVARG_FIBER_SDP3_NOT_TX_DISABLE) == 1 &&
> +	    rte_kvargs_process(kvlist, IXGBE_DEVARG_FIBER_SDP3_NOT_TX_DISABLE,
> +			       devarg_handle_int, &sdp3_no_tx_disable) == 0 &&
> +	    sdp3_no_tx_disable == 1)
> +		adapter->sdp3_no_tx_disable = 1;
> +
> +	rte_kvargs_free(kvlist);
> +}
> +
>  /*
>   * This function is based on code in ixgbe_attach() in base/ixgbe.c.
>   * It returns 0 on success.
> @@ -1095,6 +1127,8 @@ eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)
>  	}
> 
>  	rte_atomic32_clear(&ad->link_thread_running);
> +	ixgbe_parse_devargs(eth_dev->data->dev_private,
> +			    pci_dev->device.devargs);
>  	rte_eth_copy_pci_info(eth_dev, pci_dev);
>  	eth_dev->data->dev_flags |= RTE_ETH_DEV_AUTOFILL_QUEUE_XSTATS;
> 
> @@ -4261,7 +4295,8 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
>  		return rte_eth_linkstatus_set(dev, &link);
>  	}
> 
> -	if (ixgbe_get_media_type(hw) == ixgbe_media_type_fiber) {
> +	if (ixgbe_get_media_type(hw) == ixgbe_media_type_fiber &&
> +	    !ad->sdp3_no_tx_disable) {
>  		esdp_reg = IXGBE_READ_REG(hw, IXGBE_ESDP);
>  		if ((esdp_reg & IXGBE_ESDP_SDP3))
>  			link_up = 0;
> @@ -8250,6 +8285,8 @@ ixgbe_dev_macsec_register_disable(struct rte_eth_dev *dev)
>  RTE_PMD_REGISTER_PCI(net_ixgbe, rte_ixgbe_pmd);
>  RTE_PMD_REGISTER_PCI_TABLE(net_ixgbe, pci_id_ixgbe_map);
>  RTE_PMD_REGISTER_KMOD_DEP(net_ixgbe, "* igb_uio | uio_pci_generic | vfio-pci");
> +RTE_PMD_REGISTER_PARAM_STRING(net_ixgbe,
> +			      IXGBE_DEVARG_FIBER_SDP3_NOT_TX_DISABLE "=<0|1>");
>  RTE_PMD_REGISTER_PCI(net_ixgbe_vf, rte_ixgbevf_pmd);
>  RTE_PMD_REGISTER_PCI_TABLE(net_ixgbe_vf, pci_id_ixgbevf_map);
>  RTE_PMD_REGISTER_KMOD_DEP(net_ixgbe_vf, "* igb_uio | vfio-pci");
> diff --git a/drivers/net/ixgbe/ixgbe_ethdev.h b/drivers/net/ixgbe/ixgbe_ethdev.h
> index 69e0e82a5b..cc6049a66a 100644
> --- a/drivers/net/ixgbe/ixgbe_ethdev.h
> +++ b/drivers/net/ixgbe/ixgbe_ethdev.h
> @@ -501,6 +501,9 @@ struct ixgbe_adapter {
>  	/* For RSS reta table update */
>  	uint8_t rss_reta_updated;
> 
> +	/* Used for limiting SDP3 TX_DISABLE checks */
> +	uint8_t sdp3_no_tx_disable;
> +
>  	/* Used for VF link sync with PF's physical and logical (by checking
>  	 * mailbox status) link status.
>  	 */
> --
> 2.25.1

RE: [PATCH v6 1/2] net/ixgbe: Limit SDP3 check of TX_DISABLE to appropriate devices

[Jeff Daly]

> -----Original Message-----
> From: Wang, Haiyue <haiyue.wang@intel.com>
> Sent: Tuesday, April 12, 2022 9:22 PM
> To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> Cc: stable@dpdk.org; Xiaolong Ye <xiaolong.ye@intel.com>; Xiao Zhang
> <xiao.zhang@intel.com>; Zhao1, Wei <wei.zhao1@intel.com>; Lunyuan Cui
> <lunyuanx.cui@intel.com>
> Subject: RE: [PATCH v6 1/2] net/ixgbe: Limit SDP3 check of TX_DISABLE to
> appropriate devices
> 
> Caution: This is an external email. Please take care when clicking links or
> opening attachments.
> 
> 
> > -----Original Message-----
> > From: Jeff Daly <jeffd@silicom-usa.com>
> > Sent: Wednesday, April 13, 2022 01:42
> > To: dev@dpdk.org
> > Cc: stable@dpdk.org; Wang, Haiyue <haiyue.wang@intel.com>; Xiaolong Ye
> > <xiaolong.ye@intel.com>; Xiao Zhang <xiao.zhang@intel.com>; Zhao1, Wei
> > <wei.zhao1@intel.com>; Lunyuan Cui <lunyuanx.cui@intel.com>
> > Subject: [PATCH v6 1/2] net/ixgbe: Limit SDP3 check of TX_DISABLE to
> > appropriate devices
> >
> > 1ca05831b9b added a check that SDP3 (used as a TX_DISABLE output to
> > the SFP cage on these cards) is not asserted to avoid incorrectly
> > reporting link up when the SFP's laser is turned off.
> >
> > ff8162cb957 limited this workaround to fiber ports
> >
> > This patch:
> > * Adds devarg 'fiber_sdp3_no_tx_disable' not all fiber ixgbe devs use
> >   SDP3 as TX_DISABLE
> >
> > Fixes: 1ca05831b9b ("net/ixgbe: fix link status")
> > Fixes: ff8162cb957 ("net/ixgbe: fix link status")
> > Cc: stable@dpdk.org
> 
> This is new for soc for BIG change, not cc to stable.
> 
> >
> > Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> > ---
> >  drivers/net/ixgbe/ixgbe_ethdev.c | 39
> > +++++++++++++++++++++++++++++++-  drivers/net/ixgbe/ixgbe_ethdev.h |
> > 3 +++
> >  2 files changed, 41 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c
> > b/drivers/net/ixgbe/ixgbe_ethdev.c
> > index 2da3f67bbc..f31bbb7895 100644
> > --- a/drivers/net/ixgbe/ixgbe_ethdev.c
> > +++ b/drivers/net/ixgbe/ixgbe_ethdev.c
> > @@ -128,6 +128,13 @@
> >  #define IXGBE_EXVET_VET_EXT_SHIFT              16
> >  #define IXGBE_DMATXCTL_VT_MASK                 0xFFFF0000
> >
> > +#define IXGBE_DEVARG_FIBER_SDP3_NOT_TX_DISABLE
> "fiber_sdp3_no_tx_disable"
> > +
> 
> 'platform' may be a good arg for the soc related change.
> 
> dpdk-testpmd -a af:10.0,platform=xxx - -i
> 
> enum ixgbe_platform_type {
>         ixgbe_platform_unknown = 0,
>         ixgbe_platform_soc_atom, ??? You can specify it.
> 
> 
> 
> enum ixgbe_media_type ixgbe_get_platform_type(xxx) {
>         return xxx;
> }
> 

This patchset is not explicitly for SoC platform support.  *Any* implementation may or may not use
SDP3 as TX_DISABLE.   The previous version of the patch added a check for the mac being an 82599 that
uses fiber SFP rather than just a fiber SFP.  Our platform specifically can be fiber SFP, but TX_DISABLE 
is not SDP3.  However, our platform may not be the only implementation that doesn't use SDP3 this way.
It does seem that *most* implementations out there do use SDP3 this way, so our platform would be
the setting this option to 1 to skip this check while any others would work the same as before.

> 
> > +static const char * const ixgbe_valid_arguments[] = {
> > +     IXGBE_DEVARG_FIBER_SDP3_NOT_TX_DISABLE,
> > +     NULL
> > +};
> > +
> >  #define IXGBEVF_DEVARG_PFLINK_FULLCHK                "pflink_fullchk"
> >
> >  static const char * const ixgbevf_valid_arguments[] = { @@ -348,6
> > +355,8 @@ static int ixgbe_dev_udp_tunnel_port_del(struct rte_eth_dev
> > *dev,  static int ixgbe_filter_restore(struct rte_eth_dev *dev);
> > static void ixgbe_l2_tunnel_conf(struct rte_eth_dev *dev);  static int
> > ixgbe_wait_for_link_up(struct ixgbe_hw *hw);
> > +static int devarg_handle_int(__rte_unused const char *key, const char
> *value,
> > +                          void *extra_args);
> >
> >  /*
> >   * Define VF Stats MACRO for Non "cleared on read" register @@
> > -1032,6 +1041,29 @@ ixgbe_swfw_lock_reset(struct ixgbe_hw *hw)
> >       ixgbe_release_swfw_semaphore(hw, mask);  }
> >
> > +static void
> > +ixgbe_parse_devargs(struct ixgbe_adapter *adapter,
> > +                   struct rte_devargs *devargs) {
> > +     struct rte_kvargs *kvlist;
> > +     uint16_t sdp3_no_tx_disable;
> > +
> > +     if (devargs == NULL)
> > +             return;
> > +
> > +     kvlist = rte_kvargs_parse(devargs->args, ixgbe_valid_arguments);
> > +     if (kvlist == NULL)
> > +             return;
> > +
> > +     if (rte_kvargs_count(kvlist,
> IXGBE_DEVARG_FIBER_SDP3_NOT_TX_DISABLE) == 1 &&
> > +         rte_kvargs_process(kvlist,
> IXGBE_DEVARG_FIBER_SDP3_NOT_TX_DISABLE,
> > +                            devarg_handle_int, &sdp3_no_tx_disable) == 0 &&
> > +         sdp3_no_tx_disable == 1)
> > +             adapter->sdp3_no_tx_disable = 1;
> > +
> > +     rte_kvargs_free(kvlist);
> > +}
> > +
> >  /*
> >   * This function is based on code in ixgbe_attach() in base/ixgbe.c.
> >   * It returns 0 on success.
> > @@ -1095,6 +1127,8 @@ eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev,
> void *init_params __rte_unused)
> >       }
> >
> >       rte_atomic32_clear(&ad->link_thread_running);
> > +     ixgbe_parse_devargs(eth_dev->data->dev_private,
> > +                         pci_dev->device.devargs);
> >       rte_eth_copy_pci_info(eth_dev, pci_dev);
> >       eth_dev->data->dev_flags |= RTE_ETH_DEV_AUTOFILL_QUEUE_XSTATS;
> >
> > @@ -4261,7 +4295,8 @@ ixgbe_dev_link_update_share(struct rte_eth_dev
> *dev,
> >               return rte_eth_linkstatus_set(dev, &link);
> >       }
> >
> > -     if (ixgbe_get_media_type(hw) == ixgbe_media_type_fiber) {
> > +     if (ixgbe_get_media_type(hw) == ixgbe_media_type_fiber &&
> > +         !ad->sdp3_no_tx_disable) {
> >               esdp_reg = IXGBE_READ_REG(hw, IXGBE_ESDP);
> >               if ((esdp_reg & IXGBE_ESDP_SDP3))
> >                       link_up = 0;
> > @@ -8250,6 +8285,8 @@ ixgbe_dev_macsec_register_disable(struct
> > rte_eth_dev *dev)  RTE_PMD_REGISTER_PCI(net_ixgbe, rte_ixgbe_pmd);
> > RTE_PMD_REGISTER_PCI_TABLE(net_ixgbe, pci_id_ixgbe_map);
> > RTE_PMD_REGISTER_KMOD_DEP(net_ixgbe, "* igb_uio | uio_pci_generic |
> > vfio-pci");
> > +RTE_PMD_REGISTER_PARAM_STRING(net_ixgbe,
> > +                           IXGBE_DEVARG_FIBER_SDP3_NOT_TX_DISABLE
> > +"=<0|1>");
> >  RTE_PMD_REGISTER_PCI(net_ixgbe_vf, rte_ixgbevf_pmd);
> > RTE_PMD_REGISTER_PCI_TABLE(net_ixgbe_vf, pci_id_ixgbevf_map);
> > RTE_PMD_REGISTER_KMOD_DEP(net_ixgbe_vf, "* igb_uio | vfio-pci"); diff
> > --git a/drivers/net/ixgbe/ixgbe_ethdev.h
> > b/drivers/net/ixgbe/ixgbe_ethdev.h
> > index 69e0e82a5b..cc6049a66a 100644
> > --- a/drivers/net/ixgbe/ixgbe_ethdev.h
> > +++ b/drivers/net/ixgbe/ixgbe_ethdev.h
> > @@ -501,6 +501,9 @@ struct ixgbe_adapter {
> >       /* For RSS reta table update */
> >       uint8_t rss_reta_updated;
> >
> > +     /* Used for limiting SDP3 TX_DISABLE checks */
> > +     uint8_t sdp3_no_tx_disable;
> > +
> >       /* Used for VF link sync with PF's physical and logical (by checking
> >        * mailbox status) link status.
> >        */
> > --
> > 2.25.1

RE: [PATCH v6 1/2] net/ixgbe: Limit SDP3 check of TX_DISABLE to appropriate devices

[Wang, Haiyue]

> -----Original Message-----
> From: Jeff Daly <jeffd@silicom-usa.com>
> Sent: Wednesday, April 13, 2022 23:32
> To: Wang, Haiyue <haiyue.wang@intel.com>; dev@dpdk.org
> Cc: stable@dpdk.org; Xiaolong Ye <xiaolong.ye@intel.com>; Xiao Zhang <xiao.zhang@intel.com>; Zhao1,
> Wei <wei.zhao1@intel.com>; Lunyuan Cui <lunyuanx.cui@intel.com>
> Subject: RE: [PATCH v6 1/2] net/ixgbe: Limit SDP3 check of TX_DISABLE to appropriate devices
> 
> 
> 
> > -----Original Message-----
> > From: Wang, Haiyue <haiyue.wang@intel.com>
> > Sent: Tuesday, April 12, 2022 9:22 PM
> > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> > Cc: stable@dpdk.org; Xiaolong Ye <xiaolong.ye@intel.com>; Xiao Zhang
> > <xiao.zhang@intel.com>; Zhao1, Wei <wei.zhao1@intel.com>; Lunyuan Cui
> > <lunyuanx.cui@intel.com>
> > Subject: RE: [PATCH v6 1/2] net/ixgbe: Limit SDP3 check of TX_DISABLE to
> > appropriate devices
> >
> > Caution: This is an external email. Please take care when clicking links or
> > opening attachments.
> >
> >
> > > -----Original Message-----
> > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > Sent: Wednesday, April 13, 2022 01:42
> > > To: dev@dpdk.org
> > > Cc: stable@dpdk.org; Wang, Haiyue <haiyue.wang@intel.com>; Xiaolong Ye
> > > <xiaolong.ye@intel.com>; Xiao Zhang <xiao.zhang@intel.com>; Zhao1, Wei
> > > <wei.zhao1@intel.com>; Lunyuan Cui <lunyuanx.cui@intel.com>
> > > Subject: [PATCH v6 1/2] net/ixgbe: Limit SDP3 check of TX_DISABLE to
> > > appropriate devices
> > >
> > > 1ca05831b9b added a check that SDP3 (used as a TX_DISABLE output to
> > > the SFP cage on these cards) is not asserted to avoid incorrectly
> > > reporting link up when the SFP's laser is turned off.
> > >
> > > ff8162cb957 limited this workaround to fiber ports
> > >
> > > This patch:
> > > * Adds devarg 'fiber_sdp3_no_tx_disable' not all fiber ixgbe devs use
> > >   SDP3 as TX_DISABLE
> > >
> > > Fixes: 1ca05831b9b ("net/ixgbe: fix link status")
> > > Fixes: ff8162cb957 ("net/ixgbe: fix link status")
> > > Cc: stable@dpdk.org
> >
> > This is new for soc for BIG change, not cc to stable.
> >
> > >
> > > Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> > > ---
> > >  drivers/net/ixgbe/ixgbe_ethdev.c | 39
> > > +++++++++++++++++++++++++++++++-  drivers/net/ixgbe/ixgbe_ethdev.h |
> > > 3 +++
> > >  2 files changed, 41 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c
> > > b/drivers/net/ixgbe/ixgbe_ethdev.c
> > > index 2da3f67bbc..f31bbb7895 100644
> > > --- a/drivers/net/ixgbe/ixgbe_ethdev.c
> > > +++ b/drivers/net/ixgbe/ixgbe_ethdev.c
> > > @@ -128,6 +128,13 @@
> > >  #define IXGBE_EXVET_VET_EXT_SHIFT              16
> > >  #define IXGBE_DMATXCTL_VT_MASK                 0xFFFF0000
> > >
> > > +#define IXGBE_DEVARG_FIBER_SDP3_NOT_TX_DISABLE
> > "fiber_sdp3_no_tx_disable"

> > > +
> >
> > 'platform' may be a good arg for the soc related change.
> >
> > dpdk-testpmd -a af:10.0,platform=xxx - -i
> >
> > enum ixgbe_platform_type {
> >         ixgbe_platform_unknown = 0,
> >         ixgbe_platform_soc_atom, ??? You can specify it.
> >
> >
> >
> > enum ixgbe_media_type ixgbe_get_platform_type(xxx) {
> >         return xxx;
> > }
> >
> 
> This patchset is not explicitly for SoC platform support.  *Any* implementation may or may not use
> SDP3 as TX_DISABLE.   The previous version of the patch added a check for the mac being an 82599 that
> uses fiber SFP rather than just a fiber SFP.  Our platform specifically can be fiber SFP, but
> TX_DISABLE
> is not SDP3.  However, our platform may not be the only implementation that doesn't use SDP3 this way.
> It does seem that *most* implementations out there do use SDP3 this way, so our platform would be
> the setting this option to 1 to skip this check while any others would work the same as before.
> 

OK, the reason looks good to me.

Please also update the doc:
https://doc.dpdk.org/guides/nics/ixgbe.html

And one more session before "VF Runtime Options", to describe the devarg.

> > >
> > > +     /* Used for limiting SDP3 TX_DISABLE checks */

This comment can be enhanced as your commit message said:


> > > +     uint8_t sdp3_no_tx_disable;
> > > +
> > >       /* Used for VF link sync with PF's physical and logical (by checking
> > >        * mailbox status) link status.
> > >        */
> > > --
> > > 2.25.1

[PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug

[Jeff Daly]

Currently the ixgbe driver does not ID any SFP except for the first one
plugged in. This can lead to no-link, or incorrect speed conditions.

For example:

* If link is initially established with a 1G SFP, and later a 1G/10G
multispeed part is later installed, then the MAC link setup functions are
never called to change from 1000BASE-X to 10GBASE-R mode, and the link
stays running at the slower rate.

* If link is initially established with a 1G SFP, and later a 10G only
module is later installed, no link is established, since we are still
trasnsmitting in 1000BASE-X mode to a 10GBASE-R only partner.

Refactor the SFP ID/setup, and link setup code, to more closely match the
flow of the mainline kernel driver which does not have these issues.  In
that driver a service task runs periodically to handle these operations
based on bit flags that have been set (usually via interrupt or userspace
request), and then get cleared once the requested subtask has been
completed.

Fixes: af75078fece ("first public release")
Cc: stable@dpdk.org

Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 533 +++++++++++++++++++++++--------
 drivers/net/ixgbe/ixgbe_ethdev.h |  14 +-
 2 files changed, 410 insertions(+), 137 deletions(-)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index f31bbb7895..9e720eee47 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -236,9 +236,6 @@ static int ixgbe_dev_interrupt_get_status(struct rte_eth_dev *dev);
 static int ixgbe_dev_interrupt_action(struct rte_eth_dev *dev);
 static void ixgbe_dev_interrupt_handler(void *param);
 static void ixgbe_dev_interrupt_delayed_handler(void *param);
-static void *ixgbe_dev_setup_link_thread_handler(void *param);
-static int ixgbe_dev_wait_setup_link_complete(struct rte_eth_dev *dev,
-					      uint32_t timeout_ms);
 
 static int ixgbe_add_rar(struct rte_eth_dev *dev,
 			struct rte_ether_addr *mac_addr,
@@ -775,6 +772,33 @@ static const struct rte_ixgbe_xstats_name_off rte_ixgbevf_stats_strings[] = {
 #define IXGBEVF_NB_XSTATS (sizeof(rte_ixgbevf_stats_strings) /	\
 		sizeof(rte_ixgbevf_stats_strings[0]))
 
+/**
+ * This function is the same as ixgbe_need_crosstalk_fix() in base/ixgbe_common.c
+ *
+ * ixgbe_need_crosstalk_fix - Determine if we need to do cross talk fix
+ * @hw: pointer to hardware structure
+ *
+ * Contains the logic to identify if we need to verify link for the
+ * crosstalk fix
+ **/
+static bool ixgbe_need_crosstalk_fix(struct ixgbe_hw *hw)
+{
+	/* Does FW say we need the fix */
+	if (!hw->need_crosstalk_fix)
+		return false;
+
+	/* Only consider SFP+ PHYs i.e. media type fiber */
+	switch (ixgbe_get_media_type(hw)) {
+	case ixgbe_media_type_fiber:
+	case ixgbe_media_type_fiber_qsfp:
+		break;
+	default:
+		return false;
+	}
+
+	return true;
+}
+
 /*
  * This function is the same as ixgbe_is_sfp() in base/ixgbe.h.
  */
@@ -1064,6 +1088,306 @@ ixgbe_parse_devargs(struct ixgbe_adapter *adapter,
 	rte_kvargs_free(kvlist);
 }
 
+/**
+ * ixgbe_check_sfp_cage - Find present status of SFP module
+ * @hw: pointer to hardware structure
+ *
+ * Find if a SFP module is present and if this device supports SFPs
+ **/
+enum ixgbe_sfp_cage_status ixgbe_check_sfp_cage(struct ixgbe_hw *hw)
+{
+	enum ixgbe_sfp_cage_status sfp_cage_status;
+
+	/* If we're not a fiber/fiber_qsfp, no cage to check */
+	switch (hw->mac.ops.get_media_type(hw)) {
+	case ixgbe_media_type_fiber:
+	case ixgbe_media_type_fiber_qsfp:
+		break;
+	default:
+		return IXGBE_SFP_CAGE_NOCAGE;
+	}
+
+	switch (hw->mac.type) {
+	case ixgbe_mac_82599EB:
+		sfp_cage_status = !!(IXGBE_READ_REG(hw, IXGBE_ESDP) &
+			     IXGBE_ESDP_SDP2);
+		break;
+	case ixgbe_mac_X550EM_x:
+	case ixgbe_mac_X550EM_a:
+		/* SDP0 is the active low signal PRSNT#, so invert this */
+		sfp_cage_status = !(IXGBE_READ_REG(hw, IXGBE_ESDP) &
+				  IXGBE_ESDP_SDP0);
+		break;
+	default:
+		/* Don't know how to check this device type yet */
+		sfp_cage_status = IXGBE_SFP_CAGE_UNKNOWN;
+		DEBUGOUT("IXGBE_SFP_CAGE_UNKNOWN, unknown mac type %d\n",
+			 hw->mac.type);
+		break;
+	}
+
+	DEBUGOUT("sfp status %d for mac type %d\n", sfp_cage_status, hw->mac.type);
+	return sfp_cage_status;
+}
+
+static s32
+ixgbe_sfp_id_and_setup(struct rte_eth_dev *dev)
+{
+	struct ixgbe_hw *hw =
+		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	enum ixgbe_sfp_cage_status sfp_cage_status;
+	s32 err;
+
+	/* Can't ID or setup SFP if it's not plugged in */
+	sfp_cage_status = ixgbe_check_sfp_cage(hw);
+	if (sfp_cage_status == IXGBE_SFP_CAGE_EMPTY ||
+	    sfp_cage_status == IXGBE_SFP_CAGE_NOCAGE)
+		return IXGBE_ERR_SFP_NOT_PRESENT;
+
+	/* Something's in the cage, ID it */
+	hw->phy.ops.identify_sfp(hw);
+
+	/* Unknown module type, give up */
+	if (hw->phy.sfp_type == ixgbe_sfp_type_unknown) {
+		PMD_DRV_LOG(ERR, "unknown SFP type, giving up");
+		return IXGBE_ERR_SFP_NOT_SUPPORTED;
+	}
+
+	/* This should be a redundant check, since we looked at the
+	 * PRSNT# signal from the cage above, but just in case this is
+	 * an SFP that's slow to respond to I2C pokes correctly, try it
+	 * again later
+	 */
+	if (hw->phy.sfp_type == ixgbe_sfp_type_not_present) {
+		PMD_DRV_LOG(ERR, "IDed SFP as absent but cage PRSNT# active!?");
+		return IXGBE_ERR_SFP_NOT_PRESENT;
+	}
+
+	/* SFP is present and identified, try to set it up */
+	err = hw->mac.ops.setup_sfp(hw);
+	if (err)
+		PMD_DRV_LOG(ERR, "setup_sfp() failed %d", err);
+
+	return err;
+}
+
+static void
+ixgbe_sfp_service(struct rte_eth_dev *dev)
+{
+	struct ixgbe_hw *hw =
+		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	struct ixgbe_interrupt *intr =
+		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
+	enum ixgbe_sfp_cage_status sfp_cage_status;
+	s32 err;
+	u8 sff_id;
+	bool have_int = false;
+
+	/* If there's no module cage, then there's nothing to service */
+	sfp_cage_status = ixgbe_check_sfp_cage(hw);
+	if (sfp_cage_status == IXGBE_SFP_CAGE_NOCAGE) {
+		PMD_DRV_LOG(DEBUG, "No SFP to service\n");
+		return;
+	}
+
+	/* TODO - Even for platforms where ixgbe_check_sfp_cage() gives a clear
+	 * status result, if there's no interrupts, or no interrupt for the SFP
+	 * cage present pin, even if other interrupts exist, then we still need
+	 * to poll here to set the flag.
+	 */
+#ifndef RTE_EXEC_ENV_FREEBSD
+	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
+	if (rte_intr_allow_others(intr_handle)) {
+		/* check if lsc interrupt is enabled */
+		if (dev->data->dev_conf.intr_conf.lsc)
+			have_int = true;
+	}
+#endif /* #ifdef RTE_EXEC_ENV_FREEBSD */
+
+	if (!have_int && sfp_cage_status == IXGBE_SFP_CAGE_EMPTY) {
+		intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
+		PMD_DRV_LOG(DEBUG, "No SFP, no LSC, set NEED_SFP_SETUP\n");
+	}
+
+	/* For platforms that don't have a way to read the PRESENT# signal from
+	 * the SFP cage, fallback to doing an I2C read and seeing if it's ACKed
+	 * to determine if a module is present
+	 */
+	if (sfp_cage_status == IXGBE_SFP_CAGE_UNKNOWN) {
+		PMD_DRV_LOG(DEBUG,
+			    "SFP present unknown (int? %d), try I2C read\n",
+			    have_int);
+
+		/* Rather than calling identify_sfp, which will read a lot of I2C
+		 * registers (and in a slow processor intensive fashion due to
+		 * bit-banging, just read the SFF ID register, which is at a
+		 * common address across SFP/SFP+/QSFP modules and see if
+		 * there's a NACK.  This works since we only expect a NACK if no
+		 * module is present
+		 */
+		err = ixgbe_read_i2c_eeprom(hw, IXGBE_SFF_IDENTIFIER, &sff_id);
+		if (err != IXGBE_SUCCESS) {
+			PMD_DRV_LOG(DEBUG, "Received I2C NAK from SFP, set NEED_SFP_SETUP flag\n");
+			intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
+			sfp_cage_status = IXGBE_SFP_CAGE_EMPTY;
+		} else {
+			PMD_DRV_LOG(DEBUG, "SFP ID read ACKed");
+			sfp_cage_status = IXGBE_SFP_CAGE_FULL;
+		}
+	}
+
+	if (sfp_cage_status == IXGBE_SFP_CAGE_EMPTY) {
+		PMD_DRV_LOG(DEBUG, "SFP absent, cage_status %d\n", sfp_cage_status);
+		return;
+	}
+
+	/* No setup requested?	Nothing to do */
+	if (!(intr->flags & IXGBE_FLAG_NEED_SFP_SETUP))
+		return;
+
+	err = ixgbe_sfp_id_and_setup(dev);
+	if (err) {
+		PMD_DRV_LOG(DEBUG, "failed to ID & setup SFP %d", err);
+		return;
+	}
+
+	/* Setup is done, clear the flag, but make sure link config runs for new SFP */
+	intr->flags &= ~IXGBE_FLAG_NEED_SFP_SETUP;
+	intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
+
+	/*
+	 * Since this is a new SFP, clear the old advertised speed mask so we don't
+	 * end up using an old slower rate
+	 */
+	hw->phy.autoneg_advertised = 0;
+}
+
+static void
+ixgbe_link_service(struct rte_eth_dev *dev)
+{
+	struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	struct ixgbe_interrupt *intr =
+		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
+	bool link_up, autoneg = false, have_int = false;
+	u32 speed;
+	s32 err;
+
+	/* Test if we have a LSC interrupt for this platform, if not we need to
+	 * manually check the link register since IXGBE_FLAG_NEED_LINK_CONFIG
+	 * will never be set in the interrupt handler
+	 */
+#ifndef RTE_EXEC_ENV_FREEBSD
+	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
+	if (rte_intr_allow_others(intr_handle)) {
+		/* check if lsc interrupt is enabled */
+		if (dev->data->dev_conf.intr_conf.lsc)
+			have_int = true;
+	}
+#endif /* #ifdef RTE_EXEC_ENV_FREEBSD */
+
+	/* Skip if we still need to setup an SFP, or if no link config requested
+	 */
+	if ((intr->flags & IXGBE_FLAG_NEED_SFP_SETUP) ||
+	    (!(intr->flags & IXGBE_FLAG_NEED_LINK_CONFIG) && have_int))
+		return;
+
+	if (!have_int && !(intr->flags & IXGBE_FLAG_NEED_LINK_CONFIG)) {
+		err = ixgbe_check_link(hw, &speed, &link_up, 0);
+		if (!err && !link_up) {
+			intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
+			PMD_DRV_LOG(DEBUG, "Link down, no LSC, set NEED_LINK_CONFIG\n");
+		} else {
+			return;
+		}
+	}
+
+	speed = hw->phy.autoneg_advertised;
+	if (!speed)
+		ixgbe_get_link_capabilities(hw, &speed, &autoneg);
+
+	err = ixgbe_setup_link(hw, speed, true);
+	if (err) {
+		PMD_DRV_LOG(ERR, "ixgbe_setup_link failed %d", err);
+		return;
+	}
+
+	intr->flags &= ~IXGBE_FLAG_NEED_LINK_CONFIG;
+}
+
+static void
+ixgbe_link_update_service(struct rte_eth_dev *dev)
+{
+	/* Update internal link status, waiting for link */
+	if (!ixgbe_dev_link_update(dev, 0)) {
+		ixgbe_dev_link_status_print(dev);
+		rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_LSC, NULL);
+	}
+}
+
+/*
+ * Service task thread to handle periodic tasks
+ */
+static void *
+ixgbe_dev_service_thread_handler(void *param)
+{
+	struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
+	struct ixgbe_hw *hw =
+		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	uint64_t start, ticks, service_ms;
+	uint32_t speed;
+	s32 err;
+	bool link_up;
+
+	while (1) {
+		ixgbe_sfp_service(dev);
+		ixgbe_link_service(dev);
+		ixgbe_link_update_service(dev);
+
+		/* Run the service thread handler more frequently when link is
+		 * down to reduce link up latency (every 200ms vs 1s)
+		 *
+		 * Use a number of smaller sleeps to decrease exit latency when
+		 * ixgbe_dev_stop() wants this thread to join
+		 */
+		err = ixgbe_check_link(hw, &speed, &link_up, 0);
+		if (err == IXGBE_SUCCESS && link_up)
+			service_ms = 2000;
+		else
+			service_ms = 100;
+
+		/* Call msec_delay in a loop with several smaller sleeps to
+		 * provide periodic thread cancellation points
+		 */
+		start = rte_get_timer_cycles();
+		ticks = (uint64_t)service_ms * rte_get_timer_hz() / 1E3;
+		while ((rte_get_timer_cycles() - start) < ticks)
+			msec_delay(100);
+	}
+
+	/* Never return */
+	return NULL;
+}
+
+static s32
+eth_ixgbe_check_mac_link_generic(struct ixgbe_hw *hw, ixgbe_link_speed *speed,
+				 bool *link_up, bool link_up_wait_to_complete)
+{
+	if (ixgbe_need_crosstalk_fix(hw)) {
+		enum ixgbe_sfp_cage_status sfp_cage_status;
+
+		sfp_cage_status = ixgbe_check_sfp_cage(hw);
+		if (sfp_cage_status != IXGBE_SFP_CAGE_FULL) {
+			*link_up = false;
+			*speed = IXGBE_LINK_SPEED_UNKNOWN;
+			return IXGBE_SUCCESS;
+		}
+	}
+
+	return ixgbe_check_mac_link_generic(hw, speed, link_up, link_up_wait_to_complete);
+}
+
 /*
  * This function is based on code in ixgbe_attach() in base/ixgbe.c.
  * It returns 0 on success.
@@ -1071,7 +1395,6 @@ ixgbe_parse_devargs(struct ixgbe_adapter *adapter,
 static int
 eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)
 {
-	struct ixgbe_adapter *ad = eth_dev->data->dev_private;
 	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
 	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
 	struct ixgbe_hw *hw =
@@ -1086,6 +1409,7 @@ eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)
 		IXGBE_DEV_PRIVATE_TO_FILTER_INFO(eth_dev->data->dev_private);
 	struct ixgbe_bw_conf *bw_conf =
 		IXGBE_DEV_PRIVATE_TO_BW_CONF(eth_dev->data->dev_private);
+	struct ixgbe_mac_info *mac = &hw->mac;
 	uint32_t ctrl_ext;
 	uint16_t csum;
 	int diag, i, ret;
@@ -1126,7 +1450,6 @@ eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)
 		return 0;
 	}
 
-	rte_atomic32_clear(&ad->link_thread_running);
 	ixgbe_parse_devargs(eth_dev->data->dev_private,
 			    pci_dev->device.devargs);
 	rte_eth_copy_pci_info(eth_dev, pci_dev);
@@ -1158,6 +1481,17 @@ eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)
 		return -EIO;
 	}
 
+	/* override mac_link_check to check for sfp cage full/empty */
+	switch (hw->mac.type) {
+	case ixgbe_mac_X550EM_x:
+	case ixgbe_mac_X550EM_a:
+	case ixgbe_mac_82599EB:
+		mac->ops.check_link = eth_ixgbe_check_mac_link_generic;
+		break;
+	default:
+		break;
+	}
+
 	/* pick up the PCI bus settings for reporting later */
 	ixgbe_get_bus_info(hw);
 
@@ -1581,7 +1915,6 @@ eth_ixgbevf_dev_init(struct rte_eth_dev *eth_dev)
 {
 	int diag;
 	uint32_t tc, tcs;
-	struct ixgbe_adapter *ad = eth_dev->data->dev_private;
 	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
 	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
 	struct ixgbe_hw *hw =
@@ -1624,7 +1957,6 @@ eth_ixgbevf_dev_init(struct rte_eth_dev *eth_dev)
 		return 0;
 	}
 
-	rte_atomic32_clear(&ad->link_thread_running);
 	ixgbevf_parse_devargs(eth_dev->data->dev_private,
 			      pci_dev->device.devargs);
 
@@ -2426,6 +2758,8 @@ ixgbe_dev_configure(struct rte_eth_dev *dev)
 	struct ixgbe_interrupt *intr =
 		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
 	struct ixgbe_adapter *adapter = dev->data->dev_private;
+	struct ixgbe_hw *hw =
+		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
 	int ret;
 
 	PMD_INIT_FUNC_TRACE();
@@ -2444,6 +2778,10 @@ ixgbe_dev_configure(struct rte_eth_dev *dev)
 	/* set flag to update link status after init */
 	intr->flags |= IXGBE_FLAG_NEED_LINK_UPDATE;
 
+	/* set flag to setup SFP after init */
+	if (ixgbe_is_sfp(hw))
+		intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
+
 	/*
 	 * Initialize to TRUE. If any of Rx queues doesn't meet the bulk
 	 * allocation or vector Rx preconditions we will reset it.
@@ -2463,13 +2801,24 @@ ixgbe_dev_phy_intr_setup(struct rte_eth_dev *dev)
 		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
 	uint32_t gpie;
 
-	/* only set up it on X550EM_X */
+	/* only set up it on X550EM_X (external PHY interrupt)
+	 * or on X550EM_a_* for SFP_PRSNT# de-assertion (SFP removal)
+	 */
 	if (hw->mac.type == ixgbe_mac_X550EM_x) {
 		gpie = IXGBE_READ_REG(hw, IXGBE_GPIE);
 		gpie |= IXGBE_SDP0_GPIEN_X550EM_x;
 		IXGBE_WRITE_REG(hw, IXGBE_GPIE, gpie);
 		if (hw->phy.type == ixgbe_phy_x550em_ext_t)
 			intr->mask |= IXGBE_EICR_GPI_SDP0_X550EM_x;
+	} else if (hw->mac.type == ixgbe_mac_X550EM_a) {
+		gpie = IXGBE_READ_REG(hw, IXGBE_GPIE);
+		gpie |= IXGBE_SDP0_GPIEN_X550EM_a;
+		IXGBE_WRITE_REG(hw, IXGBE_GPIE, gpie);
+		intr->mask |= IXGBE_EICR_GPI_SDP0_X550EM_a;
+	} else {
+		PMD_DRV_LOG(DEBUG,
+			    "No PHY/SFP interrupt for MAC %d, PHY %d\n",
+			    hw->mac.type, hw->phy.type);
 	}
 }
 
@@ -2592,8 +2941,11 @@ ixgbe_flow_ctrl_enable(struct rte_eth_dev *dev, struct ixgbe_hw *hw)
 static int
 ixgbe_dev_start(struct rte_eth_dev *dev)
 {
+	struct ixgbe_adapter *ad = dev->data->dev_private;
 	struct ixgbe_hw *hw =
 		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	struct ixgbe_interrupt *intr =
+		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
 	struct ixgbe_vf_info *vfinfo =
 		*IXGBE_DEV_PRIVATE_TO_P_VFDATA(dev->data->dev_private);
 	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
@@ -2614,9 +2966,6 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
 
 	PMD_INIT_FUNC_TRACE();
 
-	/* Stop the link setup handler before resetting the HW. */
-	ixgbe_dev_wait_setup_link_complete(dev, 0);
-
 	/* disable uio/vfio intr/eventfd mapping */
 	rte_intr_disable(intr_handle);
 
@@ -2732,12 +3081,6 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
 		}
 	}
 
-	if (ixgbe_is_sfp(hw) && hw->phy.multispeed_fiber) {
-		err = hw->mac.ops.setup_sfp(hw);
-		if (err)
-			goto error;
-	}
-
 	if (hw->mac.ops.get_media_type(hw) == ixgbe_media_type_copper) {
 		/* Turn on the copper */
 		ixgbe_set_phy_power(hw, true);
@@ -2849,6 +3192,20 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
 	ixgbe_l2_tunnel_conf(dev);
 	ixgbe_filter_restore(dev);
 
+	/* Spawn service thread */
+	if (ixgbe_is_sfp(hw)) {
+		intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
+		err = rte_ctrl_thread_create(&ad->service_thread_tid,
+					     "ixgbe-service-thread",
+					     NULL,
+					     ixgbe_dev_service_thread_handler,
+					     dev);
+		if (err) {
+			PMD_DRV_LOG(ERR, "service_thread err");
+			goto error;
+		}
+	}
+
 	if (tm_conf->root && !tm_conf->committed)
 		PMD_DRV_LOG(WARNING,
 			    "please call hierarchy_commit() "
@@ -2859,12 +3216,6 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
 	if (err)
 		goto error;
 
-	/*
-	 * Update link status right before return, because it may
-	 * start link configuration process in a separate thread.
-	 */
-	ixgbe_dev_link_update(dev, 0);
-
 	/* setup the macsec setting register */
 	if (macsec_setting->offload_en)
 		ixgbe_dev_macsec_register_enable(dev, macsec_setting);
@@ -2894,13 +3245,21 @@ ixgbe_dev_stop(struct rte_eth_dev *dev)
 	int vf;
 	struct ixgbe_tm_conf *tm_conf =
 		IXGBE_DEV_PRIVATE_TO_TM_CONF(dev->data->dev_private);
+	void *res;
+	s32 err;
 
 	if (hw->adapter_stopped)
 		return 0;
 
 	PMD_INIT_FUNC_TRACE();
 
-	ixgbe_dev_wait_setup_link_complete(dev, 0);
+	/* Cancel the service thread, and wait for it to join */
+	err = pthread_cancel(adapter->service_thread_tid);
+	if (err)
+		PMD_DRV_LOG(ERR, "failed to cancel service thread %d", err);
+	err = pthread_join(adapter->service_thread_tid, &res);
+	if (err)
+		PMD_DRV_LOG(ERR, "failed to join service thread %d", err);
 
 	/* disable interrupts */
 	ixgbe_disable_intr(hw);
@@ -2979,7 +3338,6 @@ ixgbe_dev_set_link_up(struct rte_eth_dev *dev)
 	} else {
 		/* Turn on the laser */
 		ixgbe_enable_tx_laser(hw);
-		ixgbe_dev_link_update(dev, 0);
 	}
 
 	return 0;
@@ -3010,7 +3368,6 @@ ixgbe_dev_set_link_down(struct rte_eth_dev *dev)
 	} else {
 		/* Turn off the laser */
 		ixgbe_disable_tx_laser(hw);
-		ixgbe_dev_link_update(dev, 0);
 	}
 
 	return 0;
@@ -4163,57 +4520,6 @@ ixgbevf_check_link(struct ixgbe_hw *hw, ixgbe_link_speed *speed,
 	return ret_val;
 }
 
-/*
- * If @timeout_ms was 0, it means that it will not return until link complete.
- * It returns 1 on complete, return 0 on timeout.
- */
-static int
-ixgbe_dev_wait_setup_link_complete(struct rte_eth_dev *dev, uint32_t timeout_ms)
-{
-#define WARNING_TIMEOUT    9000 /* 9s  in total */
-	struct ixgbe_adapter *ad = dev->data->dev_private;
-	uint32_t timeout = timeout_ms ? timeout_ms : WARNING_TIMEOUT;
-
-	while (rte_atomic32_read(&ad->link_thread_running)) {
-		msec_delay(1);
-		timeout--;
-
-		if (timeout_ms) {
-			if (!timeout)
-				return 0;
-		} else if (!timeout) {
-			/* It will not return until link complete */
-			timeout = WARNING_TIMEOUT;
-			PMD_DRV_LOG(ERR, "IXGBE link thread not complete too long time!");
-		}
-	}
-
-	return 1;
-}
-
-static void *
-ixgbe_dev_setup_link_thread_handler(void *param)
-{
-	struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
-	struct ixgbe_adapter *ad = dev->data->dev_private;
-	struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
-	struct ixgbe_interrupt *intr =
-		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
-	u32 speed;
-	bool autoneg = false;
-
-	pthread_detach(pthread_self());
-	speed = hw->phy.autoneg_advertised;
-	if (!speed)
-		ixgbe_get_link_capabilities(hw, &speed, &autoneg);
-
-	ixgbe_setup_link(hw, speed, true);
-
-	intr->flags &= ~IXGBE_FLAG_NEED_LINK_CONFIG;
-	rte_atomic32_clear(&ad->link_thread_running);
-	return NULL;
-}
-
 /*
  * In freebsd environment, nic_uio drivers do not support interrupts,
  * rte_intr_callback_register() will fail to register interrupts.
@@ -4256,8 +4562,6 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 	struct ixgbe_adapter *ad = dev->data->dev_private;
 	struct rte_eth_link link;
 	ixgbe_link_speed link_speed = IXGBE_LINK_SPEED_UNKNOWN;
-	struct ixgbe_interrupt *intr =
-		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
 	bool link_up;
 	int diag;
 	int wait = 1;
@@ -4272,9 +4576,6 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 
 	hw->mac.get_link_status = true;
 
-	if (intr->flags & IXGBE_FLAG_NEED_LINK_CONFIG)
-		return rte_eth_linkstatus_set(dev, &link);
-
 	/* check if it needs to wait to complete, if lsc interrupt is enabled */
 	if (wait_to_complete == 0 || dev->data->dev_conf.intr_conf.lsc != 0)
 		wait = 0;
@@ -4289,7 +4590,7 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 	else
 		diag = ixgbe_check_link(hw, &link_speed, &link_up, wait);
 
-	if (diag != 0) {
+	if (diag != 0 || !link_up) {
 		link.link_speed = RTE_ETH_SPEED_NUM_100M;
 		link.link_duplex = RTE_ETH_LINK_FULL_DUPLEX;
 		return rte_eth_linkstatus_set(dev, &link);
@@ -4302,32 +4603,6 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
 			link_up = 0;
 	}
 
-	if (link_up == 0) {
-		if (ixgbe_get_media_type(hw) == ixgbe_media_type_fiber) {
-			ixgbe_dev_wait_setup_link_complete(dev, 0);
-			if (rte_atomic32_test_and_set(&ad->link_thread_running)) {
-				/* To avoid race condition between threads, set
-				 * the IXGBE_FLAG_NEED_LINK_CONFIG flag only
-				 * when there is no link thread running.
-				 */
-				intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
-				if (rte_ctrl_thread_create(&ad->link_thread_tid,
-					"ixgbe-link-handler",
-					NULL,
-					ixgbe_dev_setup_link_thread_handler,
-					dev) < 0) {
-					PMD_DRV_LOG(ERR,
-						"Create link thread failed!");
-					rte_atomic32_clear(&ad->link_thread_running);
-				}
-			} else {
-				PMD_DRV_LOG(ERR,
-					"Other link thread is running now!");
-			}
-		}
-		return rte_eth_linkstatus_set(dev, &link);
-	}
-
 	link.link_status = RTE_ETH_LINK_UP;
 	link.link_duplex = RTE_ETH_LINK_FULL_DUPLEX;
 
@@ -4533,8 +4808,6 @@ ixgbe_dev_interrupt_get_status(struct rte_eth_dev *dev)
 	eicr = IXGBE_READ_REG(hw, IXGBE_EICR);
 	PMD_DRV_LOG(DEBUG, "eicr %x", eicr);
 
-	intr->flags = 0;
-
 	/* set flag for async link update */
 	if (eicr & IXGBE_EICR_LSC)
 		intr->flags |= IXGBE_FLAG_NEED_LINK_UPDATE;
@@ -4550,6 +4823,14 @@ ixgbe_dev_interrupt_get_status(struct rte_eth_dev *dev)
 	    (eicr & IXGBE_EICR_GPI_SDP0_X550EM_x))
 		intr->flags |= IXGBE_FLAG_PHY_INTERRUPT;
 
+	/* Check for loss of SFP */
+	/* TODO - For platforms that don't have this flag, do we need to set
+	 *  NEED_SFP_SETUP on LSC if we're a SFP platform?
+	 */
+	if (hw->mac.type ==  ixgbe_mac_X550EM_a &&
+	    (eicr & IXGBE_EICR_GPI_SDP0_X550EM_a))
+		intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
+
 	return 0;
 }
 
@@ -4601,11 +4882,13 @@ ixgbe_dev_link_status_print(struct rte_eth_dev *dev)
 static int
 ixgbe_dev_interrupt_action(struct rte_eth_dev *dev)
 {
+	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
 	struct ixgbe_interrupt *intr =
 		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
-	int64_t timeout;
 	struct ixgbe_hw *hw =
 		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	int64_t timeout;
 
 	PMD_DRV_LOG(DEBUG, "intr action type %d", intr->flags);
 
@@ -4640,16 +4923,14 @@ ixgbe_dev_interrupt_action(struct rte_eth_dev *dev)
 		if (rte_eal_alarm_set(timeout * 1000,
 				      ixgbe_dev_interrupt_delayed_handler, (void *)dev) < 0)
 			PMD_DRV_LOG(ERR, "Error setting alarm");
-		else {
-			/* remember original mask */
-			intr->mask_original = intr->mask;
+		else
 			/* only disable lsc interrupt */
 			intr->mask &= ~IXGBE_EIMS_LSC;
-		}
 	}
 
 	PMD_DRV_LOG(DEBUG, "enable intr immediately");
 	ixgbe_enable_intr(dev);
+	rte_intr_ack(intr_handle);
 
 	return 0;
 }
@@ -4672,8 +4953,6 @@ static void
 ixgbe_dev_interrupt_delayed_handler(void *param)
 {
 	struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
-	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
-	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
 	struct ixgbe_interrupt *intr =
 		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
 	struct ixgbe_hw *hw =
@@ -4703,13 +4982,10 @@ ixgbe_dev_interrupt_delayed_handler(void *param)
 		intr->flags &= ~IXGBE_FLAG_MACSEC;
 	}
 
-	/* restore original mask */
-	intr->mask = intr->mask_original;
-	intr->mask_original = 0;
+	if (dev->data->dev_conf.intr_conf.lsc != 0)
+		intr->mask |= IXGBE_EICR_LSC;
 
-	PMD_DRV_LOG(DEBUG, "enable intr in delayed handler S[%08x]", eicr);
 	ixgbe_enable_intr(dev);
-	rte_intr_ack(intr_handle);
 }
 
 /**
@@ -5351,9 +5627,6 @@ ixgbevf_dev_start(struct rte_eth_dev *dev)
 
 	PMD_INIT_FUNC_TRACE();
 
-	/* Stop the link setup handler before resetting the HW. */
-	ixgbe_dev_wait_setup_link_complete(dev, 0);
-
 	err = hw->mac.ops.reset_hw(hw);
 
 	/**
@@ -5433,12 +5706,6 @@ ixgbevf_dev_start(struct rte_eth_dev *dev)
 	/* Re-enable interrupt for VF */
 	ixgbevf_intr_enable(dev);
 
-	/*
-	 * Update link status right before return, because it may
-	 * start link configuration process in a separate thread.
-	 */
-	ixgbevf_dev_link_update(dev, 0);
-
 	hw->adapter_stopped = false;
 
 	return 0;
@@ -5457,8 +5724,6 @@ ixgbevf_dev_stop(struct rte_eth_dev *dev)
 
 	PMD_INIT_FUNC_TRACE();
 
-	ixgbe_dev_wait_setup_link_complete(dev, 0);
-
 	ixgbevf_intr_disable(dev);
 
 	dev->data->dev_started = 0;
diff --git a/drivers/net/ixgbe/ixgbe_ethdev.h b/drivers/net/ixgbe/ixgbe_ethdev.h
index cc6049a66a..d8d0bebd04 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.h
+++ b/drivers/net/ixgbe/ixgbe_ethdev.h
@@ -29,6 +29,7 @@
 #define IXGBE_FLAG_PHY_INTERRUPT    (uint32_t)(1 << 2)
 #define IXGBE_FLAG_MACSEC           (uint32_t)(1 << 3)
 #define IXGBE_FLAG_NEED_LINK_CONFIG (uint32_t)(1 << 4)
+#define IXGBE_FLAG_NEED_SFP_SETUP   ((uint32_t)(1 << 5))
 
 /*
  * Defines that were not part of ixgbe_type.h as they are not used by the
@@ -223,8 +224,6 @@ struct ixgbe_rte_flow_rss_conf {
 struct ixgbe_interrupt {
 	uint32_t flags;
 	uint32_t mask;
-	/*to save original mask during delayed handler */
-	uint32_t mask_original;
 };
 
 struct ixgbe_stat_mapping_registers {
@@ -510,7 +509,7 @@ struct ixgbe_adapter {
 	uint8_t pflink_fullchk;
 	uint8_t mac_ctrl_frame_fwd;
 	rte_atomic32_t link_thread_running;
-	pthread_t link_thread_tid;
+	pthread_t service_thread_tid;
 };
 
 struct ixgbe_vf_representor {
@@ -673,6 +672,15 @@ int ixgbe_syn_filter_set(struct rte_eth_dev *dev,
 			struct rte_eth_syn_filter *filter,
 			bool add);
 
+enum ixgbe_sfp_cage_status {
+	IXGBE_SFP_CAGE_EMPTY = 0,
+	IXGBE_SFP_CAGE_FULL,
+	IXGBE_SFP_CAGE_UNKNOWN = -1,
+	IXGBE_SFP_CAGE_NOCAGE = -2,
+};
+enum ixgbe_sfp_cage_status ixgbe_check_sfp_cage(struct ixgbe_hw *hw);
+
+
 /**
  * l2 tunnel configuration.
  */
-- 
2.25.1

RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug

[Wang, Haiyue]

> -----Original Message-----
> From: Jeff Daly <jeffd@silicom-usa.com>
> Sent: Wednesday, April 13, 2022 01:42
> To: dev@dpdk.org
> Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>; Wang, Haiyue <haiyue.wang@intel.com>
> Subject: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug
> 
> Currently the ixgbe driver does not ID any SFP except for the first one
> plugged in. This can lead to no-link, or incorrect speed conditions.
> 
> For example:
> 
> * If link is initially established with a 1G SFP, and later a 1G/10G
> multispeed part is later installed, then the MAC link setup functions are
> never called to change from 1000BASE-X to 10GBASE-R mode, and the link
> stays running at the slower rate.
> 
> * If link is initially established with a 1G SFP, and later a 10G only
> module is later installed, no link is established, since we are still
> trasnsmitting in 1000BASE-X mode to a 10GBASE-R only partner.
> 
> Refactor the SFP ID/setup, and link setup code, to more closely match the
> flow of the mainline kernel driver which does not have these issues.  In
> that driver a service task runs periodically to handle these operations
> based on bit flags that have been set (usually via interrupt or userspace
> request), and then get cleared once the requested subtask has been
> completed.
> 
> Fixes: af75078fece ("first public release")
> Cc: stable@dpdk.org
> 

So BIG change for new platform, DON'T CC to stable!

> Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
> Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> ---
>  drivers/net/ixgbe/ixgbe_ethdev.c | 533 +++++++++++++++++++++++--------
>  drivers/net/ixgbe/ixgbe_ethdev.h |  14 +-
>  2 files changed, 410 insertions(+), 137 deletions(-)
> 
> diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
> index f31bbb7895..9e720eee47 100644
> --- a/drivers/net/ixgbe/ixgbe_ethdev.c
> +++ b/drivers/net/ixgbe/ixgbe_ethdev.c
> @@ -236,9 +236,6 @@ static int ixgbe_dev_interrupt_get_status(struct rte_eth_dev *dev);
>  static int ixgbe_dev_interrupt_action(struct rte_eth_dev *dev);
>  static void ixgbe_dev_interrupt_handler(void *param);
>  static void ixgbe_dev_interrupt_delayed_handler(void *param);
> -static void *ixgbe_dev_setup_link_thread_handler(void *param);
> -static int ixgbe_dev_wait_setup_link_complete(struct rte_eth_dev *dev,
> -					      uint32_t timeout_ms);
> 


> +	/* TODO - Even for platforms where ixgbe_check_sfp_cage() gives a clear
> +	 * status result, if there's no interrupts, or no interrupt for the SFP
> +	 * cage present pin, even if other interrupts exist, then we still need
> +	 * to poll here to set the flag.
> +	 */
> +#ifndef RTE_EXEC_ENV_FREEBSD
> +	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
> +	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
> +	if (rte_intr_allow_others(intr_handle)) {
> +		/* check if lsc interrupt is enabled */
> +		if (dev->data->dev_conf.intr_conf.lsc)
> +			have_int = true;


> +	/* Check for loss of SFP */
> +	/* TODO - For platforms that don't have this flag, do we need to set
> +	 *  NEED_SFP_SETUP on LSC if we're a SFP platform?
> +	 */
> +	if (hw->mac.type ==  ixgbe_mac_X550EM_a &&
> +	    (eicr & IXGBE_EICR_GPI_SDP0_X550EM_a))
> +		intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
> +
>  	return 0;
>  }


TODO ?

> --
> 2.25.1

RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug

[Morten Brørup]

> From: Wang, Haiyue [mailto:haiyue.wang@intel.com]
> Sent: Wednesday, 13 April 2022 04.47
> To: Daly, Jeff; dev@dpdk.org
> Cc: stable@dpdk.org; Stephen Douthit; Yang, Qiming
> 
> > From: Jeff Daly <jeffd@silicom-usa.com>
> > Sent: Wednesday, April 13, 2022 01:42
> > To: dev@dpdk.org
> > Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>;
> Wang, Haiyue <haiyue.wang@intel.com>
> >
> > Currently the ixgbe driver does not ID any SFP except for the first
> one
> > plugged in. This can lead to no-link, or incorrect speed conditions.
> >
> > For example:
> >
> > * If link is initially established with a 1G SFP, and later a 1G/10G
> > multispeed part is later installed, then the MAC link setup functions
> are
> > never called to change from 1000BASE-X to 10GBASE-R mode, and the
> link
> > stays running at the slower rate.
> >
> > * If link is initially established with a 1G SFP, and later a 10G
> only
> > module is later installed, no link is established, since we are still
> > trasnsmitting in 1000BASE-X mode to a 10GBASE-R only partner.
> >
> > Refactor the SFP ID/setup, and link setup code, to more closely match
> the
> > flow of the mainline kernel driver which does not have these issues.
> In
> > that driver a service task runs periodically to handle these
> operations
> > based on bit flags that have been set (usually via interrupt or
> userspace
> > request), and then get cleared once the requested subtask has been
> > completed.
> >
> > Fixes: af75078fece ("first public release")
> > Cc: stable@dpdk.org
> >
> 
> So BIG change for new platform, DON'T CC to stable!

What do you mean by "new platform"? The ixgbe hardware and driver is not new.

This patch fixes a bug (with a serious impact when occurring), so it should be backported. The size of the patch does not disqualify it for backporting.

-Morten

RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug

[Wang, Haiyue]

> -----Original Message-----
> From: Morten Brørup <mb@smartsharesystems.com>
> Sent: Wednesday, April 13, 2022 14:58
> To: Wang, Haiyue <haiyue.wang@intel.com>; Daly, Jeff <jeffd@silicom-usa.com>; dev@dpdk.org
> Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>; Yang, Qiming <qiming.yang@intel.com>
> Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug
> 
> > From: Wang, Haiyue [mailto:haiyue.wang@intel.com]
> > Sent: Wednesday, 13 April 2022 04.47
> > To: Daly, Jeff; dev@dpdk.org
> > Cc: stable@dpdk.org; Stephen Douthit; Yang, Qiming
> >
> > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > Sent: Wednesday, April 13, 2022 01:42
> > > To: dev@dpdk.org
> > > Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>;
> > Wang, Haiyue <haiyue.wang@intel.com>
> > >
> > > Currently the ixgbe driver does not ID any SFP except for the first
> > one
> > > plugged in. This can lead to no-link, or incorrect speed conditions.
> > >
> > > For example:
> > >
> > > * If link is initially established with a 1G SFP, and later a 1G/10G
> > > multispeed part is later installed, then the MAC link setup functions
> > are
> > > never called to change from 1000BASE-X to 10GBASE-R mode, and the
> > link
> > > stays running at the slower rate.
> > >
> > > * If link is initially established with a 1G SFP, and later a 10G
> > only
> > > module is later installed, no link is established, since we are still
> > > trasnsmitting in 1000BASE-X mode to a 10GBASE-R only partner.
> > >
> > > Refactor the SFP ID/setup, and link setup code, to more closely match
> > the
> > > flow of the mainline kernel driver which does not have these issues.
> > In
> > > that driver a service task runs periodically to handle these
> > operations
> > > based on bit flags that have been set (usually via interrupt or
> > userspace
> > > request), and then get cleared once the requested subtask has been
> > > completed.
> > >
> > > Fixes: af75078fece ("first public release")
> > > Cc: stable@dpdk.org
> > >
> >
> > So BIG change for new platform, DON'T CC to stable!
> 
> What do you mean by "new platform"? The ixgbe hardware and driver is not new.
> 

It's soc NIC, ixgbe not support before.

> This patch fixes a bug (with a serious impact when occurring), so it should be backported. The size of
> the patch does not disqualify it for backporting.
> 
> -Morten

RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug

[Morten Brørup]

> From: Wang, Haiyue [mailto:haiyue.wang@intel.com]
> Sent: Wednesday, 13 April 2022 09.02
> 
> > From: Morten Brørup <mb@smartsharesystems.com>
> > Sent: Wednesday, April 13, 2022 14:58
> >
> > > From: Wang, Haiyue [mailto:haiyue.wang@intel.com]
> > > Sent: Wednesday, 13 April 2022 04.47
> > > To: Daly, Jeff; dev@dpdk.org
> > > Cc: stable@dpdk.org; Stephen Douthit; Yang, Qiming
> > >
> > > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > > Sent: Wednesday, April 13, 2022 01:42
> > > > To: dev@dpdk.org
> > > > Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>;
> > > Wang, Haiyue <haiyue.wang@intel.com>
> > > >
> > > > Currently the ixgbe driver does not ID any SFP except for the
> first
> > > one
> > > > plugged in. This can lead to no-link, or incorrect speed
> conditions.
> > > >
> > > > For example:
> > > >
> > > > * If link is initially established with a 1G SFP, and later a
> 1G/10G
> > > > multispeed part is later installed, then the MAC link setup
> functions
> > > are
> > > > never called to change from 1000BASE-X to 10GBASE-R mode, and the
> > > link
> > > > stays running at the slower rate.
> > > >
> > > > * If link is initially established with a 1G SFP, and later a 10G
> > > only
> > > > module is later installed, no link is established, since we are
> still
> > > > trasnsmitting in 1000BASE-X mode to a 10GBASE-R only partner.
> > > >
> > > > Refactor the SFP ID/setup, and link setup code, to more closely
> match
> > > the
> > > > flow of the mainline kernel driver which does not have these
> issues.
> > > In
> > > > that driver a service task runs periodically to handle these
> > > operations
> > > > based on bit flags that have been set (usually via interrupt or
> > > userspace
> > > > request), and then get cleared once the requested subtask has
> been
> > > > completed.
> > > >
> > > > Fixes: af75078fece ("first public release")
> > > > Cc: stable@dpdk.org
> > > >
> > >
> > > So BIG change for new platform, DON'T CC to stable!
> >
> > What do you mean by "new platform"? The ixgbe hardware and driver is
> not new.
> >
> 
> It's soc NIC, ixgbe not support before.

If the patch only fixes the driver for a new NIC that not supported by older DPDK versions, and that NIC is not going to be supported by older DPDK versions, then I agree that there is no point in backporting it or CC'ing stable.

However, if the patch could also apply to any other ixgbe NIC that is potentially supported by older DPDK versions, then it should be backported.

> 
> > This patch fixes a bug (with a serious impact when occurring), so it
> should be backported. The size of
> > the patch does not disqualify it for backporting.
> >
> > -Morten
> 

RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug

[Wang, Haiyue]

> -----Original Message-----
> From: Morten Brørup <mb@smartsharesystems.com>
> Sent: Wednesday, April 13, 2022 15:20
> To: Wang, Haiyue <haiyue.wang@intel.com>; Daly, Jeff <jeffd@silicom-usa.com>; dev@dpdk.org
> Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>; Yang, Qiming <qiming.yang@intel.com>
> Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug
> 
> > From: Wang, Haiyue [mailto:haiyue.wang@intel.com]
> > Sent: Wednesday, 13 April 2022 09.02
> >
> > > From: Morten Brørup <mb@smartsharesystems.com>
> > > Sent: Wednesday, April 13, 2022 14:58
> > >
> > > > From: Wang, Haiyue [mailto:haiyue.wang@intel.com]
> > > > Sent: Wednesday, 13 April 2022 04.47
> > > > To: Daly, Jeff; dev@dpdk.org
> > > > Cc: stable@dpdk.org; Stephen Douthit; Yang, Qiming
> > > >
> > > > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > > > Sent: Wednesday, April 13, 2022 01:42
> > > > > To: dev@dpdk.org
> > > > > Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>;
> > > > Wang, Haiyue <haiyue.wang@intel.com>
> > > > >
> > > > > Currently the ixgbe driver does not ID any SFP except for the
> > first
> > > > one
> > > > > plugged in. This can lead to no-link, or incorrect speed
> > conditions.
> > > > >
> > > > > For example:
> > > > >
> > > > > * If link is initially established with a 1G SFP, and later a
> > 1G/10G
> > > > > multispeed part is later installed, then the MAC link setup
> > functions
> > > > are
> > > > > never called to change from 1000BASE-X to 10GBASE-R mode, and the
> > > > link
> > > > > stays running at the slower rate.
> > > > >
> > > > > * If link is initially established with a 1G SFP, and later a 10G
> > > > only
> > > > > module is later installed, no link is established, since we are
> > still
> > > > > trasnsmitting in 1000BASE-X mode to a 10GBASE-R only partner.
> > > > >
> > > > > Refactor the SFP ID/setup, and link setup code, to more closely
> > match
> > > > the
> > > > > flow of the mainline kernel driver which does not have these
> > issues.
> > > > In
> > > > > that driver a service task runs periodically to handle these
> > > > operations
> > > > > based on bit flags that have been set (usually via interrupt or
> > > > userspace
> > > > > request), and then get cleared once the requested subtask has
> > been
> > > > > completed.
> > > > >
> > > > > Fixes: af75078fece ("first public release")
> > > > > Cc: stable@dpdk.org
> > > > >
> > > >
> > > > So BIG change for new platform, DON'T CC to stable!
> > >
> > > What do you mean by "new platform"? The ixgbe hardware and driver is
> > not new.
> > >
> >
> > It's soc NIC, ixgbe not support before.
> 
> If the patch only fixes the driver for a new NIC that not supported by older DPDK versions, and that
> NIC is not going to be supported by older DPDK versions, then I agree that there is no point in
> backporting it or CC'ing stable.
> 
> However, if the patch could also apply to any other ixgbe NIC that is potentially supported by older
> DPDK versions, then it should be backported.

It's hard to say, these years, I see many ixgbe link related fixes.
At least now, no big link fix for normal ixgbe NICs.

This patch still have some kind of TODOs. And this is not acceptable for
us to maintain this kind of code for released stable DPDK version. I don't
want to see many follow fixes ...

And we have two DPDK development cycle (22.07 22.11) to make it for next
stable release.

> 
> >
> > > This patch fixes a bug (with a serious impact when occurring), so it
> > should be backported. The size of
> > > the patch does not disqualify it for backporting.
> > >
> > > -Morten
> >

RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug

[Morten Brørup]

> From: Wang, Haiyue [mailto:haiyue.wang@intel.com]
> Sent: Wednesday, 13 April 2022 13.49
> 
> > From: Morten Brørup <mb@smartsharesystems.com>
> > Sent: Wednesday, April 13, 2022 15:20
> >
> > > From: Wang, Haiyue [mailto:haiyue.wang@intel.com]
> > > Sent: Wednesday, 13 April 2022 09.02
> > >
> > > > From: Morten Brørup <mb@smartsharesystems.com>
> > > > Sent: Wednesday, April 13, 2022 14:58
> > > >
> > > > > From: Wang, Haiyue [mailto:haiyue.wang@intel.com]
> > > > > Sent: Wednesday, 13 April 2022 04.47
> > > > > To: Daly, Jeff; dev@dpdk.org
> > > > > Cc: stable@dpdk.org; Stephen Douthit; Yang, Qiming
> > > > >
> > > > > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > > > > Sent: Wednesday, April 13, 2022 01:42
> > > > > > To: dev@dpdk.org
> > > > > > Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-
> usa.com>;
> > > > > Wang, Haiyue <haiyue.wang@intel.com>
> > > > > >
> > > > > > Currently the ixgbe driver does not ID any SFP except for the
> > > first
> > > > > one
> > > > > > plugged in. This can lead to no-link, or incorrect speed
> > > conditions.
> > > > > >
> > > > > > For example:
> > > > > >
> > > > > > * If link is initially established with a 1G SFP, and later a
> > > 1G/10G
> > > > > > multispeed part is later installed, then the MAC link setup
> > > functions
> > > > > are
> > > > > > never called to change from 1000BASE-X to 10GBASE-R mode, and
> the
> > > > > link
> > > > > > stays running at the slower rate.
> > > > > >
> > > > > > * If link is initially established with a 1G SFP, and later a
> 10G
> > > > > only
> > > > > > module is later installed, no link is established, since we
> are
> > > still
> > > > > > trasnsmitting in 1000BASE-X mode to a 10GBASE-R only partner.
> > > > > >
> > > > > > Refactor the SFP ID/setup, and link setup code, to more
> closely
> > > match
> > > > > the
> > > > > > flow of the mainline kernel driver which does not have these
> > > issues.
> > > > > In
> > > > > > that driver a service task runs periodically to handle these
> > > > > operations
> > > > > > based on bit flags that have been set (usually via interrupt
> or
> > > > > userspace
> > > > > > request), and then get cleared once the requested subtask has
> > > been
> > > > > > completed.
> > > > > >
> > > > > > Fixes: af75078fece ("first public release")
> > > > > > Cc: stable@dpdk.org
> > > > > >
> > > > >
> > > > > So BIG change for new platform, DON'T CC to stable!
> > > >
> > > > What do you mean by "new platform"? The ixgbe hardware and driver
> is
> > > not new.
> > > >
> > >
> > > It's soc NIC, ixgbe not support before.
> >
> > If the patch only fixes the driver for a new NIC that not supported
> by older DPDK versions, and that
> > NIC is not going to be supported by older DPDK versions, then I agree
> that there is no point in
> > backporting it or CC'ing stable.
> >
> > However, if the patch could also apply to any other ixgbe NIC that is
> potentially supported by older
> > DPDK versions, then it should be backported.
> 
> It's hard to say, these years, I see many ixgbe link related fixes.

The physical layers have always been tricky... Apparently, they still are. :-)

> At least now, no big link fix for normal ixgbe NICs.

I would not discriminate between normal and less common NICs. If they are not EOL according to Intel, the drivers should support them.

In the real world, though, driver development resources will be allocated to the important customers and/or the high volume products. So I do understand your concern! Not being directly involved in this work myself, it is easy to voice my opinion as a "backseat driver". :-)

> 
> This patch still have some kind of TODOs. And this is not acceptable
> for
> us to maintain this kind of code for released stable DPDK version. I
> don't
> want to see many follow fixes ...
> 
> And we have two DPDK development cycle (22.07 22.11) to make it for
> next
> stable release.
> 

I 100 % agree that all the TODOs should be solved first, so the driver is reliable and complete before any backporting starts. Adding follow-on fixes in multiple DPDK versions is a waste of maintainer time, and I agree with your pushback when this is the reason.

With that in mind, CC'ing stable could be postponed until the patch reaches backporting worthy quality.

> >
> > >
> > > > This patch fixes a bug (with a serious impact when occurring), so
> it
> > > should be backported. The size of
> > > > the patch does not disqualify it for backporting.
> > > >
> > > > -Morten
> > >
> 

RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug

[Jeff Daly]

> -----Original Message-----
> From: Morten Brørup <mb@smartsharesystems.com>
> Sent: Wednesday, April 13, 2022 3:20 AM
> To: Wang, Haiyue <haiyue.wang@intel.com>; Jeff Daly <jeffd@silicom-
> usa.com>; dev@dpdk.org
> Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>; Yang,
> Qiming <qiming.yang@intel.com>
> Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on
> hotplug
> 
> Caution: This is an external email. Please take care when clicking links or
> opening attachments.
> 
> 
> > From: Wang, Haiyue [mailto:haiyue.wang@intel.com]
> > Sent: Wednesday, 13 April 2022 09.02
> >
> > > From: Morten Brørup <mb@smartsharesystems.com>
> > > Sent: Wednesday, April 13, 2022 14:58
> > >
> > > > From: Wang, Haiyue [mailto:haiyue.wang@intel.com]
> > > > Sent: Wednesday, 13 April 2022 04.47
> > > > To: Daly, Jeff; dev@dpdk.org
> > > > Cc: stable@dpdk.org; Stephen Douthit; Yang, Qiming
> > > >
> > > > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > > > Sent: Wednesday, April 13, 2022 01:42
> > > > > To: dev@dpdk.org
> > > > > Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>;
> > > > Wang, Haiyue <haiyue.wang@intel.com>
> > > > >
> > > > > Currently the ixgbe driver does not ID any SFP except for the
> > first
> > > > one
> > > > > plugged in. This can lead to no-link, or incorrect speed
> > conditions.
> > > > >
> > > > > For example:
> > > > >
> > > > > * If link is initially established with a 1G SFP, and later a
> > 1G/10G
> > > > > multispeed part is later installed, then the MAC link setup
> > functions
> > > > are
> > > > > never called to change from 1000BASE-X to 10GBASE-R mode, and the
> > > > link
> > > > > stays running at the slower rate.
> > > > >
> > > > > * If link is initially established with a 1G SFP, and later a 10G
> > > > only
> > > > > module is later installed, no link is established, since we are
> > still
> > > > > trasnsmitting in 1000BASE-X mode to a 10GBASE-R only partner.
> > > > >
> > > > > Refactor the SFP ID/setup, and link setup code, to more closely
> > match
> > > > the
> > > > > flow of the mainline kernel driver which does not have these
> > issues.
> > > > In
> > > > > that driver a service task runs periodically to handle these
> > > > operations
> > > > > based on bit flags that have been set (usually via interrupt or
> > > > userspace
> > > > > request), and then get cleared once the requested subtask has
> > been
> > > > > completed.
> > > > >
> > > > > Fixes: af75078fece ("first public release")
> > > > > Cc: stable@dpdk.org
> > > > >
> > > >
> > > > So BIG change for new platform, DON'T CC to stable!
> > >
> > > What do you mean by "new platform"? The ixgbe hardware and driver is
> > not new.
> > >
> >
> > It's soc NIC, ixgbe not support before.
> 
> If the patch only fixes the driver for a new NIC that not supported by older
> DPDK versions, and that NIC is not going to be supported by older DPDK
> versions, then I agree that there is no point in backporting it or CC'ing stable.
> 
> However, if the patch could also apply to any other ixgbe NIC that is
> potentially supported by older DPDK versions, then it should be backported.
> 

This patch is not *only* for soc NIC, it's for *all* IXGBE supported NIC that have SFP cages. 
The code was also validated using a dual port 82599ES 10G SFI/SFP+ PCIe NIC adapter.

> >
> > > This patch fixes a bug (with a serious impact when occurring), so it
> > should be backported. The size of
> > > the patch does not disqualify it for backporting.
> > >
> > > -Morten
> >

RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug

[Jeff Daly]

> -----Original Message-----
> From: Wang, Haiyue <haiyue.wang@intel.com>
> Sent: Tuesday, April 12, 2022 10:47 PM
> To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>; Yang,
> Qiming <qiming.yang@intel.com>
> Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on
> hotplug
> 
> Caution: This is an external email. Please take care when clicking links or
> opening attachments.
> 
> 
> > -----Original Message-----
> > From: Jeff Daly <jeffd@silicom-usa.com>
> > Sent: Wednesday, April 13, 2022 01:42
> > To: dev@dpdk.org
> > Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>;
> Wang,
> > Haiyue <haiyue.wang@intel.com>
> > Subject: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on
> > hotplug
> >
> > Currently the ixgbe driver does not ID any SFP except for the first
> > one plugged in. This can lead to no-link, or incorrect speed conditions.
> >
> > For example:
> >
> > * If link is initially established with a 1G SFP, and later a 1G/10G
> > multispeed part is later installed, then the MAC link setup functions
> > are never called to change from 1000BASE-X to 10GBASE-R mode, and the
> > link stays running at the slower rate.
> >
> > * If link is initially established with a 1G SFP, and later a 10G only
> > module is later installed, no link is established, since we are still
> > trasnsmitting in 1000BASE-X mode to a 10GBASE-R only partner.
> >
> > Refactor the SFP ID/setup, and link setup code, to more closely match
> > the flow of the mainline kernel driver which does not have these
> > issues.  In that driver a service task runs periodically to handle
> > these operations based on bit flags that have been set (usually via
> > interrupt or userspace request), and then get cleared once the
> > requested subtask has been completed.
> >
> > Fixes: af75078fece ("first public release")
> > Cc: stable@dpdk.org
> >
> 
> So BIG change for new platform, DON'T CC to stable!
> 
> > Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
> > Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> > ---
> >  drivers/net/ixgbe/ixgbe_ethdev.c | 533
> > +++++++++++++++++++++++--------  drivers/net/ixgbe/ixgbe_ethdev.h |
> > 14 +-
> >  2 files changed, 410 insertions(+), 137 deletions(-)
> >
> > diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c
> > b/drivers/net/ixgbe/ixgbe_ethdev.c
> > index f31bbb7895..9e720eee47 100644
> > --- a/drivers/net/ixgbe/ixgbe_ethdev.c
> > +++ b/drivers/net/ixgbe/ixgbe_ethdev.c
> > @@ -236,9 +236,6 @@ static int ixgbe_dev_interrupt_get_status(struct
> > rte_eth_dev *dev);  static int ixgbe_dev_interrupt_action(struct
> > rte_eth_dev *dev);  static void ixgbe_dev_interrupt_handler(void
> > *param);  static void ixgbe_dev_interrupt_delayed_handler(void
> > *param); -static void *ixgbe_dev_setup_link_thread_handler(void
> > *param); -static int ixgbe_dev_wait_setup_link_complete(struct
> rte_eth_dev *dev,
> > -                                           uint32_t timeout_ms);
> >
> 
> 
> > +     /* TODO - Even for platforms where ixgbe_check_sfp_cage() gives a
> clear
> > +      * status result, if there's no interrupts, or no interrupt for the SFP
> > +      * cage present pin, even if other interrupts exist, then we still need
> > +      * to poll here to set the flag.
> > +      */
> > +#ifndef RTE_EXEC_ENV_FREEBSD
> > +     struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
> > +     struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
> > +     if (rte_intr_allow_others(intr_handle)) {
> > +             /* check if lsc interrupt is enabled */
> > +             if (dev->data->dev_conf.intr_conf.lsc)
> > +                     have_int = true;
> 
> 
> > +     /* Check for loss of SFP */
> > +     /* TODO - For platforms that don't have this flag, do we need to set
> > +      *  NEED_SFP_SETUP on LSC if we're a SFP platform?
> > +      */
> > +     if (hw->mac.type ==  ixgbe_mac_X550EM_a &&
> > +         (eicr & IXGBE_EICR_GPI_SDP0_X550EM_a))
> > +             intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
> > +
> >       return 0;
> >  }
> 
> 
> TODO ?
> 

These were notes that some further refinements could be possible.  I can remove the comments if that makes it more acceptable.  The first TODO was a note that (for FreeBSD mostly) w/o interrupts the code below (not shown, which checks for 'have_int') will run vs. exiting early and relying on the NEED_SFP_SETUP flag being set elsewhere.  The 2nd TODO was a question from the original developer to himself where (I believe) he was considering other implementations which may not use MOD_ABS the same that other platforms do.  I don't think this is an issue (that I know of) so removing this commend and just moving on would be the best thing, and if someone else knows of a situation where this consideration might need to be made, then another patch can be submitted to address it.

> > --
> > 2.25.1

RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug

[Jeff Daly]

> -----Original Message-----
> From: Jeff Daly <jeffd@silicom-usa.com>
> Sent: Thursday, April 14, 2022 6:50 AM
> To: Wang, Haiyue <haiyue.wang@intel.com>; dev@dpdk.org
> Cc: stable@dpdk.org; Yang, Qiming <qiming.yang@intel.com>
> Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on
> hotplug
> 
> Caution: This is an external email. Please take care when clicking links or
> opening attachments.
> 
> 
> > -----Original Message-----
> > From: Wang, Haiyue <haiyue.wang@intel.com>
> > Sent: Tuesday, April 12, 2022 10:47 PM
> > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> > Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>; Yang,
> > Qiming <qiming.yang@intel.com>
> > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking
> > on hotplug
> >
> > Caution: This is an external email. Please take care when clicking
> > links or opening attachments.
> >
> >
> > > -----Original Message-----
> > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > Sent: Wednesday, April 13, 2022 01:42
> > > To: dev@dpdk.org
> > > Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>;
> > Wang,
> > > Haiyue <haiyue.wang@intel.com>
> > > Subject: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on
> > > hotplug
> > >
> > > Currently the ixgbe driver does not ID any SFP except for the first
> > > one plugged in. This can lead to no-link, or incorrect speed conditions.
> > >
> > > For example:
> > >
> > > * If link is initially established with a 1G SFP, and later a 1G/10G
> > > multispeed part is later installed, then the MAC link setup
> > > functions are never called to change from 1000BASE-X to 10GBASE-R
> > > mode, and the link stays running at the slower rate.
> > >
> > > * If link is initially established with a 1G SFP, and later a 10G
> > > only module is later installed, no link is established, since we are
> > > still trasnsmitting in 1000BASE-X mode to a 10GBASE-R only partner.
> > >
> > > Refactor the SFP ID/setup, and link setup code, to more closely
> > > match the flow of the mainline kernel driver which does not have
> > > these issues.  In that driver a service task runs periodically to
> > > handle these operations based on bit flags that have been set
> > > (usually via interrupt or userspace request), and then get cleared
> > > once the requested subtask has been completed.
> > >
> > > Fixes: af75078fece ("first public release")
> > > Cc: stable@dpdk.org
> > >
> >
> > So BIG change for new platform, DON'T CC to stable!
> >
> > > Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
> > > Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> > > ---
> > >  drivers/net/ixgbe/ixgbe_ethdev.c | 533
> > > +++++++++++++++++++++++--------  drivers/net/ixgbe/ixgbe_ethdev.h |
> > > 14 +-
> > >  2 files changed, 410 insertions(+), 137 deletions(-)
> > >
> > > diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c
> > > b/drivers/net/ixgbe/ixgbe_ethdev.c
> > > index f31bbb7895..9e720eee47 100644
> > > --- a/drivers/net/ixgbe/ixgbe_ethdev.c
> > > +++ b/drivers/net/ixgbe/ixgbe_ethdev.c
> > > @@ -236,9 +236,6 @@ static int ixgbe_dev_interrupt_get_status(struct
> > > rte_eth_dev *dev);  static int ixgbe_dev_interrupt_action(struct
> > > rte_eth_dev *dev);  static void ixgbe_dev_interrupt_handler(void
> > > *param);  static void ixgbe_dev_interrupt_delayed_handler(void
> > > *param); -static void *ixgbe_dev_setup_link_thread_handler(void
> > > *param); -static int ixgbe_dev_wait_setup_link_complete(struct
> > rte_eth_dev *dev,
> > > -                                           uint32_t timeout_ms);
> > >
> >
> >
> > > +     /* TODO - Even for platforms where ixgbe_check_sfp_cage()
> > > + gives a
> > clear
> > > +      * status result, if there's no interrupts, or no interrupt for the SFP
> > > +      * cage present pin, even if other interrupts exist, then we still need
> > > +      * to poll here to set the flag.
> > > +      */
> > > +#ifndef RTE_EXEC_ENV_FREEBSD
> > > +     struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
> > > +     struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
> > > +     if (rte_intr_allow_others(intr_handle)) {
> > > +             /* check if lsc interrupt is enabled */
> > > +             if (dev->data->dev_conf.intr_conf.lsc)
> > > +                     have_int = true;
> >
> >
> > > +     /* Check for loss of SFP */
> > > +     /* TODO - For platforms that don't have this flag, do we need to set
> > > +      *  NEED_SFP_SETUP on LSC if we're a SFP platform?
> > > +      */
> > > +     if (hw->mac.type ==  ixgbe_mac_X550EM_a &&
> > > +         (eicr & IXGBE_EICR_GPI_SDP0_X550EM_a))
> > > +             intr->flags |= IXGBE_FLAG_NEED_SFP_SETUP;
> > > +
> > >       return 0;
> > >  }
> >
> >
> > TODO ?
> >
> 
> These were notes that some further refinements could be possible.  I can
> remove the comments if that makes it more acceptable.  The first TODO was
> a note that (for FreeBSD mostly) w/o interrupts the code below (not shown,
> which checks for 'have_int') will run vs. exiting early and relying on the
> NEED_SFP_SETUP flag being set elsewhere.  The 2nd TODO was a question
> from the original developer to himself where (I believe) he was considering
> other implementations which may not use MOD_ABS the same that other
> platforms do.  I don't think this is an issue (that I know of) so removing this
> commend and just moving on would be the best thing, and if someone else
> knows of a situation where this consideration might need to be made, then
> another patch can be submitted to address it.
> 

Ugh.  Apologies for the dumb Outlook editor not wrapping the lines properly,
if you are reading this on patches.dpdk.org it runs off the page.  Repeating below:

These were notes that some further refinements could be possible.  I can 
remove the comments if that makes it more acceptable.  The first TODO was
a note that (for FreeBSD mostly) w/o interrupts the code below (not shown, 
which checks for 'have_int') will run vs. exiting early and relying on the 
NEED_SFP_SETUP flag being set elsewhere.  The 2nd TODO was a question 
from the original developer to himself where (I believe) he was considering
other implementations which may not use MOD_ABS the same that other
platforms do.  I don't think this is an issue (that I know of) so removing this
comment and just moving on would be the best thing, and if someone else
knows of a situation where this consideration might need to be made, then
another patch can be submitted to address it.

> > > --
> > > 2.25.1

RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug

[Wang, Haiyue]

> -----Original Message-----
> From: Jeff Daly <jeffd@silicom-usa.com>
> Sent: Wednesday, April 13, 2022 01:42
> To: dev@dpdk.org
> Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>; Wang, Haiyue <haiyue.wang@intel.com>
> Subject: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug
> 
> Currently the ixgbe driver does not ID any SFP except for the first one
> plugged in. This can lead to no-link, or incorrect speed conditions.
> 
> For example:
> 
> * If link is initially established with a 1G SFP, and later a 1G/10G
> multispeed part is later installed, then the MAC link setup functions are
> never called to change from 1000BASE-X to 10GBASE-R mode, and the link
> stays running at the slower rate.
> 
> * If link is initially established with a 1G SFP, and later a 10G only
> module is later installed, no link is established, since we are still
> trasnsmitting in 1000BASE-X mode to a 10GBASE-R only partner.
> 
> Refactor the SFP ID/setup, and link setup code, to more closely match the
> flow of the mainline kernel driver which does not have these issues.  In
> that driver a service task runs periodically to handle these operations
> based on bit flags that have been set (usually via interrupt or userspace
> request), and then get cleared once the requested subtask has been
> completed.
> 
> Fixes: af75078fece ("first public release")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
> Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> ---
>  drivers/net/ixgbe/ixgbe_ethdev.c | 533 +++++++++++++++++++++++--------
>  drivers/net/ixgbe/ixgbe_ethdev.h |  14 +-
>  2 files changed, 410 insertions(+), 137 deletions(-)
> 


> 
>  struct ixgbe_stat_mapping_registers {
> @@ -510,7 +509,7 @@ struct ixgbe_adapter {
>  	uint8_t pflink_fullchk;
>  	uint8_t mac_ctrl_frame_fwd;
>  	rte_atomic32_t link_thread_running;
> -	pthread_t link_thread_tid;
> +	pthread_t service_thread_tid;

No need to rename this variable, we can separate this patch as least into
two patches:

1st, change the thread handle 'ixgbe_dev_setup_link_thread_handler' from

run-once to as periodical, to handle the original issue.

The name 'ixgbe_dev_setup_link_thread_handler' may be not suitable now,
as it is a service thread.

We can change it to "'ixgbe_link_service_thread_handler'" to reflect the
change purpose.

2nd, add the SFP hotplug in this patch.



>  };
> 
> --
> 2.25.1

RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug

[Wang, Haiyue]

> -----Original Message-----
> From: Wang, Haiyue
> Sent: Thursday, April 14, 2022 10:49
> To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>
> Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug
> 
> > -----Original Message-----
> > From: Jeff Daly <jeffd@silicom-usa.com>
> > Sent: Wednesday, April 13, 2022 01:42
> > To: dev@dpdk.org
> > Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>; Wang, Haiyue <haiyue.wang@intel.com>
> > Subject: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug
> >
> > Currently the ixgbe driver does not ID any SFP except for the first one
> > plugged in. This can lead to no-link, or incorrect speed conditions.
> >
> > For example:
> >
> > * If link is initially established with a 1G SFP, and later a 1G/10G
> > multispeed part is later installed, then the MAC link setup functions are
> > never called to change from 1000BASE-X to 10GBASE-R mode, and the link
> > stays running at the slower rate.
> >
> > * If link is initially established with a 1G SFP, and later a 10G only
> > module is later installed, no link is established, since we are still
> > trasnsmitting in 1000BASE-X mode to a 10GBASE-R only partner.
> >
> > Refactor the SFP ID/setup, and link setup code, to more closely match the
> > flow of the mainline kernel driver which does not have these issues.  In
> > that driver a service task runs periodically to handle these operations
> > based on bit flags that have been set (usually via interrupt or userspace
> > request), and then get cleared once the requested subtask has been
> > completed.
> >
> > Fixes: af75078fece ("first public release")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
> > Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> > ---
> >  drivers/net/ixgbe/ixgbe_ethdev.c | 533 +++++++++++++++++++++++--------
> >  drivers/net/ixgbe/ixgbe_ethdev.h |  14 +-
> >  2 files changed, 410 insertions(+), 137 deletions(-)
> >
> 
> 
> >
> >  struct ixgbe_stat_mapping_registers {
> > @@ -510,7 +509,7 @@ struct ixgbe_adapter {
> >  	uint8_t pflink_fullchk;
> >  	uint8_t mac_ctrl_frame_fwd;
> >  	rte_atomic32_t link_thread_running;
> > -	pthread_t link_thread_tid;
> > +	pthread_t service_thread_tid;
> 
> No need to rename this variable, 

Let's do link related service now, so we can keep it, I missed to add my
comment. ;-)

> we can separate this patch as least into
> two patches:


> 
> 1st, change the thread handle 'ixgbe_dev_setup_link_thread_handler' from
> 
> run-once to as periodical, to handle the original issue.
> 
> The name 'ixgbe_dev_setup_link_thread_handler' may be not suitable now,
> as it is a service thread.
> 
> We can change it to "'ixgbe_link_service_thread_handler'" to reflect the
> change purpose.
> 
> 2nd, add the SFP hotplug in this patch.
> 
> 
> 
> >  };
> >
> > --
> > 2.25.1

RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug

[Jeff Daly]

> -----Original Message-----
> From: Wang, Haiyue <haiyue.wang@intel.com>
> Sent: Wednesday, April 13, 2022 11:00 PM
> To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>
> Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on
> hotplug
> 
> Caution: This is an external email. Please take care when clicking links or
> opening attachments.
> 
> 
> > -----Original Message-----
> > From: Wang, Haiyue
> > Sent: Thursday, April 14, 2022 10:49
> > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> > Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>
> > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking
> > on hotplug
> >
> > > -----Original Message-----
> > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > Sent: Wednesday, April 13, 2022 01:42
> > > To: dev@dpdk.org
> > > Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>;
> > > Wang, Haiyue <haiyue.wang@intel.com>
> > > Subject: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on
> > > hotplug
> > >
> > > Currently the ixgbe driver does not ID any SFP except for the first
> > > one plugged in. This can lead to no-link, or incorrect speed conditions.
> > >
> > > For example:
> > >
> > > * If link is initially established with a 1G SFP, and later a 1G/10G
> > > multispeed part is later installed, then the MAC link setup
> > > functions are never called to change from 1000BASE-X to 10GBASE-R
> > > mode, and the link stays running at the slower rate.
> > >
> > > * If link is initially established with a 1G SFP, and later a 10G
> > > only module is later installed, no link is established, since we are
> > > still trasnsmitting in 1000BASE-X mode to a 10GBASE-R only partner.
> > >
> > > Refactor the SFP ID/setup, and link setup code, to more closely
> > > match the flow of the mainline kernel driver which does not have
> > > these issues.  In that driver a service task runs periodically to
> > > handle these operations based on bit flags that have been set
> > > (usually via interrupt or userspace request), and then get cleared
> > > once the requested subtask has been completed.
> > >
> > > Fixes: af75078fece ("first public release")
> > > Cc: stable@dpdk.org
> > >
> > > Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
> > > Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> > > ---
> > >  drivers/net/ixgbe/ixgbe_ethdev.c | 533
> > > +++++++++++++++++++++++--------  drivers/net/ixgbe/ixgbe_ethdev.h |
> > > 14 +-
> > >  2 files changed, 410 insertions(+), 137 deletions(-)
> > >
> >
> >
> > >
> > >  struct ixgbe_stat_mapping_registers { @@ -510,7 +509,7 @@ struct
> > > ixgbe_adapter {
> > >     uint8_t pflink_fullchk;
> > >     uint8_t mac_ctrl_frame_fwd;
> > >     rte_atomic32_t link_thread_running;
> > > -   pthread_t link_thread_tid;
> > > +   pthread_t service_thread_tid;
> >
> > No need to rename this variable,
> 
> Let's do link related service now, so we can keep it, I missed to add my
> comment. ;-)
> 

I don't understand this reply, are you still asking to rework the patch or not?

> > we can separate this patch as least into two patches:
> 
> 
> >
> > 1st, change the thread handle 'ixgbe_dev_setup_link_thread_handler'
> > from
> >
> > run-once to as periodical, to handle the original issue.
> >
> > The name 'ixgbe_dev_setup_link_thread_handler' may be not suitable
> > now, as it is a service thread.
> >
> > We can change it to "'ixgbe_link_service_thread_handler'" to reflect
> > the change purpose.
> >
> > 2nd, add the SFP hotplug in this patch.
> >
> >
> >
> > >  };
> > >
> > > --
> > > 2.25.1

RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug

[Wang, Haiyue]

> -----Original Message-----
> From: Jeff Daly <jeffd@silicom-usa.com>
> Sent: Thursday, April 14, 2022 18:41
> To: Wang, Haiyue <haiyue.wang@intel.com>; dev@dpdk.org
> Cc: stable@dpdk.org
> Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug
> 
> 
> 
> > -----Original Message-----
> > From: Wang, Haiyue <haiyue.wang@intel.com>
> > Sent: Wednesday, April 13, 2022 11:00 PM
> > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> > Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>
> > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on
> > hotplug
> >
> > Caution: This is an external email. Please take care when clicking links or
> > opening attachments.
> >
> >
> > > -----Original Message-----
> > > From: Wang, Haiyue
> > > Sent: Thursday, April 14, 2022 10:49
> > > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> > > Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>
> > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking
> > > on hotplug
> > >
> > > > -----Original Message-----
> > > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > > Sent: Wednesday, April 13, 2022 01:42
> > > > To: dev@dpdk.org
> > > > Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>;
> > > > Wang, Haiyue <haiyue.wang@intel.com>
> > > > Subject: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on
> > > > hotplug
> > > >
> > > > Currently the ixgbe driver does not ID any SFP except for the first
> > > > one plugged in. This can lead to no-link, or incorrect speed conditions.
> > > >
> > > > For example:
> > > >
> > > > * If link is initially established with a 1G SFP, and later a 1G/10G
> > > > multispeed part is later installed, then the MAC link setup
> > > > functions are never called to change from 1000BASE-X to 10GBASE-R
> > > > mode, and the link stays running at the slower rate.
> > > >
> > > > * If link is initially established with a 1G SFP, and later a 10G
> > > > only module is later installed, no link is established, since we are
> > > > still trasnsmitting in 1000BASE-X mode to a 10GBASE-R only partner.
> > > >
> > > > Refactor the SFP ID/setup, and link setup code, to more closely
> > > > match the flow of the mainline kernel driver which does not have
> > > > these issues.  In that driver a service task runs periodically to
> > > > handle these operations based on bit flags that have been set
> > > > (usually via interrupt or userspace request), and then get cleared
> > > > once the requested subtask has been completed.
> > > >
> > > > Fixes: af75078fece ("first public release")
> > > > Cc: stable@dpdk.org
> > > >
> > > > Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
> > > > Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> > > > ---
> > > >  drivers/net/ixgbe/ixgbe_ethdev.c | 533
> > > > +++++++++++++++++++++++--------  drivers/net/ixgbe/ixgbe_ethdev.h |
> > > > 14 +-
> > > >  2 files changed, 410 insertions(+), 137 deletions(-)
> > > >
> > >
> > >
> > > >
> > > >  struct ixgbe_stat_mapping_registers { @@ -510,7 +509,7 @@ struct
> > > > ixgbe_adapter {
> > > >     uint8_t pflink_fullchk;
> > > >     uint8_t mac_ctrl_frame_fwd;
> > > >     rte_atomic32_t link_thread_running;
> > > > -   pthread_t link_thread_tid;
> > > > +   pthread_t service_thread_tid;
> > >
> > > No need to rename this variable,
> >
> > Let's do link related service now, so we can keep it, I missed to add my
> > comment. ;-)
> >
> 
> I don't understand this reply, are you still asking to rework the patch or not?
> 

Different thing.

1. This var can be kept to trace the created thread. (change less code to keep
   the patch clean.)
2. Yes, two patches.

> > > we can separate this patch as least into two patches:
> >
> >
> > >
> > > 1st, change the thread handle 'ixgbe_dev_setup_link_thread_handler'
> > > from
> > >
> > > run-once to as periodical, to handle the original issue.
> > >
> > > The name 'ixgbe_dev_setup_link_thread_handler' may be not suitable
> > > now, as it is a service thread.
> > >
> > > We can change it to "'ixgbe_link_service_thread_handler'" to reflect
> > > the change purpose.
> > >
> > > 2nd, add the SFP hotplug in this patch.
> > >
> > >
> > >
> > > >  };
> > > >
> > > > --
> > > > 2.25.1

RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug

[Jeff Daly]

> -----Original Message-----
> From: Wang, Haiyue <haiyue.wang@intel.com>
> Sent: Thursday, April 14, 2022 8:11 AM
> To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> Cc: stable@dpdk.org
> Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on
> hotplug
> 
> Caution: This is an external email. Please take care when clicking links or
> opening attachments.
> 
> 
> > -----Original Message-----
> > From: Jeff Daly <jeffd@silicom-usa.com>
> > Sent: Thursday, April 14, 2022 18:41
> > To: Wang, Haiyue <haiyue.wang@intel.com>; dev@dpdk.org
> > Cc: stable@dpdk.org
> > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking
> > on hotplug
> >
> >
> >
> > > -----Original Message-----
> > > From: Wang, Haiyue <haiyue.wang@intel.com>
> > > Sent: Wednesday, April 13, 2022 11:00 PM
> > > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> > > Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>
> > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking
> > > on hotplug
> > >
> > > Caution: This is an external email. Please take care when clicking
> > > links or opening attachments.
> > >
> > >
> > > > -----Original Message-----
> > > > From: Wang, Haiyue
> > > > Sent: Thursday, April 14, 2022 10:49
> > > > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> > > > Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>
> > > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and
> > > > linking on hotplug
> > > >
> > > > > -----Original Message-----
> > > > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > > > Sent: Wednesday, April 13, 2022 01:42
> > > > > To: dev@dpdk.org
> > > > > Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>;
> > > > > Wang, Haiyue <haiyue.wang@intel.com>
> > > > > Subject: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking
> > > > > on hotplug
> > > > >
> > > > > Currently the ixgbe driver does not ID any SFP except for the
> > > > > first one plugged in. This can lead to no-link, or incorrect speed
> conditions.
> > > > >
> > > > > For example:
> > > > >
> > > > > * If link is initially established with a 1G SFP, and later a
> > > > > 1G/10G multispeed part is later installed, then the MAC link
> > > > > setup functions are never called to change from 1000BASE-X to
> > > > > 10GBASE-R mode, and the link stays running at the slower rate.
> > > > >
> > > > > * If link is initially established with a 1G SFP, and later a
> > > > > 10G only module is later installed, no link is established,
> > > > > since we are still trasnsmitting in 1000BASE-X mode to a 10GBASE-R
> only partner.
> > > > >
> > > > > Refactor the SFP ID/setup, and link setup code, to more closely
> > > > > match the flow of the mainline kernel driver which does not have
> > > > > these issues.  In that driver a service task runs periodically
> > > > > to handle these operations based on bit flags that have been set
> > > > > (usually via interrupt or userspace request), and then get
> > > > > cleared once the requested subtask has been completed.
> > > > >
> > > > > Fixes: af75078fece ("first public release")
> > > > > Cc: stable@dpdk.org
> > > > >
> > > > > Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
> > > > > Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> > > > > ---
> > > > >  drivers/net/ixgbe/ixgbe_ethdev.c | 533
> > > > > +++++++++++++++++++++++--------
> > > > > +++++++++++++++++++++++drivers/net/ixgbe/ixgbe_ethdev.h |
> > > > > 14 +-
> > > > >  2 files changed, 410 insertions(+), 137 deletions(-)
> > > > >
> > > >
> > > >
> > > > >
> > > > >  struct ixgbe_stat_mapping_registers { @@ -510,7 +509,7 @@
> > > > > struct ixgbe_adapter {
> > > > >     uint8_t pflink_fullchk;
> > > > >     uint8_t mac_ctrl_frame_fwd;
> > > > >     rte_atomic32_t link_thread_running;
> > > > > -   pthread_t link_thread_tid;
> > > > > +   pthread_t service_thread_tid;
> > > >
> > > > No need to rename this variable,
> > >
> > > Let's do link related service now, so we can keep it, I missed to
> > > add my comment. ;-)
> > >
> >
> > I don't understand this reply, are you still asking to rework the patch or
> not?
> >
> 
> Different thing.
> 
> 1. This var can be kept to trace the created thread. (change less code to keep
>    the patch clean.)
> 2. Yes, two patches.
> 

ok, I guess I'm just being thick-headed here, but I still don't understand why you are saying it should be split into 
2 patches.  if I understand *what* you are asking, you're saying make the original thread periodic to continuously
do ixgbe_link_setup() ?  I believe the problem with the setup is that the sfp_type is only detected once
at initialization time and if nothing is in the cage then the code just returns IXGBE_SUCCESS, in which case making 
this task periodic is useless.  the whole issue of hotplug is only addressed by the entire patch which 1) makes the
task periodic, 2) changes the actions of the task to look for whether the cage has something in it and whether its
been changed and needs to be configured again.


> > > > we can separate this patch as least into two patches:
> > >
> > >
> > > >
> > > > 1st, change the thread handle 'ixgbe_dev_setup_link_thread_handler'
> > > > from
> > > >
> > > > run-once to as periodical, to handle the original issue.
> > > >
> > > > The name 'ixgbe_dev_setup_link_thread_handler' may be not suitable
> > > > now, as it is a service thread.
> > > >
> > > > We can change it to "'ixgbe_link_service_thread_handler'" to
> > > > reflect the change purpose.
> > > >
> > > > 2nd, add the SFP hotplug in this patch.
> > > >
> > > >
> > > >
> > > > >  };
> > > > >
> > > > > --
> > > > > 2.25.1

RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug

[Wang, Haiyue]

> -----Original Message-----
> From: Jeff Daly <jeffd@silicom-usa.com>
> Sent: Tuesday, April 19, 2022 05:55
> To: Wang, Haiyue <haiyue.wang@intel.com>; dev@dpdk.org
> Cc: stable@dpdk.org
> Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug
> 
> 
> 
> > -----Original Message-----
> > From: Wang, Haiyue <haiyue.wang@intel.com>
> > Sent: Thursday, April 14, 2022 8:11 AM
> > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> > Cc: stable@dpdk.org
> > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on
> > hotplug
> >
> > Caution: This is an external email. Please take care when clicking links or
> > opening attachments.
> >
> >
> > > -----Original Message-----
> > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > Sent: Thursday, April 14, 2022 18:41
> > > To: Wang, Haiyue <haiyue.wang@intel.com>; dev@dpdk.org
> > > Cc: stable@dpdk.org
> > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking
> > > on hotplug
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Wang, Haiyue <haiyue.wang@intel.com>
> > > > Sent: Wednesday, April 13, 2022 11:00 PM
> > > > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> > > > Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>
> > > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking
> > > > on hotplug
> > > >
> > > > Caution: This is an external email. Please take care when clicking
> > > > links or opening attachments.
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Wang, Haiyue
> > > > > Sent: Thursday, April 14, 2022 10:49
> > > > > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> > > > > Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>
> > > > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and
> > > > > linking on hotplug
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > > > > Sent: Wednesday, April 13, 2022 01:42
> > > > > > To: dev@dpdk.org
> > > > > > Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>;
> > > > > > Wang, Haiyue <haiyue.wang@intel.com>
> > > > > > Subject: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking
> > > > > > on hotplug
> > > > > >
> > > > > > Currently the ixgbe driver does not ID any SFP except for the
> > > > > > first one plugged in. This can lead to no-link, or incorrect speed
> > conditions.
> > > > > >
> > > > > > For example:
> > > > > >
> > > > > > * If link is initially established with a 1G SFP, and later a
> > > > > > 1G/10G multispeed part is later installed, then the MAC link
> > > > > > setup functions are never called to change from 1000BASE-X to
> > > > > > 10GBASE-R mode, and the link stays running at the slower rate.
> > > > > >
> > > > > > * If link is initially established with a 1G SFP, and later a
> > > > > > 10G only module is later installed, no link is established,
> > > > > > since we are still trasnsmitting in 1000BASE-X mode to a 10GBASE-R
> > only partner.
> > > > > >
> > > > > > Refactor the SFP ID/setup, and link setup code, to more closely
> > > > > > match the flow of the mainline kernel driver which does not have
> > > > > > these issues.  In that driver a service task runs periodically
> > > > > > to handle these operations based on bit flags that have been set
> > > > > > (usually via interrupt or userspace request), and then get
> > > > > > cleared once the requested subtask has been completed.
> > > > > >
> > > > > > Fixes: af75078fece ("first public release")
> > > > > > Cc: stable@dpdk.org
> > > > > >
> > > > > > Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
> > > > > > Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> > > > > > ---
> > > > > >  drivers/net/ixgbe/ixgbe_ethdev.c | 533
> > > > > > +++++++++++++++++++++++--------
> > > > > > +++++++++++++++++++++++drivers/net/ixgbe/ixgbe_ethdev.h |
> > > > > > 14 +-
> > > > > >  2 files changed, 410 insertions(+), 137 deletions(-)
> > > > > >
> > > > >
> > > > >
> > > > > >
> > > > > >  struct ixgbe_stat_mapping_registers { @@ -510,7 +509,7 @@
> > > > > > struct ixgbe_adapter {
> > > > > >     uint8_t pflink_fullchk;
> > > > > >     uint8_t mac_ctrl_frame_fwd;
> > > > > >     rte_atomic32_t link_thread_running;
> > > > > > -   pthread_t link_thread_tid;
> > > > > > +   pthread_t service_thread_tid;
> > > > >
> > > > > No need to rename this variable,
> > > >
> > > > Let's do link related service now, so we can keep it, I missed to
> > > > add my comment. ;-)
> > > >
> > >
> > > I don't understand this reply, are you still asking to rework the patch or
> > not?
> > >
> >
> > Different thing.
> >
> > 1. This var can be kept to trace the created thread. (change less code to keep
> >    the patch clean.)
> > 2. Yes, two patches.
> >
> 
> ok, I guess I'm just being thick-headed here, but I still don't understand why you are saying it
> should be split into
> 2 patches.  if I understand *what* you are asking, you're saying make the original thread periodic to
> continuously

Well, ...

Your patch merges the original 'ixgbe_setup_link' task into one,
this will make us hard to review the whole design. So what I said
is: firstly, let's change the thread to a service thread to handle
the 'ixgbe_setup_link' subtask firstly. Which is 'ixgbe_link_service'
in your whole patch.

Small patch is good for us to review, and try to do one thing.

Hope this time, I can make myself clear. ;-)

> do ixgbe_link_setup() ?  I believe the problem with the setup is that the sfp_type is only detected
> once
> at initialization time and if nothing is in the cage then the code just returns IXGBE_SUCCESS, in
> which case making
> this task periodic is useless.  the whole issue of hotplug is only addressed by the entire patch which
> 1) makes the
> task periodic, 2) changes the actions of the task to look for whether the cage has something in it and
> whether its
> been changed and needs to be configured again.
> 
> 
> > > > > we can separate this patch as least into two patches:
> > > >
> > > >
> > > > >
> > > > > 1st, change the thread handle 'ixgbe_dev_setup_link_thread_handler'
> > > > > from
> > > > >
> > > > > run-once to as periodical, to handle the original issue.
> > > > >
> > > > > The name 'ixgbe_dev_setup_link_thread_handler' may be not suitable
> > > > > now, as it is a service thread.
> > > > >
> > > > > We can change it to "'ixgbe_link_service_thread_handler'" to
> > > > > reflect the change purpose.
> > > > >
> > > > > 2nd, add the SFP hotplug in this patch.
> > > > >
> > > > >
> > > > >
> > > > > >  };
> > > > > >
> > > > > > --
> > > > > > 2.25.1

RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug

[Jeff Daly]

> -----Original Message-----
> From: Wang, Haiyue <haiyue.wang@intel.com>
> Sent: Monday, April 18, 2022 10:05 PM
> To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> Cc: stable@dpdk.org; Yang, Qiming <qiming.yang@intel.com>
> Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on
> hotplug
> 
> Caution: This is an external email. Please take care when clicking links or
> opening attachments.
> 
> 
> > -----Original Message-----
> > From: Jeff Daly <jeffd@silicom-usa.com>
> > Sent: Tuesday, April 19, 2022 05:55
> > To: Wang, Haiyue <haiyue.wang@intel.com>; dev@dpdk.org
> > Cc: stable@dpdk.org
> > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking
> > on hotplug
> >
> >
> >
> > > -----Original Message-----
> > > From: Wang, Haiyue <haiyue.wang@intel.com>
> > > Sent: Thursday, April 14, 2022 8:11 AM
> > > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> > > Cc: stable@dpdk.org
> > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking
> > > on hotplug
> > >
> > > Caution: This is an external email. Please take care when clicking
> > > links or opening attachments.
> > >
> > >
> > > > -----Original Message-----
> > > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > > Sent: Thursday, April 14, 2022 18:41
> > > > To: Wang, Haiyue <haiyue.wang@intel.com>; dev@dpdk.org
> > > > Cc: stable@dpdk.org
> > > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and
> > > > linking on hotplug
> > > >
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Wang, Haiyue <haiyue.wang@intel.com>
> > > > > Sent: Wednesday, April 13, 2022 11:00 PM
> > > > > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> > > > > Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>
> > > > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and
> > > > > linking on hotplug
> > > > >
> > > > > Caution: This is an external email. Please take care when
> > > > > clicking links or opening attachments.
> > > > >
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Wang, Haiyue
> > > > > > Sent: Thursday, April 14, 2022 10:49
> > > > > > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> > > > > > Cc: stable@dpdk.org; Stephen Douthit
> > > > > > <stephend@silicom-usa.com>
> > > > > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and
> > > > > > linking on hotplug
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > > > > > Sent: Wednesday, April 13, 2022 01:42
> > > > > > > To: dev@dpdk.org
> > > > > > > Cc: stable@dpdk.org; Stephen Douthit
> > > > > > > <stephend@silicom-usa.com>; Wang, Haiyue
> > > > > > > <haiyue.wang@intel.com>
> > > > > > > Subject: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and
> > > > > > > linking on hotplug
> > > > > > >
> > > > > > > Currently the ixgbe driver does not ID any SFP except for
> > > > > > > the first one plugged in. This can lead to no-link, or
> > > > > > > incorrect speed
> > > conditions.
> > > > > > >
> > > > > > > For example:
> > > > > > >
> > > > > > > * If link is initially established with a 1G SFP, and later
> > > > > > > a 1G/10G multispeed part is later installed, then the MAC
> > > > > > > link setup functions are never called to change from
> > > > > > > 1000BASE-X to 10GBASE-R mode, and the link stays running at the
> slower rate.
> > > > > > >
> > > > > > > * If link is initially established with a 1G SFP, and later
> > > > > > > a 10G only module is later installed, no link is
> > > > > > > established, since we are still trasnsmitting in 1000BASE-X
> > > > > > > mode to a 10GBASE-R
> > > only partner.
> > > > > > >
> > > > > > > Refactor the SFP ID/setup, and link setup code, to more
> > > > > > > closely match the flow of the mainline kernel driver which
> > > > > > > does not have these issues.  In that driver a service task
> > > > > > > runs periodically to handle these operations based on bit
> > > > > > > flags that have been set (usually via interrupt or userspace
> > > > > > > request), and then get cleared once the requested subtask has
> been completed.
> > > > > > >
> > > > > > > Fixes: af75078fece ("first public release")
> > > > > > > Cc: stable@dpdk.org
> > > > > > >
> > > > > > > Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
> > > > > > > Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> > > > > > > ---
> > > > > > >  drivers/net/ixgbe/ixgbe_ethdev.c | 533
> > > > > > > +++++++++++++++++++++++--------
> > > > > > > +++++++++++++++++++++++drivers/net/ixgbe/ixgbe_ethdev.h |
> > > > > > > 14 +-
> > > > > > >  2 files changed, 410 insertions(+), 137 deletions(-)
> > > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >  struct ixgbe_stat_mapping_registers { @@ -510,7 +509,7 @@
> > > > > > > struct ixgbe_adapter {
> > > > > > >     uint8_t pflink_fullchk;
> > > > > > >     uint8_t mac_ctrl_frame_fwd;
> > > > > > >     rte_atomic32_t link_thread_running;
> > > > > > > -   pthread_t link_thread_tid;
> > > > > > > +   pthread_t service_thread_tid;
> > > > > >
> > > > > > No need to rename this variable,
> > > > >
> > > > > Let's do link related service now, so we can keep it, I missed
> > > > > to add my comment. ;-)
> > > > >
> > > >
> > > > I don't understand this reply, are you still asking to rework the
> > > > patch or
> > > not?
> > > >
> > >
> > > Different thing.
> > >
> > > 1. This var can be kept to trace the created thread. (change less code to
> keep
> > >    the patch clean.)
> > > 2. Yes, two patches.
> > >
> >
> > ok, I guess I'm just being thick-headed here, but I still don't
> > understand why you are saying it should be split into
> > 2 patches.  if I understand *what* you are asking, you're saying make
> > the original thread periodic to continuously
> 
> Well, ...
> 
> Your patch merges the original 'ixgbe_setup_link' task into one, this will
> make us hard to review the whole design. So what I said
> is: firstly, let's change the thread to a service thread to handle the
> 'ixgbe_setup_link' subtask firstly. Which is 'ixgbe_link_service'
> in your whole patch.
> 

still not 100%, are you suggesting that the original ixgbe_dev_setup_link_thread_handler()
which currently is not periodic and only really calls ixgbe_setup_link() be changed to be a 
periodic task that essentially does what the patch's ixgbe_link_service() function does which
would only be checking whether link config is needed and if so calls ixgbe_setup_link() as 
before?

if I'm following the code correctly, it ends up going down to ixgbe_check_mac_link_generic()
which looks at SDP0 (in the case of needing xtalk fix) which incorrectly will set sfp_cage_full
when in fact the PRESENT# signal (or MOD_ABS#) is active low.  the code is extremely convoluted
in it's effort to be smart so I may be missing something, but I *believe* that what we end up
with is completely unnecessary probing of i2c busses looking for SFPs that don't exist.  even when
making it periodic, I don't think it's going to end up with working code.

> Small patch is good for us to review, and try to do one thing.
> 
> Hope this time, I can make myself clear. ;-)
> 
> > do ixgbe_link_setup() ?  I believe the problem with the setup is that
> > the sfp_type is only detected once at initialization time and if
> > nothing is in the cage then the code just returns IXGBE_SUCCESS, in
> > which case making this task periodic is useless.  the whole issue of
> > hotplug is only addressed by the entire patch which
> > 1) makes the
> > task periodic, 2) changes the actions of the task to look for whether
> > the cage has something in it and whether its been changed and needs to
> > be configured again.
> >
> >
> > > > > > we can separate this patch as least into two patches:
> > > > >
> > > > >
> > > > > >
> > > > > > 1st, change the thread handle
> 'ixgbe_dev_setup_link_thread_handler'
> > > > > > from
> > > > > >
> > > > > > run-once to as periodical, to handle the original issue.
> > > > > >
> > > > > > The name 'ixgbe_dev_setup_link_thread_handler' may be not
> > > > > > suitable now, as it is a service thread.
> > > > > >
> > > > > > We can change it to "'ixgbe_link_service_thread_handler'" to
> > > > > > reflect the change purpose.
> > > > > >
> > > > > > 2nd, add the SFP hotplug in this patch.
> > > > > >
> > > > > >
> > > > > >
> > > > > > >  };
> > > > > > >
> > > > > > > --
> > > > > > > 2.25.1

RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug

[Wang, Haiyue]

> -----Original Message-----
> From: Jeff Daly <jeffd@silicom-usa.com>
> Sent: Wednesday, April 20, 2022 01:34
> To: Wang, Haiyue <haiyue.wang@intel.com>; dev@dpdk.org
> Cc: stable@dpdk.org; Yang, Qiming <qiming.yang@intel.com>
> Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug
> 
> 
> 
> > -----Original Message-----
> > From: Wang, Haiyue <haiyue.wang@intel.com>
> > Sent: Monday, April 18, 2022 10:05 PM
> > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> > Cc: stable@dpdk.org; Yang, Qiming <qiming.yang@intel.com>
> > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on
> > hotplug
> >
> > Caution: This is an external email. Please take care when clicking links or
> > opening attachments.
> >
> >
> > > -----Original Message-----
> > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > Sent: Tuesday, April 19, 2022 05:55
> > > To: Wang, Haiyue <haiyue.wang@intel.com>; dev@dpdk.org
> > > Cc: stable@dpdk.org
> > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking
> > > on hotplug
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Wang, Haiyue <haiyue.wang@intel.com>
> > > > Sent: Thursday, April 14, 2022 8:11 AM
> > > > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> > > > Cc: stable@dpdk.org
> > > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking
> > > > on hotplug
> > > >
> > > > Caution: This is an external email. Please take care when clicking
> > > > links or opening attachments.
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > > > Sent: Thursday, April 14, 2022 18:41
> > > > > To: Wang, Haiyue <haiyue.wang@intel.com>; dev@dpdk.org
> > > > > Cc: stable@dpdk.org
> > > > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and
> > > > > linking on hotplug
> > > > >
> > > > >
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Wang, Haiyue <haiyue.wang@intel.com>
> > > > > > Sent: Wednesday, April 13, 2022 11:00 PM
> > > > > > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> > > > > > Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>
> > > > > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and
> > > > > > linking on hotplug
> > > > > >
> > > > > > Caution: This is an external email. Please take care when
> > > > > > clicking links or opening attachments.
> > > > > >
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Wang, Haiyue
> > > > > > > Sent: Thursday, April 14, 2022 10:49
> > > > > > > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> > > > > > > Cc: stable@dpdk.org; Stephen Douthit
> > > > > > > <stephend@silicom-usa.com>
> > > > > > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and
> > > > > > > linking on hotplug
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > > > > > > Sent: Wednesday, April 13, 2022 01:42
> > > > > > > > To: dev@dpdk.org
> > > > > > > > Cc: stable@dpdk.org; Stephen Douthit
> > > > > > > > <stephend@silicom-usa.com>; Wang, Haiyue
> > > > > > > > <haiyue.wang@intel.com>
> > > > > > > > Subject: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and
> > > > > > > > linking on hotplug
> > > > > > > >
> > > > > > > > Currently the ixgbe driver does not ID any SFP except for
> > > > > > > > the first one plugged in. This can lead to no-link, or
> > > > > > > > incorrect speed
> > > > conditions.
> > > > > > > >
> > > > > > > > For example:
> > > > > > > >
> > > > > > > > * If link is initially established with a 1G SFP, and later
> > > > > > > > a 1G/10G multispeed part is later installed, then the MAC
> > > > > > > > link setup functions are never called to change from
> > > > > > > > 1000BASE-X to 10GBASE-R mode, and the link stays running at the
> > slower rate.
> > > > > > > >
> > > > > > > > * If link is initially established with a 1G SFP, and later
> > > > > > > > a 10G only module is later installed, no link is
> > > > > > > > established, since we are still trasnsmitting in 1000BASE-X
> > > > > > > > mode to a 10GBASE-R
> > > > only partner.
> > > > > > > >
> > > > > > > > Refactor the SFP ID/setup, and link setup code, to more
> > > > > > > > closely match the flow of the mainline kernel driver which
> > > > > > > > does not have these issues.  In that driver a service task
> > > > > > > > runs periodically to handle these operations based on bit
> > > > > > > > flags that have been set (usually via interrupt or userspace
> > > > > > > > request), and then get cleared once the requested subtask has
> > been completed.
> > > > > > > >
> > > > > > > > Fixes: af75078fece ("first public release")
> > > > > > > > Cc: stable@dpdk.org
> > > > > > > >
> > > > > > > > Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
> > > > > > > > Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> > > > > > > > ---
> > > > > > > >  drivers/net/ixgbe/ixgbe_ethdev.c | 533
> > > > > > > > +++++++++++++++++++++++--------
> > > > > > > > +++++++++++++++++++++++drivers/net/ixgbe/ixgbe_ethdev.h |
> > > > > > > > 14 +-
> > > > > > > >  2 files changed, 410 insertions(+), 137 deletions(-)
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > >  struct ixgbe_stat_mapping_registers { @@ -510,7 +509,7 @@
> > > > > > > > struct ixgbe_adapter {
> > > > > > > >     uint8_t pflink_fullchk;
> > > > > > > >     uint8_t mac_ctrl_frame_fwd;
> > > > > > > >     rte_atomic32_t link_thread_running;
> > > > > > > > -   pthread_t link_thread_tid;
> > > > > > > > +   pthread_t service_thread_tid;
> > > > > > >
> > > > > > > No need to rename this variable,
> > > > > >
> > > > > > Let's do link related service now, so we can keep it, I missed
> > > > > > to add my comment. ;-)
> > > > > >
> > > > >
> > > > > I don't understand this reply, are you still asking to rework the
> > > > > patch or
> > > > not?
> > > > >
> > > >
> > > > Different thing.
> > > >
> > > > 1. This var can be kept to trace the created thread. (change less code to
> > keep
> > > >    the patch clean.)
> > > > 2. Yes, two patches.
> > > >
> > >
> > > ok, I guess I'm just being thick-headed here, but I still don't
> > > understand why you are saying it should be split into
> > > 2 patches.  if I understand *what* you are asking, you're saying make
> > > the original thread periodic to continuously
> >
> > Well, ...
> >
> > Your patch merges the original 'ixgbe_setup_link' task into one, this will
> > make us hard to review the whole design. So what I said
> > is: firstly, let's change the thread to a service thread to handle the
> > 'ixgbe_setup_link' subtask firstly. Which is 'ixgbe_link_service'
> > in your whole patch.
> >
> 
> still not 100%, are you suggesting that the original ixgbe_dev_setup_link_thread_handler()
> which currently is not periodic and only really calls ixgbe_setup_link() be changed to be a
> periodic task that essentially does what the patch's ixgbe_link_service() function does which
> would only be checking whether link config is needed and if so calls ixgbe_setup_link() as
> before?
> 
> if I'm following the code correctly, it ends up going down to ixgbe_check_mac_link_generic()
> which looks at SDP0 (in the case of needing xtalk fix) which incorrectly will set sfp_cage_full
> when in fact the PRESENT# signal (or MOD_ABS#) is active low.  the code is extremely convoluted
> in it's effort to be smart so I may be missing something, but I *believe* that what we end up
> with is completely unnecessary probing of i2c busses looking for SFPs that don't exist.  even when
> making it periodic, I don't think it's going to end up with working code.


I'm lost ...

The 'periodic' is the service thread running mode, but the subtask is
ONLY called when be scheduled, like:

if (!(adapter->flags & IXGBE_FLAG_NEED_LINK_CONFIG))
		return;

> 
> > Small patch is good for us to review, and try to do one thing.
> >
> > Hope this time, I can make myself clear. ;-)
> >
> > > do ixgbe_link_setup() ?  I believe the problem with the setup is that
> > > the sfp_type is only detected once at initialization time and if
> > > nothing is in the cage then the code just returns IXGBE_SUCCESS, in
> > > which case making this task periodic is useless.  the whole issue of
> > > hotplug is only addressed by the entire patch which
> > > 1) makes the
> > > task periodic, 2) changes the actions of the task to look for whether
> > > the cage has something in it and whether its been changed and needs to
> > > be configured again.
> > >
> > >
> > > > > > > we can separate this patch as least into two patches:
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > 1st, change the thread handle
> > 'ixgbe_dev_setup_link_thread_handler'
> > > > > > > from
> > > > > > >
> > > > > > > run-once to as periodical, to handle the original issue.
> > > > > > >
> > > > > > > The name 'ixgbe_dev_setup_link_thread_handler' may be not
> > > > > > > suitable now, as it is a service thread.
> > > > > > >
> > > > > > > We can change it to "'ixgbe_link_service_thread_handler'" to
> > > > > > > reflect the change purpose.
> > > > > > >
> > > > > > > 2nd, add the SFP hotplug in this patch.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > >  };
> > > > > > > >
> > > > > > > > --
> > > > > > > > 2.25.1

RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug

[Jeff Daly]

> -----Original Message-----
> From: Wang, Haiyue <haiyue.wang@intel.com>
> Sent: Tuesday, April 19, 2022 9:09 PM
> To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> Cc: stable@dpdk.org; Yang, Qiming <qiming.yang@intel.com>
> Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on
> hotplug
> 
> Caution: This is an external email. Please take care when clicking links or
> opening attachments.
> 
> 
> > -----Original Message-----
> > From: Jeff Daly <jeffd@silicom-usa.com>
> > Sent: Wednesday, April 20, 2022 01:34
> > To: Wang, Haiyue <haiyue.wang@intel.com>; dev@dpdk.org
> > Cc: stable@dpdk.org; Yang, Qiming <qiming.yang@intel.com>
> > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking
> > on hotplug
> >
> >
> >
> > > -----Original Message-----
> > > From: Wang, Haiyue <haiyue.wang@intel.com>
> > > Sent: Monday, April 18, 2022 10:05 PM
> > > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> > > Cc: stable@dpdk.org; Yang, Qiming <qiming.yang@intel.com>
> > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking
> > > on hotplug
> > >
> > > Caution: This is an external email. Please take care when clicking
> > > links or opening attachments.
> > >
> > >
> > > > -----Original Message-----
> > > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > > Sent: Tuesday, April 19, 2022 05:55
> > > > To: Wang, Haiyue <haiyue.wang@intel.com>; dev@dpdk.org
> > > > Cc: stable@dpdk.org
> > > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and
> > > > linking on hotplug
> > > >
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Wang, Haiyue <haiyue.wang@intel.com>
> > > > > Sent: Thursday, April 14, 2022 8:11 AM
> > > > > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> > > > > Cc: stable@dpdk.org
> > > > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and
> > > > > linking on hotplug
> > > > >
> > > > > Caution: This is an external email. Please take care when
> > > > > clicking links or opening attachments.
> > > > >
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > > > > Sent: Thursday, April 14, 2022 18:41
> > > > > > To: Wang, Haiyue <haiyue.wang@intel.com>; dev@dpdk.org
> > > > > > Cc: stable@dpdk.org
> > > > > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and
> > > > > > linking on hotplug
> > > > > >
> > > > > >
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Wang, Haiyue <haiyue.wang@intel.com>
> > > > > > > Sent: Wednesday, April 13, 2022 11:00 PM
> > > > > > > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> > > > > > > Cc: stable@dpdk.org; Stephen Douthit
> > > > > > > <stephend@silicom-usa.com>
> > > > > > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and
> > > > > > > linking on hotplug
> > > > > > >
> > > > > > > Caution: This is an external email. Please take care when
> > > > > > > clicking links or opening attachments.
> > > > > > >
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Wang, Haiyue
> > > > > > > > Sent: Thursday, April 14, 2022 10:49
> > > > > > > > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> > > > > > > > Cc: stable@dpdk.org; Stephen Douthit
> > > > > > > > <stephend@silicom-usa.com>
> > > > > > > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection
> > > > > > > > and linking on hotplug
> > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > > > > > > > Sent: Wednesday, April 13, 2022 01:42
> > > > > > > > > To: dev@dpdk.org
> > > > > > > > > Cc: stable@dpdk.org; Stephen Douthit
> > > > > > > > > <stephend@silicom-usa.com>; Wang, Haiyue
> > > > > > > > > <haiyue.wang@intel.com>
> > > > > > > > > Subject: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and
> > > > > > > > > linking on hotplug
> > > > > > > > >
> > > > > > > > > Currently the ixgbe driver does not ID any SFP except
> > > > > > > > > for the first one plugged in. This can lead to no-link,
> > > > > > > > > or incorrect speed
> > > > > conditions.
> > > > > > > > >
> > > > > > > > > For example:
> > > > > > > > >
> > > > > > > > > * If link is initially established with a 1G SFP, and
> > > > > > > > > later a 1G/10G multispeed part is later installed, then
> > > > > > > > > the MAC link setup functions are never called to change
> > > > > > > > > from 1000BASE-X to 10GBASE-R mode, and the link stays
> > > > > > > > > running at the
> > > slower rate.
> > > > > > > > >
> > > > > > > > > * If link is initially established with a 1G SFP, and
> > > > > > > > > later a 10G only module is later installed, no link is
> > > > > > > > > established, since we are still trasnsmitting in
> > > > > > > > > 1000BASE-X mode to a 10GBASE-R
> > > > > only partner.
> > > > > > > > >
> > > > > > > > > Refactor the SFP ID/setup, and link setup code, to more
> > > > > > > > > closely match the flow of the mainline kernel driver
> > > > > > > > > which does not have these issues.  In that driver a
> > > > > > > > > service task runs periodically to handle these
> > > > > > > > > operations based on bit flags that have been set
> > > > > > > > > (usually via interrupt or userspace request), and then
> > > > > > > > > get cleared once the requested subtask has
> > > been completed.
> > > > > > > > >
> > > > > > > > > Fixes: af75078fece ("first public release")
> > > > > > > > > Cc: stable@dpdk.org
> > > > > > > > >
> > > > > > > > > Signed-off-by: Stephen Douthit
> > > > > > > > > <stephend@silicom-usa.com>
> > > > > > > > > Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> > > > > > > > > ---
> > > > > > > > >  drivers/net/ixgbe/ixgbe_ethdev.c | 533
> > > > > > > > > +++++++++++++++++++++++--------
> > > > > > > > > +++++++++++++++++++++++drivers/net/ixgbe/ixgbe_ethdev.h
> > > > > > > > > +++++++++++++++++++++++|
> > > > > > > > > 14 +-
> > > > > > > > >  2 files changed, 410 insertions(+), 137 deletions(-)
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > >  struct ixgbe_stat_mapping_registers { @@ -510,7 +509,7
> > > > > > > > > @@ struct ixgbe_adapter {
> > > > > > > > >     uint8_t pflink_fullchk;
> > > > > > > > >     uint8_t mac_ctrl_frame_fwd;
> > > > > > > > >     rte_atomic32_t link_thread_running;
> > > > > > > > > -   pthread_t link_thread_tid;
> > > > > > > > > +   pthread_t service_thread_tid;
> > > > > > > >
> > > > > > > > No need to rename this variable,
> > > > > > >
> > > > > > > Let's do link related service now, so we can keep it, I
> > > > > > > missed to add my comment. ;-)
> > > > > > >
> > > > > >
> > > > > > I don't understand this reply, are you still asking to rework
> > > > > > the patch or
> > > > > not?
> > > > > >
> > > > >
> > > > > Different thing.
> > > > >
> > > > > 1. This var can be kept to trace the created thread. (change
> > > > > less code to
> > > keep
> > > > >    the patch clean.)
> > > > > 2. Yes, two patches.
> > > > >
> > > >
> > > > ok, I guess I'm just being thick-headed here, but I still don't
> > > > understand why you are saying it should be split into
> > > > 2 patches.  if I understand *what* you are asking, you're saying
> > > > make the original thread periodic to continuously
> > >
> > > Well, ...
> > >
> > > Your patch merges the original 'ixgbe_setup_link' task into one,
> > > this will make us hard to review the whole design. So what I said
> > > is: firstly, let's change the thread to a service thread to handle
> > > the 'ixgbe_setup_link' subtask firstly. Which is 'ixgbe_link_service'
> > > in your whole patch.
> > >
> >
> > still not 100%, are you suggesting that the original
> > ixgbe_dev_setup_link_thread_handler()
> > which currently is not periodic and only really calls
> > ixgbe_setup_link() be changed to be a periodic task that essentially
> > does what the patch's ixgbe_link_service() function does which would
> > only be checking whether link config is needed and if so calls
> ixgbe_setup_link() as before?
> >
> > if I'm following the code correctly, it ends up going down to
> > ixgbe_check_mac_link_generic() which looks at SDP0 (in the case of
> > needing xtalk fix) which incorrectly will set sfp_cage_full when in
> > fact the PRESENT# signal (or MOD_ABS#) is active low.  the code is
> > extremely convoluted in it's effort to be smart so I may be missing
> > something, but I *believe* that what we end up with is completely
> unnecessary probing of i2c busses looking for SFPs that don't exist.  even
> when making it periodic, I don't think it's going to end up with working code.
> 
> 
> I'm lost ...
> 
> The 'periodic' is the service thread running mode, but the subtask is ONLY
> called when be scheduled, like:
> 
> if (!(adapter->flags & IXGBE_FLAG_NEED_LINK_CONFIG))
>                 return;
> 

this makes no sense.  the *only* time right now when IXGBE_FLAG_NEED_LINK_CONFIG is set is
immediately before the thread is created!  there's no 'hotplug' part of the patch vs fixing the 
'original issue' (your words, below).  the 'original issue' is that the code doesn't work for hotplug
(SFP cages), so there's no breaking up the patch to fix any original issue first.  the only issue is
that for SFP cages, the code doesn't work.  

> >
> > > Small patch is good for us to review, and try to do one thing.
> > >
> > > Hope this time, I can make myself clear. ;-)
> > >
> > > > do ixgbe_link_setup() ?  I believe the problem with the setup is
> > > > that the sfp_type is only detected once at initialization time and
> > > > if nothing is in the cage then the code just returns
> > > > IXGBE_SUCCESS, in which case making this task periodic is useless.
> > > > the whole issue of hotplug is only addressed by the entire patch
> > > > which
> > > > 1) makes the
> > > > task periodic, 2) changes the actions of the task to look for
> > > > whether the cage has something in it and whether its been changed
> > > > and needs to be configured again.
> > > >
> > > >
> > > > > > > > we can separate this patch as least into two patches:
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > 1st, change the thread handle
> > > 'ixgbe_dev_setup_link_thread_handler'
> > > > > > > > from
> > > > > > > >
> > > > > > > > run-once to as periodical, to handle the original issue.
> > > > > > > >
> > > > > > > > The name 'ixgbe_dev_setup_link_thread_handler' may be not
> > > > > > > > suitable now, as it is a service thread.
> > > > > > > >
> > > > > > > > We can change it to "'ixgbe_link_service_thread_handler'"
> > > > > > > > to reflect the change purpose.
> > > > > > > >
> > > > > > > > 2nd, add the SFP hotplug in this patch.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > >  };
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > 2.25.1

RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug

[Wang, Haiyue]

> -----Original Message-----
> From: Jeff Daly <jeffd@silicom-usa.com>
> Sent: Friday, April 22, 2022 01:31
> To: Wang, Haiyue <haiyue.wang@intel.com>; dev@dpdk.org
> Cc: stable@dpdk.org; Yang, Qiming <qiming.yang@intel.com>
> Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug
> 
> 
> 
> > -----Original Message-----
> > From: Wang, Haiyue <haiyue.wang@intel.com>
> > Sent: Tuesday, April 19, 2022 9:09 PM
> > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> > Cc: stable@dpdk.org; Yang, Qiming <qiming.yang@intel.com>
> > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on
> > hotplug
> >
> > Caution: This is an external email. Please take care when clicking links or
> > opening attachments.
> >
> >
> > > -----Original Message-----
> > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > Sent: Wednesday, April 20, 2022 01:34
> > > To: Wang, Haiyue <haiyue.wang@intel.com>; dev@dpdk.org
> > > Cc: stable@dpdk.org; Yang, Qiming <qiming.yang@intel.com>
> > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking
> > > on hotplug
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Wang, Haiyue <haiyue.wang@intel.com>
> > > > Sent: Monday, April 18, 2022 10:05 PM
> > > > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> > > > Cc: stable@dpdk.org; Yang, Qiming <qiming.yang@intel.com>
> > > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking
> > > > on hotplug
> > > >
> > > > Caution: This is an external email. Please take care when clicking
> > > > links or opening attachments.
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > > > Sent: Tuesday, April 19, 2022 05:55
> > > > > To: Wang, Haiyue <haiyue.wang@intel.com>; dev@dpdk.org
> > > > > Cc: stable@dpdk.org
> > > > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and
> > > > > linking on hotplug
> > > > >
> > > > >
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Wang, Haiyue <haiyue.wang@intel.com>
> > > > > > Sent: Thursday, April 14, 2022 8:11 AM
> > > > > > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> > > > > > Cc: stable@dpdk.org
> > > > > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and
> > > > > > linking on hotplug
> > > > > >
> > > > > > Caution: This is an external email. Please take care when
> > > > > > clicking links or opening attachments.
> > > > > >
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > > > > > Sent: Thursday, April 14, 2022 18:41
> > > > > > > To: Wang, Haiyue <haiyue.wang@intel.com>; dev@dpdk.org
> > > > > > > Cc: stable@dpdk.org
> > > > > > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and
> > > > > > > linking on hotplug
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Wang, Haiyue <haiyue.wang@intel.com>
> > > > > > > > Sent: Wednesday, April 13, 2022 11:00 PM
> > > > > > > > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> > > > > > > > Cc: stable@dpdk.org; Stephen Douthit
> > > > > > > > <stephend@silicom-usa.com>
> > > > > > > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and
> > > > > > > > linking on hotplug
> > > > > > > >
> > > > > > > > Caution: This is an external email. Please take care when
> > > > > > > > clicking links or opening attachments.
> > > > > > > >
> > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Wang, Haiyue
> > > > > > > > > Sent: Thursday, April 14, 2022 10:49
> > > > > > > > > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> > > > > > > > > Cc: stable@dpdk.org; Stephen Douthit
> > > > > > > > > <stephend@silicom-usa.com>
> > > > > > > > > Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection
> > > > > > > > > and linking on hotplug
> > > > > > > > >
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > > > > > > > > Sent: Wednesday, April 13, 2022 01:42
> > > > > > > > > > To: dev@dpdk.org
> > > > > > > > > > Cc: stable@dpdk.org; Stephen Douthit
> > > > > > > > > > <stephend@silicom-usa.com>; Wang, Haiyue
> > > > > > > > > > <haiyue.wang@intel.com>
> > > > > > > > > > Subject: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and
> > > > > > > > > > linking on hotplug
> > > > > > > > > >
> > > > > > > > > > Currently the ixgbe driver does not ID any SFP except
> > > > > > > > > > for the first one plugged in. This can lead to no-link,
> > > > > > > > > > or incorrect speed
> > > > > > conditions.
> > > > > > > > > >
> > > > > > > > > > For example:
> > > > > > > > > >
> > > > > > > > > > * If link is initially established with a 1G SFP, and
> > > > > > > > > > later a 1G/10G multispeed part is later installed, then
> > > > > > > > > > the MAC link setup functions are never called to change
> > > > > > > > > > from 1000BASE-X to 10GBASE-R mode, and the link stays
> > > > > > > > > > running at the
> > > > slower rate.
> > > > > > > > > >
> > > > > > > > > > * If link is initially established with a 1G SFP, and
> > > > > > > > > > later a 10G only module is later installed, no link is
> > > > > > > > > > established, since we are still trasnsmitting in
> > > > > > > > > > 1000BASE-X mode to a 10GBASE-R
> > > > > > only partner.
> > > > > > > > > >
> > > > > > > > > > Refactor the SFP ID/setup, and link setup code, to more
> > > > > > > > > > closely match the flow of the mainline kernel driver
> > > > > > > > > > which does not have these issues.  In that driver a
> > > > > > > > > > service task runs periodically to handle these
> > > > > > > > > > operations based on bit flags that have been set
> > > > > > > > > > (usually via interrupt or userspace request), and then
> > > > > > > > > > get cleared once the requested subtask has
> > > > been completed.
> > > > > > > > > >
> > > > > > > > > > Fixes: af75078fece ("first public release")
> > > > > > > > > > Cc: stable@dpdk.org
> > > > > > > > > >
> > > > > > > > > > Signed-off-by: Stephen Douthit
> > > > > > > > > > <stephend@silicom-usa.com>
> > > > > > > > > > Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> > > > > > > > > > ---
> > > > > > > > > >  drivers/net/ixgbe/ixgbe_ethdev.c | 533
> > > > > > > > > > +++++++++++++++++++++++--------
> > > > > > > > > > +++++++++++++++++++++++drivers/net/ixgbe/ixgbe_ethdev.h
> > > > > > > > > > +++++++++++++++++++++++|
> > > > > > > > > > 14 +-
> > > > > > > > > >  2 files changed, 410 insertions(+), 137 deletions(-)
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >  struct ixgbe_stat_mapping_registers { @@ -510,7 +509,7
> > > > > > > > > > @@ struct ixgbe_adapter {
> > > > > > > > > >     uint8_t pflink_fullchk;
> > > > > > > > > >     uint8_t mac_ctrl_frame_fwd;
> > > > > > > > > >     rte_atomic32_t link_thread_running;
> > > > > > > > > > -   pthread_t link_thread_tid;
> > > > > > > > > > +   pthread_t service_thread_tid;
> > > > > > > > >
> > > > > > > > > No need to rename this variable,
> > > > > > > >
> > > > > > > > Let's do link related service now, so we can keep it, I
> > > > > > > > missed to add my comment. ;-)
> > > > > > > >
> > > > > > >
> > > > > > > I don't understand this reply, are you still asking to rework
> > > > > > > the patch or
> > > > > > not?
> > > > > > >
> > > > > >
> > > > > > Different thing.
> > > > > >
> > > > > > 1. This var can be kept to trace the created thread. (change
> > > > > > less code to
> > > > keep
> > > > > >    the patch clean.)
> > > > > > 2. Yes, two patches.
> > > > > >
> > > > >
> > > > > ok, I guess I'm just being thick-headed here, but I still don't
> > > > > understand why you are saying it should be split into
> > > > > 2 patches.  if I understand *what* you are asking, you're saying
> > > > > make the original thread periodic to continuously
> > > >
> > > > Well, ...
> > > >
> > > > Your patch merges the original 'ixgbe_setup_link' task into one,
> > > > this will make us hard to review the whole design. So what I said
> > > > is: firstly, let's change the thread to a service thread to handle
> > > > the 'ixgbe_setup_link' subtask firstly. Which is 'ixgbe_link_service'
> > > > in your whole patch.
> > > >
> > >
> > > still not 100%, are you suggesting that the original
> > > ixgbe_dev_setup_link_thread_handler()
> > > which currently is not periodic and only really calls
> > > ixgbe_setup_link() be changed to be a periodic task that essentially
> > > does what the patch's ixgbe_link_service() function does which would
> > > only be checking whether link config is needed and if so calls
> > ixgbe_setup_link() as before?
> > >
> > > if I'm following the code correctly, it ends up going down to
> > > ixgbe_check_mac_link_generic() which looks at SDP0 (in the case of
> > > needing xtalk fix) which incorrectly will set sfp_cage_full when in
> > > fact the PRESENT# signal (or MOD_ABS#) is active low.  the code is
> > > extremely convoluted in it's effort to be smart so I may be missing
> > > something, but I *believe* that what we end up with is completely
> > unnecessary probing of i2c busses looking for SFPs that don't exist.  even
> > when making it periodic, I don't think it's going to end up with working code.
> >
> >
> > I'm lost ...
> >
> > The 'periodic' is the service thread running mode, but the subtask is ONLY
> > called when be scheduled, like:
> >
> > if (!(adapter->flags & IXGBE_FLAG_NEED_LINK_CONFIG))
> >                 return;
> >
> 
> this makes no sense.  the *only* time right now when IXGBE_FLAG_NEED_LINK_CONFIG is set is
> immediately before the thread is created!  there's no 'hotplug' part of the patch vs fixing the
> 'original issue' (your words, below).  the 'original issue' is that the code doesn't work for hotplug
> (SFP cages), so there's no breaking up the patch to fix any original issue first.  the only issue is
> that for SFP cages, the code doesn't work.

No!

Your code is right, but looks like you still be lost in your hotplug issue.

+static void *
+ixgbe_dev_service_thread_handler(void *param)
+{
+	struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
+	struct ixgbe_hw *hw =
+		IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	uint64_t start, ticks, service_ms;
+	uint32_t speed;
+	s32 err;
+	bool link_up;
+
+	while (1) {
+		ixgbe_sfp_service(dev);
+		ixgbe_link_service(dev); 
                 ^
                 |
                 +- 1st patch, just handle the 'ixgbe_link_service_thread_handler' issue.

I've no words to comment again, hope this makes thing clear.

> 
> > >
> > > > Small patch is good for us to review, and try to do one thing.
> > > >
> > > > Hope this time, I can make myself clear. ;-)
> > > >
> > > > > do ixgbe_link_setup() ?  I believe the problem with the setup is
> > > > > that the sfp_type is only detected once at initialization time and
> > > > > if nothing is in the cage then the code just returns
> > > > > IXGBE_SUCCESS, in which case making this task periodic is useless.
> > > > > the whole issue of hotplug is only addressed by the entire patch
> > > > > which
> > > > > 1) makes the
> > > > > task periodic, 2) changes the actions of the task to look for
> > > > > whether the cage has something in it and whether its been changed
> > > > > and needs to be configured again.
> > > > >
> > > > >
> > > > > > > > > we can separate this patch as least into two patches:
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 1st, change the thread handle
> > > > 'ixgbe_dev_setup_link_thread_handler'
> > > > > > > > > from
> > > > > > > > >
> > > > > > > > > run-once to as periodical, to handle the original issue.
> > > > > > > > >
> > > > > > > > > The name 'ixgbe_dev_setup_link_thread_handler' may be not
> > > > > > > > > suitable now, as it is a service thread.
> > > > > > > > >
> > > > > > > > > We can change it to "'ixgbe_link_service_thread_handler'"
> > > > > > > > > to reflect the change purpose.
> > > > > > > > >
> > > > > > > > > 2nd, add the SFP hotplug in this patch.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > >  };
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > 2.25.1

RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug

[Zhang, Qi Z]

> -----Original Message-----
> From: Jeff Daly <jeffd@silicom-usa.com>
> Sent: Wednesday, April 13, 2022 1:42 AM
> To: dev@dpdk.org
> Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>; Wang,
> Haiyue <haiyue.wang@intel.com>
> Subject: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug
> 
> Currently the ixgbe driver does not ID any SFP except for the first one plugged
> in. This can lead to no-link, or incorrect speed conditions.

Does kernel driver has the same issue for this?

> 
> For example:
> 
> * If link is initially established with a 1G SFP, and later a 1G/10G multispeed
> part is later installed, then the MAC link setup functions are never called to
> change from 1000BASE-X to 10GBASE-R mode, and the link stays running at the
> slower rate.
> 
> * If link is initially established with a 1G SFP, and later a 10G only module is
> later installed, no link is established, since we are still trasnsmitting in
> 1000BASE-X mode to a 10GBASE-R only partner.
> 
> Refactor the SFP ID/setup, and link setup code, to more closely match the flow
> of the mainline kernel driver which does not have these issues.  In that driver a
> service task runs periodically to handle these operations based on bit flags that
> have been set (usually via interrupt or userspace request), and then get cleared
> once the requested subtask has been completed.

If kernel driver don't have this issue, Is this the same way that kernel driver handle this issue?

Btw, could you break down the patch for easy review?

Thanks
Qi

RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on hotplug

[Jeff Daly]

> -----Original Message-----
> From: Zhang, Qi Z <qi.z.zhang@intel.com>
> Sent: Wednesday, May 11, 2022 9:27 PM
> To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org
> Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>
> Subject: RE: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on
> hotplug
> 
> Caution: This is an external email. Please take care when clicking links or
> opening attachments.
> 
> 
> > -----Original Message-----
> > From: Jeff Daly <jeffd@silicom-usa.com>
> > Sent: Wednesday, April 13, 2022 1:42 AM
> > To: dev@dpdk.org
> > Cc: stable@dpdk.org; Stephen Douthit <stephend@silicom-usa.com>;
> Wang,
> > Haiyue <haiyue.wang@intel.com>
> > Subject: [PATCH v6 2/2] net/ixgbe: Fix SFP detection and linking on
> > hotplug
> >
> > Currently the ixgbe driver does not ID any SFP except for the first
> > one plugged in. This can lead to no-link, or incorrect speed conditions.
> 
> Does kernel driver has the same issue for this?
> 

No, the kernel driver does the correct thing.

> >
> > For example:
> >
> > * If link is initially established with a 1G SFP, and later a 1G/10G
> > multispeed part is later installed, then the MAC link setup functions
> > are never called to change from 1000BASE-X to 10GBASE-R mode, and the
> > link stays running at the slower rate.
> >
> > * If link is initially established with a 1G SFP, and later a 10G only
> > module is later installed, no link is established, since we are still
> > trasnsmitting in 1000BASE-X mode to a 10GBASE-R only partner.
> >
> > Refactor the SFP ID/setup, and link setup code, to more closely match
> > the flow of the mainline kernel driver which does not have these
> > issues.  In that driver a service task runs periodically to handle
> > these operations based on bit flags that have been set (usually via
> > interrupt or userspace request), and then get cleared once the requested
> subtask has been completed.
> 
> If kernel driver don't have this issue, Is this the same way that kernel driver
> handle this issue?
>

The history can probably be searched back for, but yes the mechanism
implemented in this patch tries to follow the kernel's method for handling
hotplug.  The patch has been split into 3 chunks, starting with:

https://patchwork.dpdk.org/project/dpdk/patch/ffbdaf3aa487241a51bb6512bbb4701da17e69fe.1652988826.git.jeffd@silicom-usa.com/

it was refactored according to the wishes of the prior maintainer, so it's not
exactly the same as this one which is why it was submitted as a new one, 
not a revision of this.  

HTH.
 
> Btw, could you break down the patch for easy review?
> 
> Thanks
> Qi