From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9839742B9A; Thu, 25 May 2023 12:10:52 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B243742D77; Thu, 25 May 2023 12:10:08 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id 7D1EE410F9 for ; Thu, 25 May 2023 12:09:58 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 34PA2UjS020269 for ; Thu, 25 May 2023 03:09:57 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=kRHigjH98bdhwoGGG41niOq+Tw9a6wNDtexyz4Y2ZTM=; b=DNRlbAMS5LxQbcNt2BkxsKkSEJnZYQZmFtUGdXBne/VTMd/fWHHeGHuh3HFO1Nj2AfcM Q2JWeelFm68rySBRg2W8uN3VSZn+DVff+cOVXPwshnEyo90AZXeXQ8M75V3FG1EUtwHt ZSw0lO2LeW3NxmSA0Qe63LLhniD2XTk/CV9IDFaRwFPQL2rFwPl4El2e7GMcrHljxeW9 lHRuPhyUjFt0wlqOYhHEaGgw5Pv2keZYbwpfZbP1sj+fQN2+Ak7X2L8/irjLv3cItFNo pWn1MvvLAp9kmWc/ZY3WPe5xkhycXbEqFKJsm54n6qStSorjd8cjJjyVuNjNUq2FhTgp ZQ== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0a-0016f401.pphosted.com (PPS) with ESMTPS id 3qt5jng0kq-18 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Thu, 25 May 2023 03:09:57 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Thu, 25 May 2023 03:09:44 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Thu, 25 May 2023 03:09:44 -0700 Received: from hyd1588t430.caveonetworks.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id A83565B6D16; Thu, 25 May 2023 03:00:07 -0700 (PDT) From: Nithin Dabilpuram To: Nithin Kumar Dabilpuram , Kiran Kumar K , Sunil Kumar Kori , Satha Rao CC: , , Harman Kalra Subject: [PATCH v3 17/32] common/cnxk: sync between mbox up and down messages Date: Thu, 25 May 2023 15:28:49 +0530 Message-ID: <20230525095904.3967080-17-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230525095904.3967080-1-ndabilpuram@marvell.com> References: <20230411091144.1087887-1-ndabilpuram@marvell.com> <20230525095904.3967080-1-ndabilpuram@marvell.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Proofpoint-ORIG-GUID: QTW16jWatVSZM6yxd3cx78BWSgM2qYrN X-Proofpoint-GUID: QTW16jWatVSZM6yxd3cx78BWSgM2qYrN X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.573,FMLib:17.11.176.26 definitions=2023-05-25_06,2023-05-24_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Harman Kalra An issue is observed where if PF is with DPDK and VF as kernel netdev does not responds to link events. It was due to recent design change in kernel where sender checks whether previous interrupt is received before triggering current interrupt by waiting for mailbox data register to become zero. Signed-off-by: Harman Kalra --- drivers/common/cnxk/roc_dev.c | 20 ++++++++- drivers/common/cnxk/roc_mbox.c | 64 +++++++++++++++++++++-------- drivers/common/cnxk/roc_mbox.h | 15 +++++++ drivers/common/cnxk/roc_mbox_priv.h | 6 ++- 4 files changed, 84 insertions(+), 21 deletions(-) diff --git a/drivers/common/cnxk/roc_dev.c b/drivers/common/cnxk/roc_dev.c index 5e4e564ebe..e5a5cd7c10 100644 --- a/drivers/common/cnxk/roc_dev.c +++ b/drivers/common/cnxk/roc_dev.c @@ -195,7 +195,8 @@ af_pf_wait_msg(struct dev *dev, uint16_t vf, int num_msg) vf_msg->rc = msg->rc; vf_msg->pcifunc = msg->pcifunc; /* Send to VF */ - mbox_msg_send(&dev->mbox_vfpf_up, vf); + mbox_msg_send_up(&dev->mbox_vfpf_up, vf); + mbox_wait_for_zero(&dev->mbox_vfpf_up, vf); } } @@ -498,6 +499,7 @@ pf_vf_mbox_send_up_msg(struct dev *dev, void *rec_msg) /* Send to VF */ mbox_msg_send(vf_mbox, vf); + mbox_wait_for_zero(&dev->mbox_vfpf_up, vf); } } @@ -631,6 +633,7 @@ static void roc_pf_vf_mbox_irq(void *param) { struct dev *dev = param; + uint64_t mbox_data; uint64_t intr; intr = plt_read64(dev->bar2 + RVU_VF_INT); @@ -640,6 +643,13 @@ roc_pf_vf_mbox_irq(void *param) plt_write64(intr, dev->bar2 + RVU_VF_INT); plt_base_dbg("Irq 0x%" PRIx64 "(pf:%d,vf:%d)", intr, dev->pf, dev->vf); + /* Reading for UP/DOWN message, next message sending will be delayed + * by 1ms until this region is zeroed mbox_wait_for_zero() + */ + mbox_data = plt_read64(dev->bar2 + RVU_VF_VFPF_MBOX0); + if (mbox_data) + plt_write64(!mbox_data, dev->bar2 + RVU_VF_VFPF_MBOX0); + /* First process all configuration messages */ process_msgs(dev, dev->mbox); @@ -651,6 +661,7 @@ static void roc_af_pf_mbox_irq(void *param) { struct dev *dev = param; + uint64_t mbox_data; uint64_t intr; intr = plt_read64(dev->bar2 + RVU_PF_INT); @@ -660,6 +671,13 @@ roc_af_pf_mbox_irq(void *param) plt_write64(intr, dev->bar2 + RVU_PF_INT); plt_base_dbg("Irq 0x%" PRIx64 "(pf:%d,vf:%d)", intr, dev->pf, dev->vf); + /* Reading for UP/DOWN message, next message sending will be delayed + * by 1ms until this region is zeroed mbox_wait_for_zero() + */ + mbox_data = plt_read64(dev->bar2 + RVU_PF_PFAF_MBOX0); + if (mbox_data) + plt_write64(!mbox_data, dev->bar2 + RVU_PF_PFAF_MBOX0); + /* First process all configuration messages */ process_msgs(dev, dev->mbox); diff --git a/drivers/common/cnxk/roc_mbox.c b/drivers/common/cnxk/roc_mbox.c index 7dcd188ca7..5338a960d9 100644 --- a/drivers/common/cnxk/roc_mbox.c +++ b/drivers/common/cnxk/roc_mbox.c @@ -10,18 +10,6 @@ #include "roc_api.h" #include "roc_priv.h" -#define RVU_AF_AFPF_MBOX0 (0x02000) -#define RVU_AF_AFPF_MBOX1 (0x02008) - -#define RVU_PF_PFAF_MBOX0 (0xC00) -#define RVU_PF_PFAF_MBOX1 (0xC08) - -#define RVU_PF_VFX_PFVF_MBOX0 (0x0000) -#define RVU_PF_VFX_PFVF_MBOX1 (0x0008) - -#define RVU_VF_VFPF_MBOX0 (0x0000) -#define RVU_VF_VFPF_MBOX1 (0x0008) - /* RCLK, SCLK in MHz */ uint16_t dev_rclk_freq; uint16_t dev_sclk_freq; @@ -194,10 +182,31 @@ mbox_alloc_msg_rsp(struct mbox *mbox, int devid, int size, int size_rsp) /** * @internal - * Send a mailbox message + * Synchronization between UP and DOWN messages */ -void -mbox_msg_send(struct mbox *mbox, int devid) +bool +mbox_wait_for_zero(struct mbox *mbox, int devid) +{ + uint64_t data; + + data = plt_read64((volatile void *)(mbox->reg_base + + (mbox->trigger | (devid << mbox->tr_shift)))); + + /* If data is non-zero wait for ~1ms and return to caller + * whether data has changed to zero or not after the wait. + */ + if (data) + usleep(1000); + else + return true; + + data = plt_read64((volatile void *)(mbox->reg_base + + (mbox->trigger | (devid << mbox->tr_shift)))); + return data == 0; +} + +static void +mbox_msg_send_data(struct mbox *mbox, int devid, uint8_t data) { struct mbox_dev *mdev = &mbox->dev[devid]; struct mbox_hdr *tx_hdr = @@ -223,9 +232,28 @@ mbox_msg_send(struct mbox *mbox, int devid) /* The interrupt should be fired after num_msgs is written * to the shared memory */ - plt_write64(1, (volatile void *)(mbox->reg_base + - (mbox->trigger | - (devid << mbox->tr_shift)))); + plt_write64(data, (volatile void *)(mbox->reg_base + + (mbox->trigger | (devid << mbox->tr_shift)))); +} + +/** + * @internal + * Send a mailbox message + */ +void +mbox_msg_send(struct mbox *mbox, int devid) +{ + mbox_msg_send_data(mbox, devid, MBOX_DOWN_MSG); +} + +/** + * @internal + * Send an UP mailbox message + */ +void +mbox_msg_send_up(struct mbox *mbox, int devid) +{ + mbox_msg_send_data(mbox, devid, MBOX_UP_MSG); } /** diff --git a/drivers/common/cnxk/roc_mbox.h b/drivers/common/cnxk/roc_mbox.h index 3d5746b9b8..93c5451c0f 100644 --- a/drivers/common/cnxk/roc_mbox.h +++ b/drivers/common/cnxk/roc_mbox.h @@ -35,6 +35,21 @@ struct mbox_msghdr { int __io rc; /* Msg processed response code */ }; +#define RVU_AF_AFPF_MBOX0 (0x02000) +#define RVU_AF_AFPF_MBOX1 (0x02008) + +#define RVU_PF_PFAF_MBOX0 (0xC00) +#define RVU_PF_PFAF_MBOX1 (0xC08) + +#define RVU_PF_VFX_PFVF_MBOX0 (0x0000) +#define RVU_PF_VFX_PFVF_MBOX1 (0x0008) + +#define RVU_VF_VFPF_MBOX0 (0x0000) +#define RVU_VF_VFPF_MBOX1 (0x0008) + +#define MBOX_DOWN_MSG 1 +#define MBOX_UP_MSG 2 + /* Mailbox message types */ #define MBOX_MSG_MASK 0xFFFF #define MBOX_MSG_INVALID 0xFFFE diff --git a/drivers/common/cnxk/roc_mbox_priv.h b/drivers/common/cnxk/roc_mbox_priv.h index 4fafca6f72..354c8fa52a 100644 --- a/drivers/common/cnxk/roc_mbox_priv.h +++ b/drivers/common/cnxk/roc_mbox_priv.h @@ -71,10 +71,12 @@ struct mbox { const char *mbox_id2name(uint16_t id); int mbox_id2size(uint16_t id); void mbox_reset(struct mbox *mbox, int devid); -int mbox_init(struct mbox *mbox, uintptr_t hwbase, uintptr_t reg_base, - int direction, int ndevsi, uint64_t intr_offset); +int mbox_init(struct mbox *mbox, uintptr_t hwbase, uintptr_t reg_base, int direction, int ndevsi, + uint64_t intr_offset); void mbox_fini(struct mbox *mbox); void mbox_msg_send(struct mbox *mbox, int devid); +void mbox_msg_send_up(struct mbox *mbox, int devid); +bool mbox_wait_for_zero(struct mbox *mbox, int devid); int mbox_wait_for_rsp(struct mbox *mbox, int devid); int mbox_wait_for_rsp_tmo(struct mbox *mbox, int devid, uint32_t tmo); int mbox_get_rsp(struct mbox *mbox, int devid, void **msg); -- 2.25.1