From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0EFB6A04E1; Tue, 22 Sep 2020 10:55:21 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 594BC1DB56; Tue, 22 Sep 2020 10:51:03 +0200 (CEST) Received: from dispatch1-us1.ppe-hosted.com (dispatch1-us1.ppe-hosted.com [67.231.154.164]) by dpdk.org (Postfix) with ESMTP id 81AFF1D72A for ; Tue, 22 Sep 2020 10:50:20 +0200 (CEST) Received: from mx1-us1.ppe-hosted.com (unknown [10.110.50.144]) by dispatch1-us1.ppe-hosted.com (PPE Hosted ESMTP Server) with ESMTP id 27C9F20050 for ; Tue, 22 Sep 2020 08:50:20 +0000 (UTC) Received: from us4-mdac16-40.at1.mdlocal (unknown [10.110.51.55]) by mx1-us1.ppe-hosted.com (PPE Hosted ESMTP Server) with ESMTP id 2764F800A4 for ; Tue, 22 Sep 2020 08:50:20 +0000 (UTC) X-Virus-Scanned: Proofpoint Essentials engine Received: from mx1-us1.ppe-hosted.com (unknown [10.110.50.12]) by mx1-us1.ppe-hosted.com (PPE Hosted ESMTP Server) with ESMTPS id A935F4004D for ; Tue, 22 Sep 2020 08:50:19 +0000 (UTC) Received: from webmail.solarflare.com (uk.solarflare.com [193.34.186.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mx1-us1.ppe-hosted.com (PPE Hosted ESMTP Server) with ESMTPS id 71DBB4005B for ; Tue, 22 Sep 2020 08:50:19 +0000 (UTC) Received: from ukex01.SolarFlarecom.com (10.17.10.4) by ukex01.SolarFlarecom.com (10.17.10.4) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 22 Sep 2020 09:50:10 +0100 Received: from opal.uk.solarflarecom.com (10.17.10.1) by ukex01.SolarFlarecom.com (10.17.10.4) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Tue, 22 Sep 2020 09:50:10 +0100 Received: from ukv-loginhost.uk.solarflarecom.com (ukv-loginhost.uk.solarflarecom.com [10.17.10.39]) by opal.uk.solarflarecom.com (8.13.8/8.13.8) with ESMTP id 08M8oAo8004679 for ; Tue, 22 Sep 2020 09:50:10 +0100 Received: from ukv-loginhost.uk.solarflarecom.com (localhost [127.0.0.1]) by ukv-loginhost.uk.solarflarecom.com (Postfix) with ESMTP id 39A131613AB for ; Tue, 22 Sep 2020 09:50:10 +0100 (BST) From: Andrew Rybchenko To: Date: Tue, 22 Sep 2020 09:49:17 +0100 Message-ID: <1600764594-14752-24-git-send-email-arybchenko@solarflare.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1600764594-14752-1-git-send-email-arybchenko@solarflare.com> References: <1600764594-14752-1-git-send-email-arybchenko@solarflare.com> MIME-Version: 1.0 Content-Type: text/plain X-TM-AS-Product-Ver: SMEX-12.5.0.1300-8.6.1012-25674.003 X-TM-AS-Result: No-2.782700-8.000000-10 X-TMASE-MatchedRID: 2nX43281tfLxS+DKaBymNNm6V9lyTcKhJGh48zigYxUm3tlnC/E6VOZ5 Gn23AeDZuA9fFHhyLzywgcHDNo5AtPI1YbpS1+avPwKTD1v8YV5MkOX0UoduuVVkJxysad/IJGZ 0d4KSzVJEj+W35/xeanyAzXnKF8SjhMgg0NxeFawqsMfMfrOZRUcA1Ouvduu8ddaBGkyiPh23Wj IVml7BRzQ41nj8u/2a+saxFRK561NJxzdMxVoC9g97mDMXdNW3ce/io1zKJlVrpD6YHU4QOuw1g mAhG2GAM4hLs4cdh7GQ74bS+K8hS7MywGOaB4QQUIJW4+Og/QJKgIbix5+XxB1rVWTdGrE4SPxT RJaTawxCgxT/GGibYSO0knbdm9tXMGg+wgnY/ellpwNsTvdlKf79kuWeDe6KclEZ0EDaOYjeaXa qx8P4MkzMGb6zS1tZjTE+m11oKuAeBRC8fzEM9CyKzJY7d2nbpxoLgv7S3sAHWPn2mj7oRIU5i7 0LbWcciYudapQzIa+/0KHo86FaOv5VZrGTCJG/bWsCUkrA4EmtggmrIZxcdc288EjhWgdoo0YGr aGLZ0BXG1puYu9iTSwPha7vXEnJs1dXYh6zp8fiHyvyXeXh5hfNah8m7SiNRjNrjV0arFKRvCTB x6I65SBj027VCXoAq9J6dPzwLxKvvxILmKK/HBRFJJyf5BJe3QfwsVk0UbsIoUKaF27lxQCvAdu 6XkiNNecfeclMFjvtmioV/te8SjBa687SM1peRGyCFMKXit9vxM0FGYND1rJMkoidxllEAgCYm7 lujVi5AnIl9ahuLADlJLZdGMoUz5pk4iF6iVS+9SjSFk+GXMz/jdQvUwlHnqg/VrSZEiM= X-TM-AS-User-Approved-Sender: Yes X-TM-AS-User-Blocked-Sender: No X-TMASE-Result: 10--2.782700-8.000000 X-TMASE-Version: SMEX-12.5.0.1300-8.6.1012-25674.003 X-MDID: 1600764620-ZmmGFg6PY5Lb Subject: [dpdk-dev] [PATCH 23/60] common/sfc_efx/base: add event queue module for Riverhead X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Events are significantly reworked on Riverhead, so it is better to implement own set of callbacks to simplify future development and avoid inheritance of legacy code. Signed-off-by: Andrew Rybchenko Reviewed-by: Andy Moreton --- drivers/common/sfc_efx/base/efx_ev.c | 22 ++ drivers/common/sfc_efx/base/efx_impl.h | 4 +- drivers/common/sfc_efx/base/efx_mcdi.c | 19 +- drivers/common/sfc_efx/base/meson.build | 1 + drivers/common/sfc_efx/base/rhead_ev.c | 265 +++++++++++++++++++++++ drivers/common/sfc_efx/base/rhead_impl.h | 73 +++++++ 6 files changed, 379 insertions(+), 5 deletions(-) create mode 100644 drivers/common/sfc_efx/base/rhead_ev.c diff --git a/drivers/common/sfc_efx/base/efx_ev.c b/drivers/common/sfc_efx/base/efx_ev.c index 4d11c531ce..edc1b182c9 100644 --- a/drivers/common/sfc_efx/base/efx_ev.c +++ b/drivers/common/sfc_efx/base/efx_ev.c @@ -109,6 +109,22 @@ static const efx_ev_ops_t __efx_ev_ef10_ops = { }; #endif /* EFX_OPTS_EF10() */ +#if EFSYS_OPT_RIVERHEAD +static const efx_ev_ops_t __efx_ev_rhead_ops = { + rhead_ev_init, /* eevo_init */ + rhead_ev_fini, /* eevo_fini */ + rhead_ev_qcreate, /* eevo_qcreate */ + rhead_ev_qdestroy, /* eevo_qdestroy */ + rhead_ev_qprime, /* eevo_qprime */ + rhead_ev_qpost, /* eevo_qpost */ + rhead_ev_qpoll, /* eevo_qpoll */ + rhead_ev_qmoderate, /* eevo_qmoderate */ +#if EFSYS_OPT_QSTATS + rhead_ev_qstats_update, /* eevo_qstats_update */ +#endif +}; +#endif /* EFSYS_OPT_RIVERHEAD */ + __checkReturn efx_rc_t efx_ev_init( @@ -150,6 +166,12 @@ efx_ev_init( break; #endif /* EFSYS_OPT_MEDFORD2 */ +#if EFSYS_OPT_RIVERHEAD + case EFX_FAMILY_RIVERHEAD: + eevop = &__efx_ev_rhead_ops; + break; +#endif /* EFSYS_OPT_RIVERHEAD */ + default: EFSYS_ASSERT(0); rc = ENOTSUP; diff --git a/drivers/common/sfc_efx/base/efx_impl.h b/drivers/common/sfc_efx/base/efx_impl.h index 47e4dcb01f..f6b0850a65 100644 --- a/drivers/common/sfc_efx/base/efx_impl.h +++ b/drivers/common/sfc_efx/base/efx_impl.h @@ -1408,7 +1408,7 @@ efx_mcdi_get_workarounds( __out_opt uint32_t *implementedp, __out_opt uint32_t *enabledp); -#if EFX_OPTS_EF10() +#if EFSYS_OPT_RIVERHEAD || EFX_OPTS_EF10() LIBEFX_INTERNAL extern __checkReturn efx_rc_t @@ -1428,7 +1428,7 @@ efx_mcdi_fini_evq( __in efx_nic_t *enp, __in uint32_t instance); -#endif /* EFX_OPTS_EF10() */ +#endif /* EFSYS_OPT_RIVERHEAD || EFX_OPTS_EF10() */ #endif /* EFSYS_OPT_MCDI */ diff --git a/drivers/common/sfc_efx/base/efx_mcdi.c b/drivers/common/sfc_efx/base/efx_mcdi.c index 69d2327839..841dacc1c8 100644 --- a/drivers/common/sfc_efx/base/efx_mcdi.c +++ b/drivers/common/sfc_efx/base/efx_mcdi.c @@ -2443,7 +2443,20 @@ efx_mcdi_phy_module_get_info( return (rc); } +#if EFSYS_OPT_RIVERHEAD || EFX_OPTS_EF10() + +#define INIT_EVQ_MAXNBUFS MC_CMD_INIT_EVQ_V2_IN_DMA_ADDR_MAXNUM + #if EFX_OPTS_EF10() +# if (INIT_EVQ_MAXNBUFS < EF10_EVQ_MAXNBUFS) +# error "INIT_EVQ_MAXNBUFS too small" +# endif +#endif /* EFX_OPTS_EF10 */ +#if EFSYS_OPT_RIVERHEAD +# if (INIT_EVQ_MAXNBUFS < RHEAD_EVQ_MAXNBUFS) +# error "INIT_EVQ_MAXNBUFS too small" +# endif +#endif /* EFSYS_OPT_RIVERHEAD */ __checkReturn efx_rc_t efx_mcdi_init_evq( @@ -2459,7 +2472,7 @@ efx_mcdi_init_evq( const efx_nic_cfg_t *encp = efx_nic_cfg_get(enp); efx_mcdi_req_t req; EFX_MCDI_DECLARE_BUF(payload, - MC_CMD_INIT_EVQ_V2_IN_LEN(EF10_EVQ_MAXNBUFS), + MC_CMD_INIT_EVQ_V2_IN_LEN(INIT_EVQ_MAXNBUFS), MC_CMD_INIT_EVQ_V2_OUT_LEN); boolean_t interrupting; int ev_cut_through; @@ -2472,7 +2485,7 @@ efx_mcdi_init_evq( efx_rc_t rc; npages = efx_evq_nbufs(enp, nevs); - if (npages > EF10_EVQ_MAXNBUFS) { + if (npages > INIT_EVQ_MAXNBUFS) { rc = EINVAL; goto fail1; } @@ -2667,6 +2680,6 @@ efx_mcdi_fini_evq( return (rc); } -#endif /* EFX_OPTS_EF10() */ +#endif /* EFSYS_OPT_RIVERHEAD || EFX_OPTS_EF10() */ #endif /* EFSYS_OPT_MCDI */ diff --git a/drivers/common/sfc_efx/base/meson.build b/drivers/common/sfc_efx/base/meson.build index ea2517ba26..8e50f82154 100644 --- a/drivers/common/sfc_efx/base/meson.build +++ b/drivers/common/sfc_efx/base/meson.build @@ -52,6 +52,7 @@ sources = [ 'hunt_nic.c', 'medford_nic.c', 'medford2_nic.c', + 'rhead_ev.c', 'rhead_intr.c', 'rhead_nic.c', ] diff --git a/drivers/common/sfc_efx/base/rhead_ev.c b/drivers/common/sfc_efx/base/rhead_ev.c new file mode 100644 index 0000000000..fa74e408ae --- /dev/null +++ b/drivers/common/sfc_efx/base/rhead_ev.c @@ -0,0 +1,265 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * + * Copyright(c) 2019-2020 Xilinx, Inc. + * Copyright(c) 2018-2019 Solarflare Communications Inc. + */ + +#include "efx.h" +#include "efx_impl.h" + +#if EFSYS_OPT_RIVERHEAD + +/* + * Non-interrupting event queue requires interrrupting event queue to + * refer to for wake-up events even if wake ups are never used. + * It could be even non-allocated event queue. + */ +#define EFX_RHEAD_ALWAYS_INTERRUPTING_EVQ_INDEX (0) + + + __checkReturn efx_rc_t +rhead_ev_init( + __in efx_nic_t *enp) +{ + _NOTE(ARGUNUSED(enp)) + + return (0); +} + + void +rhead_ev_fini( + __in efx_nic_t *enp) +{ + _NOTE(ARGUNUSED(enp)) +} + + __checkReturn efx_rc_t +rhead_ev_qcreate( + __in efx_nic_t *enp, + __in unsigned int index, + __in efsys_mem_t *esmp, + __in size_t ndescs, + __in uint32_t id, + __in uint32_t us, + __in uint32_t flags, + __in efx_evq_t *eep) +{ + uint32_t irq; + efx_rc_t rc; + + _NOTE(ARGUNUSED(id)) /* buftbl id managed by MC */ + + /* Set up the handler table */ + eep->ee_rx = NULL; /* FIXME */ + eep->ee_tx = NULL; /* FIXME */ + eep->ee_driver = NULL; /* FIXME */ + eep->ee_drv_gen = NULL; /* FIXME */ + eep->ee_mcdi = NULL; /* FIXME */ + + /* Set up the event queue */ + /* INIT_EVQ expects function-relative vector number */ + if ((flags & EFX_EVQ_FLAGS_NOTIFY_MASK) == + EFX_EVQ_FLAGS_NOTIFY_INTERRUPT) { + irq = index; + } else if (index == EFX_RHEAD_ALWAYS_INTERRUPTING_EVQ_INDEX) { + irq = index; + flags = (flags & ~EFX_EVQ_FLAGS_NOTIFY_MASK) | + EFX_EVQ_FLAGS_NOTIFY_INTERRUPT; + } else { + irq = EFX_RHEAD_ALWAYS_INTERRUPTING_EVQ_INDEX; + } + + /* + * Interrupts may be raised for events immediately after the queue is + * created. See bug58606. + */ + rc = efx_mcdi_init_evq(enp, index, esmp, ndescs, irq, us, flags, + B_FALSE); + if (rc != 0) + goto fail1; + + return (0); + +fail1: + EFSYS_PROBE1(fail1, efx_rc_t, rc); + + return (rc); +} + + void +rhead_ev_qdestroy( + __in efx_evq_t *eep) +{ + efx_nic_t *enp = eep->ee_enp; + + EFSYS_ASSERT(enp->en_family == EFX_FAMILY_RIVERHEAD); + + (void) efx_mcdi_fini_evq(enp, eep->ee_index); +} + + __checkReturn efx_rc_t +rhead_ev_qprime( + __in efx_evq_t *eep, + __in unsigned int count) +{ + efx_nic_t *enp = eep->ee_enp; + uint32_t rptr; + efx_dword_t dword; + + rptr = count & eep->ee_mask; + + EFX_POPULATE_DWORD_2(dword, ERF_GZ_EVQ_ID, eep->ee_index, + ERF_GZ_IDX, rptr); + /* EVQ_INT_PRIME lives function control window only on Riverhead */ + EFX_BAR_WRITED(enp, ER_GZ_EVQ_INT_PRIME, &dword, B_FALSE); + + return (0); +} + + void +rhead_ev_qpost( + __in efx_evq_t *eep, + __in uint16_t data) +{ + _NOTE(ARGUNUSED(eep, data)) + + /* Not implemented yet */ + EFSYS_ASSERT(B_FALSE); +} + +/* + * Poll event queue in batches. Size of the batch is equal to cache line + * size divided by event size. + * + * Event queue is written by NIC and read by CPU. If CPU starts reading + * of events on the cache line, read all remaining events in a tight + * loop while event is present. + */ +#define EF100_EV_BATCH 8 + +/* + * Check if event is present. + * + * Riverhead EvQs use a phase bit to indicate the presence of valid events, + * by flipping the phase bit on each wrap of the write index. + */ +#define EF100_EV_PRESENT(_qword, _phase_bit) \ + (EFX_QWORD_FIELD((_qword), ESF_GZ_EV_EVQ_PHASE) == _phase_bit) + + void +rhead_ev_qpoll( + __in efx_evq_t *eep, + __inout unsigned int *countp, + __in const efx_ev_callbacks_t *eecp, + __in_opt void *arg) +{ + efx_qword_t ev[EF100_EV_BATCH]; + unsigned int batch; + unsigned int phase_bit; + unsigned int total; + unsigned int count; + unsigned int index; + size_t offset; + + EFSYS_ASSERT3U(eep->ee_magic, ==, EFX_EVQ_MAGIC); + EFSYS_ASSERT(countp != NULL); + EFSYS_ASSERT(eecp != NULL); + + count = *countp; + do { + /* Read up until the end of the batch period */ + batch = EF100_EV_BATCH - (count & (EF100_EV_BATCH - 1)); + phase_bit = (count & (eep->ee_mask + 1)) != 0; + offset = (count & eep->ee_mask) * sizeof (efx_qword_t); + for (total = 0; total < batch; ++total) { + EFSYS_MEM_READQ(eep->ee_esmp, offset, &(ev[total])); + + if (!EF100_EV_PRESENT(ev[total], phase_bit)) + break; + + EFSYS_PROBE3(event, unsigned int, eep->ee_index, + uint32_t, EFX_QWORD_FIELD(ev[total], EFX_DWORD_1), + uint32_t, EFX_QWORD_FIELD(ev[total], EFX_DWORD_0)); + + offset += sizeof (efx_qword_t); + } + + /* Process the batch of events */ + for (index = 0; index < total; ++index) { + boolean_t should_abort; + uint32_t code; + + EFX_EV_QSTAT_INCR(eep, EV_ALL); + + code = EFX_QWORD_FIELD(ev[index], ESF_GZ_E_TYPE); + switch (code) { + default: + EFSYS_PROBE3(bad_event, + unsigned int, eep->ee_index, + uint32_t, + EFX_QWORD_FIELD(ev[index], EFX_DWORD_1), + uint32_t, + EFX_QWORD_FIELD(ev[index], EFX_DWORD_0)); + + EFSYS_ASSERT(eecp->eec_exception != NULL); + (void) eecp->eec_exception(arg, + EFX_EXCEPTION_EV_ERROR, code); + should_abort = B_TRUE; + } + if (should_abort) { + /* Ignore subsequent events */ + total = index + 1; + + /* + * Poison batch to ensure the outer + * loop is broken out of. + */ + EFSYS_ASSERT(batch <= EF100_EV_BATCH); + batch += (EF100_EV_BATCH << 1); + EFSYS_ASSERT(total != batch); + break; + } + } + + /* + * There is no necessity to clear processed events since + * phase bit which is flipping on each write index wrap + * is used for event presence indication. + */ + + count += total; + + } while (total == batch); + + *countp = count; +} + + __checkReturn efx_rc_t +rhead_ev_qmoderate( + __in efx_evq_t *eep, + __in unsigned int us) +{ + _NOTE(ARGUNUSED(eep, us)) + + return (ENOTSUP); +} + + +#if EFSYS_OPT_QSTATS + void +rhead_ev_qstats_update( + __in efx_evq_t *eep, + __inout_ecount(EV_NQSTATS) efsys_stat_t *stat) +{ + unsigned int id; + + for (id = 0; id < EV_NQSTATS; id++) { + efsys_stat_t *essp = &stat[id]; + + EFSYS_STAT_INCR(essp, eep->ee_stat[id]); + eep->ee_stat[id] = 0; + } +} +#endif /* EFSYS_OPT_QSTATS */ + +#endif /* EFSYS_OPT_RIVERHEAD */ diff --git a/drivers/common/sfc_efx/base/rhead_impl.h b/drivers/common/sfc_efx/base/rhead_impl.h index b95302a13f..dc8a40cd2a 100644 --- a/drivers/common/sfc_efx/base/rhead_impl.h +++ b/drivers/common/sfc_efx/base/rhead_impl.h @@ -12,6 +12,13 @@ extern "C" { #endif +/* + * Riverhead requires physically contiguos event rings (so, just one + * DMA address is sufficient to represent it), but MCDI interface is still + * in terms of 4k size 4k-aligned DMA buffers. + */ +#define RHEAD_EVQ_MAXNBUFS 32 + #define RHEAD_EVQ_MAXNEVS 16384 #define RHEAD_EVQ_MINNEVS 256 @@ -98,6 +105,72 @@ rhead_nic_unprobe( __in efx_nic_t *enp); +/* EV */ + +LIBEFX_INTERNAL +extern __checkReturn efx_rc_t +rhead_ev_init( + __in efx_nic_t *enp); + +LIBEFX_INTERNAL +extern void +rhead_ev_fini( + __in efx_nic_t *enp); + +LIBEFX_INTERNAL +extern __checkReturn efx_rc_t +rhead_ev_qcreate( + __in efx_nic_t *enp, + __in unsigned int index, + __in efsys_mem_t *esmp, + __in size_t ndescs, + __in uint32_t id, + __in uint32_t us, + __in uint32_t flags, + __in efx_evq_t *eep); + +LIBEFX_INTERNAL +extern void +rhead_ev_qdestroy( + __in efx_evq_t *eep); + +LIBEFX_INTERNAL +extern __checkReturn efx_rc_t +rhead_ev_qprime( + __in efx_evq_t *eep, + __in unsigned int count); + +LIBEFX_INTERNAL +extern void +rhead_ev_qpost( + __in efx_evq_t *eep, + __in uint16_t data); + +LIBEFX_INTERNAL +extern void +rhead_ev_qpoll( + __in efx_evq_t *eep, + __inout unsigned int *countp, + __in const efx_ev_callbacks_t *eecp, + __in_opt void *arg); + +LIBEFX_INTERNAL +extern __checkReturn efx_rc_t +rhead_ev_qmoderate( + __in efx_evq_t *eep, + __in unsigned int us); + +#if EFSYS_OPT_QSTATS + +LIBEFX_INTERNAL +extern void +rhead_ev_qstats_update( + __in efx_evq_t *eep, + __inout_ecount(EV_NQSTATS) efsys_stat_t *stat); + +#endif /* EFSYS_OPT_QSTATS */ + + /* INTR */ LIBEFX_INTERNAL -- 2.17.1