From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6721446B2F; Wed, 9 Jul 2025 19:36:56 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2D1BF40E17; Wed, 9 Jul 2025 19:36:25 +0200 (CEST) Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by mails.dpdk.org (Postfix) with ESMTP id 0A74840DD5 for ; Wed, 9 Jul 2025 19:36:22 +0200 (CEST) Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-236377f00easo2571355ad.1 for ; Wed, 09 Jul 2025 10:36:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20230601.gappssmtp.com; s=20230601; t=1752082581; x=1752687381; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2mRgOL2wuyaMqmGKIgzWpqPitShZTG6mDfIm+nd/50s=; b=HJCXnCZR8L4WrvRyOoQsgTx3HEEhu8K3rTra1sdPimhZjkDO3DI2qN+ddBPhczCY2e A/ixsnHUrjZIrIR3aOtFw4Xg0t3r3pNm0CiRHNdpaq3P+aqcRUjpql1Rimd6scpt6ZJ7 ilDw5kk2mQ4+x5H3hPvSV8JP11mkIbkge8vdGoUXF9ynLjxwFajrTK+DkGxp5/V8YAtL njGvq0FX3MD1gcmeye5bODLY/WMfIzBCYa2OORSD9JofqC/8VULxoS2fvTgKbqahgPAl DrXRQut5omk8cQkylg2X9jOk/cqFLsrhTNyn/Hdqn0TDhD2ZRYUTtj4Hxkf0AS7ippvb FsFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1752082581; x=1752687381; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2mRgOL2wuyaMqmGKIgzWpqPitShZTG6mDfIm+nd/50s=; b=s93EXGoQmWC13E8Qr8K5vWl2Q/rzAt6HT9F2SYuWYhcoQmeeVJqrMajNsH7ulVzLWg 6UZZ+z0fxqTEkE9waa3yox8nMzW49+gynrkbuQwrDUZaiXBlVfdO92hf7LhZGaXG2dps AXoOzcna4JAeS1cMQq0DXZnmpI9h0vgMV/6tCCh871JxCClZVDMTc41Um9DKw7oUq/Iw UfUXs3ILvCKxFp64VIKrY/+/WtOkaXOT6vOwZTGJPz98op88TDBvmQNvjBbPXIftB4KZ 6fDl+SdDJwHT9lYnj6rJk42x2KYa+kuQdeXIrxGRuGnR8J63veBs6JM8ZIR8hk31QxY0 Kv0w== X-Gm-Message-State: AOJu0YzgbKq7QB6lj9sy9/HqSIl5JGpPklsu3FqH+ekBRPlKb0Xh3kTg vrqyWs9OW/amA6k5sYcng5EG6umCvKGm2uvtxjnbRWXX+9IfFSUaDjJmNq+346EdEb+rDk9+68k J0hNE X-Gm-Gg: ASbGncuphblp3XdfIK0lbbBaFPda83qzgSor0mYnz11cdusmMYJyjfSQnBORrXVU503 3NFylIp1RjusoXuZHF2g0WQ14Dq7+qbILnzHc6EMHum4SiGczBLuWe8arHhF8SGKwn4V2mmO9GW 3Qk64ugbI4MHprxA0kRwVR09U+KjQAClVXashxpOhshNJC6Quxg3iF5+XaVF/Kz6JXP9E0ZpUEG MLNlTdQa/291bDgsTt+MTt0USvkRE9jvojtat8zW2jixwl0JyH2iWFGFPs86TLhdkr1ddwXRGuS vmoJeYPxlquMJxXSdlZPa1kWO7a6RKKQTaJhtj+rICScDuW+gW1dPTke0I472uau/Bh7DtSBvdY vST2iloYFYcutstFs6T1CoHOKtWAhikU65fP/ X-Google-Smtp-Source: AGHT+IE3XxHeaBl4HyEfHZ+4F16ZrkgN50Iuuwkc5rh/yE6ZdXneN6pv1ARIOk51bHHXoveRzPKZIA== X-Received: by 2002:a17:902:dad1:b0:234:d7b2:2abd with SMTP id d9443c01a7336-23de245ff8bmr9791105ad.19.1752082580772; Wed, 09 Jul 2025 10:36:20 -0700 (PDT) Received: from hermes.lan (204-195-96-226.wavecable.com. [204.195.96.226]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-23c845b7467sm138525475ad.230.2025.07.09.10.36.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Jul 2025 10:36:20 -0700 (PDT) From: Stephen Hemminger To: dev@dpdk.org Cc: Stephen Hemminger , Bruce Richardson , Thomas Monjalon , Ferruh Yigit , Andrew Rybchenko , Anatoly Burakov Subject: [RFC v2 06/12] ethdev: add port mirroring feature Date: Wed, 9 Jul 2025 10:33:32 -0700 Message-ID: <20250709173611.6390-7-stephen@networkplumber.org> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20250709173611.6390-1-stephen@networkplumber.org> References: <0250411234927.114568-1-stephen@networkplumber.org> <20250709173611.6390-1-stephen@networkplumber.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This adds new feature port mirroring to the ethdev layer. Signed-off-by: Stephen Hemminger --- config/rte_config.h | 1 + lib/ethdev/ethdev_driver.h | 6 + lib/ethdev/ethdev_private.c | 58 +++++- lib/ethdev/ethdev_private.h | 3 + lib/ethdev/ethdev_trace.h | 17 ++ lib/ethdev/ethdev_trace_points.c | 6 + lib/ethdev/meson.build | 1 + lib/ethdev/rte_ethdev.c | 17 +- lib/ethdev/rte_ethdev.h | 23 ++- lib/ethdev/rte_ethdev_core.h | 6 +- lib/ethdev/rte_mirror.c | 344 +++++++++++++++++++++++++++++++ lib/ethdev/rte_mirror.h | 113 ++++++++++ 12 files changed, 577 insertions(+), 18 deletions(-) create mode 100644 lib/ethdev/rte_mirror.c create mode 100644 lib/ethdev/rte_mirror.h diff --git a/config/rte_config.h b/config/rte_config.h index 05344e2619..76eb2aa417 100644 --- a/config/rte_config.h +++ b/config/rte_config.h @@ -69,6 +69,7 @@ #define RTE_MAX_QUEUES_PER_PORT 1024 #define RTE_ETHDEV_QUEUE_STAT_CNTRS 16 /* max 256 */ #define RTE_ETHDEV_RXTX_CALLBACKS 1 +#define RTE_ETHDEV_MIRROR 1 #define RTE_MAX_MULTI_HOST_CTRLS 4 /* cryptodev defines */ diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h index 2b4d2ae9c3..586f9b7a2b 100644 --- a/lib/ethdev/ethdev_driver.h +++ b/lib/ethdev/ethdev_driver.h @@ -85,12 +85,18 @@ struct __rte_cache_aligned rte_eth_dev { * received packets before passing them to the user */ RTE_ATOMIC(struct rte_eth_rxtx_callback *) post_rx_burst_cbs[RTE_MAX_QUEUES_PER_PORT]; + + /** Receive mirrors */ + RTE_ATOMIC(struct rte_eth_mirror *) rx_mirror; /** * User-supplied functions called from tx_burst to pre-process * received packets before passing them to the driver for transmission */ RTE_ATOMIC(struct rte_eth_rxtx_callback *) pre_tx_burst_cbs[RTE_MAX_QUEUES_PER_PORT]; + /** Transmit mirrors */ + RTE_ATOMIC(struct rte_eth_mirror *) tx_mirror; + enum rte_eth_dev_state state; /**< Flag indicating the port state */ void *security_ctx; /**< Context for security ops */ }; diff --git a/lib/ethdev/ethdev_private.c b/lib/ethdev/ethdev_private.c index 184cf33f99..d7d99a09f2 100644 --- a/lib/ethdev/ethdev_private.c +++ b/lib/ethdev/ethdev_private.c @@ -280,6 +280,8 @@ eth_dev_fp_ops_setup(struct rte_eth_fp_ops *fpo, fpo->tx_descriptor_status = dev->tx_descriptor_status; fpo->recycle_tx_mbufs_reuse = dev->recycle_tx_mbufs_reuse; fpo->recycle_rx_descriptors_refill = dev->recycle_rx_descriptors_refill; + fpo->rx_mirror = (struct rte_eth_mirror * __rte_atomic *)(uintptr_t)&dev->rx_mirror; + fpo->tx_mirror = (struct rte_eth_mirror * __rte_atomic *)(uintptr_t)&dev->tx_mirror; fpo->rxq.data = dev->data->rx_queues; fpo->rxq.clbk = (void * __rte_atomic *)(uintptr_t)dev->post_rx_burst_cbs; @@ -481,17 +483,53 @@ eth_dev_tx_queue_config(struct rte_eth_dev *dev, uint16_t nb_queues) } static int -ethdev_handle_request(const struct ethdev_mp_request *req) +ethdev_handle_request(const struct ethdev_mp_request *req, size_t len) { + len -= sizeof(*req); + switch (req->operation) { case ETH_REQ_START: + if (len != 0) + return -EINVAL; + return rte_eth_dev_start(req->port_id); case ETH_REQ_STOP: + if (len != 0) + return -EINVAL; return rte_eth_dev_stop(req->port_id); + case ETH_REQ_RESET: + if (len != 0) + return -EINVAL; + return rte_eth_dev_reset(req->port_id); + + case ETH_REQ_ADD_MIRROR: + if (len != sizeof(struct rte_eth_mirror_conf)) { + RTE_ETHDEV_LOG_LINE(ERR, + "add mirror conf wrong size %zu", len); + return -EINVAL; + } + + const struct rte_eth_mirror_conf *conf + = (const struct rte_eth_mirror_conf *) req->config; + + return rte_eth_add_mirror(req->port_id, conf); + + case ETH_REQ_REMOVE_MIRROR: + if (len != sizeof(uint16_t)) { + RTE_ETHDEV_LOG_LINE(ERR, + "mirror remove wrong size %zu", len); + return -EINVAL; + } + + uint16_t target = *(const uint16_t *) req->config; + return rte_eth_remove_mirror(req->port_id, target); + default: - return -EINVAL; + RTE_ETHDEV_LOG_LINE(ERR, + "Unknown mp request operation %u", req->operation); + return -ENOTSUP; } } @@ -504,23 +542,25 @@ static_assert(sizeof(struct ethdev_mp_response) <= RTE_MP_MAX_PARAM_LEN, int ethdev_server(const struct rte_mp_msg *mp_msg, const void *peer) { - const struct ethdev_mp_request *req - = (const struct ethdev_mp_request *)mp_msg->param; - struct rte_mp_msg mp_resp = { .name = ETHDEV_MP, }; struct ethdev_mp_response *resp; + const struct ethdev_mp_request *req; resp = (struct ethdev_mp_response *)mp_resp.param; mp_resp.len_param = sizeof(*resp); - resp->res_op = req->operation; /* recv client requests */ - if (mp_msg->len_param != sizeof(*req)) + if (mp_msg->len_param < (int)sizeof(*req)) { + RTE_ETHDEV_LOG_LINE(ERR, "invalid request from secondary"); resp->err_value = -EINVAL; - else - resp->err_value = ethdev_handle_request(req); + } else { + req = (const struct ethdev_mp_request *)mp_msg->param; + resp->res_op = req->operation; + resp->err_value = ethdev_handle_request(req, mp_msg->len_param); + + } return rte_mp_reply(&mp_resp, peer); } diff --git a/lib/ethdev/ethdev_private.h b/lib/ethdev/ethdev_private.h index f58f161871..d2fdc20057 100644 --- a/lib/ethdev/ethdev_private.h +++ b/lib/ethdev/ethdev_private.h @@ -85,6 +85,9 @@ int eth_dev_tx_queue_config(struct rte_eth_dev *dev, uint16_t nb_queues); enum ethdev_mp_operation { ETH_REQ_START, ETH_REQ_STOP, + ETH_REQ_RESET, + ETH_REQ_ADD_MIRROR, + ETH_REQ_REMOVE_MIRROR, }; struct ethdev_mp_request { diff --git a/lib/ethdev/ethdev_trace.h b/lib/ethdev/ethdev_trace.h index 482befc209..e137afcbf7 100644 --- a/lib/ethdev/ethdev_trace.h +++ b/lib/ethdev/ethdev_trace.h @@ -1035,6 +1035,23 @@ RTE_TRACE_POINT( rte_trace_point_emit_int(ret); ) +RTE_TRACE_POINT( + rte_eth_trace_add_mirror, + RTE_TRACE_POINT_ARGS(uint16_t port_id, + const struct rte_eth_mirror_conf *conf, int ret), + rte_trace_point_emit_u16(port_id); + rte_trace_point_emit_u16(conf->target); + rte_trace_point_emit_int(ret); +) + +RTE_TRACE_POINT( + rte_eth_trace_remove_mirror, + RTE_TRACE_POINT_ARGS(uint16_t port_id, uint16_t target_id, int ret), + rte_trace_point_emit_u16(port_id); + rte_trace_point_emit_u16(target_id); + rte_trace_point_emit_int(ret); +) + RTE_TRACE_POINT( rte_eth_trace_rx_queue_info_get, RTE_TRACE_POINT_ARGS(uint16_t port_id, uint16_t queue_id, diff --git a/lib/ethdev/ethdev_trace_points.c b/lib/ethdev/ethdev_trace_points.c index 071c508327..fa1fd21809 100644 --- a/lib/ethdev/ethdev_trace_points.c +++ b/lib/ethdev/ethdev_trace_points.c @@ -389,6 +389,12 @@ RTE_TRACE_POINT_REGISTER(rte_eth_trace_remove_rx_callback, RTE_TRACE_POINT_REGISTER(rte_eth_trace_remove_tx_callback, lib.ethdev.remove_tx_callback) +RTE_TRACE_POINT_REGISTER(rte_eth_trace_add_mirror, + lib.ethdev.add_mirror) + +RTE_TRACE_POINT_REGISTER(rte_eth_trace_remove_mirror, + lib.ethdev.remove_mirror) + RTE_TRACE_POINT_REGISTER(rte_eth_trace_rx_queue_info_get, lib.ethdev.rx_queue_info_get) diff --git a/lib/ethdev/meson.build b/lib/ethdev/meson.build index f1d2586591..3672b6a35b 100644 --- a/lib/ethdev/meson.build +++ b/lib/ethdev/meson.build @@ -11,6 +11,7 @@ sources = files( 'rte_ethdev_cman.c', 'rte_ethdev_telemetry.c', 'rte_flow.c', + 'rte_mirror.c', 'rte_mtr.c', 'rte_tm.c', 'sff_telemetry.c', diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c index 41363af2c3..6602a771bd 100644 --- a/lib/ethdev/rte_ethdev.c +++ b/lib/ethdev/rte_ethdev.c @@ -14,6 +14,8 @@ #include #include #include +#include +#include #include #include #include @@ -2041,13 +2043,16 @@ rte_eth_dev_reset(uint16_t port_id) if (dev->dev_ops->dev_reset == NULL) return -ENOTSUP; - ret = rte_eth_dev_stop(port_id); - if (ret != 0) { - RTE_ETHDEV_LOG_LINE(ERR, - "Failed to stop device (port %u) before reset: %s - ignore", - port_id, rte_strerror(-ret)); + if (rte_eal_process_type() == RTE_PROC_PRIMARY) { + ret = rte_eth_dev_stop(port_id); + if (ret != 0) + RTE_ETHDEV_LOG_LINE(ERR, + "Failed to stop device (port %u) before reset: %s - ignore", + port_id, rte_strerror(-ret)); + ret = eth_err(port_id, dev->dev_ops->dev_reset(dev)); + } else { + ret = ethdev_request(port_id, ETH_REQ_STOP, NULL, 0); } - ret = eth_err(port_id, dev->dev_ops->dev_reset(dev)); rte_ethdev_trace_reset(port_id, ret); diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h index f9fb6ae549..4bd75e7afb 100644 --- a/lib/ethdev/rte_ethdev.h +++ b/lib/ethdev/rte_ethdev.h @@ -170,6 +170,7 @@ #include "rte_ethdev_trace_fp.h" #include "rte_dev_info.h" +#include "rte_mirror.h" #ifdef __cplusplus extern "C" { @@ -1466,7 +1467,6 @@ enum rte_eth_tunnel_type { RTE_ETH_TUNNEL_TYPE_ECPRI, RTE_ETH_TUNNEL_TYPE_MAX, }; - #ifdef __cplusplus } #endif @@ -6334,6 +6334,17 @@ rte_eth_rx_burst(uint16_t port_id, uint16_t queue_id, nb_rx = p->rx_pkt_burst(qd, rx_pkts, nb_pkts); +#ifdef RTE_ETHDEV_MIRROR + if (p->rx_mirror) { + const struct rte_eth_mirror *mirror; + + mirror = rte_atomic_load_explicit(p->rx_mirror, rte_memory_order_relaxed); + if (unlikely(mirror != NULL)) + rte_eth_mirror_burst(port_id, queue_id, RTE_ETH_MIRROR_DIRECTION_INGRESS, + rx_pkts, nb_rx, mirror); + } +#endif + #ifdef RTE_ETHDEV_RXTX_CALLBACKS { void *cb; @@ -6692,6 +6703,16 @@ rte_eth_tx_burst(uint16_t port_id, uint16_t queue_id, } #endif +#ifdef RTE_ETHDEV_MIRROR + if (p->tx_mirror) { + const struct rte_eth_mirror *mirror; + + mirror = rte_atomic_load_explicit(p->tx_mirror, rte_memory_order_relaxed); + if (unlikely(mirror != NULL)) + rte_eth_mirror_burst(port_id, queue_id, RTE_ETH_MIRROR_DIRECTION_EGRESS, + tx_pkts, nb_pkts, mirror); + } +#endif nb_pkts = p->tx_pkt_burst(qd, tx_pkts, nb_pkts); rte_ethdev_trace_tx_burst(port_id, queue_id, (void **)tx_pkts, nb_pkts); diff --git a/lib/ethdev/rte_ethdev_core.h b/lib/ethdev/rte_ethdev_core.h index e55fb42996..2ef3b659f7 100644 --- a/lib/ethdev/rte_ethdev_core.h +++ b/lib/ethdev/rte_ethdev_core.h @@ -101,7 +101,8 @@ struct __rte_cache_aligned rte_eth_fp_ops { eth_rx_descriptor_status_t rx_descriptor_status; /** Refill Rx descriptors with the recycling mbufs. */ eth_recycle_rx_descriptors_refill_t recycle_rx_descriptors_refill; - uintptr_t reserved1[2]; + uintptr_t reserved1; + RTE_ATOMIC(struct rte_eth_mirror *) *rx_mirror; /**@}*/ /**@{*/ @@ -121,7 +122,8 @@ struct __rte_cache_aligned rte_eth_fp_ops { eth_recycle_tx_mbufs_reuse_t recycle_tx_mbufs_reuse; /** Get the number of used Tx descriptors. */ eth_tx_queue_count_t tx_queue_count; - uintptr_t reserved2[1]; + RTE_ATOMIC(struct rte_eth_mirror *) *tx_mirror; + uintptr_t reserved2; /**@}*/ }; diff --git a/lib/ethdev/rte_mirror.c b/lib/ethdev/rte_mirror.c new file mode 100644 index 0000000000..97318176b2 --- /dev/null +++ b/lib/ethdev/rte_mirror.c @@ -0,0 +1,344 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2025 Stephen Hemminger + */ + +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "rte_ethdev.h" +#include "rte_mirror.h" +#include "ethdev_driver.h" +#include "ethdev_private.h" +#include "ethdev_trace.h" + +/* Upper bound of packet bursts redirected */ +#define RTE_MIRROR_BURST_SIZE 64 + +/* spinlock for setting up mirror ports */ +static rte_spinlock_t mirror_port_lock = RTE_SPINLOCK_INITIALIZER; + +/* dynamically assigned offload flag to indicate ingress vs egress */ +static uint64_t mirror_origin_flag; +static int mirror_origin_offset = -1; +static uint64_t mirror_ingress_flag; +static uint64_t mirror_egress_flag; + +static uint64_t mbuf_timestamp_dynflag; +static int mbuf_timestamp_offset = -1; + +/* register dynamic mbuf fields, done on first mirror creation */ +static int +ethdev_dyn_mirror_register(void) +{ + const struct rte_mbuf_dynfield field_desc = { + .name = RTE_MBUF_DYNFIELD_MIRROR_ORIGIN, + .size = sizeof(rte_mbuf_origin_t), + .align = sizeof(rte_mbuf_origin_t), + }; + struct rte_mbuf_dynflag flag_desc = { + .name = RTE_MBUF_DYNFLAG_MIRROR_ORIGIN, + }; + int offset; + + if (rte_mbuf_dyn_tx_timestamp_register(&mbuf_timestamp_offset, &mbuf_timestamp_dynflag) < 0) { + RTE_ETHDEV_LOG_LINE(ERR, "Failed to register timestamp flag"); + return -1; + } + + offset = rte_mbuf_dynfield_register(&field_desc); + if (offset < 0) { + RTE_ETHDEV_LOG_LINE(ERR, "Failed to register mbuf origin field"); + return -1; + } + mirror_origin_offset = offset; + + offset = rte_mbuf_dynflag_register(&flag_desc); + if (offset < 0) { + RTE_ETHDEV_LOG_LINE(ERR, "Failed to register mbuf origin flag"); + return -1; + } + mirror_origin_flag = RTE_BIT64(offset); + + strlcpy(flag_desc.name, RTE_MBUF_DYNFLAG_MIRROR_INGRESS, + sizeof(flag_desc.name)); + offset = rte_mbuf_dynflag_register(&flag_desc); + if (offset < 0) { + RTE_ETHDEV_LOG_LINE(ERR, "Failed to register mbuf ingress flag"); + return -1; + } + mirror_ingress_flag = RTE_BIT64(offset); + + strlcpy(flag_desc.name, RTE_MBUF_DYNFLAG_MIRROR_EGRESS, + sizeof(flag_desc.name)); + offset = rte_mbuf_dynflag_register(&flag_desc); + if (offset < 0) { + RTE_ETHDEV_LOG_LINE(ERR, "Failed to register mbuf egress flag"); + return -1; + } + mirror_egress_flag = RTE_BIT64(offset); + + return 0; +} + +/** + * Structure used to hold information mirror port mirrors for a + * queue on Rx and Tx. + */ +struct rte_eth_mirror { + RTE_ATOMIC(struct rte_eth_mirror *) next; + struct rte_eth_mirror_conf conf; +}; + +/* Add a new mirror entry to the list. */ +static int +ethdev_insert_mirror(struct rte_eth_mirror **top, + const struct rte_eth_mirror_conf *conf) +{ + struct rte_eth_mirror *mirror = rte_zmalloc(NULL, sizeof(*mirror), 0); + if (mirror == NULL) + return -ENOMEM; + + mirror->conf = *conf; + mirror->next = *top; + + rte_atomic_store_explicit(top, mirror, rte_memory_order_relaxed); + return 0; +} + +RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_add_mirror, 25.11) +int +rte_eth_add_mirror(uint16_t port_id, const struct rte_eth_mirror_conf *conf) +{ +#ifndef RTE_ETHDEV_MIRROR + return -ENOTSUP; +#endif + RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV); + + struct rte_eth_dev *dev = &rte_eth_devices[port_id]; + struct rte_eth_dev_info dev_info; + + if (conf == NULL) { + RTE_ETHDEV_LOG_LINE(ERR, "Missing configuration information"); + return -EINVAL; + } + + if (conf->direction == 0 || + conf->direction > (RTE_ETH_MIRROR_DIRECTION_INGRESS | RTE_ETH_MIRROR_DIRECTION_EGRESS)) { + RTE_ETHDEV_LOG_LINE(ERR, "Invalid direction %#x", conf->direction); + return -EINVAL; + } + + if (conf->snaplen < RTE_ETHER_HDR_LEN) { + RTE_ETHDEV_LOG_LINE(ERR, "invalid snap len"); + return -EINVAL; + } + + if (conf->mp == NULL) { + RTE_ETHDEV_LOG_LINE(ERR, "not a valid mempool"); + return -EINVAL; + } + + if (conf->flags & ~RTE_ETH_MIRROR_FLAG_MASK) { + RTE_ETHDEV_LOG_LINE(ERR, "unsupported flags"); + return -EINVAL; + } + + /* Checks that target exists */ + int ret = rte_eth_dev_info_get(conf->target, &dev_info); + if (ret != 0) + return ret; + + /* Loopback mirror could create packet storm */ + if (conf->target == port_id) { + RTE_ETHDEV_LOG_LINE(ERR, "Cannot mirror port to self"); + return -EINVAL; + } + + /* Because multiple directions and multiple queues will all going to the mirror port + * need the transmit path to be lockfree. + */ + if (!(dev_info.tx_offload_capa & RTE_ETH_TX_OFFLOAD_MT_LOCKFREE)) { + RTE_ETHDEV_LOG_LINE(ERR, "Mirror needs lockfree transmit"); + return -ENOTSUP; + } + + if (rte_eal_process_type() == RTE_PROC_PRIMARY) { + /* Register dynamic fields once */ + if (mirror_origin_offset < 0) { + ret = ethdev_dyn_mirror_register(); + if (ret < 0) + return ret; + } + + rte_spinlock_lock(&mirror_port_lock); + ret = 0; + if (conf->direction & RTE_ETH_MIRROR_DIRECTION_INGRESS) + ret = ethdev_insert_mirror(&dev->rx_mirror, conf); + if (ret == 0 && (conf->direction & RTE_ETH_MIRROR_DIRECTION_EGRESS)) + ret = ethdev_insert_mirror(&dev->tx_mirror, conf); + rte_spinlock_unlock(&mirror_port_lock); + } else { + /* in secondary, proxy to primary */ + ret = ethdev_request(port_id, ETH_REQ_ADD_MIRROR, conf, sizeof(*conf)); + if (ret != 0) + return ret; + } + + rte_eth_trace_add_mirror(port_id, conf, ret); + return ret; +} + +static bool +ethdev_delete_mirror(struct rte_eth_mirror **top, uint16_t target_id) +{ + struct rte_eth_mirror *mirror; + + while ((mirror = *top) != NULL) { + if (mirror->conf.target == target_id) + goto found; + top = &mirror->next; + } + /* not found in list */ + return false; + +found: + /* unlink from list */ + rte_atomic_store_explicit(top, mirror->next, rte_memory_order_relaxed); + + /* Defer freeing the mirror until after one second + * to allow for active threads that are using it. + * Assumes no PMD takes more than one second to transmit a burst. + * Alternative would be RCU, but RCU in DPDK is optional. + */ + rte_eal_alarm_set(US_PER_S, rte_free, mirror); + return true; +} + +RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_remove_mirror, 25.11) +int +rte_eth_remove_mirror(uint16_t port_id, uint16_t target_id) +{ +#ifndef RTE_ETHDEV_MIRROR + return -ENOTSUP; +#endif + int ret = 0; + + RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV); + struct rte_eth_dev *dev = &rte_eth_devices[port_id]; + + RTE_ETH_VALID_PORTID_OR_ERR_RET(target_id, -ENODEV); + + if (rte_eal_process_type() == RTE_PROC_PRIMARY) { + bool found; + + rte_spinlock_lock(&mirror_port_lock); + found = ethdev_delete_mirror(&dev->rx_mirror, target_id); + found |= ethdev_delete_mirror(&dev->tx_mirror, target_id); + rte_spinlock_unlock(&mirror_port_lock); + if (!found) + ret = -ENOENT; /* no mirror present */ + } else { + ret = ethdev_request(port_id, ETH_REQ_REMOVE_MIRROR, + &target_id, sizeof(target_id)); + } + + rte_eth_trace_remove_mirror(port_id, target_id, ret); + return ret; +} + +static inline void +eth_dev_mirror(uint16_t port_id, uint16_t queue_id, uint8_t direction, + struct rte_mbuf **pkts, uint16_t nb_pkts, + const struct rte_eth_mirror_conf *conf) +{ + struct rte_mbuf *tosend[RTE_MIRROR_BURST_SIZE]; + unsigned int count = 0; + unsigned int i; + + for (i = 0; i < nb_pkts; i++) { + struct rte_mbuf *m = pkts[i]; + struct rte_mbuf *mc; + + if (conf->flags & RTE_ETH_MIRROR_INDIRECT_FLAG) { + mc = rte_pktmbuf_alloc(conf->mp); + if (unlikely(mc == NULL)) + continue; + + /* Make both mbuf's point to the same data */ + rte_pktmbuf_attach(mc, m); + } else { + mc = rte_pktmbuf_copy(m, conf->mp, 0, conf->snaplen); + /* TODO: dropped stats? */ + if (unlikely(mc == NULL)) + continue; + } + + /* Put info about origin of the packet */ + if (conf->flags & RTE_ETH_MIRROR_ORIGIN_FLAG) { + struct rte_mbuf_origin *origin + = RTE_MBUF_DYNFIELD(mc, mirror_origin_offset, rte_mbuf_origin_t *); + origin->original_len = m->pkt_len; + origin->port_id = port_id; + origin->queue_id = queue_id; + mc->ol_flags |= mirror_origin_flag; + } + + /* Insert timestamp into packet */ + if (conf->flags & RTE_ETH_MIRROR_TIMESTAMP_FLAG) { + *RTE_MBUF_DYNFIELD(m, mbuf_timestamp_offset, rte_mbuf_timestamp_t *) + = rte_get_tsc_cycles(); + mc->ol_flags |= mbuf_timestamp_dynflag; + } + + mc->ol_flags &= ~(mirror_ingress_flag | mirror_egress_flag); + if (direction & RTE_ETH_MIRROR_DIRECTION_INGRESS) + mc->ol_flags |= mirror_ingress_flag; + else if (direction & RTE_ETH_MIRROR_DIRECTION_EGRESS) + mc->ol_flags |= mirror_egress_flag; + + tosend[count++] = mc; + } + + uint16_t nsent = rte_eth_tx_burst(conf->target, 0, tosend, count); + if (unlikely(nsent < count)) { + uint16_t drop = count - nsent; + + /* TODO: need some stats here? */ + rte_pktmbuf_free_bulk(pkts + nsent, drop); + } +} + +RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_mirror_burst, 25.11) +void +rte_eth_mirror_burst(uint16_t port_id, uint16_t queue_id, uint8_t direction, + struct rte_mbuf **pkts, uint16_t nb_pkts, + const struct rte_eth_mirror *mirror) +{ + unsigned int i; + + for (i = 0; i < nb_pkts; i += RTE_MIRROR_BURST_SIZE) { + uint16_t left = nb_pkts - i; + uint16_t burst = RTE_MIN(left, RTE_MIRROR_BURST_SIZE); + + eth_dev_mirror(port_id, queue_id, direction, + pkts + i, burst, &mirror->conf); + } +} diff --git a/lib/ethdev/rte_mirror.h b/lib/ethdev/rte_mirror.h new file mode 100644 index 0000000000..27a684b4ae --- /dev/null +++ b/lib/ethdev/rte_mirror.h @@ -0,0 +1,113 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2025 Stephen Hemminger + */ + +#ifndef RTE_MIRROR_H_ +#define RTE_MIRROR_H_ + +#include + +#include +#include + +#ifdef __cplusplus +extern "C" { +#endif + +/** + * @file + * Ethdev port mirroring + * + * This interface provides the ability to duplicate packets to another port. + */ + +/* Definitions for ethdev analyzer direction */ +#define RTE_ETH_MIRROR_DIRECTION_INGRESS 1 +#define RTE_ETH_MIRROR_DIRECTION_EGRESS 2 + +/** + * @warning + * @b EXPERIMENTAL: this structure may change without prior notice. + * + * This dynamic field is added to mbuf's when they are copied to + * the port mirror. + */ +typedef struct rte_mbuf_origin { + uint32_t original_len; /**< Packet length before copy */ + uint16_t port_id; /**< Port where packet originated */ + uint16_t queue_id; /**< Queue used for Tx or Rx */ +} rte_mbuf_origin_t; + +/** + * @warning + * @b EXPERIMENTAL: this structure may change without prior notice. + * + * Structure used to configure ethdev Switched Port Analyzer (MIRROR) + */ +struct rte_eth_mirror_conf { + struct rte_mempool *mp; /**< Memory pool for copies, If NULL then cloned. */ + uint32_t snaplen; /**< Upper limit on number of bytes to copy */ + uint32_t flags; /**< bitmask of RTE_ETH_MIRROR_XXX_FLAG's */ + uint16_t target; /**< Destination port */ + uint8_t direction; /**< bitmask of RTE_ETH_MIRROR_DIRECTION_XXX */ +}; + +#define RTE_ETH_MIRROR_TIMESTAMP_FLAG 1 /**< insert timestamp into mirrored packet */ +#define RTE_ETH_MIRROR_ORIGIN_FLAG 2 /**< insert rte_mbuf_origin into mirrored packet */ +#define RTE_ETH_MIRROR_INDIRECT_FLAG 4 /**< use rte_mbuf_attach rather than copy */ + +#define RTE_ETH_MIRROR_FLAG_MASK 7 + +/** + * @warning + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice + * + * Create a Switched Port Analyzer (MIRROR) instance. + * + * @param port_id + * The port identifier of the source Ethernet device. + * @param conf + * Settings for this MIRROR instance.. + * @return + * Negative errno value on error, 0 on success. + */ +__rte_experimental +int +rte_eth_add_mirror(uint16_t port_id, const struct rte_eth_mirror_conf *conf); + +/** + * @warning + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice + * + * Break port mirrorning. + * After this call no more packets will be sent the target port. + * + * @param port_id + * The port identifier of the source Ethernet device. + * @param target_id + * The identifier of the destination port. + * @return + * Negative errno value on error, 0 on success. + */ +__rte_experimental +int rte_eth_remove_mirror(uint16_t port_id, uint16_t target_id); + + +/** + * @warning + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice + * + * Helper routine for rte_eth_rx_burst() and rte_eth_tx_burst(). + * Do not use directly. + */ +struct rte_eth_mirror; +__rte_experimental +void rte_eth_mirror_burst(uint16_t port_id, uint16_t quque_id, uint8_t dir, + struct rte_mbuf **pkts, uint16_t nb_pkts, + const struct rte_eth_mirror *mirror); + +#ifdef __cplusplus +} +#endif + +#endif /* RTE_MIRROR_H_ */ -- 2.47.2