From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id A5C1B4F9B for ; Mon, 2 Jul 2018 07:44:26 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 01 Jul 2018 22:44:24 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,298,1526367600"; d="scan'208";a="53378953" Received: from dpdk51.sh.intel.com ([10.67.110.190]) by orsmga007.jf.intel.com with ESMTP; 01 Jul 2018 22:44:23 -0700 From: Qi Zhang To: thomas@monjalon.net, anatoly.burakov@intel.com Cc: konstantin.ananyev@intel.com, dev@dpdk.org, bruce.richardson@intel.com, ferruh.yigit@intel.com, benjamin.h.shelton@intel.com, narender.vangati@intel.com, Qi Zhang Date: Mon, 2 Jul 2018 13:44:36 +0800 Message-Id: <20180702054450.29269-6-qi.z.zhang@intel.com> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20180702054450.29269-1-qi.z.zhang@intel.com> References: <20180607123849.14439-1-qi.z.zhang@intel.com> <20180702054450.29269-1-qi.z.zhang@intel.com> Subject: [dpdk-dev] [PATCH v8 05/19] ethdev: support attach or detach share device from secondary X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jul 2018 05:44:27 -0000 This patch cover the multi-process hotplug case when a share device attach/detach request be issued from secondary process device attach on secondary: a) seconary send sync request to primary. b) primary receive the request and attach the new device if failed goto i). c) primary forward attach sync request to all secondary. d) secondary receive request and attach device and send reply. e) primary check the reply if all success go to j). f) primary send attach rollback sync request to all secondary. g) secondary receive the request and detach device and send reply. h) primary receive the reply and detach device as rollback action. i) send fail reply to secondary, goto k). j) send success reply to secondary. k) secondary process receive reply of step a) and return. device detach on secondary: a) secondary send sync request to primary b) primary receive the request and perform pre-detach check, if device is locked, goto j). c) primary send pre-detach sync request to all secondary. d) secondary perform pre-detach check and send reply. e) primary check the reply if any fail goto j). f) primary send detach sync request to all secondary g) secondary detach the device and send reply h) primary detach the device. i) send success reply to secondary, goto k). j) send fail reply to secondary. k) secondary process receive reply of step a) and return. Signed-off-by: Qi Zhang Reviewed-by: Anatoly Burakov --- lib/librte_ethdev/ethdev_mp.c | 179 ++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 173 insertions(+), 6 deletions(-) diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c index 1d148cd5e..8d13da591 100644 --- a/lib/librte_ethdev/ethdev_mp.c +++ b/lib/librte_ethdev/ethdev_mp.c @@ -5,8 +5,44 @@ #include #include "rte_ethdev_driver.h" + #include "ethdev_mp.h" #include "ethdev_lock.h" +#include "ethdev_private.h" + +/** + * + * secondary to primary request. + * start from function eth_dev_request_to_primary. + * + * device attach on secondary: + * a) seconary send sycn request to primary + * b) primary receive the request and attach the new device thread, + * if failed goto i). + * c) primary forward attach request to all secondary as sync request + * d) secondary receive request and attach device and send reply. + * e) primary check the reply if all success go to j). + * f) primary send attach rollback sync request to all secondary. + * g) secondary receive the request and detach device and send reply. + * h) primary receive the reply and detach device as rollback action. + * i) send fail sync reply to secondary, goto k). + * j) send success sync reply to secondary. + * k) secondary process receive reply of step a) and return. + * + * device detach on secondary: + * a) secondary send detach sync request to primary + * b) primary receive the request and perform pre-detach check, if device + * is locked, goto j). + * c) primary send pre-detach sync request to all secondary. + * d) secondary perform pre-detach check and send reply. + * e) primary check the reply if any fail goto j). + * f) primary send detach sync request to all secondary + * g) secondary detach the device and send reply + * h) primary detach the device. + * i) send success sync reply to secondary, goto k). + * j) send fail sync reply to secondary. + * k) secondary process receive reply of step a) and return. + */ #define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */ @@ -84,11 +120,122 @@ static int attach_on_secondary(const char *devargs, uint16_t port_id) } static int -handle_secondary_request(const struct rte_mp_msg *msg, const void *peer) +send_response_to_secondary(const struct eth_dev_mp_req *req, + int result, + const void *peer) +{ + struct rte_mp_msg mp_resp; + struct eth_dev_mp_req *resp = + (struct eth_dev_mp_req *)mp_resp.param; + int ret; + + memset(&mp_resp, 0, sizeof(mp_resp)); + mp_resp.len_param = sizeof(*resp); + strcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST); + memcpy(resp, req, sizeof(*req)); + resp->result = result; + + ret = rte_mp_reply(&mp_resp, peer); + if (ret) + ethdev_log(ERR, "failed to send response to secondary\n"); + + return ret; +} + +int eth_dev_request_to_secondary(struct eth_dev_mp_req *req); + +static void +__handle_secondary_request(void *param) +{ + struct mp_reply_bundle *bundle = param; + const struct rte_mp_msg *msg = &bundle->msg; + const struct eth_dev_mp_req *req = + (const struct eth_dev_mp_req *)msg->param; + struct eth_dev_mp_req tmp_req; + uint16_t port_id; + int ret = 0; + + tmp_req = *req; + + if (req->t == REQ_TYPE_ATTACH) { + ret = do_eth_dev_attach(req->devargs, &port_id); + if (ret) + goto finish; + + tmp_req.port_id = port_id; + ret = eth_dev_request_to_secondary(&tmp_req); + } else if (req->t == REQ_TYPE_DETACH) { + if (!rte_eth_dev_is_valid_port(req->port_id)) { + ret = -EINVAL; + goto finish; + } + + ret = process_lock_callbacks(req->port_id); + if (ret) + goto finish; + + tmp_req.t = REQ_TYPE_PRE_DETACH; + ret = eth_dev_request_to_secondary(&tmp_req); + if (ret) + goto finish; + + if (!tmp_req.result) { + tmp_req.t = REQ_TYPE_DETACH; + ret = eth_dev_request_to_secondary(&tmp_req); + if (ret) + goto finish; + + ret = do_eth_dev_detach(req->port_id); + } else { + ret = tmp_req.result; + } + } else { + ethdev_log(ERR, "unsupported secondary to primary request\n"); + ret = -ENOTSUP; + } + +finish: + ret = send_response_to_secondary(&tmp_req, ret, bundle->peer); + if (ret) + ethdev_log(ERR, "failed to send response to secondary\n"); + + free(bundle->peer); + free(bundle); +} + +static int +handle_secondary_request(const struct rte_mp_msg *msg, + const void *peer) { - RTE_SET_USED(msg); - RTE_SET_USED(peer); - return -ENOTSUP; + struct mp_reply_bundle *bundle; + const struct eth_dev_mp_req *req = + (const struct eth_dev_mp_req *)msg->param; + int ret = 0; + + bundle = malloc(sizeof(*bundle)); + if (bundle == NULL) { + ethdev_log(ERR, "not enough memory\n"); + return send_response_to_secondary(req, -ENOMEM, peer); + } + + bundle->msg = *msg; + /** + * We need to send reply on interrupt thread, but peer can't be + * parsed directly, so this is a temporal hack, need to be fixed + * when it is ready. + */ + bundle->peer = strdup(peer); + + /** + * We are at IPC callback thread, sync IPC is not allowed due to + * dead lock, so we delegate the task to interrupt thread. + */ + ret = rte_eal_alarm_set(1, __handle_secondary_request, bundle); + if (ret) { + ethdev_log(ERR, "failed to add mp task\n"); + return send_response_to_secondary(req, ret, peer); + } + return 0; } static void __handle_primary_request(void *param) @@ -183,8 +330,28 @@ handle_primary_request(const struct rte_mp_msg *msg, const void *peer) int eth_dev_request_to_primary(struct eth_dev_mp_req *req) { - RTE_SET_USED(req); - return -ENOTSUP; + struct rte_mp_msg mp_req; + struct rte_mp_reply mp_reply; + struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0}; + struct eth_dev_mp_req *resp; + int ret; + + memset(&mp_req, 0, sizeof(mp_req)); + memcpy(mp_req.param, req, sizeof(*req)); + mp_req.len_param = sizeof(*req); + strlcpy(mp_req.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name)); + + ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts); + if (ret) { + ethdev_log(ERR, "cannot send request to primary"); + return ret; + } + + resp = (struct eth_dev_mp_req *)mp_reply.msgs[0].param; + req->result = resp->result; + req->port_id = resp->port_id; + + return ret; } /** -- 2.13.6