From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3578EA04DB; Thu, 15 Oct 2020 08:08:42 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 6CD221DC5D; Thu, 15 Oct 2020 08:08:40 +0200 (CEST) Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by dpdk.org (Postfix) with ESMTP id CE11E1DA84 for ; Thu, 15 Oct 2020 08:08:36 +0200 (CEST) IronPort-SDR: YzRxSG/uFp3pK4jDO7WFeMj38AxMurJlK6rNUny77PFg/yHdvMed543G7DHxdXYaiaMUp8vgcO 4UbU7OUEzlgw== X-IronPort-AV: E=McAfee;i="6000,8403,9774"; a="230460075" X-IronPort-AV: E=Sophos;i="5.77,377,1596524400"; d="scan'208";a="230460075" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Oct 2020 23:08:35 -0700 IronPort-SDR: lti59Q1kWHAgaZjLSwS0+nmNvwPXRvfldi1Jog5kJ0zaVcbVcCPQwVulF6735kDcWfnx5qgIdX 4De5O4b6z87g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,377,1596524400"; d="scan'208";a="356879345" Received: from orsmsx606.amr.corp.intel.com ([10.22.229.19]) by FMSMGA003.fm.intel.com with ESMTP; 14 Oct 2020 23:08:34 -0700 Received: from orsmsx601.amr.corp.intel.com (10.22.229.14) by ORSMSX606.amr.corp.intel.com (10.22.229.19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 14 Oct 2020 23:08:34 -0700 Received: from ORSEDG601.ED.cps.intel.com (10.7.248.6) by orsmsx601.amr.corp.intel.com (10.22.229.14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5 via Frontend Transport; Wed, 14 Oct 2020 23:08:34 -0700 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (104.47.55.170) by edgegateway.intel.com (134.134.137.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.1713.5; Wed, 14 Oct 2020 23:08:33 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=YPSIDq9JUu0ddg8mWYss6XQd6D09ZCkMtibwVfSRqLDpe7sSiX8CqiePqtEy2NOn80Dv+vOkrNFcpiHAfdmzrLJQ8JoHfVxrhXb2mHWERhXZr82HspnY6YS8xCYBwRP5ydRpwNWGooPnUS5D+K8oBEGPgTJ/J0/3sUwbfX+8GTkoTB3reUdA4c3G6IPwQvsxGDGF4GM/W9JlwYTxQfA42vFj30EJIiqEog6JQ1SFxB9yEfPAVq5olfwTUvSfhTrKbIap7dg0Z/AMRJCIapuJCuGhn2V5bB0eh91gS6xil8tPKseTZV4sLbb+kNJxK4uZ8ZetEz90DyxsJMOZdHonkQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xw9JksNbiSrZ5cQJVKVDhyexIG36ZcGUUv1ky9XqdE0=; b=cw9ryfqFEruOnj6GUPSEgo6KaRhWg29MlkSqrP4Tg+y2a1io6AA/UlJcoX3l563JPLN1JiLQxS9IYwWPLwkH/tsajeIu73lndRZVAhR55MhDqC5rFnKgrtMBBylWeIEwEayzc0djVrt9hXbLuiH3LOaQOto0hsQj7BMvpsfXAzJqqs4AcX5WWmfSXChcJ8c+CUwdr0BKFD0gsqdZg9saCwQ8ftEyoXlmoxXHCCAr5AHl9sI7wd0jNFlrXZlUrf2c44NxV1FeG8MxgBGMT960jtBe9OfKo5XdZyQPq+m/Iv1iZYger0M6s0RqAFmYCT3P3a8M4IDDdE31lVPngkob+Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xw9JksNbiSrZ5cQJVKVDhyexIG36ZcGUUv1ky9XqdE0=; b=e7eBN48Q0llu29UFQPSYW5aX7AlEag79qi5vTasmsZ1MyaR7gCskziRd7lRVLrywyPwYkZX1FJ5cfjhP0xGLNenv0433lI85Iee2s2cfy4AxMnFNwxQR/Km1DziD+dIMK0Lxx5hZEmp0bI+Msgv39tKORWIM+vEqSx91JYPkFEA= Received: from BYAPR11MB2901.namprd11.prod.outlook.com (2603:10b6:a03:91::23) by BYAPR11MB3607.namprd11.prod.outlook.com (2603:10b6:a03:b2::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3455.21; Thu, 15 Oct 2020 06:08:30 +0000 Received: from BYAPR11MB2901.namprd11.prod.outlook.com ([fe80::8d75:dadd:9af1:81ab]) by BYAPR11MB2901.namprd11.prod.outlook.com ([fe80::8d75:dadd:9af1:81ab%7]) with mapi id 15.20.3455.029; Thu, 15 Oct 2020 06:08:30 +0000 From: "Xu, Rosen" To: "Zhang, Tianfei" , "dev@dpdk.org" , "Huang, Wei" Thread-Topic: [PATCH v2 4/4] raw/ifpga/base: enhance driver reliability in multi-process Thread-Index: AQHWlUDCsPxo8m+OtkCZxPhD31iwBqmYSIqQ Date: Thu, 15 Oct 2020 06:08:29 +0000 Message-ID: References: <1600846213-18093-1-git-send-email-tianfei.zhang@intel.com> <1601257218-6606-1-git-send-email-tianfei.zhang@intel.com> <1601257218-6606-5-git-send-email-tianfei.zhang@intel.com> In-Reply-To: <1601257218-6606-5-git-send-email-tianfei.zhang@intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-version: 11.5.1.3 dlp-reaction: no-action x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiMGMyNDRmY2MtMTkxOS00MzNiLThmMzQtNmJmM2ViNGZiYmNkIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoiSTBDTVNnSjhTamRwSUJhY0RwOFFqdklRb1RZemEwSWVzUmM5aEVqVXRNM1ZESm1qbTBreFJkbUJZaFRSTktcL1EifQ== x-ctpclassification: CTP_NT authentication-results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [192.198.147.213] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: bbd03be9-5c7e-4045-90ab-08d870d0bd12 x-ms-traffictypediagnostic: BYAPR11MB3607: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:2089; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: REc9Lu/mBlaP5hPICkA3Z7a/ExMkq2X/cQmTyI2OLVkocT3zIj4BvnSZYVWiAqsou1Z+j+2tRmujV5NlQipItW4tskbhxdB/Jm2TXNP75UYjxx/ZTcmAdoBvStyaKZPGGhR/M9s2AdSi1xDmE0HpPfwMq24cliBVOfEMjQlHO8purQUdcY51F/YOTcEkssJBlYJeL1QM/QAeJNYz1AfyCOyISG5GeDbLEJpUM20kKEAkjOcI4+/wvWaMahFizdteFHc1lw7eU9qyCByOi5jEWp9VkqZz9Jma3A9rNOCzM4HUdUgR1lLsYuSJhF9drBTYzqNkTMBHl8ywz3CFtIR2kA== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR11MB2901.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(346002)(136003)(39860400002)(396003)(376002)(5660300002)(7696005)(186003)(26005)(110136005)(6506007)(8936002)(9686003)(6636002)(8676002)(53546011)(83380400001)(33656002)(316002)(66556008)(76116006)(66476007)(64756008)(55016002)(66946007)(52536014)(71200400001)(86362001)(30864003)(2906002)(478600001)(66446008)(579004)(559001); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata: 4Kn0PZmhDfJ7CsYg73YcaaPEtQbue4Nt8rR32tLXmicLxbCtIKo2X6f0YYK8HVx29qAr8CF5Hb40wxWpZMAXez2ymXUbb8c/ltW6ZSvVuldq02K/zFxkzaVf5/FPMCPqdVwotMYgf6rP3FotMbeTT8D+fxP2RjqAKKZVGQQIqY7HoRmZ4Mfl8SAdwVgzh5k5Sy9LixS1NUzUrihaIi5eyjsc6vlKC/XrbhY4uzAvRtqe8k6d6Ruz5aJ8MJkkPtg1nimH09qqQj0ErZr7ZBbMWwWfi7oWjtFJzse8OUIxxMBHHFrxr4C6a/jdslvpunmpoV1tOgR9I+rrR0PK2I4lPf522S26iUnmzMkytk8SS3Bdth142uqUcjDixg9tYvX7hAcldfXKejSKO5wgCrY1WWZoyFawYdNHdh5TQLv3Y9TGQhsnsE7M3p0qliiMUkKj2lPOfaMF3wiLfHGJVQnno2Axk+WX2vgySInlLBi5wrPNA3/zooG20/8EmCIAUlBRISWvBI1e0zwYZ4jer9phrilfVN/PI5bV5yTlN6fK7HB2uE+v2W/WlEmE/5Sd93JwJQ+8kKxij5USgnE2egbFvTuRZVDSm3r5AepkJNr5VysK35+UxbhdYaLuuxFS0QJCyGVyXvcPgl5RNZyqhD6dUA== Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB2901.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: bbd03be9-5c7e-4045-90ab-08d870d0bd12 X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Oct 2020 06:08:29.8405 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 0v0P8/9yw/gsRcipjzvHeC2x8V1sLOCatYhAngc9chEqH5jeRMYyRr/6tffqo018qAMYL78sJx33mNhV0FG0Yw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR11MB3607 X-OriginatorOrg: intel.com Subject: Re: [dpdk-dev] [PATCH v2 4/4] raw/ifpga/base: enhance driver reliability in multi-process X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi, > -----Original Message----- > From: Zhang, Tianfei > Sent: Monday, September 28, 2020 9:40 > To: dev@dpdk.org; Xu, Rosen ; Huang, Wei > > Cc: Zhang, Tianfei > Subject: [PATCH v2 4/4] raw/ifpga/base: enhance driver reliability in mul= ti- > process >=20 > From: Wei Huang >=20 > Current hardware protection is based on pthread mutex which work just for > situation of multi-thread in one process. In multi-process environment, > hardware state machine would be corrupted by concurrent access, that > means original pthread mutex mechanism need be enhanced. >=20 > The major modifications in this patch are list below: > 1. Create a mutex for adapter in shared memory named > "mutex.IFPGA:domain:bus:dev.func" when device is probed. > 2. Create a shared memory named "IFPGA:domain:bus:dev.func" > during opae adapter is initializing. There is a reference count in shared > memory. Shared memory will be destroyed once reference count turned to > zero. > 3. Two mutexs are created in shared memory and initialized with flag > PTHREAD_PROCESS_SHARED. One for SPI and the other for I2C. They will be > passed to SPI and I2C driver subsequently. > 4. DTB data in flash will be cached in shared memory. Then > MAX10 driver can read DTB from shared memory instead of flash. This avoid > confliction of concurrent flash access between hardware and software. >=20 > Signed-off-by: Wei Huang > Signed-off-by: Tianfei zhang > --- > v2: fix typo in commit log. 'master' is not misspelled, it's used in orig= inal code. > There will be a separate patch to clean up the language. > --- > drivers/raw/ifpga/base/ifpga_fme.c | 52 +++- > drivers/raw/ifpga/base/meson.build | 12 + > drivers/raw/ifpga/base/opae_hw_api.c | 250 ++++++++++++++++++ > drivers/raw/ifpga/base/opae_hw_api.h | 27 +- > drivers/raw/ifpga/base/opae_i2c.c | 9 +- > drivers/raw/ifpga/base/opae_i2c.h | 1 + > drivers/raw/ifpga/base/opae_intel_max10.c | 152 ++++++----- > drivers/raw/ifpga/base/opae_spi.c | 4 + > drivers/raw/ifpga/base/opae_spi.h | 5 + > drivers/raw/ifpga/base/opae_spi_transaction.c | 15 +- > 10 files changed, 456 insertions(+), 71 deletions(-) >=20 > diff --git a/drivers/raw/ifpga/base/ifpga_fme.c > b/drivers/raw/ifpga/base/ifpga_fme.c > index 9057087b5..540bb1110 100644 > --- a/drivers/raw/ifpga/base/ifpga_fme.c > +++ b/drivers/raw/ifpga/base/ifpga_fme.c > @@ -919,6 +919,25 @@ static int spi_self_checking(struct > intel_max10_device *dev) > return 0; > } >=20 > +static void init_spi_share_data(struct ifpga_fme_hw *fme, > + struct altera_spi_device *spi) > +{ > + struct ifpga_hw *hw =3D (struct ifpga_hw *)fme->parent; > + opae_share_data *sd =3D NULL; > + > + if (hw && hw->adapter && hw->adapter->shm.ptr) { > + dev_info(NULL, "transfer share data to spi\n"); > + sd =3D (opae_share_data *)hw->adapter->shm.ptr; > + spi->mutex =3D &sd->spi_mutex; > + spi->dtb_sz_ptr =3D &sd->dtb_size; > + spi->dtb =3D sd->dtb; > + } else { > + spi->mutex =3D NULL; > + spi->dtb_sz_ptr =3D NULL; > + spi->dtb =3D NULL; > + } > +} > + > static int fme_spi_init(struct ifpga_feature *feature) { > struct ifpga_fme_hw *fme =3D (struct ifpga_fme_hw *)feature->parent; > @@ -935,6 +954,7 @@ static int fme_spi_init(struct ifpga_feature *feature= ) > spi_master =3D altera_spi_alloc(feature->addr, TYPE_SPI); > if (!spi_master) > return -ENODEV; > + init_spi_share_data(fme, spi_master); >=20 > altera_spi_init(spi_master); >=20 > @@ -945,7 +965,6 @@ static int fme_spi_init(struct ifpga_feature *feature= ) > goto spi_fail; > } >=20 > - > fme->max10_dev =3D max10; >=20 > /* SPI self test */ > @@ -1084,11 +1103,15 @@ static int fme_nios_spi_init(struct ifpga_feature > *feature) > spi_master =3D altera_spi_alloc(feature->addr, TYPE_NIOS_SPI); > if (!spi_master) > return -ENODEV; > + init_spi_share_data(fme, spi_master); >=20 > /** > * 1. wait A10 NIOS initial finished and > * release the SPI master to Host > */ > + if (spi_master->mutex) > + pthread_mutex_lock(spi_master->mutex); > + > ret =3D nios_spi_wait_init_done(spi_master); > if (ret !=3D 0) { > dev_err(fme, "FME NIOS_SPI init fail\n"); @@ -1101,6 > +1124,9 @@ static int fme_nios_spi_init(struct ifpga_feature *feature) > if (nios_spi_check_error(spi_master)) > dev_info(fme, "NIOS_SPI INIT done, but found some > error\n"); >=20 > + if (spi_master->mutex) > + pthread_mutex_unlock(spi_master->mutex); > + > /* 3. init the spi master*/ > altera_spi_init(spi_master); >=20 > @@ -1112,11 +1138,12 @@ static int fme_nios_spi_init(struct ifpga_feature > *feature) > goto release_dev; > } >=20 > + fme->max10_dev =3D max10; > + > max10->bus =3D hw->pci_data->bus; >=20 > fme_get_board_interface(fme); >=20 > - fme->max10_dev =3D max10; > mgr->sensor_list =3D &max10->opae_sensor_list; >=20 > /* SPI self test */ > @@ -1178,6 +1205,25 @@ static int i2c_mac_rom_test(struct altera_i2c_dev > *dev) > return 0; > } >=20 > +static void init_i2c_mutex(struct ifpga_fme_hw *fme) { > + struct ifpga_hw *hw =3D (struct ifpga_hw *)fme->parent; > + struct altera_i2c_dev *i2c_dev; > + opae_share_data *sd =3D NULL; > + > + if (fme->i2c_master) { > + i2c_dev =3D (struct altera_i2c_dev *)fme->i2c_master; > + if (hw && hw->adapter && hw->adapter->shm.ptr) { > + dev_info(NULL, "use multi-process mutex in i2c\n"); > + sd =3D (opae_share_data *)hw->adapter->shm.ptr; > + i2c_dev->mutex =3D &sd->i2c_mutex; > + } else { > + dev_info(NULL, "use multi-thread mutex in i2c\n"); > + i2c_dev->mutex =3D &i2c_dev->lock; > + } > + } > +} > + > static int fme_i2c_init(struct ifpga_feature *feature) { > struct feature_fme_i2c *i2c; > @@ -1191,6 +1237,8 @@ static int fme_i2c_init(struct ifpga_feature *featu= re) > if (!fme->i2c_master) > return -ENODEV; >=20 > + init_i2c_mutex(fme); > + > /* MAC ROM self test */ > i2c_mac_rom_test(fme->i2c_master); >=20 > diff --git a/drivers/raw/ifpga/base/meson.build > b/drivers/raw/ifpga/base/meson.build > index b13e13e89..da2d6e33c 100644 > --- a/drivers/raw/ifpga/base/meson.build > +++ b/drivers/raw/ifpga/base/meson.build > @@ -23,6 +23,18 @@ sources =3D [ > 'opae_eth_group.c', > ] >=20 > +rtdep =3D dependency('librt', required: false) if not rtdep.found() > + rtdep =3D cc.find_library('librt', required: false) endif if not > +rtdep.found() > + build =3D false > + reason =3D 'missing dependency, "librt"' > + subdir_done() > +endif > + > +ext_deps +=3D rtdep > + > base_lib =3D static_library('ifpga_rawdev_base', sources, > dependencies: static_rte_eal, > c_args: cflags) > diff --git a/drivers/raw/ifpga/base/opae_hw_api.c > b/drivers/raw/ifpga/base/opae_hw_api.c > index c969dfed3..600afdea1 100644 > --- a/drivers/raw/ifpga/base/opae_hw_api.c > +++ b/drivers/raw/ifpga/base/opae_hw_api.c > @@ -2,6 +2,10 @@ > * Copyright(c) 2010-2018 Intel Corporation > */ >=20 > +#include > +#include > +#include > +#include > #include "opae_hw_api.h" > #include "opae_debug.h" > #include "ifpga_api.h" > @@ -305,6 +309,244 @@ static struct opae_adapter_ops *match_ops(struct > opae_adapter *adapter) > return NULL; > } >=20 > +static void opae_mutex_init(pthread_mutex_t *mutex) { > + pthread_mutexattr_t mattr; > + > + pthread_mutexattr_init(&mattr); > + pthread_mutexattr_settype(&mattr, PTHREAD_MUTEX_RECURSIVE); > + pthread_mutexattr_setpshared(&mattr, > PTHREAD_PROCESS_SHARED); > + pthread_mutexattr_setrobust(&mattr, PTHREAD_MUTEX_ROBUST); > + pthread_mutexattr_setprotocol(&mattr, PTHREAD_PRIO_INHERIT); > + pthread_mutex_init(mutex, &mattr); > + pthread_mutexattr_destroy(&mattr); > +} > + > +static int opae_shm_open(char *shm_name, u32 size, int *new_shm) { > + int shm_id; > + int ret; > + > + shm_id =3D shm_open(shm_name, O_CREAT | O_EXCL | O_RDWR, > 0666); > + if (shm_id =3D=3D -1) { > + if (errno =3D=3D EEXIST) { > + dev_info(NULL, "shared memory %s already exist\n", > + shm_name); > + shm_id =3D shm_open(shm_name, O_RDWR, 0666); > + } else { > + dev_err(NULL, "failed to create shared > memory %s\n", > + shm_name); > + return -1; > + } > + } else { > + *new_shm =3D 1; > + ret =3D ftruncate(shm_id, size); > + if (ret =3D=3D -1) { > + dev_err(NULL, > + "failed to set shared memory size > to %u\n", > + size); > + ret =3D shm_unlink(shm_name); > + if (ret =3D=3D -1) { > + dev_err(NULL, > + "failed to unlink shared > memory %s\n", > + shm_name); > + } > + return -1; > + } > + } > + > + return shm_id; > +} > + > +static pthread_mutex_t *opae_adapter_mutex_open(struct opae_adapter > +*adapter) { > + char shm_name[32]; > + void *ptr; > + int shm_id; > + int new_shm =3D 0; > + > + if (!adapter->data) > + return NULL; > + adapter->lock =3D NULL; > + > + snprintf(shm_name, sizeof(shm_name), "/mutex.IFPGA:%s", > adapter->name); > + shm_id =3D opae_shm_open(shm_name, sizeof(pthread_mutex_t), > &new_shm); > + if (shm_id =3D=3D -1) { > + dev_err(NULL, "failed to open shared memory %s\n", > shm_name); > + } else { > + dev_info(NULL, "shared memory %s id is %d\n", > + shm_name, shm_id); > + ptr =3D mmap(NULL, sizeof(pthread_mutex_t), > + PROT_READ | PROT_WRITE, MAP_SHARED, > + shm_id, 0); > + adapter->lock =3D (pthread_mutex_t *)ptr; > + if (ptr) { > + dev_info(NULL, > + "shared memory %s address is %p\n", > + shm_name, ptr); > + if (new_shm) > + opae_mutex_init(adapter->lock); > + } else { > + dev_err(NULL, "failed to map shared memory %s\n", > + shm_name); > + } > + } > + > + return adapter->lock; > +} > + > +static void opae_adapter_mutex_close(struct opae_adapter *adapter) { > + char shm_name[32]; > + int ret; > + > + if (!adapter->lock) > + return; > + > + snprintf(shm_name, sizeof(shm_name), "/mutex.IFPGA:%s", > +adapter->name); > + > + ret =3D munmap(adapter->lock, sizeof(pthread_mutex_t)); > + if (ret =3D=3D -1) > + dev_err(NULL, "failed to unmap shared memory %s\n", > shm_name); > + else > + adapter->lock =3D NULL; > +} > + > +/** > + * opae_adapter_lock - lock this adapter > + * @adapter: adapter to lock. > + * @timeout: maximum time to wait for lock done > + * -1 wait until the lock is available > + * 0 do not wait and return immediately > + * t positive time in second to wait > + * > + * Return: 0 on success, otherwise error code. > + */ > +int opae_adapter_lock(struct opae_adapter *adapter, int timeout) { > + struct timespec t; > + int ret =3D -EINVAL; > + > + if (adapter && adapter->lock) { > + if (timeout < 0) { > + ret =3D pthread_mutex_lock(adapter->lock); > + } else if (timeout =3D=3D 0) { > + ret =3D pthread_mutex_trylock(adapter->lock); > + } else { > + clock_gettime(CLOCK_REALTIME, &t); > + t.tv_sec +=3D timeout; > + ret =3D pthread_mutex_timedlock(adapter->lock, &t); > + } > + } > + return ret; > +} > + > +/** > + * opae_adapter_unlock - unlock this adapter > + * @adapter: adapter to unlock. > + * > + * Return: 0 on success, otherwise error code. > + */ > +int opae_adapter_unlock(struct opae_adapter *adapter) { > + int ret =3D -EINVAL; > + > + if (adapter && adapter->lock) > + ret =3D pthread_mutex_unlock(adapter->lock); > + > + return ret; > +} > + > +static void opae_adapter_shm_init(struct opae_adapter *adapter) { > + opae_share_data *sd; > + > + if (!adapter->shm.ptr) > + return; > + > + sd =3D (opae_share_data *)adapter->shm.ptr; > + dev_info(NULL, "initialize shared memory\n"); > + opae_mutex_init(&sd->spi_mutex); > + opae_mutex_init(&sd->i2c_mutex); > + sd->ref_cnt =3D 0; > + sd->dtb_size =3D SHM_BLK_SIZE; > +} > + > +static void *opae_adapter_shm_alloc(struct opae_adapter *adapter) { > + char shm_name[32]; > + opae_share_data *sd; > + u32 size =3D sizeof(opae_share_data); > + int shm_id; > + int new_shm =3D 0; > + > + if (!adapter->data) > + return NULL; > + > + snprintf(shm_name, sizeof(shm_name), "/IFPGA:%s", adapter- > >name); > + adapter->shm.ptr =3D NULL; > + > + opae_adapter_lock(adapter, -1); > + shm_id =3D opae_shm_open(shm_name, size, &new_shm); > + if (shm_id =3D=3D -1) { > + dev_err(NULL, "failed to open shared memory %s\n", > shm_name); > + } else { > + dev_info(NULL, "shared memory %s id is %d\n", > + shm_name, shm_id); > + adapter->shm.id =3D shm_id; > + adapter->shm.size =3D size; > + adapter->shm.ptr =3D mmap(NULL, size, PROT_READ | > PROT_WRITE, > + MAP_SHARED, > shm_id, 0); > + if (adapter->shm.ptr) { > + dev_info(NULL, > + "shared memory %s address is %p\n", > + shm_name, adapter->shm.ptr); > + if (new_shm) > + opae_adapter_shm_init(adapter); > + sd =3D (opae_share_data *)adapter->shm.ptr; > + sd->ref_cnt++; > + } else { > + dev_err(NULL, "failed to map shared memory %s\n", > + shm_name); > + } > + } > + opae_adapter_unlock(adapter); > + > + return adapter->shm.ptr; > +} > + > +static void opae_adapter_shm_free(struct opae_adapter *adapter) { > + char shm_name[32]; > + opae_share_data *sd; > + u32 ref_cnt; > + int ret; > + > + if (!adapter->shm.ptr) > + return; > + > + sd =3D (opae_share_data *)adapter->shm.ptr; > + snprintf(shm_name, sizeof(shm_name), "/IFPGA:%s", adapter- > >name); > + > + opae_adapter_lock(adapter, -1); > + ref_cnt =3D --sd->ref_cnt; > + ret =3D munmap(adapter->shm.ptr, adapter->shm.size); > + if (ret =3D=3D -1) > + dev_err(NULL, "failed to unmap shared memory %s\n", > shm_name); > + else > + adapter->shm.ptr =3D NULL; > + > + if (ref_cnt =3D=3D 0) { > + dev_info(NULL, "unlink shared memory %s\n", shm_name); > + ret =3D shm_unlink(shm_name); > + if (ret =3D=3D -1) { > + dev_err(NULL, "failed to unlink shared > memory %s\n", > + shm_name); > + } > + } > + opae_adapter_unlock(adapter); > +} > + > /** > * opae_adapter_init - init opae_adapter data structure > * @adapter: pointer of opae_adapter data structure @@ -324,6 +566,12 > @@ int opae_adapter_init(struct opae_adapter *adapter, > adapter->name =3D name; > adapter->ops =3D match_ops(adapter); >=20 > + if (!opae_adapter_mutex_open(adapter)) > + return -ENOMEM; > + > + if (!opae_adapter_shm_alloc(adapter)) > + return -ENOMEM; > + > return 0; > } >=20 > @@ -359,6 +607,8 @@ void opae_adapter_destroy(struct opae_adapter > *adapter) { > if (adapter && adapter->ops && adapter->ops->destroy) > adapter->ops->destroy(adapter); > + opae_adapter_shm_free(adapter); > + opae_adapter_mutex_close(adapter); > } >=20 > /** > diff --git a/drivers/raw/ifpga/base/opae_hw_api.h > b/drivers/raw/ifpga/base/opae_hw_api.h > index cf8ff93a6..e99ee4564 100644 > --- a/drivers/raw/ifpga/base/opae_hw_api.h > +++ b/drivers/raw/ifpga/base/opae_hw_api.h > @@ -265,12 +265,36 @@ TAILQ_HEAD(opae_accelerator_list, > opae_accelerator); #define opae_adapter_for_each_acc(adatper, acc) \ > TAILQ_FOREACH(acc, &adapter->acc_list, node) >=20 > +#define SHM_PREFIX "/IFPGA:" > +#define SHM_BLK_SIZE 0x2000 > + > +typedef struct { > + union { > + u8 byte[SHM_BLK_SIZE]; > + struct { > + pthread_mutex_t spi_mutex; > + pthread_mutex_t i2c_mutex; > + u32 ref_cnt; /* reference count of shared memory > */ > + u32 dtb_size; /* actual length of DTB data in byte */ > + }; > + }; > + u8 dtb[SHM_BLK_SIZE]; /* DTB data */ > +} opae_share_data; > + > +typedef struct { > + int id; /* shared memory id returned by shm_open */ > + u32 size; /* size of shared memory in byte */ > + void *ptr; /* start address of shared memory */ > +} opae_share_memory; > + > struct opae_adapter { > const char *name; > struct opae_manager *mgr; > struct opae_accelerator_list acc_list; > struct opae_adapter_ops *ops; > void *data; > + pthread_mutex_t *lock; /* multi-process mutex for IFPGA */ > + opae_share_memory shm; > }; >=20 > /* OPAE Adapter APIs */ > @@ -280,7 +304,8 @@ void *opae_adapter_data_alloc(enum > opae_adapter_type type); int opae_adapter_init(struct opae_adapter > *adapter, > const char *name, void *data); > #define opae_adapter_free(adapter) opae_free(adapter) > - > +int opae_adapter_lock(struct opae_adapter *adapter, int timeout); int > +opae_adapter_unlock(struct opae_adapter *adapter); > int opae_adapter_enumerate(struct opae_adapter *adapter); void > opae_adapter_destroy(struct opae_adapter *adapter); static inline struct > opae_manager * diff --git a/drivers/raw/ifpga/base/opae_i2c.c > b/drivers/raw/ifpga/base/opae_i2c.c > index 846d751f5..598eab574 100644 > --- a/drivers/raw/ifpga/base/opae_i2c.c > +++ b/drivers/raw/ifpga/base/opae_i2c.c > @@ -30,7 +30,7 @@ int i2c_read(struct altera_i2c_dev *dev, int flags, > unsigned int slave_addr, > int i =3D 0; > int ret; >=20 > - pthread_mutex_lock(&dev->lock); > + pthread_mutex_lock(dev->mutex); >=20 > if (flags & I2C_FLAG_ADDR16) > msgbuf[i++] =3D offset >> 8; > @@ -60,7 +60,7 @@ int i2c_read(struct altera_i2c_dev *dev, int flags, > unsigned int slave_addr, > ret =3D i2c_transfer(dev, msg, 2); >=20 > exit: > - pthread_mutex_unlock(&dev->lock); > + pthread_mutex_unlock(dev->mutex); > return ret; > } >=20 > @@ -72,7 +72,7 @@ int i2c_write(struct altera_i2c_dev *dev, int flags, > unsigned int slave_addr, > int ret; > int i =3D 0; >=20 > - pthread_mutex_lock(&dev->lock); > + pthread_mutex_lock(dev->mutex); >=20 > if (!dev->xfer) { > ret =3D -ENODEV; > @@ -100,7 +100,7 @@ int i2c_write(struct altera_i2c_dev *dev, int flags, > unsigned int slave_addr, >=20 > opae_free(buf); > exit: > - pthread_mutex_unlock(&dev->lock); > + pthread_mutex_unlock(dev->mutex); > return ret; > } >=20 > @@ -496,6 +496,7 @@ struct altera_i2c_dev *altera_i2c_probe(void *base) >=20 > if (pthread_mutex_init(&dev->lock, NULL)) > return NULL; > + dev->mutex =3D &dev->lock; >=20 > altera_i2c_hardware_init(dev); >=20 > diff --git a/drivers/raw/ifpga/base/opae_i2c.h > b/drivers/raw/ifpga/base/opae_i2c.h > index 266e127b7..4f6b0b28b 100644 > --- a/drivers/raw/ifpga/base/opae_i2c.h > +++ b/drivers/raw/ifpga/base/opae_i2c.h > @@ -94,6 +94,7 @@ struct altera_i2c_dev { > u8 *buf; > int (*xfer)(struct altera_i2c_dev *dev, struct i2c_msg *msg, int num); > pthread_mutex_t lock; > + pthread_mutex_t *mutex; /* multi-process mutex from adapter */ > }; >=20 > /** > diff --git a/drivers/raw/ifpga/base/opae_intel_max10.c > b/drivers/raw/ifpga/base/opae_intel_max10.c > index 8e23ca18a..1a526ea54 100644 > --- a/drivers/raw/ifpga/base/opae_intel_max10.c > +++ b/drivers/raw/ifpga/base/opae_intel_max10.c > @@ -138,84 +138,116 @@ static int enable_nor_flash(struct > intel_max10_device *dev, bool on) >=20 > static int init_max10_device_table(struct intel_max10_device *max10) { > + struct altera_spi_device *spi =3D NULL; > struct max10_compatible_id *id; > struct fdt_header hdr; > char *fdt_root =3D NULL; > - > + u32 dtb_magic =3D 0; > u32 dt_size, dt_addr, val; > - int ret; > - > - ret =3D max10_sys_read(max10, DT_AVAIL_REG, &val); > - if (ret) { > - dev_err(max10 "cannot read DT_AVAIL_REG\n"); > - return ret; > - } > + int ret =3D 0; >=20 > - if (!(val & DT_AVAIL)) { > - dev_err(max10 "DT not available\n"); > + spi =3D (struct altera_spi_device *)max10->spi_master; > + if (!spi) { > + dev_err(max10, "spi master is not set\n"); > return -EINVAL; > } > + if (spi->dtb) > + dtb_magic =3D *(u32 *)spi->dtb; > + > + if (dtb_magic !=3D 0xEDFE0DD0) { > + dev_info(max10, "read DTB from NOR flash\n"); > + ret =3D max10_sys_read(max10, DT_AVAIL_REG, &val); > + if (ret) { > + dev_err(max10 "cannot read DT_AVAIL_REG\n"); > + return ret; > + } >=20 > - ret =3D max10_sys_read(max10, DT_BASE_ADDR_REG, &dt_addr); > - if (ret) { > - dev_info(max10 "cannot get base addr of device table\n"); > - return ret; > - } > - > - ret =3D enable_nor_flash(max10, true); > - if (ret) { > - dev_err(max10 "fail to enable flash\n"); > - return ret; > - } > + if (!(val & DT_AVAIL)) { > + dev_err(max10 "DT not available\n"); > + return -EINVAL; > + } >=20 > - ret =3D altera_nor_flash_read(max10, dt_addr, &hdr, sizeof(hdr)); > - if (ret) { > - dev_err(max10 "read fdt header fail\n"); > - goto done; > - } > + ret =3D max10_sys_read(max10, DT_BASE_ADDR_REG, > &dt_addr); > + if (ret) { > + dev_info(max10 "cannot get base addr of device > table\n"); > + return ret; > + } >=20 > - ret =3D fdt_check_header(&hdr); > - if (ret) { > - dev_err(max10 "check fdt header fail\n"); > - goto done; > - } > + ret =3D enable_nor_flash(max10, true); > + if (ret) { > + dev_err(max10 "fail to enable flash\n"); > + return ret; > + } >=20 > - dt_size =3D fdt_totalsize(&hdr); > - if (dt_size > DFT_MAX_SIZE) { > - dev_err(max10 "invalid device table size\n"); > - ret =3D -EINVAL; > - goto done; > - } > + ret =3D altera_nor_flash_read(max10, dt_addr, &hdr, > sizeof(hdr)); > + if (ret) { > + dev_err(max10 "read fdt header fail\n"); > + goto disable_nor_flash; > + } >=20 > - fdt_root =3D opae_malloc(dt_size); > - if (!fdt_root) { > - ret =3D -ENOMEM; > - goto done; > - } > + ret =3D fdt_check_header(&hdr); > + if (ret) { > + dev_err(max10 "check fdt header fail\n"); > + goto disable_nor_flash; > + } >=20 > - ret =3D altera_nor_flash_read(max10, dt_addr, fdt_root, dt_size); > - if (ret) { > - dev_err(max10 "cannot read device table\n"); > - goto done; > - } > + dt_size =3D fdt_totalsize(&hdr); > + if (dt_size > DFT_MAX_SIZE) { > + dev_err(max10 "invalid device table size\n"); > + ret =3D -EINVAL; > + goto disable_nor_flash; > + } >=20 > - id =3D max10_match_compatible(fdt_root); > - if (!id) { > - dev_err(max10 "max10 compatible not found\n"); > - ret =3D -ENODEV; > - goto done; > - } > + fdt_root =3D opae_malloc(dt_size); > + if (!fdt_root) { > + ret =3D -ENOMEM; > + goto disable_nor_flash; > + } >=20 > - max10->flags |=3D MAX10_FLAGS_DEVICE_TABLE; > + ret =3D altera_nor_flash_read(max10, dt_addr, fdt_root, > dt_size); > + if (ret) { > + opae_free(fdt_root); > + fdt_root =3D NULL; > + dev_err(max10 "cannot read device table\n"); > + goto disable_nor_flash; > + } >=20 > - max10->id =3D id; > - max10->fdt_root =3D fdt_root; > + if (spi->dtb) { > + if (*spi->dtb_sz_ptr < dt_size) { > + dev_warn(max10, > + "share memory for dtb is > smaller than required %u\n", > + dt_size); > + } else { > + *spi->dtb_sz_ptr =3D dt_size; > + } > + /* store dtb data into share memory */ > + memcpy(spi->dtb, fdt_root, *spi->dtb_sz_ptr); > + } >=20 > -done: > - ret =3D enable_nor_flash(max10, false); > +disable_nor_flash: > + enable_nor_flash(max10, false); > + } else { > + if (*spi->dtb_sz_ptr > 0) { > + dev_info(max10, "read DTB from shared memory\n"); > + fdt_root =3D opae_malloc(*spi->dtb_sz_ptr); > + if (fdt_root) > + memcpy(fdt_root, spi->dtb, *spi- > >dtb_sz_ptr); > + else > + ret =3D -ENOMEM; > + } > + } >=20 > - if (ret && fdt_root) > - opae_free(fdt_root); > + if (fdt_root) { > + id =3D max10_match_compatible(fdt_root); > + if (!id) { > + dev_err(max10 "max10 compatible not found\n"); > + ret =3D -ENODEV; > + } else { > + max10->flags |=3D MAX10_FLAGS_DEVICE_TABLE; > + max10->id =3D id; > + max10->fdt_root =3D fdt_root; > + } > + } >=20 > return ret; > } > diff --git a/drivers/raw/ifpga/base/opae_spi.c > b/drivers/raw/ifpga/base/opae_spi.c > index bfdc83e6c..9efeecb79 100644 > --- a/drivers/raw/ifpga/base/opae_spi.c > +++ b/drivers/raw/ifpga/base/opae_spi.c > @@ -285,11 +285,15 @@ void altera_spi_init(struct altera_spi_device > *spi_dev) > spi_dev->num_chipselect, > spi_dev->spi_param.clock_phase); >=20 > + if (spi_dev->mutex) > + pthread_mutex_lock(spi_dev->mutex); > /* clear */ > spi_reg_write(spi_dev, ALTERA_SPI_CONTROL, 0); > spi_reg_write(spi_dev, ALTERA_SPI_STATUS, 0); > /* flush rxdata */ > spi_flush_rx(spi_dev); > + if (spi_dev->mutex) > + pthread_mutex_unlock(spi_dev->mutex); > } >=20 > void altera_spi_release(struct altera_spi_device *dev) diff --git > a/drivers/raw/ifpga/base/opae_spi.h b/drivers/raw/ifpga/base/opae_spi.h > index 73a227673..af11656e4 100644 > --- a/drivers/raw/ifpga/base/opae_spi.h > +++ b/drivers/raw/ifpga/base/opae_spi.h > @@ -77,6 +77,10 @@ struct altera_spi_device { > int (*reg_read)(struct altera_spi_device *dev, u32 reg, u32 *val); > int (*reg_write)(struct altera_spi_device *dev, u32 reg, > u32 value); > + /* below are data to be shared in multiple process */ > + pthread_mutex_t *mutex; /* to be passed to spi_transaction_dev > */ > + unsigned int *dtb_sz_ptr; /* to be used in init_max10_device_table > */ > + unsigned char *dtb; /* to be used in init_max10_device_table */ > }; >=20 > #define HEADER_LEN 8 > @@ -103,6 +107,7 @@ struct spi_transaction_dev { > int chipselect; > struct spi_tran_buffer *buffer; > pthread_mutex_t lock; > + pthread_mutex_t *mutex; /* multi-process mutex from adapter */ > }; >=20 > struct spi_tran_header { > diff --git a/drivers/raw/ifpga/base/opae_spi_transaction.c > b/drivers/raw/ifpga/base/opae_spi_transaction.c > index d13d2fbc8..006cdb4c1 100644 > --- a/drivers/raw/ifpga/base/opae_spi_transaction.c > +++ b/drivers/raw/ifpga/base/opae_spi_transaction.c > @@ -434,11 +434,11 @@ int spi_transaction_read(struct > spi_transaction_dev *dev, unsigned int addr, { > int ret; >=20 > - pthread_mutex_lock(&dev->lock); > + pthread_mutex_lock(dev->mutex); > ret =3D do_transaction(dev, addr, size, data, > (size > SPI_REG_BYTES) ? > SPI_TRAN_SEQ_READ : SPI_TRAN_NON_SEQ_READ); > - pthread_mutex_unlock(&dev->lock); > + pthread_mutex_unlock(dev->mutex); >=20 > return ret; > } > @@ -448,11 +448,11 @@ int spi_transaction_write(struct > spi_transaction_dev *dev, unsigned int addr, { > int ret; >=20 > - pthread_mutex_lock(&dev->lock); > + pthread_mutex_lock(dev->mutex); > ret =3D do_transaction(dev, addr, size, data, > (size > SPI_REG_BYTES) ? > SPI_TRAN_SEQ_WRITE : SPI_TRAN_NON_SEQ_WRITE); > - pthread_mutex_unlock(&dev->lock); > + pthread_mutex_unlock(dev->mutex); >=20 > return ret; > } > @@ -479,6 +479,13 @@ struct spi_transaction_dev > *spi_transaction_init(struct altera_spi_device *dev, > dev_err(spi_tran_dev, "fail to init mutex lock\n"); > goto err; > } > + if (dev->mutex) { > + dev_info(NULL, "use multi-process mutex in spi\n"); > + spi_tran_dev->mutex =3D dev->mutex; > + } else { > + dev_info(NULL, "use multi-thread mutex in spi\n"); > + spi_tran_dev->mutex =3D &spi_tran_dev->lock; > + } >=20 > return spi_tran_dev; >=20 > -- > 2.17.1 Acked-by: Rosen Xu