From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DE863A0C43; Fri, 24 Sep 2021 07:56:38 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 62B854122D; Fri, 24 Sep 2021 07:56:38 +0200 (CEST) Received: from mail1.bemta25.messagelabs.com (mail1.bemta25.messagelabs.com [195.245.230.66]) by mails.dpdk.org (Postfix) with ESMTP id 5E43340687 for ; Fri, 24 Sep 2021 07:56:36 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rbbn.com; s=rbbnselector03122020; t=1632462996; i=@rbbn.com; bh=cJLf9KeYQbxMV3BZaRx0i2butsRNbEKRJuUyRVPtYlw=; h=From:To:Subject:Date:Message-ID:References:In-Reply-To: Content-Type:MIME-Version; b=fEEqngKPRXnneN4aOYMaINkfwTqZrPjev6cDH+Mi0s8Fj/lPkbvNiBMH6yPJaJ5Hz 01fsa9YDyU54QU4Q+WXlBKsIW7JCKfDcX8H5GniNZFGwL9jJzcQaWFWkr1jjxa7Isd sYLGvZJviQQUzjHgu5QXklM5J7FxpegWj0DzBQGwyE7ycsjxourN+8MYEyQzV1l4yJ uDwQWpMc5SO3ioknUSSzAZ3MyNGOvgHN0NlBR6sga0R4MGCmSV11xRtNezCiJ1ZEKo G2GAPMrU3fgkQXIvLEg1LObjYnx/CndmfG3rcbo+tEwjmX9TgOZFghHPumIveWvcrg LpQSbij88Rqbw== Received: from [100.112.197.4] (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256 bits)) by server-2.bemta.az-b.eu-west-1.aws.symcld.net id A9/0B-21482-3986D416; Fri, 24 Sep 2021 05:56:35 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrJJsWRWlGSWpSXmKPExsWSoW+xRndShm+ iwfmz1hbdH1qYLN592s5kMXPBZ0YHZo9fC5ayeize85IpgCmKNTMvKb8igTXj8b9/TAXHNjBW NM/fzNTA+HIBYxcjFwejwFJmiXfPr7BAOCtZJD4/O8sO4axilOic/wesjEVgN7NE6/tDrCCOk MBcJonnx2azQTj3GSWu9h4Ecjg52ARUJG62TmEGsUUEiiTOvpvNDmILC1hKLJr+mR0ibiVx5H E7nN384CALiM0ioCpxeusksDivQKzE8vmzGCEWrGeUWDj7LlgRp0CcxI5VO8EWMAqISXw/tYY JxGYWEJe49WQ+mC0hICCxZM95ZghbVOLl43+sEA91MEpsblzMBpGQlbg0v5sRwvaVuHv3BVSz lsTj7zuh4jkS834+ZYGwVSV23b4MVSMnsar3IVRcXmLaovfsELaMxIMb26Hm/2GVmNqZDrJYQ uAcs8T/ptlQCQOJed+OsEEk3vBKrH9zkG0Co+4sJF/MYuQAsvMkru/MmQUODUGJkzOfsECU6E gs2P2JDcLWlli28DUzjH3mwGMmZPEFjOyrGM2TijLTM0pyEzNzdA0NDHQNDY10DS3NdM1M9BK rdJP0Ukt1y1OLS3QN9RLLi/WKK3OTc1L08lJLNjEC01lKwRGHHYz3X3/QO8QoycGkJMp7iNU3 UYgvKT+lMiOxOCO+qDQntfgQowwHh5IEL2cqUE6wKDU9tSItMweYWmHSEhw8SiK8FSBp3uKCx NzizHSI1ClGb44JL+cuYubo6lkIJD+uWgIkv4PJm+9B5KQju7czC7Hk5eelSonzfksHGiEAMi KjNA9uASxHXGKUlRLmZWRgYBDiKUgtys0sQZV/xSjOwagkzPsUZApPZl4J3B2vgE5kAjrx+Vw fkBNLEhFSUg1MbX8m16ofXP5VafK63JpNPHLnvSws1nlMMjnGO/994+p7TMm9bly3EmVPO/y1 ehS4xPz6+lSVZTLfFxrqBBjeDzTryhHPL01c+dWQc8FPx92x6S9CGBQaWjVlzled1F0sk+Jwj P3FJhmRbyqzyidun31gNafQtpPfbXseJj9gUntqyW68TKpS8LmBqzS/b/DlOxJZK451pT4R5T +6Yk3EjhOOC5YtyWe9lMt0Vft17o+nxxw/RhXkZZxIad7yWevSu/Apflwl20MyUt5vO8fVY31 Aiq/Uf1l9Mmt1c5XdiwCf7b63qityVxzKeCv7p/74ds5Zi/er7L7xvVcjk1neKk+589YJye03 1ryYoV+jxFKckWioxVxUnAgAN0ITiYwEAAA= X-Env-Sender: sodey@rbbn.com X-Msg-Ref: server-15.tower-287.messagelabs.com!1632462993!46483!1 X-Originating-IP: [104.47.56.172] X-SYMC-ESS-Client-Auth: mailfrom-relay-check=pass X-StarScan-Received: X-StarScan-Version: 9.81.4; banners=rbbn.com,-,- X-VirusChecked: Checked Received: (qmail 13289 invoked from network); 24 Sep 2021 05:56:34 -0000 Received: from mail-co1nam11lp2172.outbound.protection.outlook.com (HELO NAM11-CO1-obe.outbound.protection.outlook.com) (104.47.56.172) by server-15.tower-287.messagelabs.com with ECDHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 24 Sep 2021 05:56:34 -0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hUDeydctymHil90ThVHYd5kOSl2Lsf4HRmh+KNJh4g9q3RpPGRQgFtePQsUy/Eo7V7qD0K22plmViSfOj73D/kkoLJVBTGHhiLUC+Q8GvQm4HogQTlA2lnk5ktphUhEPceNcwE8+7LmUl4trVqKj3g7nIAG869ZAP3VxHGn44nf3oB4oRi0PZPXPmpYsIb3pKIltM6HD3mD9GM8R6teKDv2qm9AdiCc+dagYiDV0X65BHVxtluISa8eS61LlcDGq/4RQRPNSznDIAaRwKo3mGEsEDwvimwQe/HgzaAzfjsJ+h/IQxiReZjZ6cXTKtWBujgxlrbUQYdIxzolNjMv4jA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=SN+vCDiLzGXfV2hm7Arm34zdr3USXdeaKt1ZA7J6GDc=; b=eV2PHmH8TrxF2QfVD08B98LPYHdb2yxodNvW6YKKY+O1R0b9qM+HdGKkgRwImqJKAq3T9qpxLa+yYAkKU0cRV8zM3P0sLO6aYs/q8D7qUAxjPgskGH1yMiNeJV4teJDifrnRUt33oGqKVpTNe+JJ1NGrC1AnhgA6gVOhURqQo7hE3Vq6mJ7rlL6bDHOLpTJXf9UqQ2Jx5KSbrwzcVGxMSlAwQChKAQzfhPzY/+61Ztudddr3PnOzRUH5iwpNao4ONtL7bczVyhKbPhkUE6zgl/GqZBgvUWdpUPgcznFVjAvi/CqHiNLEIbpcJY6jchR2wVDHDzYzgZ+5fj3BDcg4GA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=rbbn.com; dmarc=pass action=none header.from=rbbn.com; dkim=pass header.d=rbbn.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=SonusNetworks.onmicrosoft.com; s=selector2-SonusNetworks-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SN+vCDiLzGXfV2hm7Arm34zdr3USXdeaKt1ZA7J6GDc=; b=kVtB/6xHfkQ1h60kJJ3aHjbaWI93GZ6sO6f4HjXwTN2G6W8ZJ5AIvNyc+6TnitHDczP7OETgp7OFwys+Dm9PEJ1dQzddJfG946KOIMEeY30MbWwq3YdCYFDqqegv6kfVeMn76Xb7ufR3HE3UIcprMiik/KA6kFM20dTivdTuvqU= Received: from DM6PR03MB4777.namprd03.prod.outlook.com (2603:10b6:5:18b::26) by DM6PR03MB4699.namprd03.prod.outlook.com (2603:10b6:5:183::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.14; Fri, 24 Sep 2021 05:56:30 +0000 Received: from DM6PR03MB4777.namprd03.prod.outlook.com ([fe80::61f5:26e2:117b:cc25]) by DM6PR03MB4777.namprd03.prod.outlook.com ([fe80::61f5:26e2:117b:cc25%4]) with mapi id 15.20.4544.015; Fri, 24 Sep 2021 05:56:30 +0000 From: "Dey, Souvik" To: "Wu, Jingjing" , "dev@dpdk.org" , "Xing, Beilei" Thread-Topic: Not able to start IAVF PMD with dpdk 20.11.3 Thread-Index: Adewj8KUV+1wMsfFRoyxRcOsPLJIPgATLnugAAr37jA= Date: Fri, 24 Sep 2021 05:56:30 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 190991a4-1fb8-4063-31c9-08d97f200e70 x-ms-traffictypediagnostic: DM6PR03MB4699: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:285; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: fm9NIFOljYr2miEoDXosaqsjYP0p3MabFs/jofh2OfxlHAdQyAV4cGogtel++M/lKqXWgI36ByjZCvwKA25oTk1SFhNLs3TXoNiPtkSBAIW/ruZTdULuPqIubmmK8wFvnnMhIENlbP/0wQ8LW6APMNZJUE22kH4tdeopk2ISet7H5YZk1U3Stiq7b/shRyO7jkpBZZ/Z6B8WkFuL9+CaP4GEsmmmNV6nclTNdJ6gjchtmWnlBMN2PREKB7mI5UA4C09zH7VnwaNR0XCCrmMpykhFXaOYYabnrWKhYyGgBJsgHYytAB5kJ2PfdCAYevo06QQZ4r7lD80LWEBN702NLS+gE5KOpXcBGr3Orfvd2ZeZKPkXYdWYUgB5TURd3qie3rEM0RIud1IeEHx4N6MRAFoDLA1Mn98WnQgvHSHn6/2FCM799Z64YW2CAISkgrtsuUFKVTY7adQ0EwLmmYHBFpi78MYQ7ldtrozSgppAxJ/sWpqiooe+i7+OcVO/sv40ENK83sBJbjKj5O4rTeNTnx3gWYx4sPJYtm6nZr7NvIKUSUSBvSl+/axJCxdgQSAU3xP9/5X0KGP/a0yiAWq6O4/0t4BINLhEOD76j12OJTfEf7MYT87k7QpeSmk+3vZV1iOeBDpk4PzgwB0ztaivkv07pmRrJ/TxdSzbWY/9A4JMJ+n6pZCB1KbBNUuFt4WimVBwDM93RXDNfhWyOt5Urw== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM6PR03MB4777.namprd03.prod.outlook.com; PTR:; CAT:NONE; SFS:(366004)(66446008)(66556008)(64756008)(2906002)(66946007)(6506007)(8936002)(52536014)(66476007)(33656002)(53546011)(83380400001)(122000001)(38100700002)(71200400001)(26005)(316002)(8676002)(508600001)(55016002)(5660300002)(76116006)(186003)(7696005)(9686003)(86362001)(38070700005)(110136005); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?rij2kTMK7cRQ050udE9atYdBVf1yyvG1Hym8AdtTVW7Ux3MpS+WTnluWZPC1?= =?us-ascii?Q?OLXGECBlNYuAljElPp4oh2aWf/IXgBfZpHSyo32pdyD5F7eT4E3GlhXE6r5x?= =?us-ascii?Q?s1l/A3KcQI6m5m2LMgCbR262uFUH1mtrL/cZXZYsEEPQeUYOuBV+Gf6WH17D?= =?us-ascii?Q?3HG7pPe0K/LGCt3Q+6RQfqI+x946VbzQKfcl2n41AMopuQfAf77ebW9w38KX?= =?us-ascii?Q?vrIuu5eR+m3UT4Nk3VvNIXuAlrtqwrmx1/+XZlb9mF4HxusLF/U8wOuUIfvN?= =?us-ascii?Q?VvXc1qDnKTVFFFIZcRt3jhancelRHY19cE+FHhqZS1Om6cN7tHc33S42V/ki?= =?us-ascii?Q?oWvYY4nthpeb0TVeiJq6NefjvD/lJFWA0R7xoBq89FcKPMYXsnD7TbU6whTI?= =?us-ascii?Q?NQSUF+eTNJI+8FGGlC9BJatzt/RKhkCVLJTNl9oacF0goCm850TkXBwOBt6a?= =?us-ascii?Q?xOaY3ea7VKwvLVlGHayHHIt9HZQ05R7qgBqiKFA2eADcYB8XZ7PaPmZ7g+oP?= =?us-ascii?Q?zMHgcpIL7QpkzU84G+WkuhNxWjlY+DXtBQBLR/xiHeyrf62vCOuuEi3m/as8?= =?us-ascii?Q?hgZtRmmpRWKs3SGSPai1xRXAv3vGZhNexmFKvZJShsIntTgmq6ILxGeSdM0K?= =?us-ascii?Q?gqCriB5oqLkJsNiphb5KpNlLjTQC4HsYAEN0fFmupTEsmhewYTQaCm8uNsys?= =?us-ascii?Q?QSIq8Ny/YPFxoQRSuuyvRanVo4b0oPnqppKwPI4Uq0cM/6U9cdr1+HqVIAeS?= =?us-ascii?Q?sj1RhrwkBJAijDr88LGfQ6Ny7gHXsk/EXTik0J8wPWMFLviMZrz1PRCDT42T?= =?us-ascii?Q?8p4ufnKM3ai/kJvvL19Lk/Kd/crmKWDZXW0DtV8UACtwH+7goXWTXfWOVJF0?= =?us-ascii?Q?PHNDUiGDfsduuachDkYddCUG3VQcPOW/b5uDbWc71nlT0Oi3bPAv2R9Hqk3p?= =?us-ascii?Q?ujstg+bY4ANECxMQ8+YNmZLxX5+RhFq07V1v88OPq9udr+TV0oueubei9D+L?= =?us-ascii?Q?SMrCi4Tdok4FT4JTgRa8ycW7dXiacfQvKKbgaI8Wy83tpDXStUNoTE7Sq9eo?= =?us-ascii?Q?AaQt7/Ne7Kxx+q1lbYKkISC/7PXFKv/ZdR3id6GOwc5FS4RU3fsjDh+Cs1mT?= =?us-ascii?Q?hO0Va4eRVuOXyeh6ARtWrQ7Ta29Tf5VzeMpFGWeONFPDSvExcc72zgnwMjS1?= =?us-ascii?Q?x9DtUyZ+S9WVZuEsRESZjOQfZeN30h4pmGPV2y1A1PveNQHPJoq+cpsFKiBD?= =?us-ascii?Q?vUnbX0DDvi7j/MQ+ww/fjjdBpz787ezAYePpjuUTAhtnNLcX0p1qDlp0uoOi?= =?us-ascii?Q?hA+YhD5bAH3ltWxJzS9C68B6?= x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: rbbn.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM6PR03MB4777.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 190991a4-1fb8-4063-31c9-08d97f200e70 X-MS-Exchange-CrossTenant-originalarrivaltime: 24 Sep 2021 05:56:30.6111 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 29a671dc-ed7e-4a54-b1e5-8da1eb495dc3 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: xRnwEEJOKIA47OuNsWh/g5Omc7y3NUI5qNCX0NEpeXHPMZt48/AxkaNOyZykQ/Sw X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR03MB4699 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: Re: [dpdk-dev] Not able to start IAVF PMD with dpdk 20.11.3 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" With vfio_pci , I am getting the following error and bind itself is failin= g, due to probe fail with EINVAL. [root@connexip swe]# ./dpdk-devbind.py -b vfio_pci 08:00.0 Error: bind failed for 0000:08:00.0 - Cannot open /sys/bus/pci/drivers/vfi= o_pci/bind Sep 24 01:50:29 connexip kernel: [ 323.376403] vfio-pci: probe of 0000:08= :00.0 failed with error -22 Sep 24 01:50:29 connexip kernel: [ 323.376440] vfio-pci: probe of 0000:08= :00.0 failed with error -22 I am using kernel 4.19.194 from debian buster. From: Wu, Jingjing Sent: Thursday, September 23, 2021 8:41 PM To: Dey, Souvik ; dev@dpdk.org; Xing, Beilei Subject: [EXTERNAL] RE: Not able to start IAVF PMD with dpdk 20.11.3 Could you have a try to switch from uio_pci_generic to vfio_pci? From: Dey, Souvik > Sent: Thursday, September 23, 2021 11:29 PM To: dev@dpdk.org; Xing, Beilei >; Wu, Jingjing > Subject: Not able to start IAVF PMD with dpdk 20.11.3 Hi All, Trying to test E810 Sr-IOV based nic card with 20.11.3 dpdk. I am run= ning the host kernel driver and only the VF is attached to the VM where I = am running dpdk. But when I attaching the VF to the testpmd it is throwing= the below error. Is there are way to move forward here ? [root@connexip linuxadmin]# ./dpdk-testpmd -c 0xf -n 4 -a 08:00.0 -- -i --= rxq=3D16 --txq=3D16 EAL: Detected 4 lcore(s) EAL: Detected 1 NUMA nodes EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'PA' EAL: No available hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: Probe PCI driver: net_iavf (8086:1889) device: 0000:08:00.0 (socket 0= ) EAL: Error reading from file descriptor 27: Input/output error EAL: No legacy callbacks, legacy socket not created Interactive-mode selected testpmd: create a new mbuf pool :=20n=3D171456, size=3D2176, so= cket=3D0 testpmd: preferred mempool ops selected: ring_mp_mc Warning! port-topology=3Dpaired and odd forward ports number, the last por= t will pair with itself. Configuring Port 0 (socket 0) EAL: Error reading from file descriptor 27: Input/output error iavf_execute_vf_cmd(): No response for cmd 28 iavf_disable_vlan_strip(): Failed to execute command of OP_DISABLE_VLAN_ST= RIPPING iavf_execute_vf_cmd(): No response for cmd 24 iavf_configure_rss_lut(): Failed to execute command of OP_CONFIG_RSS_LUT iavf_dev_configure(): configure rss failed Port0 dev_configure =3D -1 Fail to configure port 0 EAL: Error - exiting with code: 1 Cause: Start ports failed [root@connexip linuxadmin]# I do see the matching list for driver but is this required even if I use k= ernel driver on the host and only IAVF PMD on the guest ? DPDK Kernel Driver OS Default DDP COMMS DDP Wireless DDP Firmware 20.11 1.3.2 1.3.20 1.3.24 N/A 2.3 21.02 1.4.11 1.3.24 1.3.28 1.3.4 2.4 Host details: Red Hat Enterprise Linux release 8.2 (Ootpa) [root@localhost ~]# modinfo ice filename: /lib/modules/4.18.0-193.19.1.el8_2.x86_64/kernel/drivers/n= et/ethernet/intel/ice/ice.ko.xz firmware: intel/ice/ddp/ice.pkg (ice-1.3.4.0.pkg) version: 0.8.1-k license: GPL v2 description: Intel(R) Ethernet Connection E800 Series Linux Driver author: Intel Corporation, linux.nics@intel.com rhelversion: 8.2 Using vfio-pci to connect the VF to the VM. Guest Details: OS : debian buster Dpdk : 20.11.3 Interface bound to uio_pci_generic [root@connexip linuxadmin]# /opt/sonus/bin/np/swe/dpdk-devbind.py --status= Network devices using DPDK-compatible driver =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 0000:01:00.0 'Virtio network device 1041' drv=3Duio_pci_generic unused=3Dv= irtio_pci 0000:08:00.0 'Ethernet Adaptive Virtual Function 1889' drv=3Duio_pci_gener= ic unused=3Di40evf 0000:09:00.0 'Ethernet Adaptive Virtual Function 1889' drv=3Duio_pci_gener= ic unused=3Di40evf Network devices using kernel driver =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 0000:02:00.0 'Virtio network device 1041' if=3Dha0 drv=3Dvirtio-pci unused= =3Dvirtio_pci,uio_pci_generic *Active* Notice: This e-mail together with any attachments may contain information = of Ribbon Communications Inc. and its Affiliates that is confidential and/= or proprietary for the sole use of the intended recipient. Any review, dis= closure, reliance or distribution by others or forwarding without express = permission is strictly prohibited. If you are not the intended recipient, = please notify the sender immediately and then delete all copies, including= any attachments. Notice: This e-mail together with any attachments may contain information = of Ribbon Communications Inc. and its Affiliates that is confidential and/= or proprietary for the sole use of the intended recipient. Any review, dis= closure, reliance or distribution by others or forwarding without express = permission is strictly prohibited. If you are not the intended recipient, = please notify the sender immediately and then delete all copies, including= any attachments.