Re: [dpdk-dev] vfio-pci blocking issue

Re: [dpdk-dev] vfio-pci blocking issue

[Xu, Qian Q]

We have found the blocking issue about vfio-pci in RC2, which blocked many test cases. It is a blocking issue for 17.11 release. Please help check.

Add more failed case as below: it will also cause the VF port can't be probed.

?  Components version
dpdk 17.11 rc2
?  Defect component version (release version or repo/branch/commit ID)
?  Environment components version (QEMU, OVS, etc)
?  HW platform info
fvl10g
?  Test Steps and Test Result
1. Created VFs and bind to vfio-pci
./usertools/dpdk-devbind.py --bind=igb_uio 81:00.0 81:00.1
echo 1 > /sys/bus/pci/devices/0000:81:00.0/max_vfs
echo 1 > /sys/bus/pci/devices/0000:81:00.1/max_vfs
./usertools/dpdk-devbind.py -s
0000:05:02.0 'XL710/X710 Virtual Function 154c' if= drv=i40evf unused=igb_uio
0000:05:06.0 'XL710/X710 Virtual Function 154c' if= drv=i40evf unused=igb_uio
./usertools/dpdk-devbind.py --bind=vfio-pci 81:02.0 81:06.0
./usertools/dpdk-devbind.py -s
0000:81:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=igb_uio unused=vfio-pci
0000:81:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=igb_uio unused=vfio-pci
0000:81:02.0 'XL710/X710 Virtual Function 154c' drv=vfio-pci unused=igb_uio
0000:81:06.0 'XL710/X710 Virtual Function 154c' drv=vfio-pci unused=igb_uio
2. up PF with testpmd,
./x86_64-native-linuxapp-gcc/app/testpmd -l 23-27 -n 4 --socket-mem=1024,1024 --file-prefix=pf -w 81:00.0 -w 81:00.1 - -i --rxq=4 --txq=4
3. start VFs with testpmd,
./x86_64-native-linuxapp-gcc/app/testpmd -l 28-32 -n 4 --socket-mem=1024,1024 --file-prefix=vf -w 81:02.0 -w 81:06.0 - -i --rxq=4 --txq=4
EAL: Detected 88 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: vfio-noiommu mode configured
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:81:02.0 on NUMA socket 1
EAL: probe driver: 8086:154c net_i40e_vf
EAL: Requested device 0000:81:02.0 cannot be used
EAL: PCI device 0000:81:06.0 on NUMA socket 1
EAL: probe driver: 8086:154c net_i40e_vf
EAL: Requested device 0000:81:06.0 cannot be used
EAL: No probed ethernet devices
Interactive-mode selected
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=179456, size=2176, socket=0
USER1: create a new mbuf pool <mbuf_pool_socket_1>: n=179456, size=2176, socket=1
Done
testpmd>
it can be seen "No probed ethernet devices"


From: Lin, Xueqin
Sent: Monday, October 30, 2017 1:59 PM
To: gaetan.rivet@6wind.com; Zhang, Helin <helin.zhang@intel.com>
Cc: Xu, Qian Q <qian.q.xu@intel.com>; Pei, Yulong <yulong.pei@intel.com>; Peng, Yuan <yuan.peng@intel.com>
Subject: vfio-pci

Hi Rivet,

Some cases need to bind port to vfio-pci, for example, test inline inpsec.

1.      modprobe vfio-pci

2.      bind port to vfio-pci

./usertools/dpdk-devbind.py -b vfio-pci 81:00.0 81:00.1

3.      Run ipsec
sudo ./build/ipsec-secgw -l 20,21   -w 81:00.0 -w 81:00.1 --vdev "crypto_null" --log-level 8 --socket-mem 1024,1   -- -p 0xf -P -u 0x2 -j 9000\
        --config="(0,0,20),(1,0,21)" -f ./enc.cfg
Find error as below, vfio-pci can't be used.

EAL:   probe driver: 8086:10fb net_ixgbe
EAL: Requested device 0000:81:00.0 cannot be used
EAL: PCI device 0000:81:00.1 on NUMA socket 1
EAL:   probe driver: 8086:10fb net_ixgbe
EAL: Requested device 0000:81:00.1 cannot be used
EAL: Search driver crypto_null to probe device crypto_null
CRYPTODEV: [crypto_null] - Creating cryptodev crypto_null

Find your patch results in the error, block lots of vfio-pci cases, which is urgent.
DPDK-4026<https://jira01.devtools.intel.com/browse/DPDK-4026> should meet same issue.
If any wrong to use vfio-pci, could you give a guide?

commit 279b581c897d4bf01f2b35c7327e767400c3ee80
Author: Gaetan Rivet <gaetan.rivet@6wind.com<mailto:gaetan.rivet@6wind.com>>
Date:   Thu Oct 26 12:05:52 2017 +0200

    vfio: expose functions

    The following symbols are used by vfio implementations within the PCI bus.
    They need to be publicly available for the PCI bus to be outside the
    EAL.


Best regards,
Xueqin

Re: [dpdk-dev] vfio-pci blocking issue

[Gaëtan Rivet]

Hi Xueqin,

Thank you for the thorough test report.
Multiple filings for this bug, please refer to the thread

http://dpdk.org/ml/archives/dev/2017-October/080746.html

Which will focus discussion. A first fix and a new proposition are
available, please tell us if it works for you or if there are
any other problem with either of the solutions.

On Mon, Oct 30, 2017 at 07:25:20AM +0000, Xu, Qian Q wrote:
> We have found the blocking issue about vfio-pci in RC2, which blocked many test cases. It is a blocking issue for 17.11 release. Please help check.
> 
> Add more failed case as below: it will also cause the VF port can't be probed.
> 
> ?  Components version
> dpdk 17.11 rc2
> ?  Defect component version (release version or repo/branch/commit ID)
> ?  Environment components version (QEMU, OVS, etc)
> ?  HW platform info
> fvl10g
> ?  Test Steps and Test Result
> 1. Created VFs and bind to vfio-pci
> ./usertools/dpdk-devbind.py --bind=igb_uio 81:00.0 81:00.1
> echo 1 > /sys/bus/pci/devices/0000:81:00.0/max_vfs
> echo 1 > /sys/bus/pci/devices/0000:81:00.1/max_vfs
> ./usertools/dpdk-devbind.py -s
> 0000:05:02.0 'XL710/X710 Virtual Function 154c' if= drv=i40evf unused=igb_uio
> 0000:05:06.0 'XL710/X710 Virtual Function 154c' if= drv=i40evf unused=igb_uio
> ./usertools/dpdk-devbind.py --bind=vfio-pci 81:02.0 81:06.0
> ./usertools/dpdk-devbind.py -s
> 0000:81:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=igb_uio unused=vfio-pci
> 0000:81:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=igb_uio unused=vfio-pci
> 0000:81:02.0 'XL710/X710 Virtual Function 154c' drv=vfio-pci unused=igb_uio
> 0000:81:06.0 'XL710/X710 Virtual Function 154c' drv=vfio-pci unused=igb_uio
> 2. up PF with testpmd,
> ./x86_64-native-linuxapp-gcc/app/testpmd -l 23-27 -n 4 --socket-mem=1024,1024 --file-prefix=pf -w 81:00.0 -w 81:00.1 - -i --rxq=4 --txq=4
> 3. start VFs with testpmd,
> ./x86_64-native-linuxapp-gcc/app/testpmd -l 28-32 -n 4 --socket-mem=1024,1024 --file-prefix=vf -w 81:02.0 -w 81:06.0 - -i --rxq=4 --txq=4
> EAL: Detected 88 lcore(s)
> EAL: Some devices want iova as va but pa will be used because.. EAL: vfio-noiommu mode configured
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: PCI device 0000:81:02.0 on NUMA socket 1
> EAL: probe driver: 8086:154c net_i40e_vf
> EAL: Requested device 0000:81:02.0 cannot be used
> EAL: PCI device 0000:81:06.0 on NUMA socket 1
> EAL: probe driver: 8086:154c net_i40e_vf
> EAL: Requested device 0000:81:06.0 cannot be used
> EAL: No probed ethernet devices
> Interactive-mode selected
> USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=179456, size=2176, socket=0
> USER1: create a new mbuf pool <mbuf_pool_socket_1>: n=179456, size=2176, socket=1
> Done
> testpmd>
> it can be seen "No probed ethernet devices"
> 
> 
> From: Lin, Xueqin
> Sent: Monday, October 30, 2017 1:59 PM
> To: gaetan.rivet@6wind.com; Zhang, Helin <helin.zhang@intel.com>
> Cc: Xu, Qian Q <qian.q.xu@intel.com>; Pei, Yulong <yulong.pei@intel.com>; Peng, Yuan <yuan.peng@intel.com>
> Subject: vfio-pci
> 
> Hi Rivet,
> 
> Some cases need to bind port to vfio-pci, for example, test inline inpsec.
> 
> 1.      modprobe vfio-pci
> 
> 2.      bind port to vfio-pci
> 
> ./usertools/dpdk-devbind.py -b vfio-pci 81:00.0 81:00.1
> 
> 3.      Run ipsec
> sudo ./build/ipsec-secgw -l 20,21   -w 81:00.0 -w 81:00.1 --vdev "crypto_null" --log-level 8 --socket-mem 1024,1   -- -p 0xf -P -u 0x2 -j 9000\
>         --config="(0,0,20),(1,0,21)" -f ./enc.cfg
> Find error as below, vfio-pci can't be used.
> 
> EAL:   probe driver: 8086:10fb net_ixgbe
> EAL: Requested device 0000:81:00.0 cannot be used
> EAL: PCI device 0000:81:00.1 on NUMA socket 1
> EAL:   probe driver: 8086:10fb net_ixgbe
> EAL: Requested device 0000:81:00.1 cannot be used
> EAL: Search driver crypto_null to probe device crypto_null
> CRYPTODEV: [crypto_null] - Creating cryptodev crypto_null
> 
> Find your patch results in the error, block lots of vfio-pci cases, which is urgent.
> DPDK-4026<https://jira01.devtools.intel.com/browse/DPDK-4026> should meet same issue.
> If any wrong to use vfio-pci, could you give a guide?
> 
> commit 279b581c897d4bf01f2b35c7327e767400c3ee80
> Author: Gaetan Rivet <gaetan.rivet@6wind.com<mailto:gaetan.rivet@6wind.com>>
> Date:   Thu Oct 26 12:05:52 2017 +0200
> 
>     vfio: expose functions
> 
>     The following symbols are used by vfio implementations within the PCI bus.
>     They need to be publicly available for the PCI bus to be outside the
>     EAL.
> 
> 
> Best regards,
> Xueqin
> 

-- 
Gaëtan Rivet
6WIND

Re: [dpdk-dev] vfio-pci blocking issue

[Lin, Xueqin]

Hi Rivet,
Using your patch to include "eal_vfio.h" header file, vfio-pci could work normally on my environment. Thanks.
http://dpdk.org/ml/archives/dev/2017-October/080746.html

Best regards,
Xueqin

-----Original Message-----
From: Gaëtan Rivet [mailto:gaetan.rivet@6wind.com] 
Sent: Monday, October 30, 2017 4:14 PM
To: Xu, Qian Q <qian.q.xu@intel.com>
Cc: Lin, Xueqin <xueqin.lin@intel.com>; Zhang, Helin <helin.zhang@intel.com>; Thomas Monjalon <thomas@monjalon.net>; Pei, Yulong <yulong.pei@intel.com>; Peng, Yuan <yuan.peng@intel.com>; dev@dpdk.org; Glynn, Michael J <michael.j.glynn@intel.com>; Liu, Yu Y <yu.y.liu@intel.com>
Subject: Re: vfio-pci blocking issue

Hi Xueqin,

Thank you for the thorough test report.
Multiple filings for this bug, please refer to the thread

http://dpdk.org/ml/archives/dev/2017-October/080746.html

Which will focus discussion. A first fix and a new proposition are available, please tell us if it works for you or if there are any other problem with either of the solutions.

On Mon, Oct 30, 2017 at 07:25:20AM +0000, Xu, Qian Q wrote:
> We have found the blocking issue about vfio-pci in RC2, which blocked many test cases. It is a blocking issue for 17.11 release. Please help check.
> 
> Add more failed case as below: it will also cause the VF port can't be probed.
> 
> ?  Components version
> dpdk 17.11 rc2
> ?  Defect component version (release version or repo/branch/commit ID) 
> ?  Environment components version (QEMU, OVS, etc) ?  HW platform info 
> fvl10g ?  Test Steps and Test Result 1. Created VFs and bind to 
> vfio-pci ./usertools/dpdk-devbind.py --bind=igb_uio 81:00.0 81:00.1 
> echo 1 > /sys/bus/pci/devices/0000:81:00.0/max_vfs
> echo 1 > /sys/bus/pci/devices/0000:81:00.1/max_vfs
> ./usertools/dpdk-devbind.py -s
> 0000:05:02.0 'XL710/X710 Virtual Function 154c' if= drv=i40evf 
> unused=igb_uio
> 0000:05:06.0 'XL710/X710 Virtual Function 154c' if= drv=i40evf 
> unused=igb_uio ./usertools/dpdk-devbind.py --bind=vfio-pci 81:02.0 
> 81:06.0 ./usertools/dpdk-devbind.py -s
> 0000:81:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' 
> drv=igb_uio unused=vfio-pci
> 0000:81:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' 
> drv=igb_uio unused=vfio-pci
> 0000:81:02.0 'XL710/X710 Virtual Function 154c' drv=vfio-pci 
> unused=igb_uio
> 0000:81:06.0 'XL710/X710 Virtual Function 154c' drv=vfio-pci 
> unused=igb_uio 2. up PF with testpmd, 
> ./x86_64-native-linuxapp-gcc/app/testpmd -l 23-27 -n 4 
> --socket-mem=1024,1024 --file-prefix=pf -w 81:00.0 -w 81:00.1 - -i 
> --rxq=4 --txq=4 3. start VFs with testpmd, 
> ./x86_64-native-linuxapp-gcc/app/testpmd -l 28-32 -n 4 
> --socket-mem=1024,1024 --file-prefix=vf -w 81:02.0 -w 81:06.0 - -i 
> --rxq=4 --txq=4
> EAL: Detected 88 lcore(s)
> EAL: Some devices want iova as va but pa will be used because.. EAL: 
> vfio-noiommu mode configured
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: PCI device 0000:81:02.0 on NUMA socket 1
> EAL: probe driver: 8086:154c net_i40e_vf
> EAL: Requested device 0000:81:02.0 cannot be used
> EAL: PCI device 0000:81:06.0 on NUMA socket 1
> EAL: probe driver: 8086:154c net_i40e_vf
> EAL: Requested device 0000:81:06.0 cannot be used
> EAL: No probed ethernet devices
> Interactive-mode selected
> USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=179456, 
> size=2176, socket=0
> USER1: create a new mbuf pool <mbuf_pool_socket_1>: n=179456, 
> size=2176, socket=1 Done
> testpmd>
> it can be seen "No probed ethernet devices"
> 
> 
> From: Lin, Xueqin
> Sent: Monday, October 30, 2017 1:59 PM
> To: gaetan.rivet@6wind.com; Zhang, Helin <helin.zhang@intel.com>
> Cc: Xu, Qian Q <qian.q.xu@intel.com>; Pei, Yulong 
> <yulong.pei@intel.com>; Peng, Yuan <yuan.peng@intel.com>
> Subject: vfio-pci
> 
> Hi Rivet,
> 
> Some cases need to bind port to vfio-pci, for example, test inline inpsec.
> 
> 1.      modprobe vfio-pci
> 
> 2.      bind port to vfio-pci
> 
> ./usertools/dpdk-devbind.py -b vfio-pci 81:00.0 81:00.1
> 
> 3.      Run ipsec
> sudo ./build/ipsec-secgw -l 20,21   -w 81:00.0 -w 81:00.1 --vdev "crypto_null" --log-level 8 --socket-mem 1024,1   -- -p 0xf -P -u 0x2 -j 9000\
>         --config="(0,0,20),(1,0,21)" -f ./enc.cfg Find error as below, 
> vfio-pci can't be used.
> 
> EAL:   probe driver: 8086:10fb net_ixgbe
> EAL: Requested device 0000:81:00.0 cannot be used
> EAL: PCI device 0000:81:00.1 on NUMA socket 1
> EAL:   probe driver: 8086:10fb net_ixgbe
> EAL: Requested device 0000:81:00.1 cannot be used
> EAL: Search driver crypto_null to probe device crypto_null
> CRYPTODEV: [crypto_null] - Creating cryptodev crypto_null
> 
> Find your patch results in the error, block lots of vfio-pci cases, which is urgent.
> DPDK-4026<https://jira01.devtools.intel.com/browse/DPDK-4026> should meet same issue.
> If any wrong to use vfio-pci, could you give a guide?
> 
> commit 279b581c897d4bf01f2b35c7327e767400c3ee80
> Author: Gaetan Rivet <gaetan.rivet@6wind.com<mailto:gaetan.rivet@6wind.com>>
> Date:   Thu Oct 26 12:05:52 2017 +0200
> 
>     vfio: expose functions
> 
>     The following symbols are used by vfio implementations within the PCI bus.
>     They need to be publicly available for the PCI bus to be outside the
>     EAL.
> 
> 
> Best regards,
> Xueqin
> 

--
Gaëtan Rivet
6WIND

Re: [dpdk-dev] vfio-pci blocking issue

[Ferruh Yigit]

On 10/30/2017 1:14 AM, Gaëtan Rivet wrote:
> Hi Xueqin,
> 
> Thank you for the thorough test report.
> Multiple filings for this bug, please refer to the thread
> 
> http://dpdk.org/ml/archives/dev/2017-October/080746.html
> 
> Which will focus discussion. A first fix and a new proposition are
> available, please tell us if it works for you or if there are
> any other problem with either of the solutions.

Hi Gaetan,

Another issue related moving pci, szedata2 driver build seems broken [1] because
it uses a pci function which moved now. Can you please check it?

Also the pci_get_sysfs_path API moved from eal seems still in the eal .map file.


[1]
rte_eth_szedata2.o: In function `szedata2_eth_pci_probe':
/tmp/dpdk_maintain/self/dpdk/drivers/net/szedata2/rte_eth_szedata2.c:(.text+0x432):
undefined reference to `pci_get_sysfs_path'


Thanks,
ferruh

> 
> On Mon, Oct 30, 2017 at 07:25:20AM +0000, Xu, Qian Q wrote:
>> We have found the blocking issue about vfio-pci in RC2, which blocked many test cases. It is a blocking issue for 17.11 release. Please help check.
>>
>> Add more failed case as below: it will also cause the VF port can't be probed.
>>
>> ?  Components version
>> dpdk 17.11 rc2
>> ?  Defect component version (release version or repo/branch/commit ID)
>> ?  Environment components version (QEMU, OVS, etc)
>> ?  HW platform info
>> fvl10g
>> ?  Test Steps and Test Result
>> 1. Created VFs and bind to vfio-pci
>> ./usertools/dpdk-devbind.py --bind=igb_uio 81:00.0 81:00.1
>> echo 1 > /sys/bus/pci/devices/0000:81:00.0/max_vfs
>> echo 1 > /sys/bus/pci/devices/0000:81:00.1/max_vfs
>> ./usertools/dpdk-devbind.py -s
>> 0000:05:02.0 'XL710/X710 Virtual Function 154c' if= drv=i40evf unused=igb_uio
>> 0000:05:06.0 'XL710/X710 Virtual Function 154c' if= drv=i40evf unused=igb_uio
>> ./usertools/dpdk-devbind.py --bind=vfio-pci 81:02.0 81:06.0
>> ./usertools/dpdk-devbind.py -s
>> 0000:81:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=igb_uio unused=vfio-pci
>> 0000:81:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=igb_uio unused=vfio-pci
>> 0000:81:02.0 'XL710/X710 Virtual Function 154c' drv=vfio-pci unused=igb_uio
>> 0000:81:06.0 'XL710/X710 Virtual Function 154c' drv=vfio-pci unused=igb_uio
>> 2. up PF with testpmd,
>> ./x86_64-native-linuxapp-gcc/app/testpmd -l 23-27 -n 4 --socket-mem=1024,1024 --file-prefix=pf -w 81:00.0 -w 81:00.1 - -i --rxq=4 --txq=4
>> 3. start VFs with testpmd,
>> ./x86_64-native-linuxapp-gcc/app/testpmd -l 28-32 -n 4 --socket-mem=1024,1024 --file-prefix=vf -w 81:02.0 -w 81:06.0 - -i --rxq=4 --txq=4
>> EAL: Detected 88 lcore(s)
>> EAL: Some devices want iova as va but pa will be used because.. EAL: vfio-noiommu mode configured
>> EAL: Probing VFIO support...
>> EAL: VFIO support initialized
>> EAL: PCI device 0000:81:02.0 on NUMA socket 1
>> EAL: probe driver: 8086:154c net_i40e_vf
>> EAL: Requested device 0000:81:02.0 cannot be used
>> EAL: PCI device 0000:81:06.0 on NUMA socket 1
>> EAL: probe driver: 8086:154c net_i40e_vf
>> EAL: Requested device 0000:81:06.0 cannot be used
>> EAL: No probed ethernet devices
>> Interactive-mode selected
>> USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=179456, size=2176, socket=0
>> USER1: create a new mbuf pool <mbuf_pool_socket_1>: n=179456, size=2176, socket=1
>> Done
>> testpmd>
>> it can be seen "No probed ethernet devices"
>>
>>
>> From: Lin, Xueqin
>> Sent: Monday, October 30, 2017 1:59 PM
>> To: gaetan.rivet@6wind.com; Zhang, Helin <helin.zhang@intel.com>
>> Cc: Xu, Qian Q <qian.q.xu@intel.com>; Pei, Yulong <yulong.pei@intel.com>; Peng, Yuan <yuan.peng@intel.com>
>> Subject: vfio-pci
>>
>> Hi Rivet,
>>
>> Some cases need to bind port to vfio-pci, for example, test inline inpsec.
>>
>> 1.      modprobe vfio-pci
>>
>> 2.      bind port to vfio-pci
>>
>> ./usertools/dpdk-devbind.py -b vfio-pci 81:00.0 81:00.1
>>
>> 3.      Run ipsec
>> sudo ./build/ipsec-secgw -l 20,21   -w 81:00.0 -w 81:00.1 --vdev "crypto_null" --log-level 8 --socket-mem 1024,1   -- -p 0xf -P -u 0x2 -j 9000\
>>         --config="(0,0,20),(1,0,21)" -f ./enc.cfg
>> Find error as below, vfio-pci can't be used.
>>
>> EAL:   probe driver: 8086:10fb net_ixgbe
>> EAL: Requested device 0000:81:00.0 cannot be used
>> EAL: PCI device 0000:81:00.1 on NUMA socket 1
>> EAL:   probe driver: 8086:10fb net_ixgbe
>> EAL: Requested device 0000:81:00.1 cannot be used
>> EAL: Search driver crypto_null to probe device crypto_null
>> CRYPTODEV: [crypto_null] - Creating cryptodev crypto_null
>>
>> Find your patch results in the error, block lots of vfio-pci cases, which is urgent.
>> DPDK-4026<https://jira01.devtools.intel.com/browse/DPDK-4026> should meet same issue.
>> If any wrong to use vfio-pci, could you give a guide?
>>
>> commit 279b581c897d4bf01f2b35c7327e767400c3ee80
>> Author: Gaetan Rivet <gaetan.rivet@6wind.com<mailto:gaetan.rivet@6wind.com>>
>> Date:   Thu Oct 26 12:05:52 2017 +0200
>>
>>     vfio: expose functions
>>
>>     The following symbols are used by vfio implementations within the PCI bus.
>>     They need to be publicly available for the PCI bus to be outside the
>>     EAL.
>>
>>
>> Best regards,
>> Xueqin
>>
>