[dpdk-ci] CI Test Cases

[dpdk-ci] CI Test Cases

[Jeremy Plsek]

Hi,

Based on last weeks meeting, I've created a document to help start
documenting what test cases
should be worked on for cross platform support between NIC vendors and
which test cases can be used now in the Community Lab.

This list of test cases that should be worked on can be used when
discussing the budget with the board.

This also gives the Community Lab a list of tests that can be
implemented in the mean time.

https://docs.google.com/document/d/1WbRoIq0f5SmHM6D5sLbeh7hxRJdE2ppwB5JECHllLjg/edit?usp=sharing
(This document is publicly editable. If I notice the document changes
outside the scope of the goal, I'll try to revert it and give
individual access instead.)

If creating this list over email is wanted instead, then we can do it
in this thread instead of the document. (And I'll update the document
as emails come in.)

Thanks.
-- 
Jeremy Plsek
UNH InterOperability Laboratory

Re: [dpdk-ci] CI Test Cases

[David Marchand]

[-- Attachment #1: Type: text/plain, Size: 2218 bytes --]

Hello Jeremy,

On Mon, Mar 18, 2019 at 4:56 PM Jeremy Plsek <jplsek@iol.unh.edu> wrote:

> Based on last weeks meeting, I've created a document to help start
> documenting what test cases
> should be worked on for cross platform support between NIC vendors and
> which test cases can be used now in the Community Lab.
>
> This list of test cases that should be worked on can be used when
> discussing the budget with the board.
>
> This also gives the Community Lab a list of tests that can be
> implemented in the mean time.
>
>
> https://docs.google.com/document/d/1WbRoIq0f5SmHM6D5sLbeh7hxRJdE2ppwB5JECHllLjg/edit?usp=sharing
> (This document is publicly editable. If I notice the document changes
> outside the scope of the goal, I'll try to revert it and give
> individual access instead.)
>
> If creating this list over email is wanted instead, then we can do it
> in this thread instead of the document. (And I'll update the document
> as emails come in.)
>

Well, for now, I prefer emails because I did not follow the CI discussions
so far and I might be off topic.

Here I go, the list is large, I tried to write from the basic to the most
advanced/less used.

- Non regression tests on basic startup (we can reuse the dpdk ut and/or
enhance them), this part would be hw agnostic:
  - core checks: playing with cpus online/offline + dpdk coremask/list +
checking cpu affinity of all dpdk threads
  - memory checks: a big topic because we have multiple modes at the moment
  - startup devices blacklist/whitelist
  - hotplug (unplug) of devices

- Non regression tests on eth drivers (testpmd seems the best candidate),
this must be ran against a lot of nics/drivers:
  - reception of unicast packets: check that we receive packets for primary
and secondary addresses (and only for them, meaning with checks that
undesired packets are dropped following updates)
  - promisc support
  - idem with multicast packets
  - link status checks
  - rte_eth_link_up/down checks
  - mtu checks
  - rss checks
  - ptypes classification
  - stats checks
  - rte_flow checks
  - port restart checks + associated configuration persistency
  - flow control
  - rx/tx offloads
  - rx/tx descriptor status


-- 
David Marchand

[-- Attachment #2: Type: text/html, Size: 3181 bytes --]

Re: [dpdk-ci] CI Test Cases

[Thomas Monjalon]

18/03/2019 16:55, Jeremy Plsek:
> This also gives the Community Lab a list of tests that can be
> implemented in the mean time.
> 
> https://docs.google.com/document/d/1WbRoIq0f5SmHM6D5sLbeh7hxRJdE2ppwB5JECHllLjg/edit?usp=sharing
> (This document is publicly editable. If I notice the document changes
> outside the scope of the goal, I'll try to revert it and give
> individual access instead.)

After the meeting of today, Ferruh, David and me have added more tests
to discuss next time:

	- Coverage gcov with unit tests
	- devtools/check-symbol-maps.sh
	- devtools/validate-abi.sh
	- Spell checks
	- clang-analyzer
	- cppcheck

And we have enumerated the items for the compilation tests:

Target CPU:
	Aarch64
	i686
	x86_32
	x86_64
	Power64
OS:
	FreeBSD 10.4
	FreeBSD 11
	FreeBSD 12.0
	Fedora 28
	Fedora 29
	CentOS 7.6
	Ubuntu 16.04
	Ubuntu 18.04
	Ubuntu 19.04
	Debian 9 stable
	Debian 10 testing
	OpenSuse 12-sp3
	OpenSuse 15
	Archlinux - latest Linux/gcc/clang
	Windows 10 (only meson+clang)
Make options (config in test-build.sh):
	examples (included in test-build.sh)
	HTML doc (included in test-build.sh)
	PDF doc - make doc-guides-pdf
	+shared
	+debug
	+ASSERT
	+LIBBSD
	+ARM64_MEMCPY
	+ARCH_STRICT_ALIGN
	+ALLOW_INV_SOCKET_ID
	+ALWAYS_PANIC_ON_ERROR
	+ENABLE_AVX512
	+PROFILE_WITH_VTUNE
	+TX_PREPARE_NOOP
	+USE_HPET
	+IBVERBS_LINK_DLOPEN
Meson options (config in meson_options.txt):
	examples (included in test-meson-builds.sh)
	enable_docs=true
	allow_invalid_socket_id=true
	use_hpet=true - should it be dropped?
	ibverbs_link=dlopen
	per_library_versions=false
Toolchain:
	Latest gcc
	Latest clang
Dependencies:
	make
	meson
	ninja
	doxygen
	sphinx
	latex
	inkscape
	libnuma
	libpcap
	libelf
	libarchive
	zlib
	libjansson
	openssl
	rdma-core
	musdk
	szedata2
	ipsec_mb
	libsso-snow3g
	libsso-kasumi
	libsso-zuc
	libqos
	isa-l
	libmnl
	libvirt