Re: [dpdk-dev] [RFC 0/4] Include resources in tests

[Jan Viktorin <viktorin@rehivetech.com>]

On Fri, 29 Apr 2016 15:42:27 +0100
Bruce Richardson <bruce.richardson@intel.com> wrote:

> On Fri, Apr 29, 2016 at 03:11:32PM +0200, Jan Viktorin wrote:
> > Hello,
> > 
> > this patch set introduces a mechanism to include a resource (in general a blob)
> > into the test binary. This allows to make tests less dependent on the target
> > testing environment. The first use case is testing of PCI bus scan by changing
> > the hard-coded path (/sys/bus/pci/devices) to something different and provide
> > a fake tree of devices with the test. It can help with testing of device-tree
> > parsing as I've proposed in [1] where such mechanism was missing at that time.
> > I'd like to use such framework for the SoC infra testing as well.
> > 
> > The patch set introduces a struct resource into the app/test. The resource is
> > generic to include any kind of binary data. The binary data can be created in
> > C or linked as an object file (created by objcopy). I am not sure where to
> > place the objcopy logic and how to perform guessing of the objcopy arguments
> > as they are pretty non-standard.
> > 
> > To include a complex resource (a file hierarchy), the last patch implements
> > an archive extraction logic. So, it is possible to include a tar archive and
> > unpack it before a test starts. Any ideas how to do this in a better way are
> > welcome.
> > 
> > [1] http://comments.gmane.org/gmane.comp.networking.dpdk.devel/36545
> > 
> > Regards
> > Jan Viktorin
> >   
> 
> Hi Jan,
> 
> this looks really interesting, especially since just yesterday I was looking at
> taking the million-entry lpm test routing table out of the C code and into a
> separate resource file in this case an ini file.

If it is an automatic test case which can show some regressions over
time then it is a good example of the resource usage.

> 
> In terms of a solution, I'm not convinced of the placing of the blobs inside the
> test binary. I think a better solution would be to allow the different autotests
> to take parameters from the commandline, so that the user can specify the path
> to the file to use for the test. What would be your opinion of such a scheme?

I think that we are already at this stage. You can read parameters e.g.
from the environment vars (so in fact, no changes to the DPDK testing
code are needed). The thing is that I want to say "run tests" and don't
care about any parameters to receive the result.

It might be possible to pass parameters to the tests (optionally) to
reuse the testing code base for different cases. However, initially,
the test (in my opinion) should run on any architecture on any device
without any configuration and return consistent results.

How can I test probing of PCI devices on my PC when I have different
network card then the author of the test? I cannot or I have to pass
parameters which is not what I want as it complicates every single
such test (and I have to understand those tests or understand the
parameters syntax even though the code is unrelated to my work). I just
want to check that the DPDK works as expected - without regressions.

And what about the case when I have no such card (running automatic
tests on a build server)?

I don't like including the resources in binaries but I didn't find
any better way yet. I need to easily install the tests together with
the testing data (resources) to an embedded device and perform the
tests on the target system. When you are testing x86 code on your x86
you don't have to care too much about the location of resources. But I
do. And there is no standard way (at the moment) how to install the
resources together with the testing code.

What I like on this solution is that the DPDK git repository would
contain the testing data as a real file hierarchy (e.g. the
fake /sys/bus/...). After build and install of DPDK, the file hierachy
is packed (and compressed if needed) together with the tests. So moving
to the target platform works without any other changes to the
testing/install infrastructure. I could potentially run PCI tests on my
ARM board without any PCI available and don't have to ignore the
failures of those tests (as they simply pass).

Regards
Jan

> 
> /Bruce



-- 
  Jan Viktorin                E-mail: Viktorin@RehiveTech.com
  System Architect            Web:    www.RehiveTech.com
  RehiveTech
  Brno, Czech Republic