mirror of
https://github.com/kernelkit/infix.git
synced 2026-07-27 11:13:02 +02:00
Tobias Waldekranz
508e2287b8
Update docs to match the recent developments in the test system. Spin off the architecture description to its own document, and focus testing.md on the day-to-day usage.
262 lines
8.3 KiB
Markdown
262 lines
8.3 KiB
Markdown
Regression Testing with Infamy
|
|
==============================
|
|
|
|
Infix comes with a test suite that is intended to provide end-to-end
|
|
verification of supported features. Generally speaking, this means
|
|
that one or more DUTs are configured over NETCONF; the resulting
|
|
network is then black-box tested by injecting and inspecting network
|
|
traffic at various points.
|
|
|
|
This document is intended to be a practical guide on how to run,
|
|
develop and debug tests. There is a separate document describing the
|
|
[Test System Architecture](test-arch.md).
|
|
|
|
|
|
Modes of Testing
|
|
----------------
|
|
|
|
### Virtual Devices
|
|
|
|
By default, tests are run on a topology made up of virtual Infix nodes
|
|
using [Qeneth][]. To run the full regression test suite: build Infix
|
|
for `x86_64`:
|
|
|
|
$ make x86_64_defconfig
|
|
$ make
|
|
$ make test
|
|
|
|
### Physical Devices
|
|
|
|
To run the tests on a preexisting topology from the host's network
|
|
namespace, specify the `host` `TEST_MODE`:
|
|
|
|
$ make TEST_MODE=host test
|
|
|
|
This typically used when testing on physical hardware. By default the
|
|
topology will be sourced from `/etc/infamy.dot`, but this can be
|
|
overwritten by setting the `TOPOLOGY` variable:
|
|
|
|
$ make TEST_MODE=host TOPOLOGY=~/my-topology.dot test
|
|
|
|
|
|
### `make run` Devices
|
|
|
|
Some tests only require a single DUT. These can therefore be run
|
|
against an Infix image started from `make run`. This requires that the
|
|
instance is configured to use TAP networking.
|
|
|
|
When the instance is running, you can open a separate terminal and run
|
|
the subset of the test suite that can be mapped to it:
|
|
|
|
$ make TEST_MODE=run test
|
|
|
|
|
|
Interactive Usage
|
|
-----------------
|
|
|
|
When developing and debugging tests, the overhead of repeatedly
|
|
setting up and tearing down the test environment can quickly start to
|
|
weigh you down. In these situation, you can start an interactive test
|
|
environment:
|
|
|
|
$ make test-sh
|
|
Info: Generating topology
|
|
Info: Generating node YAML
|
|
Info: Generating executables
|
|
Info: Launching dut1
|
|
Info: Launching dut2
|
|
Info: Launching dut3
|
|
Info: Launching dut4
|
|
11:42:52 infamy0:test #
|
|
|
|
The example above uses the default mode (`qeneth`), but the `host` and
|
|
`run` modes also support the `test-sh` target.
|
|
|
|
It takes a little while to start up, but then we have a shell prompt
|
|
inside the container running Infamy. It's a very limited environment,
|
|
but it has enough to easily run single tests, connect to the virtual
|
|
devices, and *step* your code.
|
|
|
|
|
|
### Running Subsets of Tests
|
|
|
|
Each test case is a separate executable, which can be run without
|
|
arguments:
|
|
|
|
11:42:53 infamy0:test # ./case/infix_dhcp/dhcp_basic.py
|
|
|
|
To run a suite of tests, e.g., only the DHCP client tests, pass the
|
|
suite as an argument to [9PM][]:
|
|
|
|
11:42:53 infamy0:test # ./9pm/9pm.py case/infix_dhcp/all.yaml
|
|
|
|
|
|
### Connecting to Infamy
|
|
|
|
The test system runs in a Docker container, so to get a shell prompt in
|
|
*another terminal* you need to connect to that container. Infamy comes
|
|
with a helper script for this:
|
|
|
|
$ ./test/shell
|
|
11:42:53 infamy0:test #
|
|
|
|
By default it connect to the latest started Infamy instance. If you for
|
|
some reason run multiple instances of Infamy the `shell` script takes an
|
|
optional argument "system", which is the hostname of the container you
|
|
want to connect to:
|
|
|
|
$ ./test/shell infamy2
|
|
11:42:53 infamy2:test #
|
|
|
|
|
|
### Connecting to a DUT
|
|
|
|
All DUTs in a virtual Infamy topology are emulated in Qemu instances
|
|
managed by qeneth. If you want to watch what's happening on one of the
|
|
target systems, e.g., tail a log file or run `tcpdump` during the test,
|
|
there is another helper script in Infamy for this:
|
|
|
|
$ ./test/console 1
|
|
Trying 127.0.0.1...
|
|
Connected to 127.0.0.1.
|
|
|
|
Infix -- a Network Operating System v23.11.0-226-g0c144da (console)
|
|
infix-00-00-00 login: admin
|
|
Password:
|
|
.-------.
|
|
| . . | Infix -- a Network Operating System
|
|
|-. v .-| https://kernelkit.github.io
|
|
'-'---'-'
|
|
|
|
Run the command 'cli' for interactive OAM
|
|
|
|
admin@infix-00-00-00:~$
|
|
|
|
From here we can observe `dut1` freely while running tests.
|
|
|
|
> The `console` script uses `telnet` to connect to a port forwarded to
|
|
> `localhost` by the Docker container. To exit Telnet, use Ctrl-] and
|
|
> then 'q' followed by enter. This can be customized in `~/.telnetrc`
|
|
|
|
Like the `shell` script, `console` takes an optional "system" argument
|
|
in case you run multiple instances of Infamy:
|
|
|
|
$ ./test/console 1 infamy2
|
|
|
|
You can also connect to the console of a DUT from within a `shell`:
|
|
|
|
$ ./test/shell
|
|
11:42:54 infamy0:test # qeneth status
|
|
11:42:54 infamy0:test # qeneth console dut1
|
|
login: admin
|
|
password: *****
|
|
admin@infix-00-00-00:~$
|
|
|
|
> **Note:** disconnect from the qeneth console by using bringing up the
|
|
> old Telnet "menu" using Ctrl-], compared to standard Telnet this is
|
|
> the BusyBox version so you press 'e' + enter instead of 'q' to quit.
|
|
|
|
|
|
### Attaching to MACVLANs
|
|
|
|
To fully isolate the host's interfaces from one another, many tests
|
|
will stack a MACVLAN on an interface, which is then placed in a
|
|
separate network namespace.
|
|
|
|
It is often useful to attach to those namespaces, so that you can
|
|
interactively inject traffic into the test setup.
|
|
|
|
First, connect to the Infamy instance in question:
|
|
|
|
$ ./test/shell
|
|
11:43:19 infamy0:test #
|
|
|
|
Then, attach to the MACVLAN namespace by running the `nsenter` helper
|
|
script from the test directory, supplying the base interface name as
|
|
the first argument:
|
|
|
|
11:43:19 infamy0:test # ./nsenter d1b
|
|
11:43:20 infamy0(d1b):test #
|
|
|
|
By default, an interactive shell is started, but you can also supply
|
|
another command:
|
|
|
|
11:43:19 infamy0:test # ./nsenter d1b ip -br addr
|
|
lo UNKNOWN 127.0.0.1/8 ::1/128
|
|
iface@if7 UP 10.0.0.1/24 fe80::38d0:88ff:fe77:b7cd/64
|
|
|
|
You can now freely debug the network activity of your test and the
|
|
responses from the DUT.
|
|
|
|
|
|
### Using the Python Debugger
|
|
|
|
The built in `breakpoint()` function in Python is very useful when you
|
|
want to run a test case to a certain point at which you might want to
|
|
interactively inspect either the test's or the device's state.
|
|
|
|
Simply insert a call to `breakpoint()` at the point of interest in
|
|
your test and run it as normal. Once Python executes the call, it will
|
|
drop you into the Python debugger:
|
|
|
|
11:42:58 infamy0:test # ./case/infix_dhcp/dhcp_basic.py
|
|
# Starting (2024-02-10 11:42:59)
|
|
# Probing dut1 on port d1a for IPv6LL mgmt address ...
|
|
# Connecting to mgmt IP fe80::ff:fe00:0%d1a:830 ...
|
|
ok 1 - Initialize
|
|
> /home/jocke/src/infix/test/case/infix_dhcp/dhcp_basic.py(44)<module>()
|
|
(Pdb)
|
|
|
|
At this point you have full access to the test's state, but it is also
|
|
an opportunity to inspect the state of the DUTs (e.g. via their
|
|
console or over SSH).
|
|
|
|
It is also possible to run a test under Pdb from the get-go, if you
|
|
want to setup breakpoints without modifying the source, or simply step
|
|
through the code:
|
|
|
|
11:42:58 infamy0:test # python -m pdb case/infix_dhcp/dhcp_basic.py
|
|
|
|
|
|
### Deterministic Topology Mappings
|
|
|
|
By default, mappings from logical to physical topologies are not
|
|
stable across test case executions. This can be very frustrating when
|
|
debugging a failing test, since logical nodes are suffled around
|
|
between phyical nodes. In such cases, supplying a `PYTHONHASHSEED`
|
|
variable (set to any 32-bit unsigned integer) when launching the test
|
|
environment will make sure that topology mappings are deterministic:
|
|
|
|
$ make PYTHONHASHSEED=0 test-sh
|
|
|
|
If a seed is not supplied, a random value is chosen. This seed is
|
|
logged by the `meta/reproducible.py` test case when running a test
|
|
suite:
|
|
|
|
$ make test
|
|
Info: Generating topology
|
|
Info: Generating node YAML
|
|
Info: Generating executables
|
|
Info: Launching dut1
|
|
Info: Launching dut2
|
|
Info: Launching dut3
|
|
Info: Launching dut4
|
|
9PM - Simplicity is the ultimate sophistication
|
|
|
|
Starting test 0002-reproducible.py
|
|
2024-05-03 10:40:30 # Starting (2024-05-03 10:40:30)
|
|
2024-05-03 10:40:30 # Specify PYTHONHASHSEED=3773822171 to reproduce this test environment
|
|
2024-05-03 10:40:30 ok 1 - $PYTHONHASHSEED is set
|
|
2024-05-03 10:40:30 # Exiting (2024-05-03 10:40:30)
|
|
2024-05-03 10:40:30 1..1
|
|
...
|
|
|
|
This is useful because this value can then be used to rerun a test (or
|
|
the whole suite) with identical topology mappings:
|
|
|
|
$ make PYTHONHASHSEED=3773822171 INFIX_TESTS=case/ietf_system/hostname.py test
|
|
|
|
|
|
[9PM]: https://github.com/rical/9pm
|
|
[Qeneth]: https://github.com/wkz/qeneth
|