diff --git a/doc/test-arch.md b/doc/test-arch.md new file mode 100644 index 00000000..ad0ff14c --- /dev/null +++ b/doc/test-arch.md @@ -0,0 +1,246 @@ +Test System Architecture +======================== + +Tenets +------ + +- **Keep overhead to a minimum**. Tests should be fast to both write + and run. Ideally, the developer should _want_ to add tests early in + the development cycle because they instinctively feel that that is + the quickest route to arrive at a correct and robust implementation. + +- **Both physical and virtual hardware matters**. Infix is primarily + deployed on physical hardware, so being able to run the test suite + on real devices is crucial to guarantee a high quality product. At + the same time, there is much value in running the same suite on + virtual hardware, as it makes it easy to catch regressions early. + It is also much more practical and economical to build large virtual + networks than physical ones. + +- **Avoid CLI scipting & scraping**. Reliably interacting with a DUT + over a serial line in a robust way is _very_ hard to get right. + Given that we have a proper API (RESTCONF), we should leverage that + when testing. Front-ends can be tested by other means. + + +Overview +-------- + +![Infix Testing Architecture](img/testing-overview.svg) + +The test system is made up of several independent components, which +are typically used in concert to run a full test suite. + +### Test Cases + +A test case is an executable, receiving the physical topology as a +positional argument, which produces [TAP][] compliant output on its +`stdout`. I.e., it is executed in the following manner: + + test-case [OPTS] + +Test cases are typically written in Python, using the +[Infamy](#infamy) library. Ultimately though, it can be implemented +in any language, as long as it matches the calling convention above. + +### Infamy + +Rather than having each test case come up with its own implementation +of how to map topologies, how to push NETCONF data to a device, etc., +we provide a library of functions to take care of all that, dubbed +"Infamy". When adding a new test case, ask yourself if any parts of +it might belong in Infamy as a generalized component that can be +reused by other tests. + +Some of the core functions provided by Infamy are: + +- Mapping a logical topology to a physical one +- Finding and attaching to a device over an Ethernet interface, using + NETCONF +- Pushing/pulling NETCONF data to/from a device +- Generating TAP compliant output + +### 9PM + +To run multiple tests, we employ [9PM][]. It let's us define test +suites as simple YAML files. Suites can also be hierarchically +structured, with a suite being made up of other suites, etc. + +It also validates the TAP output, making sure to catch early exits +from a case, and produces a nice summary report. + +### `/test/env` + +A good way to ensure that nobody ever runs the test suite is to make +it _really_ hard to do so. `/test/env`'s job is instead to make it +very _easy_ to create a reproducible environment in which tests can be +executed. + +Several technologies are leveraged to accomplish this: + +- **Containers**: The entire execution is optionally done inside a + standardized `docker` container environment. This ensures that the + software needed to run the test suite is always available, no matter + which distribution the user is running on their machine. + +- **Python Virtual Environments**: To make sure that the expected + versions of all Python packages are available, the execution is + wrapped inside a `venv`. This is true for containerized executions, + where the container comes with a pre-installed environment, but it + can also be sourced from the host system when running outside of the + container. + +- **Virtual Test Topology**: Using [Qeneth][], the environment can + optionally be started with a virtual topology of DUTs to run the + tests on. + +> `docker` is the only supported container environment when running +> tests in the host's network namespace. When running on a virtual +> Qeneth topology, `podman` may also be used by installing the +> `podman-docker` package from your host system's distro. + + +Physical and Logical Topologies +------------------------------- + +Imagine that we want to create a test with three DUTs; one acting as a +DHCP server, and the other two as DHCP clients - with all three having +a management connection to the host PC running the test. In other +words, the test requires a _logical_ topology like the one below. + +Example Logical Topology + +```dot +graph "dhcp-client-server" { + host [ + label="host | { c1 | srv | c2 }", + kind="controller", + ]; + + server [ + label="{ mgmt } | server | { c1 | c2 }", + kind="infix", + ]; + client1 [ + label="{ mgmt } | client1 | { srv }", + kind="infix", + ]; + client2 [ + label="{ mgmt } | client2 | { srv }", + kind="infix", + ]; + + host:srv -- server:mgmt + host:c1 -- client1:mgmt + host:c2 -- client2:mgmt + + server:c1 -- client1:srv; + server:c2 -- client2:srv; +} +``` + +When running in a virtualized environment, one could simply create a +setup that matches the test's logical topology. But in scenarios when +devices are physical systems, connected by real copper cables, this is +not possible (unless you have some wicked L1 relay matrix thingy). + +Instead, the test implementation does not concern itself with the +exact nodes used to run the test, only that the _logical_ topology can +be _mapped_ to some subset of the _physical_ topology. In +mathematical terms, the physical topology must contain a subgraph that +is _isomorphic_ to the logical topology. + +Standing on the shoulders of giants (i.e. people with mathematics +degrees), we can deploy well-known algorithms to find such subgraphs. +Continuing our example, let's say we want to run our DHCP test on the +_physical_ topology below. + +Example Physical Topology + +```dot +graph "quad-ring" { + host [ + label="host | { d1a | d1b | d1c | d2a | d2b | d2c | d3a | d3b | d3c | d4a | d4b | d4c }", + kind="controller", + ]; + + dut1 [ + label="{ e1 | e2 | e3 } | dut1 | { e4 | e5 }", + kind="infix", + ]; + dut2 [ + label="{ e1 | e2 | e3 } | dut2 | { e4 | e5 }", + kind="infix", + ]; + dut3 [ + label="{ e1 | e2 | e3 } | dut3 | { e4 | e5 }", + kind="infix", + ]; + dut4 [ + label="{ e1 | e2 | e3 } | dut4 | { e4 | e5 }", + kind="infix", + ]; + + host:d1a -- dut1:e1 + host:d1b -- dut1:e2 + host:d1c -- dut1:e3 + + host:d2a -- dut2:e1 + host:d2b -- dut2:e2 + host:d2c -- dut2:e3 + + host:d3a -- dut3:e1 + host:d3b -- dut3:e2 + host:d3c -- dut3:e3 + + host:d4a -- dut4:e1 + host:d4b -- dut4:e2 + host:d4c -- dut4:e3 + + dut1:e5 -- dut2:e4 + dut2:e5 -- dut3:e4 + dut3:e5 -- dut4:e4 + dut4:e5 -- dut1:e4 +} +``` + +Our test (in fact, all tests) receives the physical topology as an +input parameter, and then maps the desired logical topology onto it, +producing a mapping from logical nodes and ports to their physical +counterparts. + +```dot +{ + "client1": "dut1", + "client1:mgmt": "dut1:e1", + "client1:srv": "dut1:e4", + "client2": "dut3", + "client2:mgmt": "dut3:e2", + "client2:srv": "dut3:e5", + "host": "host", + "host:c1": "host:d1a", + "host:c2": "host:d3b", + "host:srv": "host:d4c", + "server": "dut4", + "server:c1": "dut4:e5", + "server:c2": "dut4:e4", + "server:mgmt": "dut4:e3" +} +``` + +With this information, the test knows that, in this particular +environment, the server should be managed via the port called `d4c` on +the node called `host`; that the port connected to the server on +`client1` is `e4` on `dut1`, etc. Thereby separating the +implementation of the test from any specific physical setup. + +Testcases are not required to use a logical topology; they may choose +to accept whatever physical topology its given, and dynamically +determine the DUTs to use for testing. As an example, an STP test +could accept an arbitrary physical topology, run the STP algorithm on +it offline, enable STP on all DUTs, and then verify that the resulting +spanning tree matches the expected one. + +[9PM]: https://github.com/rical/9pm +[Qeneth]: https://github.com/wkz/qeneth +[TAP]: https://testanything.org/ diff --git a/doc/testing.md b/doc/testing.md index ecd56a65..bf827436 100644 --- a/doc/testing.md +++ b/doc/testing.md @@ -7,290 +7,57 @@ that one or more DUTs are configured over NETCONF; the resulting network is then black-box tested by injecting and inspecting network traffic at various points. +This document is intended to be a practical guide on how to run, +develop and debug tests. There is a separate document describing the +[Test System Architecture](test-arch.md). -TL;DR ------ -Build Infix and run the test suite on a set of virtual Infix nodes. +Modes of Testing +---------------- + +### Virtual Devices + +By default, tests are run on a topology made up of virtual Infix nodes +using [Qeneth][]. To run the full regression test suite: build Infix +for `x86_64`: $ make x86_64_defconfig $ make $ make test -To run a subset of tests, e.g., only the DHCP client tests: +### Physical Devices - $ make test INFIX_TESTS=case/infix_dhcp/all.yaml +To run the tests on a preexisting topology from the host's network +namespace, specify the `host` `TEST_MODE`: -> **Note:** see the below section [Quick Start Guide][] for how to -> connect to the test system, debug and develop tests, and more. + $ make TEST_MODE=host test + +This typically used when testing on physical hardware. By default the +topology will be sourced from `/etc/infamy.dot`, but this can be +overwritten by setting the `TOPOLOGY` variable: + + $ make TEST_MODE=host TOPOLOGY=~/my-topology.dot test -Tenets ------- +### `make run` Devices -- **Keep overhead to a minimum**. Tests should be fast to both write - and run. Ideally, the developer should _want_ to add tests early in - the development cycle because they instinctively feel that that is - the quickest route to arrive at a correct and robust implementation. +Some tests only require a single DUT. These can therefore be run +against an Infix image started from `make run`. This requires that the +instance is configured to use TAP networking. -- **Both physical and virtual hardware matters**. Infix is primarily - deployed on physical hardware, so being able to run the test suite - on real devices is crucial to guarantee a high quality product. At - the same time, there is much value in running the same suite on - virtual hardware, as it makes it easy to catch regressions early. - It is also much more practical and economical to build large virtual - networks than physical ones. +When the instance is running, you can open a separate terminal and run +the subset of the test suite that can be mapped to it: -- **Avoid CLI scipting & scraping**. Reliably interacting with a DUT - over a serial line in a robust way is _very_ hard to get right. - Given that we have a proper API (RESTCONF), we should leverage that - when testing. Front-ends can be tested by other means. + $ make TEST_MODE=run test -Architectural Overview ----------------------- - -![Infix Testing Architecture](img/testing-overview.svg) - -The test system is made up of several independent components, which -are typically used in concert to run a full test suite. - -### Test Cases - -A test case is an executable, receiving the physical topology as a -positional argument, which produces [TAP][] compliant output on its -`stdout`. I.e., it is executed in the following manner: - - test-case [OPTS] - -Test cases are typically written in Python, using the -[Infamy](#infamy) library. Ultimately though, it can be implemented -in any language, as long as it matches the calling convention above. - -### Infamy - -Rather than having each test case come up with its own implementation -of how to map topologies, how to push NETCONF data to a device, etc., -we provide a library of functions to take care of all that, dubbed -"Infamy". When adding a new test case, ask yourself if any parts of -it might belong in Infamy as a generalized component that can be -reused by other tests. - -Some of the core functions provided by Infamy are: - -- Mapping a logical topology to a physical one -- Finding and attaching to a device over an Ethernet interface, using - NETCONF -- Pushing/pulling NETCONF data to/from a device -- Generating TAP compliant output - -### 9PM - -To run multiple tests, we employ [9PM][]. It let's us define test -suites as simple YAML files. Suites can also be hierarchically -structured, with a suite being made up of other suites, etc. - -It also validates the TAP output, making sure to catch early exits -from a case, and produces a nice summary report. - -### `/test/env` - -A good way to ensure that nobody ever runs the test suite is to make -it _really_ hard to do so. `/test/env`'s job is instead to make it -very _easy_ to create a reproducible environment in which tests can be -executed. - -Several technologies are leveraged to accomplish this: - -- **Containers**: The entire execution is optionally done inside a - standardized `docker` container environment. This ensures that the - software needed to run the test suite is always available, no matter - which distribution the user is running on their machine. - -- **Python Virtual Environments**: To make sure that the expected - versions of all Python packages are available, the execution is - wrapped inside a `venv`. This is true for containerized executions, - where the container comes with a pre-installed environment, but it - can also be sourced from the host system when running outside of the - container. - -- **Virtual Test Topology**: Using [Qeneth][], the environment can - optionally be started with a virtual topology of DUTs to run the - tests on. - -> `docker` is the only supported container environment when running -> tests in the host's network namespace. When running on a virtual -> Qeneth topology, `podman` may also be used by installing the -> `podman-docker` package from your host system's distro. - Interactive Usage ----------------- -Some tests only require a single DUT. These can therefore be run -against an Infix image started from `make run`. When the instance is -running, you can open a separate terminal and run `make test-run`, to -run the subset of the test suite that can be mapped to it. - -Both `make test` and `make test-run` targets have a respective target -with a `-sh` suffix. These can be used to start an interactive session -in the reproducible environment, which is usually much easier to work -with during a debugging session. - -Inside the reproducible environment, a wrapper for Qeneth is created for -the running network's directory. E.g., calling `qeneth status` inside a -`make test-sh` environment show the expected status information. - -> **Note:** for more information and help writing a test, see the below -> [Quick Start Guide][]. - - -Physical and Logical Topologies -------------------------------- - -Imagine that we want to create a test with three DUTs; one acting as a -DHCP server, and the other two as DHCP clients - with all three having -a management connection to the host PC running the test. In other -words, the test requires a _logical_ topology like the one below. - -Example Logical Topology - -```dot -graph "dhcp-client-server" { - host [ - label="host | { c1 | srv | c2 }", - kind="controller", - ]; - - server [ - label="{ mgmt } | server | { c1 | c2 }", - kind="infix", - ]; - client1 [ - label="{ mgmt } | client1 | { srv }", - kind="infix", - ]; - client2 [ - label="{ mgmt } | client2 | { srv }", - kind="infix", - ]; - - host:srv -- server:mgmt - host:c1 -- client1:mgmt - host:c2 -- client2:mgmt - - server:c1 -- client1:srv; - server:c2 -- client2:srv; -} -``` - -When running in a virtualized environment, one could simply create a -setup that matches the test's logical topology. But in scenarios when -devices are physical systems, connected by real copper cables, this is -not possible (unless you have some wicked L1 relay matrix thingy). - -Instead, the test implementation does not concern itself with the -exact nodes used to run the test, only that the _logical_ topology can -be _mapped_ to some subset of the _physical_ topology. In -mathematical terms, the physical topology must contain a subgraph that -is _isomorphic_ to the logical topology. - -Standing on the shoulders of giants (i.e. people with mathematics -degrees), we can deploy well-known algorithms to find such subgraphs. -Continuing our example, let's say we want to run our DHCP test on the -_physical_ topology below. - -Example Physical Topology - -```dot -graph "quad-ring" { - host [ - label="host | { d1a | d1b | d1c | d2a | d2b | d2c | d3a | d3b | d3c | d4a | d4b | d4c }", - kind="controller", - ]; - - dut1 [ - label="{ e1 | e2 | e3 } | dut1 | { e4 | e5 }", - kind="infix", - ]; - dut2 [ - label="{ e1 | e2 | e3 } | dut2 | { e4 | e5 }", - kind="infix", - ]; - dut3 [ - label="{ e1 | e2 | e3 } | dut3 | { e4 | e5 }", - kind="infix", - ]; - dut4 [ - label="{ e1 | e2 | e3 } | dut4 | { e4 | e5 }", - kind="infix", - ]; - - host:d1a -- dut1:e1 - host:d1b -- dut1:e2 - host:d1c -- dut1:e3 - - host:d2a -- dut2:e1 - host:d2b -- dut2:e2 - host:d2c -- dut2:e3 - - host:d3a -- dut3:e1 - host:d3b -- dut3:e2 - host:d3c -- dut3:e3 - - host:d4a -- dut4:e1 - host:d4b -- dut4:e2 - host:d4c -- dut4:e3 - - dut1:e5 -- dut2:e4 - dut2:e5 -- dut3:e4 - dut3:e5 -- dut4:e4 - dut4:e5 -- dut1:e4 -} -``` - -Our test (in fact, all tests) receives the physical topology as an -input parameter, and then maps the desired logical topology onto it, -producing a mapping from logical nodes and ports to their physical -counterparts. - -```dot -{ - "client1": "dut1", - "client1:mgmt": "dut1:e1", - "client1:srv": "dut1:e4", - "client2": "dut3", - "client2:mgmt": "dut3:e2", - "client2:srv": "dut3:e5", - "host": "host", - "host:c1": "host:d1a", - "host:c2": "host:d3b", - "host:srv": "host:d4c", - "server": "dut4", - "server:c1": "dut4:e5", - "server:c2": "dut4:e4", - "server:mgmt": "dut4:e3" -} -``` - -With this information, the test knows that, in this particular -environment, the server should be managed via the port called `d4c` on -the node called `host`; that the port connected to the server on -`client1` is `e4` on `dut1`, etc. Thereby separating the -implementation of the test from any specific physical setup. - -Testcases are not required to use a logical topology; they may choose -to accept whatever physical topology its given, and dynamically -determine the DUTs to use for testing. As an example, an STP test -could accept an arbitrary physical topology, run the STP algorithm on -it offline, enable STP on all DUTs, and then verify that the resulting -spanning tree matches the expected one. - - -Quick Start Guide ------------------ - -When developing a test, instead of blindly coding in Python and running -`make test` over and over, start a test shell: +When developing and debugging tests, the overhead of repeatedly +setting up and tearing down the test environment can quickly start to +weigh you down. In these situation, you can start an interactive test +environment: $ make test-sh Info: Generating topology @@ -302,13 +69,28 @@ When developing a test, instead of blindly coding in Python and running Info: Launching dut4 11:42:52 infamy0:test # +The example above uses the default mode (`qeneth`), but the `host` and +`run` modes also support the `test-sh` target. + It takes a little while to start up, but then we have a shell prompt inside the container running Infamy. It's a very limited environment, but it has enough to easily run single tests, connect to the virtual -devices, and *step* your code. Let's run a test: +devices, and *step* your code. + + +### Running Subsets of Tests + +Each test case is a separate executable, which can be run without +arguments: 11:42:53 infamy0:test # ./case/infix_dhcp/dhcp_basic.py +To run a suite of tests, e.g., only the DHCP client tests, pass the +suite as an argument to [9PM][]: + + 11:42:53 infamy0:test # ./9pm/9pm.py case/infix_dhcp/all.yaml + + ### Connecting to Infamy The test system runs in a Docker container, so to get a shell prompt in @@ -316,6 +98,7 @@ The test system runs in a Docker container, so to get a shell prompt in with a helper script for this: $ ./test/shell + 11:42:53 infamy0:test # By default it connect to the latest started Infamy instance. If you for some reason run multiple instances of Infamy the `shell` script takes an @@ -323,6 +106,8 @@ optional argument "system", which is the hostname of the container you want to connect to: $ ./test/shell infamy2 + 11:42:53 infamy2:test # + ### Connecting to a DUT @@ -372,10 +157,47 @@ You can also connect to the console of a DUT from within a `shell`: > the BusyBox version so you press 'e' + enter instead of 'q' to quit. -### Debug a Test +### Attaching to MACVLANs -First, add a Python `breakpoint()` to your test and run it from your -`make test-sh` Infamy instance: +To fully isolate the host's interfaces from one another, many tests +will stack a MACVLAN on an interface, which is then placed in a +separate network namespace. + +It is often useful to attach to those namespaces, so that you can +interactively inject traffic into the test setup. + +First, connect to the Infamy instance in question: + + $ ./test/shell + 11:43:19 infamy0:test # + +Then, attach to the MACVLAN namespace by running the `nsenter` helper +script from the test directory, supplying the base interface name as +the first argument: + + 11:43:19 infamy0:test # ./nsenter d1b + 11:43:20 infamy0(d1b):test # + +By default, an interactive shell is started, but you can also supply +another command: + + 11:43:19 infamy0:test # ./nsenter d1b ip -br addr + lo UNKNOWN 127.0.0.1/8 ::1/128 + iface@if7 UP 10.0.0.1/24 fe80::38d0:88ff:fe77:b7cd/64 + +You can now freely debug the network activity of your test and the +responses from the DUT. + + +### Using the Python Debugger + +The built in `breakpoint()` function in Python is very useful when you +want to run a test case to a certain point at which you might want to +interactively inspect either the test's or the device's state. + +Simply insert a call to `breakpoint()` at the point of interest in +your test and run it as normal. Once Python executes the call, it will +drop you into the Python debugger: 11:42:58 infamy0:test # ./case/infix_dhcp/dhcp_basic.py # Starting (2024-02-10 11:42:59) @@ -385,32 +207,15 @@ First, add a Python `breakpoint()` to your test and run it from your > /home/jocke/src/infix/test/case/infix_dhcp/dhcp_basic.py(44)() (Pdb) -You are now in the Python debugger, Pdb. +At this point you have full access to the test's state, but it is also +an opportunity to inspect the state of the DUTs (e.g. via their +console or over SSH). -### Debug a Test in a MacVLAN +It is also possible to run a test under Pdb from the get-go, if you +want to setup breakpoints without modifying the source, or simply step +through the code: -Most tests use some sort of network connection to the DUTs. From a test -shell you cannot see or debug that connection by default. So you need -to employ a little trick. - -Open another terminal, and start a test shell in your Infamy instance: - - $ ./test/shell - 11:43:19 infamy0:test # - -MacVLAN interfaces created by Infamy run in another network namespace, -to enter it we can look for a sleeper process: - - 11:43:20 infamy0:test # nsenter -n U -t $(pidof sleep infinity) - -You are now one step further down: - - infamy0:/home/jocke/src/infix/test# ip -br a - lo UNKNOWN 127.0.0.1/8 ::1/128 - iface@if7 UP 10.0.0.1/24 fe80::38d0:88ff:fe77:b7cd/64 - -You can now freely debug the network activity of your test and the -responses from the DUT. + 11:42:58 infamy0:test # python -m pdb case/infix_dhcp/dhcp_basic.py ### Deterministic Topology Mappings @@ -454,4 +259,3 @@ the whole suite) with identical topology mappings: [9PM]: https://github.com/rical/9pm [Qeneth]: https://github.com/wkz/qeneth -[TAP]: https://testanything.org/