Test System Architecture ======================== Tenets ------ - **Keep overhead to a minimum**. Tests should be fast to both write and run. Ideally, the developer should _want_ to add tests early in the development cycle because they instinctively feel that that is the quickest route to arrive at a correct and robust implementation. - **Both physical and virtual hardware matters**. Infix is primarily deployed on physical hardware, so being able to run the test suite on real devices is crucial to guarantee a high quality product. At the same time, there is much value in running the same suite on virtual hardware, as it makes it easy to catch regressions early. It is also much more practical and economical to build large virtual networks than physical ones. - **Avoid CLI scripting & scraping**. Reliably interacting with a DUT over a serial line in a robust way is _very_ hard to get right. Given that we have a proper API (RESTCONF), we should leverage that when testing. Front-ends can be tested by other means. Overview -------- ![Infix Testing Architecture](img/testing-overview.svg) The test system is made up of several independent components, which are typically used in concert to run a full test suite. ### Test Cases A test case is an executable, receiving the physical topology as a positional argument, which produces [TAP][] compliant output on its `stdout`. I.e., it is executed in the following manner: test-case [OPTS] Test cases are typically written in Python, using the [Infamy](#infamy) library. Ultimately though, it can be implemented in any language, as long as it matches the calling convention above. ### Infamy Rather than having each test case come up with its own implementation of how to map topologies, how to push NETCONF data to a device, etc., we provide a library of functions to take care of all that, dubbed "Infamy". When adding a new test case, ask yourself if any parts of it might belong in Infamy as a generalized component that can be reused by other tests. Some of the core functions provided by Infamy are: - Mapping a logical topology to a physical one - Finding and attaching to a device over an Ethernet interface, using NETCONF - Pushing/pulling NETCONF data to/from a device - Generating TAP compliant output ### 9PM To run multiple tests, we employ [9PM][]. It let's us define test suites as simple YAML files. Suites can also be hierarchically structured, with a suite being made up of other suites, etc. It also validates the TAP output, making sure to catch early exits from a case, and produces a nice summary report. ### `/test/env` A good way to ensure that nobody ever runs the test suite is to make it _really_ hard to do so. `/test/env`'s job is instead to make it very _easy_ to create a reproducible environment in which tests can be executed. Several technologies are leveraged to accomplish this: - **Containers**: The entire execution is optionally done inside a standardized `docker` container environment. This ensures that the software needed to run the test suite is always available, no matter which distribution the user is running on their machine. - **Python Virtual Environments**: To make sure that the expected versions of all Python packages are available, the execution is wrapped inside a `venv`. This is true for containerized executions, where the container comes with a pre-installed environment, but it can also be sourced from the host system when running outside of the container. - **Virtual Test Topology**: Using [Qeneth][], the environment can optionally be started with a virtual topology of DUTs to run the tests on. > `docker` is the only supported container environment when running > tests in the host's network namespace. When running on a virtual > Qeneth topology, `podman` may also be used by installing the > `podman-docker` package from your host system's distro. Physical and Logical Topologies ------------------------------- Imagine that we want to create a test with three DUTs; one acting as a DHCP server, and the other two as DHCP clients - with all three having a management connection to the host PC running the test. In other words, the test requires a _logical_ topology like the one below. ![Example Logical Topology](img/testing-log.svg){ align=right width="360" } ```dot graph "dhcp-client-server" { host [ label="host | { c1 | srv | c2 }", kind="controller", ]; server [ label="{ mgmt } | server | { c1 | c2 }", kind="infix", ]; client1 [ label="{ mgmt } | client1 | { srv }", kind="infix", ]; client2 [ label="{ mgmt } | client2 | { srv }", kind="infix", ]; host:srv -- server:mgmt host:c1 -- client1:mgmt host:c2 -- client2:mgmt server:c1 -- client1:srv; server:c2 -- client2:srv; } ``` When running in a virtualized environment, one could simply create a setup that matches the test's logical topology. But in scenarios when devices are physical systems, connected by real copper cables, this is not possible (unless you have some wicked L1 relay matrix thingy). Instead, the test implementation does not concern itself with the exact nodes used to run the test, only that the _logical_ topology can be _mapped_ to some subset of the _physical_ topology. In mathematical terms, the physical topology must contain a subgraph that is _isomorphic_ to the logical topology. Standing on the shoulders of giants (i.e. people with mathematics degrees), we can deploy well-known algorithms to find such subgraphs. Continuing our example, let's say we want to run our DHCP test on the _physical_ topology below. ![Example Physical Topology](img/testing-phy.svg){ align=right width="360" } ```dot graph "quad-ring" { host [ label="host | { d1a | d1b | d1c | d2a | d2b | d2c | d3a | d3b | d3c | d4a | d4b | d4c }", kind="controller", ]; dut1 [ label="{ e1 | e2 | e3 } | dut1 | { e4 | e5 }", kind="infix", ]; dut2 [ label="{ e1 | e2 | e3 } | dut2 | { e4 | e5 }", kind="infix", ]; dut3 [ label="{ e1 | e2 | e3 } | dut3 | { e4 | e5 }", kind="infix", ]; dut4 [ label="{ e1 | e2 | e3 } | dut4 | { e4 | e5 }", kind="infix", ]; host:d1a -- dut1:e1 host:d1b -- dut1:e2 host:d1c -- dut1:e3 host:d2a -- dut2:e1 host:d2b -- dut2:e2 host:d2c -- dut2:e3 host:d3a -- dut3:e1 host:d3b -- dut3:e2 host:d3c -- dut3:e3 host:d4a -- dut4:e1 host:d4b -- dut4:e2 host:d4c -- dut4:e3 dut1:e5 -- dut2:e4 dut2:e5 -- dut3:e4 dut3:e5 -- dut4:e4 dut4:e5 -- dut1:e4 } ``` Our test (in fact, all tests) receives the physical topology as an input parameter, and then maps the desired logical topology onto it, producing a mapping from logical nodes and ports to their physical counterparts. ```dot { "client1": "dut1", "client1:mgmt": "dut1:e1", "client1:srv": "dut1:e4", "client2": "dut3", "client2:mgmt": "dut3:e2", "client2:srv": "dut3:e5", "host": "host", "host:c1": "host:d1a", "host:c2": "host:d3b", "host:srv": "host:d4c", "server": "dut4", "server:c1": "dut4:e5", "server:c2": "dut4:e4", "server:mgmt": "dut4:e3" } ``` With this information, the test knows that, in this particular environment, the server should be managed via the port called `d4c` on the node called `host`; that the port connected to the server on `client1` is `e4` on `dut1`, etc. Thereby separating the implementation of the test from any specific physical setup. Testcases are not required to use a logical topology; they may choose to accept whatever physical topology its given, and dynamically determine the DUTs to use for testing. As an example, an STP test could accept an arbitrary physical topology, run the STP algorithm on it offline, enable STP on all DUTs, and then verify that the resulting spanning tree matches the expected one. Integration to Infix -------------------- ### Test Mode Infix supports a *test mode* that makes devices safe and predictable to test against. When active, each boot starts from a known-good `test-config` (equivalent to the failure/secure config) instead of the stored startup or factory configuration. This prevents any persistent configuration from interfering with tests and avoids L2 loops that a factory config might create. It also enables two extra RPCs used by Infamy: - `infix-test:test/reset` — reload the `test-config` into the running datastore (called by `env.attach()` at the start of each test) - `infix-test:test/override-startup` — boot from the startup datastore once, instead of the `test-config` (used by tests that verify save-and-reboot behaviour) Test mode is activated by the presence of an empty file named `test-mode` on the device's `aux` partition (`/mnt/aux/test-mode`). ### Test Mode on Virtual Devices When the test environment is started with Qeneth, the `inject-test-mode` script (`test/inject-test-mode`) is run automatically. It copies the base disk image and writes the `test-mode` marker into the `aux` partition of that copy, which is then used as the Qeneth backing image. ### Test Mode on Physical Devices For a physical device, log in as `root` (or `admin`) and create the marker file on the `aux` partition, then reboot: ``` # touch /mnt/aux/test-mode # reboot ``` Alternatively, if you want to prepare the disk image before flashing, use the `inject-test-mode` script: ``` $ test/inject-test-mode -b infix-.img -o infix--test.img ``` Flash `infix--test.img` to the device instead of the plain image. The device will boot into test mode on first power-on. > [!NOTE] > Test mode persists across reboots until the marker file is removed > (`rm /mnt/aux/test-mode`). A device in test mode always starts from > the `test-config`, so any configuration changes are lost on reboot > unless `startup_override()` is called first (or the marker is removed > and a normal startup config is saved). [9PM]: https://github.com/rical/9pm [Qeneth]: https://github.com/wkz/qeneth [TAP]: https://testanything.org/