Merge pull request #702 from kernelkit/jovatn/test-spec-minor-fixes

Jovatn/test spec minor fixes
This commit is contained in:
jovatn
2024-10-14 08:12:20 +02:00
committed by GitHub
35 changed files with 165 additions and 93 deletions
+7 -6
View File
@@ -1,12 +1,13 @@
=== USB configuration
==== Description
This will test if the configuration is consistent with hardware state,
if the USB ports are _locked_ (forbidden for use) and _unlocked_
(Ready for use) when they should. Also test if it is the case during
reboot. This does not the test actual use of the USB port, just check
that the configured state is consistent with the hardware state.
This test checks if the configuration is consistent with hardware state,
and verifies whether the USB ports are correctly _locked_ (restricted from
use) and _unlocked_ (available for use) when they should. It also verifies
this behavior during reboot. This test does not involve the actual use of
the USB port; it only ensures the configured state is consistent with the
hardware state.
If this pass you can be certeain of that the configuration of the USB
If this pass you can be certain that the configuration of the USB
port is handled correctly.
==== Topology
+9 -8
View File
@@ -1,15 +1,16 @@
#!/usr/bin/env python3
"""
USB configuration
"""USB configuration
This will test if the configuration is consistent with hardware state,
if the USB ports are _locked_ (forbidden for use) and _unlocked_
(Ready for use) when they should. Also test if it is the case during
reboot. This does not the test actual use of the USB port, just check
that the configured state is consistent with the hardware state.
This test checks if the configuration is consistent with hardware state,
and verifies whether the USB ports are correctly _locked_ (restricted from
use) and _unlocked_ (available for use) when they should. It also verifies
this behavior during reboot. This test does not involve the actual use of
the USB port; it only ensures the configured state is consistent with the
hardware state.
If this pass you can be certeain of that the configuration of the USB
If this pass you can be certain that the configuration of the USB
port is handled correctly.
"""
import infamy
import copy
@@ -1,6 +1,6 @@
=== USB configuration with two USB ports
==== Description
Test that the configuration is consistent
Test that the USB locked/unlocked configuration is consistent
when having two USB ports.
==== Topology
@@ -2,7 +2,7 @@
"""
USB configuration with two USB ports
Test that the configuration is consistent
Test that the USB locked/unlocked configuration is consistent
when having two USB ports.
"""
import infamy
@@ -1,6 +1,6 @@
=== Bridge forwarding dual DUTs
==== Description
Ping through two bridges on two different DUTs,
Ping through two bridges on two different DUTs.
==== Topology
ifdef::topdoc[]
@@ -17,7 +17,7 @@
"""
Bridge forwarding dual DUTs
Ping through two bridges on two different DUTs,
Ping through two bridges on two different DUTs.
"""
import infamy
@@ -1,6 +1,6 @@
=== Bridge VLAN separation
==== Description
Test that two VLAN is correctly separated in the bridge
Test that two VLANs are correctly separated in the bridge
==== Topology
ifdef::topdoc[]
@@ -17,8 +17,8 @@ endif::topdoc[]
==== Test sequence
. Initialize
. Configure DUTs
. Verify ing 10.0.0.3 from host:data10
. Verify ing 10.0.0.4 from host:data11
. Verify ping 10.0.0.3 from host:data10
. Verify ping 10.0.0.4 from host:data11
. Verify ping not possible host:data10->10.0.0.4, host:data11->10.0.0.3, host:data10->10.0.0.2, host:data11->10.0.0.1
@@ -17,7 +17,7 @@
"""
Bridge VLAN separation
Test that two VLAN is correctly separated in the bridge
Test that two VLANs are correctly separated in the bridge
"""
import infamy
@@ -153,10 +153,10 @@ with infamy.Test() as test:
ns20.addip("10.0.0.3")
ns21.addip("10.0.0.4")
with test.step("Verify ing 10.0.0.3 from host:data10"):
with test.step("Verify ping 10.0.0.3 from host:data10"):
ns10.must_reach("10.0.0.3")
with test.step("Verify ing 10.0.0.4 from host:data11"):
with test.step("Verify ping 10.0.0.4 from host:data11"):
ns11.must_reach("10.0.0.4")
with test.step("Verify ping not possible host:data10->10.0.0.4, host:data11->10.0.0.3, host:data10->10.0.0.2, host:data11->10.0.0.1"):
@@ -1,6 +1,6 @@
=== Dual bridges
==== Description
Test that it is possible to ping a bridge through a bridge to another bridge with VETH interfaces.
Verify that it is possible to ping through a bridge to another bridge via VETH interfaces.
==== Topology
ifdef::topdoc[]
@@ -7,7 +7,7 @@
"""
Dual bridges
Test that it is possible to ping a bridge through a bridge to another bridge with VETH interfaces.
Verify that it is possible to ping through a bridge to another bridge via VETH interfaces.
"""
import infamy
@@ -1,6 +1,6 @@
=== Interface with IPv4
==== Description
Test that it is possible to set and remove the IPv4 address on a interface
Test that it is possible to set and remove the IPv4 address on an interface
==== Topology
ifdef::topdoc[]
@@ -17,7 +17,7 @@ endif::topdoc[]
==== Test sequence
. Set up topology and attach to target DUT
. Configure IPv4 address 10.10.10.20/24 on target:mgmt
. Verify '10.10.10.20/24' exist on target:mgmt
. Verify '10.10.10.20/24' exists on target:mgmt
. Remove all IPv4 addresses from target:mgmt
. Verify target:mgmt no longer has the address 10.10.10.20
@@ -2,7 +2,7 @@
"""
Interface with IPv4
Test that it is possible to set and remove the IPv4 address on a interface
Test that it is possible to set and remove the IPv4 address on an interface
"""
import copy
import infamy
@@ -39,7 +39,7 @@ with infamy.Test() as test:
target.put_config_dict("ietf-interfaces", config)
with test.step("Verify '10.10.10.20/24' exist on target:mgmt"):
with test.step("Verify '10.10.10.20/24' exists on target:mgmt"):
until(lambda: iface.address_exist(target, interface_name, new_ip_address, proto='static'))
with test.step("Remove all IPv4 addresses from target:mgmt"):
@@ -19,10 +19,10 @@ endif::topdoc[]
. Initialize
. Setup host
. Enable forwarding on target:data0 and target:data1
. Verify ping from host:data0 to 10.0.0.1
. Verify ping from host:data0 to 10.0.0.10
. Verify ping from host:data1 to 192.168.0.10
. Disable forwarding on target:data0 and target:data1
. Verfify ping does not work host:data0->10.0.0.10 and host:data1->192.168.0.10
. Verify ping does not work host:data0 to 10.0.0.10 and host:data1 to 192.168.0.10
<<<
@@ -77,7 +77,7 @@ with infamy.Test() as test:
with test.step("Enable forwarding on target:data0 and target:data1"):
config_target(target, tport0, tport1, True)
with test.step("Verify ping from host:data0 to 10.0.0.1"):
with test.step("Verify ping from host:data0 to 10.0.0.10"):
ns0.must_reach("10.0.0.10")
with test.step("Verify ping from host:data1 to 192.168.0.10"):
@@ -86,7 +86,7 @@ with infamy.Test() as test:
with test.step("Disable forwarding on target:data0 and target:data1"):
config_target(target, tport0, tport1, False)
with test.step("Verfify ping does not work host:data0->10.0.0.10 and host:data1->192.168.0.10"):
with test.step("Verify ping does not work host:data0 to 10.0.0.10 and host:data1 to 192.168.0.10"):
infamy.parallel(lambda: ns0.must_not_reach("10.0.0.10"),
lambda: ns1.must_not_reach("192.168.0.10"))
@@ -0,0 +1,44 @@
=== Verify that all interface types can be created
==== Description
This test verifies that all interface types can be created and also
checks the configuration when applied sequentially. This method takes
slightly longer than sending the entire configuration at once.
....
lo br-0 br-Q.40 br-D br-X
| | | | |
o o ethQ.10 br-Q veth0a.20 ethX.30
\ / \ | |
ethQ veth0b veth0a ethX
`---------'
....
==== Topology
ifdef::topdoc[]
image::../../test/case/ietf_interfaces/verify_all_interface_types/topology.png[Verify that all interface types can be created topology]
endif::topdoc[]
ifndef::topdoc[]
ifdef::testgroup[]
image::verify_all_interface_types/topology.png[Verify that all interface types can be created topology]
endif::testgroup[]
ifndef::testgroup[]
image::topology.png[Verify that all interface types can be created topology]
endif::testgroup[]
endif::topdoc[]
==== Test sequence
. Initialize
. Configure an empty bridge br-0
. Configure bridge br-X and associated interfaces
. Configure VETH pair
. Configure bridge br-D and associated interfaces
. Configure br-Q and associated interfaces
. Verify interface 'lo' is of type loopback
. Verify interfaces 'ethX' and 'ethQ' are of type 'ethernet' (or etherlike if running Qemu)
. Verify interfaces 'br-0', 'br-X', 'br-D' and 'br-Q' are of type 'bridge'
. Verify interfaces 'veth0a' and 'veth0b' are of type 'veth'
. Verify interfaces 'veth0a.20', 'ethQ.10', 'ethX.30', 'ethQ.10' and 'br-Q.40' are of type 'vlan'
<<<
@@ -1,10 +1,10 @@
#!/usr/bin/env python3
r"""
Verify that all interface types can be created
r"""Verify that all interface types can be created
This test verifies that all interface types can be created and also
checks the configuration when applied sequentially. This method takes
slightly longer than sending the entire configuration at once.
This test verify that all interface types can be created
and also, tesing setting the configuration in sequal (this
takes a little longer time than send it once)
....
lo br-0 br-Q.40 br-D br-X
@@ -14,6 +14,7 @@ takes a little longer time than send it once)
ethQ veth0b veth0a ethX
`---------'
....
"""
import infamy
@@ -56,7 +57,7 @@ with infamy.Test() as test:
loopback = "lo"
with test.step("Configure an empty bridge"):
with test.step("Configure an empty bridge br-0"):
target.put_config_dict("ietf-interfaces", {
"interfaces": {
"interface": [
@@ -69,7 +70,7 @@ with infamy.Test() as test:
}
})
with test.step("Configure bridge brX and associated interfaces"):
with test.step("Configure bridge br-X and associated interfaces"):
target.put_config_dict("ietf-interfaces", {
"interfaces": {
"interface": [
@@ -118,7 +119,7 @@ with infamy.Test() as test:
}
})
with test.step("Configure bridge brD and associated interfaces"):
with test.step("Configure bridge br-D and associated interfaces"):
target.put_config_dict("ietf-interfaces", {
"interfaces": {
"interface": [
@@ -192,21 +193,21 @@ with infamy.Test() as test:
with test.step("Verify interface 'lo' is of type loopback"):
verify_interface(target, "lo", "loopback")
with test.step("Verify interfaces 'ethX' and 'ethQ' is of type 'ethernet' (or etherlike if running Qemu)"):
with test.step("Verify interfaces 'ethX' and 'ethQ' are of type 'ethernet' (or etherlike if running Qemu)"):
verify_interface(target, eth_X, "etherlike")
verify_interface(target, eth_Q, "etherlike")
with test.step("Verify interfaces 'br-0', 'br-X', 'br-D' and 'br-Q' is of type 'bridge'"):
with test.step("Verify interfaces 'br-0', 'br-X', 'br-D' and 'br-Q' are of type 'bridge'"):
verify_interface(target, "br-0", "bridge")
verify_interface(target, "br-X", "bridge")
verify_interface(target, "br-Q", "bridge")
verify_interface(target, "br-D", "bridge")
with test.step("Verify interfaces 'veth0a' and 'veth0b' is of type 'veth'"):
with test.step("Verify interfaces 'veth0a' and 'veth0b' are of type 'veth'"):
verify_interface(target, "veth0a", "veth")
verify_interface(target, "veth0b", "veth")
with test.step("Verify interfaces 'veth0a.20', 'ethQ.10', 'ethX.30', 'ethQ.10' and 'br-Q.40' is of type 'vlan'"):
with test.step("Verify interfaces 'veth0a.20', 'ethQ.10', 'ethX.30', 'ethQ.10' and 'br-Q.40' are of type 'vlan'"):
verify_interface(target, "veth0a.20", "vlan")
verify_interface(target, f"{eth_X}.30", "vlan")
verify_interface(target, f"{eth_Q}.10", "vlan")
@@ -1,6 +1,6 @@
=== VLAN ping connectivity
==== Description
Very basic test if the VLAN configuration works.
Very basic test if the VLAN interface configuration works.
==== Topology
ifdef::topdoc[]
@@ -15,7 +15,7 @@ image::topology.png[VLAN ping connectivity topology]
endif::testgroup[]
endif::topdoc[]
==== Test sequence
. Configure VLAN 10 on target:data with IP 10.0.0.2
. Configure VLAN 10 interface on target:data with IP 10.0.0.2
. Waiting for links to come up
. Ping 10.0.0.2 from VLAN 10 on host:data with IP 10.0.0.1
. Remove VLAN interface from target:data, and test again (should not be able to ping)
+2 -2
View File
@@ -2,7 +2,7 @@
"""
VLAN ping connectivity
Very basic test if the VLAN configuration works.
Very basic test if the VLAN interface configuration works.
"""
import infamy
import infamy.iface as iface
@@ -25,7 +25,7 @@ def test_ping(hport, should_pass):
ns.must_not_reach("10.0.0.2")
with infamy.Test() as test:
with test.step("Configure VLAN 10 on target:data with IP 10.0.0.2"):
with test.step("Configure VLAN 10 interface on target:data with IP 10.0.0.2"):
env = infamy.Env()
target = env.attach("target", "mgmt")
_, tport = env.ltop.xlate("target", "data")
@@ -1,10 +1,10 @@
=== OSPF with multiple areas
==== Description
This test test alot of features inside OSPF using 3 areas (one NSSA area, with no summary)
to test the distribution of routes is deterministic (using cost), also test
link breaks using BFD (not implemented in infamy though)
This test evaluates various OSPF features across three areas (one NSSA area, with no summary)
to ensure that route distribution is deterministic (based on cost). It also tests link
failures using BFD, though BFD is not yet implemented in test framework (Infamy).
This test also verify broadcast and point-to-point interface type on /30 network and
This test also verifies broadcast and point-to-point interface types on /30 network, and
explicit router-id.
==== Topology
@@ -21,13 +21,13 @@ endif::testgroup[]
endif::topdoc[]
==== Test sequence
. Configure targets
. Wait for all neighbor to peer
. Wait for all neighbors to peer
. Wait for routes from OSPF on all routers
. Verify Area 0.0.0.1 on R3 is NSSA area
. Verify R1:ring2 is of type point-to-point
. Verify R4:ring1 is of type point-to-point
. Verify on R3, there are no routes beyond 10.0.23.1, just a default route
. Testing connectivitiy through NSSA area, from PC:data3 to 11.0.8.1
. Testing connectivity through NSSA area, from PC:data3 to 11.0.8.1
. Verify that the route to 10.0.0.3 from PC:data4, go through 10.0.41.2
. Break link R1:ring2 --- R4:ring1
. Verify that the route to 10.0.0.3 from PC:data4, go through 10.0.24.1
@@ -25,15 +25,15 @@
# +-------+
#
#
"""
OSPF with multiple areas
"""OSPF with multiple areas
This test test alot of features inside OSPF using 3 areas (one NSSA area, with no summary)
to test the distribution of routes is deterministic (using cost), also test
link breaks using BFD (not implemented in infamy though)
This test evaluates various OSPF features across three areas (one NSSA area, with no summary)
to ensure that route distribution is deterministic (based on cost). It also tests link
failures using BFD, though BFD is not yet implemented in test framework (Infamy).
This test also verify broadcast and point-to-point interface type on /30 network and
This test also verifies broadcast and point-to-point interface types on /30 network, and
explicit router-id.
"""
import infamy
@@ -539,7 +539,7 @@ with infamy.Test() as test:
config_target3(R3, R3ring2, R3cross, R3data),
config_target4(R4, R4ring1, R4cross, R4data))
with test.step("Wait for all neighbor to peer"):
with test.step("Wait for all neighbors to peer"):
print("Waiting for neighbors to peer")
until(lambda: route.ospf_get_neighbor(R1, "0.0.0.0", R1ring1, "1.1.1.1"), attempts=200)
until(lambda: route.ospf_get_neighbor(R1, "0.0.0.1", R1cross, "10.0.0.3"), attempts=200)
@@ -586,7 +586,7 @@ with infamy.Test() as test:
_, hport0 = env.ltop.xlate("PC", "data3")
with infamy.IsolatedMacVlan(hport0) as ns0:
with test.step("Testing connectivitiy through NSSA area, from PC:data3 to 11.0.8.1"):
with test.step("Testing connectivity through NSSA area, from PC:data3 to 11.0.8.1"):
ns0.addip("192.168.3.2")
ns0.addroute("0.0.0.0/0", "192.168.3.1")
ns0.must_reach("11.0.8.1")
+1 -1
View File
@@ -17,7 +17,7 @@ endif::topdoc[]
==== Test sequence
. Initializing
. Configure DUT
. Verify log files /var/log/bar.log and /var/log/bar.log have been created
. Verify log files /var/log/foo and /var/log/bar.log have been created
<<<
+1 -1
View File
@@ -58,7 +58,7 @@ with infamy.Test() as test:
}
})
with test.step("Verify log files /var/log/bar.log and /var/log/bar.log have been created"):
with test.step("Verify log files /var/log/foo and /var/log/bar.log have been created"):
user = tgtssh.runsh("ls /var/log/{foo,bar.log}").stdout
if "/var/log/foo" not in user:
test.fail()
+2 -2
View File
@@ -3,7 +3,7 @@
Verify that it is possible to change hostname both normal
and using format %h-%m.
The format exapnds to <default hostname>-<MAC>,
The %h-%m format expands to <default hostname>-<MAC>,
where MAC is the last three bytes of the base MAC address.
e.g. ix-01-01-01.
@@ -24,7 +24,7 @@ endif::topdoc[]
. Set up topology and attach to target DUT
. Set hostname to 'h0stn4m3'
. Verify new hostname 'h0stn4m3'
. Set hostname to to '%h-%m'
. Set hostname to '%h-%m'
. Verify hostname is %h-%m in running configuration
. Verify hostname format in operational, according to format
+2 -2
View File
@@ -5,7 +5,7 @@ Set hostname
Verify that it is possible to change hostname both normal
and using format %h-%m.
The format exapnds to <default hostname>-<MAC>,
The %h-%m format expands to <default hostname>-<MAC>,
where MAC is the last three bytes of the base MAC address.
e.g. ix-01-01-01.
@@ -34,7 +34,7 @@ with infamy.Test() as test:
running = target.get_config_dict("/ietf-system:system")
assert running["system"]["hostname"] == new
with test.step("Set hostname to to '%h-%m'"):
with test.step("Set hostname to '%h-%m'"):
target.put_config_dict("ietf-system", {
"system": {
"hostname": fmt,
@@ -0,0 +1,25 @@
=== Generate ssh key pair
==== Description
Verify that 'guest' user can fetch data using only the 'public' key
==== Topology
ifdef::topdoc[]
image::../../test/case/ietf_system/ssh_key_authentication/topology.png[Generate ssh key pair topology]
endif::topdoc[]
ifndef::topdoc[]
ifdef::testgroup[]
image::ssh_key_authentication/topology.png[Generate ssh key pair topology]
endif::testgroup[]
ifndef::testgroup[]
image::topology.png[Generate ssh key pair topology]
endif::testgroup[]
endif::topdoc[]
==== Test sequence
. Connect to the target device
. Create a guest user on the target device
. Write private key to a temporary file
. Verify it is possible to fetch syslog data using public key
<<<
+4 -4
View File
@@ -1,17 +1,17 @@
=== Set timezone
=== Set timezone using timezone name
==== Description
Verify that it is possible to set timezone using timezone names.
==== Topology
ifdef::topdoc[]
image::../../test/case/ietf_system/timezone/topology.png[Set timezone topology]
image::../../test/case/ietf_system/timezone/topology.png[Set timezone using timezone name topology]
endif::topdoc[]
ifndef::topdoc[]
ifdef::testgroup[]
image::timezone/topology.png[Set timezone topology]
image::timezone/topology.png[Set timezone using timezone name topology]
endif::testgroup[]
ifndef::testgroup[]
image::topology.png[Set timezone topology]
image::topology.png[Set timezone using timezone name topology]
endif::testgroup[]
endif::topdoc[]
==== Test sequence
+1 -1
View File
@@ -1,6 +1,6 @@
#!/usr/bin/env python3
"""
Set timezone
Set timezone using timezone name
Verify that it is possible to set timezone using timezone names.
"""
+3 -3
View File
@@ -19,12 +19,12 @@ endif::topdoc[]
. Set up topology and attach to target DUT
. Add new user 'jacky' with no NACM access
. Verify regular user jacky exists
. Verify user jacky is not in wheel group
. Verify user jacky is not in wheel group (in Linux)
. Verify user jacky password is set correctly
. Add user jacky to admin group in NACM
. Verify user jacky is now in wheel group
. Verify user jacky is now in wheel group (in Linux)
. Verify user jacky shell now is Bash
. Change user jacky to $factory$ password ...
. Change user jacky to $factory$ password
. Verify user jacky exists and has new password
. Verify user jacky can log in with SSH
+3 -3
View File
@@ -44,7 +44,7 @@ with infamy.Test() as test:
if not any(user['name'] == USER for user in users):
test.fail()
with test.step("Verify user jacky is not in wheel group"):
with test.step("Verify user jacky is not in wheel group (in Linux)"):
wheel = tgtssh.runsh("grep wheel /etc/group").stdout
if USER in wheel:
test.fail()
@@ -62,7 +62,7 @@ with infamy.Test() as test:
group["user-name"].append(USER)
target.put_config_dict("ietf-netconf-acm", nacm)
with test.step("Verify user jacky is now in wheel group"):
with test.step("Verify user jacky is now in wheel group (in Linux)"):
if not tgtssh.runsh(f"grep wheel /etc/group | grep '{USER}'"):
test.fail()
@@ -71,7 +71,7 @@ with infamy.Test() as test:
if "bash" not in user:
test.fail()
with test.step("Change user jacky to $factory$ password ..."):
with test.step("Change user jacky to $factory$ password"):
running = target.get_config_dict("/ietf-system:system")
users = running["system"]["authentication"]["user"]
+2 -2
View File
@@ -1,7 +1,7 @@
=== DHCP router
==== Description
Verify that the DHCP client receives default gatewa (DHCP option 3, router)
and that exist in operational datastore
Verify that the DHCP client receives default gateway (DHCP option 3, router)
and that route exists in operational datastore.
==== Topology
ifdef::topdoc[]
+2 -2
View File
@@ -3,8 +3,8 @@
"""
DHCP router
Verify that the DHCP client receives default gatewa (DHCP option 3, router)
and that exist in operational datastore
Verify that the DHCP client receives default gateway (DHCP option 3, router)
and that route exists in operational datastore.
"""
import time
import infamy, infamy.dhcp
+1 -1
View File
@@ -1,6 +1,6 @@
=== DHCP option 121 vs option 3
==== Description
Verify DHCP option 121 (staticroutes) is used over option 3 and that the
Verify DHCP option 121 (static routes) is used over option 3 and that the
routes exist in the operational datastore.
Installing unrelated routes from a DHCP server should not affect already
+1 -1
View File
@@ -2,7 +2,7 @@
"""
DHCP option 121 vs option 3
Verify DHCP option 121 (staticroutes) is used over option 3 and that the
Verify DHCP option 121 (static routes) is used over option 3 and that the
routes exist in the operational datastore.
Installing unrelated routes from a DHCP server should not affect already
@@ -1,7 +1,7 @@
=== Services basic
==== Description
Verify that basic services like mDNS and LLDP can be enabled and
disabled. We verify operation and non-operation by using tcpdump.
Verify that basic services, such as mDNS and LLDP, can be enabled and
disabled. Operation and non-operation are confirmed using tcpdump.
==== Topology
ifdef::topdoc[]
@@ -1,9 +1,9 @@
#!/usr/bin/env python3
"""
Services basic
"""Services basic
Verify that basic services, such as mDNS and LLDP, can be enabled and
disabled. Operation and non-operation are confirmed using tcpdump.
Verify that basic services like mDNS and LLDP can be enabled and
disabled. We verify operation and non-operation by using tcpdump.
"""
import time
import infamy