test/spec: refactor test spec & report generation

Import new 9pm version for improved test report generation.  With this
in place we can take the opportunity to also refactor and simplify the
test spec. generation.

Fixes #1129

Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
This commit is contained in:
Joachim Wiberg
2025-09-15 14:26:03 +02:00
parent fe2d0f488d
commit b4a5b35c0e
43 changed files with 427 additions and 85 deletions
@@ -1,12 +1,8 @@
---
- name: gretap_bridged
- name: GRETAP bridged with physical interface
case: test.py
opts: ["--type", "gretap"]
infamy:
title: GRETAP bridged with physical interface
- name: vxlan_bridged
- name: VXLAN bridged with physical interface
case: test.py
opts: ["--type", "vxlan"]
infamy:
title: VXLAN bridged with physical interface
+1 -1
View File
@@ -1,5 +1,5 @@
---
- name: hostname
- name: Set hostname
case: hostname/test.py
- name: add_delete_user
+15 -4
View File
@@ -1,14 +1,25 @@
:topdoc:
= Test specification
{REPLACE}
:title-page:
:topdoc:
ifdef::logo[]
:title-logo-image: {logo}
endif::[]
:toc:
:toclevels: 2
:sectnums:
:sectnumlevels: 2
:pdfmark:
:pdf-page-size: A4
:pdf-page-layout: portrait
:pdf-page-margin: [1in, 0.5in]
:keywords: regression, test, testing
:subject: Regression testing
:autofit-option:
= Test specification
{REPLACE}
<<<
include::../case/misc/Readme.adoc[]
<<<
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
+93
View File
@@ -0,0 +1,93 @@
Copyright (c) 2010-2014 by tyPoland Lukasz Dziedzic (team@latofonts.com) with Reserved Font Name "Lato"
This Font Software is licensed under the SIL Open Font License, Version 1.1.
This license is copied below, and is also available with a FAQ at:
https://openfontlicense.org
-----------------------------------------------------------
SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
-----------------------------------------------------------
PREAMBLE
The goals of the Open Font License (OFL) are to stimulate worldwide
development of collaborative font projects, to support the font creation
efforts of academic and linguistic communities, and to provide a free and
open framework in which fonts may be shared and improved in partnership
with others.
The OFL allows the licensed fonts to be used, studied, modified and
redistributed freely as long as they are not sold by themselves. The
fonts, including any derivative works, can be bundled, embedded,
redistributed and/or sold with any software provided that any reserved
names are not used by derivative works. The fonts and derivatives,
however, cannot be released under any other type of license. The
requirement for fonts to remain under this license does not apply
to any document created using the fonts or their derivatives.
DEFINITIONS
"Font Software" refers to the set of files released by the Copyright
Holder(s) under this license and clearly marked as such. This may
include source files, build scripts and documentation.
"Reserved Font Name" refers to any names specified as such after the
copyright statement(s).
"Original Version" refers to the collection of Font Software components as
distributed by the Copyright Holder(s).
"Modified Version" refers to any derivative made by adding to, deleting,
or substituting -- in part or in whole -- any of the components of the
Original Version, by changing formats or by porting the Font Software to a
new environment.
"Author" refers to any designer, engineer, programmer, technical
writer or other person who contributed to the Font Software.
PERMISSION & CONDITIONS
Permission is hereby granted, free of charge, to any person obtaining
a copy of the Font Software, to use, study, copy, merge, embed, modify,
redistribute, and sell modified and unmodified copies of the Font
Software, subject to the following conditions:
1) Neither the Font Software nor any of its individual components,
in Original or Modified Versions, may be sold by itself.
2) Original or Modified Versions of the Font Software may be bundled,
redistributed and/or sold with any software, provided that each copy
contains the above copyright notice and this license. These can be
included either as stand-alone text files, human-readable headers or
in the appropriate machine-readable metadata fields within text or
binary files as long as those fields can be easily viewed by the user.
3) No Modified Version of the Font Software may use the Reserved Font
Name(s) unless explicit written permission is granted by the corresponding
Copyright Holder. This restriction only applies to the primary font name as
presented to the users.
4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
Software shall not be used to promote, endorse or advertise any
Modified Version, except to acknowledge the contribution(s) of the
Copyright Holder(s) and the Author(s) or with their explicit written
permission.
5) The Font Software, modified or unmodified, in part or in whole,
must be distributed entirely under this license, and must not be
distributed under any other license. The requirement for fonts to
remain under this license does not apply to any document created
using the Font Software.
TERMINATION
This license becomes null and void if any of the above conditions are
not met.
DISCLAIMER
THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
OTHER DEALINGS IN THE FONT SOFTWARE.
+93
View File
@@ -0,0 +1,93 @@
Copyright 2010-2020 Adobe (http://www.adobe.com/), with Reserved Font Name 'Source'. All Rights Reserved. Source is a trademark of Adobe in the United States and/or other countries.
This Font Software is licensed under the SIL Open Font License, Version 1.1.
This license is copied below, and is also available with a FAQ at:
https://openfontlicense.org
-----------------------------------------------------------
SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
-----------------------------------------------------------
PREAMBLE
The goals of the Open Font License (OFL) are to stimulate worldwide
development of collaborative font projects, to support the font creation
efforts of academic and linguistic communities, and to provide a free and
open framework in which fonts may be shared and improved in partnership
with others.
The OFL allows the licensed fonts to be used, studied, modified and
redistributed freely as long as they are not sold by themselves. The
fonts, including any derivative works, can be bundled, embedded,
redistributed and/or sold with any software provided that any reserved
names are not used by derivative works. The fonts and derivatives,
however, cannot be released under any other type of license. The
requirement for fonts to remain under this license does not apply
to any document created using the fonts or their derivatives.
DEFINITIONS
"Font Software" refers to the set of files released by the Copyright
Holder(s) under this license and clearly marked as such. This may
include source files, build scripts and documentation.
"Reserved Font Name" refers to any names specified as such after the
copyright statement(s).
"Original Version" refers to the collection of Font Software components as
distributed by the Copyright Holder(s).
"Modified Version" refers to any derivative made by adding to, deleting,
or substituting -- in part or in whole -- any of the components of the
Original Version, by changing formats or by porting the Font Software to a
new environment.
"Author" refers to any designer, engineer, programmer, technical
writer or other person who contributed to the Font Software.
PERMISSION & CONDITIONS
Permission is hereby granted, free of charge, to any person obtaining
a copy of the Font Software, to use, study, copy, merge, embed, modify,
redistribute, and sell modified and unmodified copies of the Font
Software, subject to the following conditions:
1) Neither the Font Software nor any of its individual components,
in Original or Modified Versions, may be sold by itself.
2) Original or Modified Versions of the Font Software may be bundled,
redistributed and/or sold with any software, provided that each copy
contains the above copyright notice and this license. These can be
included either as stand-alone text files, human-readable headers or
in the appropriate machine-readable metadata fields within text or
binary files as long as those fields can be easily viewed by the user.
3) No Modified Version of the Font Software may use the Reserved Font
Name(s) unless explicit written permission is granted by the corresponding
Copyright Holder. This restriction only applies to the primary font name as
presented to the users.
4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
Software shall not be used to promote, endorse or advertise any
Modified Version, except to acknowledge the contribution(s) of the
Copyright Holder(s) and the Author(s) or with their explicit written
permission.
5) The Font Software, modified or unmodified, in part or in whole,
must be distributed entirely under this license, and must not be
distributed under any other license. The requirement for fonts to
remain under this license does not apply to any document created
using the Font Software.
TERMINATION
This license becomes null and void if any of the above conditions are
not met.
DISCLAIMER
THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
OTHER DEALINGS IN THE FONT SOFTWARE.
+79
View File
@@ -0,0 +1,79 @@
Source Sans 3 Variable Font
===========================
This download contains Source Sans 3 as both variable fonts and static fonts.
Source Sans 3 is a variable font with this axis:
wght
This means all the styles are contained in these files:
SourceSans3-VariableFont_wght.ttf
SourceSans3-Italic-VariableFont_wght.ttf
If your app fully supports variable fonts, you can now pick intermediate styles
that arent available as static fonts. Not all apps support variable fonts, and
in those cases you can use the static font files for Source Sans 3:
static/SourceSans3-ExtraLight.ttf
static/SourceSans3-Light.ttf
static/SourceSans3-Regular.ttf
static/SourceSans3-Medium.ttf
static/SourceSans3-SemiBold.ttf
static/SourceSans3-Bold.ttf
static/SourceSans3-ExtraBold.ttf
static/SourceSans3-Black.ttf
static/SourceSans3-ExtraLightItalic.ttf
static/SourceSans3-LightItalic.ttf
static/SourceSans3-Italic.ttf
static/SourceSans3-MediumItalic.ttf
static/SourceSans3-SemiBoldItalic.ttf
static/SourceSans3-BoldItalic.ttf
static/SourceSans3-ExtraBoldItalic.ttf
static/SourceSans3-BlackItalic.ttf
Get started
-----------
1. Install the font files you want to use
2. Use your app's font picker to view the font family and all the
available styles
Learn more about variable fonts
-------------------------------
https://developers.google.com/web/fundamentals/design-and-ux/typography/variable-fonts
https://variablefonts.typenetwork.com
https://medium.com/variable-fonts
In desktop apps
https://theblog.adobe.com/can-variable-fonts-illustrator-cc
https://helpx.adobe.com/nz/photoshop/using/fonts.html#variable_fonts
Online
https://developers.google.com/fonts/docs/getting_started
https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Fonts/Variable_Fonts_Guide
https://developer.microsoft.com/en-us/microsoft-edge/testdrive/demos/variable-fonts
Installing fonts
MacOS: https://support.apple.com/en-us/HT201749
Linux: https://www.google.com/search?q=how+to+install+a+font+on+gnu%2Blinux
Windows: https://support.microsoft.com/en-us/help/314960/how-to-install-or-remove-a-font-in-windows
Android Apps
https://developers.google.com/fonts/docs/android
https://developer.android.com/guide/topics/ui/look-and-feel/downloadable-fonts
License
-------
Please read the full license text (OFL.txt) to understand the permissions,
restrictions and requirements for usage, redistribution, and modification.
You can use them in your products & projects print or digital,
commercial or otherwise.
This isn't legal advice, please consider consulting a lawyer and see the full
license for all details.
+78 -54
View File
@@ -5,13 +5,17 @@ import argparse
import io
import sys
import re
import yaml
from pathlib import Path
import yaml
import graphviz
def log(msg):
"""Log message if in debug mode"""
if args.debug:
print(msg)
class TestStepVisitor(ast.NodeVisitor):
"""
A custom test step visitor to grab the test description (docstring)
@@ -70,27 +74,27 @@ class TestStepVisitor(ast.NodeVisitor):
class TestCase:
"""All test specifcation resources for a test case"""
def __init__(self, directory, rootdir=None):
"""Test specifcation resources for a test case"""
def __init__(self, directory, name="test.py", rootdir=None):
self.test_dir = Path(directory)
if rootdir:
rootdir = Path(f"{rootdir}")
self.test_dir = self.test_dir.relative_to(rootdir)
self.topology_dot = f"{directory}/topology.dot"
self.topology_image = f"{directory}/topology"
self.test_case = f"{directory}/test.py"
self.specification = f"{directory}/Readme.adoc"
self.topo_dot = f"{directory}/topology.dot"
self.topo_img = f"{directory}/topology"
self.test_path = f"{directory}/{name}"
self.spec_path = f"{directory}/{os.path.splitext(name)[0]}.adoc"
log(f"TestCase({self.test_path})")
def generate_topology(self):
def gen_topology(self):
"""Generate SVG file from the topology.dot file"""
with open(self.topology_dot, 'r', encoding='utf-8') as dot_file:
svg_file = f"{self.topology_image}.svg"
with open(self.topo_dot, 'r', encoding='utf-8') as dot_file:
svg_file = f"{self.topo_img}.svg"
try:
dot_graph = dot_file.read()
graph = graphviz.Source(dot_graph)
graph.render(self.topology_image, format='svg', cleanup=True,
quiet=True)
graph.render(self.topo_img, format='svg', cleanup=True, quiet=True)
content = ""
with open(svg_file, 'r', encoding='utf-8') as f:
@@ -102,7 +106,7 @@ class TestCase:
f.write(mod_content)
except graphviz.backend.execute.CalledProcessError as e:
msg = e.stderr.decode()
print(f"Failed rendering SVG from {self.topology_dot}: {msg}")
print(f"Failed rendering SVG from {self.topo_dot}: {msg}")
except UnboundLocalError:
print(f"Failed cleaning {svg_file}, empty or missing.")
@@ -115,9 +119,9 @@ class TestCase:
result.append(new_step)
return result
def generate_specification(self, name, case_path, spec_path, variables):
"""Generate a ASCIIDOC specification for the test case"""
with open(case_path, 'r', encoding='utf-8') as file:
def gen_spec(self, title, variables):
"""Generate a AsciiDoc specification for the test case"""
with open(self.test_path, 'r', encoding='utf-8') as file:
script_content = file.read()
parsed_script = ast.parse(script_content)
@@ -133,73 +137,91 @@ class TestCase:
test_steps = self.__replace_in(test_steps, variables)
if name is None:
name = visitor.name
if title is None:
title = visitor.name
self.generate_topology()
self.gen_topology()
with open(self.spec_path, "w", encoding='utf-8') as spec:
# This is the test name/title for the test-specification.pdf,
# it is replaced by 9PM with the 'name:' from the suite.yaml
# when it generates the test report.
spec.write(f"=== {title}\n\n")
with open(spec_path, "w", encoding='utf-8') as spec:
# When included in another document, e.g., the test-report.pdf,
# AsciiDoc needs to know where iamges are located. For other
# use-cases, e.g., browsing in GitHub '.' is implied.
spec.write(f"ifdef::topdoc[:imagesdir: {{topdoc}}../../{self.test_dir}]\n")
spec.write(f"\n=== {name}\n")
spec.write("==== Description\n")
spec.write(f"ifdef::topdoc[:imagesdir: {{topdoc}}../../{self.test_dir}]\n\n")
spec.write("==== Description\n\n")
spec.write(description + "\n\n")
spec.write("==== Topology\n")
spec.write(f"image::topology.svg[{name} topology, align=center, scaledwidth=75%]\n\n")
spec.write("==== Sequence\n")
spec.write("==== Topology\n\n")
spec.write(f"image::topology.svg[{title} topology, align=center, scaledwidth=75%]\n\n")
spec.write("==== Sequence\n\n")
spec.writelines([f". {step}\n" for step in test_steps])
spec.write("\n\n")
def parse_suite(directory, root, suitefile):
"""Parse 9pm .yaml suite file"""
test_spec = None
readme = None
first_test = True
log(f"Opening suite file {suitefile} ...")
with open(suitefile, "r", encoding='utf-8') as f:
data = yaml.safe_load(f)
for entry in data:
if entry.get("suite"):
path = os.path.dirname(entry["suite"])
parse_suite(f"{directory}/{path}", root, f"{directory}/{entry['suite']}")
elif entry.get("case"):
variables=None
if not entry.get("infamy"):
test_title = None # Set from the docstring in the test case
else:
specification = entry["infamy"].get("specification", True);
name = os.path.basename(entry["case"])
base = os.path.splitext(name)[0]
path = os.path.dirname(entry["case"])
opts = None
log(f"Parsing test case {name} in {path} ...")
if entry.get("infamy"):
specification = entry["infamy"].get("specification", True)
if not specification:
continue
test_title = entry["infamy"]["title"]
path = os.path.dirname(entry["case"])
if entry.get("name"):
test_title = entry["name"]
else:
test_title = None # Set from the docstring in the test case
if readme is None:
readme = open(f"{directory}/{path}/Readme.adoc.tmp", "w", encoding='utf-8')
test_case_id = entry["name"]
if "opts" in entry:
variables = dict((k[2:], v) for k, v in zip(entry["opts"][::2], entry["opts"][1::2]))
opts = dict((k[2:], v) for k, v in zip(entry["opts"][::2], entry["opts"][1::2]))
test_case = TestCase(f"{directory}/{path}", name, root)
test_case.gen_spec(test_title, opts)
test_file = f"{directory}/{entry['case']}"
test_spec = f"{directory}/{path}/{test_case_id}.adoc"
test_case = TestCase(f"{directory}/{path}", root)
test_case.generate_specification(test_title, test_file, test_spec, variables)
if first_test:
readme.write(f"include::{path}{test_case_id}.adoc[]\n\n")
first_test = False
else:
readme.write(f"<<<\n\n") # Page break before subsequent tests
readme.write(f"include::{path}{test_case_id}.adoc[]\n\n")
readme.write("<<<\n\n") # Page break before subsequent tests
readme.write(f"include::{base}.adoc[]\n\n")
if path != "":
lnk = f"{directory}/{path}/Readme.adoc"
readme.close()
log(f"Removing {directory}/{path}/Readme.adoc.tmp")
os.unlink(f"{directory}/{path}/Readme.adoc.tmp")
readme = None
if os.path.exists(lnk):
log(f"Checking if lnk {lnk} exists ...")
if os.path.lexists(lnk):
log(f"Removing {lnk}")
os.unlink(lnk)
os.symlink(f"{test_case_id}.adoc", lnk)
log(f"Creating symlink {base}.adoc -> {lnk}")
os.symlink(f"{base}.adoc", lnk)
if readme is not None:
readme.close()
os.rename(f"{directory}/Readme.adoc.tmp", f"{directory}/Readme.adoc")
@@ -211,22 +233,24 @@ parser.add_argument("-r", "--root-dir", required=True, help="Path that all paths
parser.add_argument("-d", "--debug", help="Show debug output", action="store_true")
args = parser.parse_args()
error_string = ""
# This is hacky, graphviz output error only to stdout and return successful(always).
# If everything goes well, output shall be empty, fail on any output
# This is hacky, graphviz output error only to stdout and *always*
# return successful. If everything goes well, output shall be empty,
# fail on any output.
ERROR_STRING = ""
if not args.debug:
output_capture = io.StringIO()
sys.stderr = output_capture
output_capture.truncate(0)
output_capture.seek(0)
parse_suite(os.path.dirname(args.suite), args.root_dir, args.suite)
if not args.debug:
sys.stdout = sys.__stdout__
if len(error_string) > 0:
print(error_string)
exit(1)
if len(ERROR_STRING) > 0:
print(ERROR_STRING)
sys.exit(1)
exit(0)
sys.exit(0)
+63 -19
View File
@@ -2,38 +2,82 @@ extends: default
font:
catalog:
merge: true
Lato:
normal: Lato/Lato-Regular.ttf
bold: Lato/Lato-Bold.ttf
italic: Lato/Lato-Italic.ttf
bold_italic: Lato/Lato-BoldItalic.ttf
SourceSansPro:
normal: SourceSansPro/static/SourceSans3-Regular.ttf
bold: SourceSansPro/static/SourceSans3-Bold.ttf
italic: SourceSansPro/static/SourceSans3-Italic.ttf
bold_italic: SourceSansPro/static/SourceSans3-BoldItalic.ttf
SourceCodePro:
normal: SourceCodePro-Regular.ttf
bold: SourceCodePro-Bold.ttf
italic: SourceCodePro-Italic.ttf
bold_italic: SourceCodePro-BoldItalic.ttf
normal: SourceCodePro/SourceCodePro-Regular.ttf
bold: SourceCodePro/SourceCodePro-Bold.ttf
italic: SourceCodePro/SourceCodePro-Italic.ttf
bold_italic: SourceCodePro/SourceCodePro-BoldItalic.ttf
base:
font_family: SourceCodePro
font_size: 11
line_height_length: 1.3
font-family: SourceSansPro
font-size: 10
line-height-length: 1.3
font-color: #1f2937
code:
font-family: SourceCodePro
font-size: 9
background-color: #ffffff
border-color: #ffffff
codespan:
font-size: 9
background-color: #ffffff
font-color: #000000
heading:
font_family: SourceCodePro
font_style: bold
font-family: Lato
font-kerning: normal
role:
underline:
text_decoration: underline
text-decoration: underline
pass:
font-color: #14532d
font-style: bold
fail:
font-color: #b91c1c
font-style: bold
skip:
font-color: #eab308
font-style: bold
link:
font-color: #2563eb
font-style: italic
text-decoration: underline
image:
caption:
align: inherit
table:
font-size: 9
# Define custom sizes for each heading level
heading_h1:
font_size: 24 # Size for H1 headings
font-family: Lato
font_size: 24
heading_h2: # Test class
font_size: 18 # Size for H2 headings
heading_h3: # test case name
font_size: 12 # Size for H3 headings
text_decoration: underline
margin_top: 30
# Test class
heading_h2:
font-family: Lato
font_size: 20
# Test case name
heading_h3:
font-family: Lato
font_size: 18
heading_h4:
font_size: 11 # Size for H4 headings
font-family: Lato
font_size: 12
+2
View File
@@ -5,6 +5,7 @@ NINEPM_PROJ_CONF ?= $(BR2_EXTERNAL_INFIX_PATH)/test/9pm-proj.yaml
spec-dir := $(test-dir)/spec
test-specification := $(BINARIES_DIR)/test-specification.pdf
test-report := $(BINARIES_DIR)/test-report.pdf
LOGO ?= $(test-dir)/../doc/logo.png[top=40%, align=right, pdfwidth=10cm]
UNIT_TESTS ?= $(test-dir)/case/all-repo.yaml $(test-dir)/case/all-unit.yaml
TESTS ?= $(test-dir)/case/all.yaml
@@ -38,6 +39,7 @@ test-spec:
$(spec-dir)/Readme.adoc.in > $(spec-dir)/Readme.adoc
@$(spec-dir)/generate_spec.py -s $(test-dir)/case/all.yaml -r $(BR2_EXTERNAL_INFIX_PATH)
@asciidoctor-pdf --failure-level INFO --theme $(spec-dir)/theme.yml \
-a logo="image:$(LOGO)" \
-a pdf-fontsdir=$(spec-dir)/fonts \
-o $(test-specification) $(spec-dir)/Readme.adoc