When stopping a container that is (stuck) in setup state, e.g., fetching
its container image, we need to send SIGTERM to the container wrapper
script rather than forwarding the 'stop' command to podman.
With this fix, and the Finit upgrade, there is no longer a 10 second
timeout before Finit sends SIGKILL to the PID. Instead, the script now
immediately react to the initial SIGTERM.
Related to issue #980
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
For tasks like upgrade we need, e.g., the image URI. Instead of
guessing where on the container command line the image is, use a
meta-data field in the container shell script.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
- More logging when deleting a container
- Update 'exec' and 'shell' commands to support running commands
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
Containers in Infix are read-only, any persistent data is either stored
in volumes or content mounts. To prevent lingering containers, as have
been seen on some test systems, we move cleanup stage from confd to use
the Finit cleanup:script mechanism.
The resulting code is simpler, shorter, and easier to debug and use.
As an added benefit, confd no longer prunes "unused" volumes by itself
on (any!) reconfiguration event, which with the new container life cycle
in Infix was quite fragile. Instead the CLI command 'container prune'
is extended with 'podman volume prune', which coincidentally the help
text already alluded to.
This patch requires Finit v4.10-rc1, or later.
Fixes#902
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
When updating a container configuration we need to rerun the setup stage
of the container. However, a run/task in Finit only runs once for each
runlevel, meaning changes at runtime of a container was broken.
This is a refactor of the container script to allow it to also run as a
pre:script in the Finit setup stage of a sysv/service. Also included is
a fix to the timeout/retry handling in case the container setup fails.
We must call `read -t 60` *before* calling `ip monitor`, otherwise it
will not be called until `ip monitor` returns.
This patch requires Finit v4.9, or later.
Fixes#930
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
- Rename local variables
- Extract image name, using basename, for podman image exists
Fixes the following syslog error message:
root@foo:~$ podman image exists docker://ghcr.io/kernelkit/curios:edge
Error: unsupported transport "docker" for looking up local images
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
Since `conmon` only logs the output from the monitored container's
`stdout`/`stderr`, we might as well get rid of `k8s-logger`, and let
the output pass through to finit, which will then pipe the messages to
syslog.
Fix#836
This is a complete redesign of how the system creates/deletes containers
from the running-config. Containers are now removed synchronously from
confd before any interfaces they may be using are removed, and created
in parallel, using a Finit task, well after confd has finished setting
up interfaces.
The logic previously provided by execd to retry container create on any
route/address changes, or periodically every 60 seconds, is now handled
by a new 'setup' command in the container wrapper script.
Additionally, container create is now split in wget/curl/podman pull of
the image and 'podman create'. This to both consolidate image fetching
and improve user feedback since most of the retry logic (above) revolves
around the image download.
Fixes#835
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
- Refactor logging to simplify code and get proper log level
- Clean up lingering directories from any extracted tarball on error
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
This patch allows running the configure script manually to create and
delete containers. The normal flow via confd has additional handling
to ensure containers are started/stopped on inictl reload.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
Instead of using $HOME, which may be a ramdisk, use /var/tmp which
podman also uses by default. Also, make sure to clean up after
ourselves.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
Disable the default "podman pull" retry value. We use execd to retry
"podman create" on failure. Wihtout this change, a single container
can block start of other containers by 3 * 20 seconds. Now we only
block max 20 seconds before we try starting the next container.
Modern versions of podman (>= 5.0) have this --retry option, but it
does not have CNI, so this is a temporary workaround.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
Should be inverted to a --verbose or --debug flag instead. After this
change we still see the full 'podman create ...' command, with all the
optionas and arguments.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
- Anonymous FTP, or URL encoded ftp://user:hostname@addr/oci.tar.gz
- HTTP/HTTPS fetched with curl, optional credentials support
- Verify download against an optional sha256 checksum
Ensure the unpacked directory name does not contain a ':', it is a
restricted character and cannot be part of the file name. If this
syntax is used we retain it as the name and retag it after load.
Fix#801
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
- Reduce the amount of queues: 3 -> 1
- Simplify post hook
- Refine execd
The resulting simplification of infix_containers_post_hook(), and
touching execd, also ensure container environment variable changes
are propagated.
Fixes#822
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
To be able to handle container restarts, incl. restart policy, at
runtime, most of the container data lives in /var/lib/containers,
which on most systems is backed by a persistent store.
As of issue #823 we no longer keep a writable layer for containers,
nor should we cache container state across reboots, all containers
are recreated at boot. This task cleans up any lingering state.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
Container support in Infix was released with v24.02, so this change may
unfortunately break a few use-cases out there. Regrettable as this is,
the default behavior, including how containers are started after boot,
break other use-cases that were considered more important.
As of this commit:
- all containers in Infix run in read-only mode, use volumes and
mounts for persistence across reboot/stop/start/upgrade
- all containers are now "recreated" at boot or related config changes,
this ensures an OCI image embedded in the Infix image, /lib/oci/, is
always used as the base for a running container
Fixes#823
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
This patch changes the way Infix DHCP and ZerocConf clients set their
routes in the system. Instead of setting them directly in the kernel
we ask FRR staticd to set them for us.
The reason for this change is to be able to override routes from these
protocols with locally set static routes. The routes are now set with
a distance of 5 and 254, respectively, while static routes by default
have a distance of 1. In contrast, kernel routes are always treated
by Frr as distance 0, i.e., they are preferred over static routes.
Finally, this patch drops the use of Linux legacy interface aliases, or
colon interfaces, in an effort to reduce confusion for end users. This
may give some odd results if using older tools like ifconfig, so we
recommend using 'show interfaces' or 'ipb a' instead.
Fixes#640
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
In 0edc2d5 and afbe5ca, just prior to v24.06.0-rc1, support for deleting
containers in the background was added. However, this also broke support
for upgrading containers.
In afbe5ca the start script for containers was renamed from NAME.sh to
S01-NAME.sh, but the container wrapper script's upgrade command was not
updated. Neither was the cleanup and post-hook callbacks in confd! So
when a container had been added to the system, the cleanup callback just
simply deleted the script, preventing it from being recreated at ugprade
This patch fixes the container identification code and also refactors
the execd code to ensure that kill scripts (for deleting in background)
and start scripts are run in the proper order *and* ensuring that execd
also does not accidentally remove the container start script.
Some cosmetic changes to the output at upgrade have also been added.
Fixes#623
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
On any change to a container network interface we should schedule a
restart of the container to activate the changes. This code triggers
also at boot, when applying the whole startup-config, which initctl
handles by queuing any create/touch events for services.
This patch depends on the two previous commtis backporing fixes to
Finit's initctl tool and an upgrade of the k8s-logger.
Fixes#375
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
Also, log the actual `podman create` arguments to the container log so
we can see what the script actually does when customers report bizarre
happenings with containers.
Fixes#370
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
This change adds limited support for container capabilities. It allows
a more fine-grained control than priviliged mode does.
Fixes#365
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
Add exception for local images, the "pull" step must be handled by the
operator, i.e., wget of the latest image to the same location as set up
in the image configuration, e.g., for oci-archive:/tmp/foo.tar.gz that
/tmp/foo.tar.gz exists when issuing the upgrade command.
Fixes#368
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
Sending SIGTERM to conmon is not a safe shutdown of a podman container.
To handle gracefully handle shutdown, restarting and provide an orderly
start of dependencies, we use the Finit sysv trick via container script
wrapper to call 'podman stop foo'.
However, since podman does not support syslog as output for containers
we employ an old FIFO trick with another program, k8s-logger, to allow
logs to reach syslog. Please note that k8s-logger must have properly
started before we call `podman start` -- this makes us fully dependent
on the 'container' wrapper script. Hence the documentation update.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
Because sysrepo callbacks are threaded and factory-default RPC is called
from a separate subscription (to prevent blocking), we cannot prevent
ietf-intefaces.c from being called before infix-containers.c, regardless
of the priority we set for our subscriptions.
When assigning a physical network interface this becomes a bit of a pain
during factory-default RPC since the physical interface is hidden from
the host network namespace.
So, when applying factory to running, we check each interface if it was
a container-network previously, if so we call on the container script in
the exit of the current dagger generation to move the interface back to
the host netns.
This affects all other functions that assume interfaces only live in the
host netns. To that end a set of new helper functions have been added
to wrap iproute2 commands in nsenter when the interface lives elsewhere.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
1. add support for extracting, and finding the directory holding the
index.json metadata file in, OCI images. An archive in Infix is either
a .tar or .tar.gz archive, which we need to unpack for this version of
podman to be able to load them.
2. new RPC '/infix-containers:oci-load' calls 'container load' of an
OCI archive (tarball), optionally gzipped. The resulting image uses
the directory name of the unpacked tarball, so the container script
offers a way to retag the image after loading it.
First class citizens in container transport are docker:// and the OCI
family of URI:s. A docker:// URL, or a local docker-archive:path, is
assumed to be well formed, in which case we leave it up to podman to
handle.
Note: 'podman import' does not fully understand OCI formats. It drops
ENTRYPOINT and COMMAND, while 'podman load' handles things a lot
better. Only letdown is it does not support nameing the image.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
- drop debug logs
- no need to restart upgraded containers, they restart automatically
- fix ordering in volume prune (when containers are removed)
- skip directories in container activation (inbox -> execd queue)
- Fix 'container pull IMG creds USER[:PASS]' bug, extra '=' variable
assignment inside infix.xml
- Fix 'container run IMG CMD ARGS' to actually put the CMD in
ENTRYPOINT and append ARGS to image. The podman run and create
commands are *not* behaving the same way
- Rename 'attach' to 'exec', execute command inside an running container
- Add 'container [shell | connect]' to start shell in -- " -- " -- " --
- Add 'container [stat | cleanup | usage stats]' commands
- Add in="tty" to commands thay may end up being interactive
- Split <ACTION> line into multiple lines for readability
- Fix container shift logic:
root@infix-00-00-00:/> show container
/usr/sbin/container: line 361: shift: shift count out of range
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
In the container configuration context:
edit file ntp.conf
set path /etc/ntp.conf
set content
The last command opens a text editor where you can paste the contents
of the file. This is stored base64 encoded in the datastore as well
as in JSON/XML.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
This patch adds a new CLI command 'container upgrade foo', where 'foo'
is the name of the container. If more than one container use the same
image, multiple upgrades must be done because a container runs not on
the 'image:tag' but on the hash of the 'image:tag' it was created from.
Rename "done" queue to "active", since we want to recreate an active
container after fetching an updated base image.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
This commit adds support for writable container volumes. A volume, when
compared to a mount, is a writable directory created when the container
starts and is automatically rsync'ed with the underlying directory it
is mounted on on first use. A mount otoh does not rsync so it results
only in an empty directory inside the container.
The podman/docker mount feature will be used later to bind mount single
files or sharing directories from the host, e.g., /sys/class/leds/ to
one of more containers.
Note: unused volumes are automatically pruned. Hence, a volume
currently cannot be moved to another container.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
This was a tough nut to crack. Turns out the trick to changing the
ENTRYPOINT is to set --entrypoint=command and then call 'podman create
... command args'. Not entirely obvious since the documented approach
is to use a JSON array as the argument: podman create
--entrypoint='["command", "args" ]'.
Admittedly, encoding this in C to transfer it via a POSIX shell script
to the command line, is not the easiest task I've undertaken. So I gave
up and found this workaround.
Worth noting, however, is that one *must* set `--entrypoint`, it is not
enough to just append the command to the 'podman create' command line.
Due to differences in docker and podman, we cannot supply the full args
to an alternate entrypoint command. But for the command to actually run
we need to override the image's ENTRYPOINT and send the command and args
as the last arguments on the command line to podman create.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
Also, ensure the deleted container is actually deleted before recreating
it with a new configuration.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
This commit adds 'manual:yes' to the container's Finit service
configuration and changes from pod: to container: prefix for a
unique namespace to prevent collision with regular services.
The prefix container: is more correct that pod:, which should
be reserved for any future pod support.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
confd: fix missing variable in container script condition
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
Various scripts and tools search for the ip command in different
places. Most distros install it to /bin/ip and symlink to /sbin
and /usr dito, but Buildroot only installs to /sbin/ip
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
Mattias Walström