Shell script:
- Factor out big portions of code into more logical helper functions
- Simplify calling setup script by checking for remote image first
- Simplify meta/sha up-to-date handling and clarify terminology
- Consistent use of -f instead of -e in file-exists checks
- Fix unsafe use of 'mktemp -u'
C code:
- Clarify meta/sha terminology: rename meta-sha256 -> meta-image-sha256
- Refactor weird archive_offset() function to local_path() helper
- Factor out helper function calc_sha()
- Check len of sha256 >= 64
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
One unique feature of Infix OS is that you can embed your OCI archive in
the rootfs.squashfs. This way your container instance does not need to
download anything from the network. To upgrade you drop in a new image
and rebuild Infix, when the new Infix boots the container script 'setup'
command will recognize that the OCI archive has changed and will reload
it into the container store.
This patch is an improvement of the way too generic 'podman image prune'
command used previously. Instead of looking for any dangling image, we
now surgically remove the old image.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
Container instances that run with mutable images, e.g., tagged with `:latest`
or similar non-versioned tags, can be upgraded without changing the config.
This commit fixes two issues found with this support:
- force container image re-fetch on upgrade, even if the file exists locally
- surgically remove old image from container store after upgrade
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
Usually, when your system is up and running properly, you want to clean
up anything unused from your previous experiments. This change alllows
that by calling the interactive 'podman image prune -a -f' command from
the CLI command 'container remove all'
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
When loading big OCI images at boot the `podman load` process completely
monopolizes all cores of an Arm Cortex-A72. It blocks on I/O, sure, but
with 'nice' we can get some attention at least to more critical services
at boot.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
This commit reverts 477f7ae and bb19d06, which intended to fix an issue
with lingering old images, see #1098. However, as detailed in #1147,
this caused severe side effects while working with multiple larger
containers. Basically, the prune operation of one container removed
images of other containers that are just being created in parallel.
Instead of using the podman prune command we can use the meta datain the
start script to pinpoint exactly which image(s) to remove, including any
downloaded OCI archives when the container instance is removed.
Fixes#1147
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
This commit adds metadata to track loaded OCI archives to allow skipping
'delete + load' of OCI images when restarting either the container or the
system as a whole. The sha256 of all loaded OCI archives is stored in a
sidecar file in our downloads directory. Then we verify the checksum of
the OCI archives against their same-named sidecar to determine if the OCI
archive is already loaded or not.
Additionally, the instance using the image is labled with metadata to detect
changes in the container configuration. This in turn allow skipping the
delete + create phase also of the instance.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
The default timeout for 'podman stop foo' is 10 seconds, which for
heavily loaded systems with intricate shutdown process is *waaaay*
too short. Increase it to the container script default 30s, which
coincidentally is also the container@.conf template's kill delay.
Fixes#1149
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
When a container's image is on an inaccessible remote server, the
container wrapper script waits in the background for any netowrk
changes to retry download of the image.
This change avoids the dangerous previous construct, and is also
easier to read: timeuot after 60 seconds unless ip monitor reads
at least one event before that.
Fixes#1124
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
From the documentation:
> 'podman image prune' removes all dangling images from local storage.
> With the all option, all unused images are deleted (i.e., images not
> in use by any container).
>
> The image prune command does not prune cache images that only use
> layers that are necessary for other images.
So, when the container script is called in the cleanup phase of the
lifetime of a container, we can use the '--all' option to ensure we also
remove this container's loaded image. In the case this happens before
a reboot of the system, there will be no old version of the image loaded
to /var/lib/containers after boot.
Issue #1098
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
Previously, upgrading Podman containers with the same tag left
behind dangling images, causing overlay storage to grow and fill
disk space.
This change ensures dangling images are cleaned up using
podman image prune. The command is run without -a, so only
unreferenced images are removed. This provides safe cleanup while
preventing unnecessary overlay growth.
Fixes#1098
Signed-off-by: Richard Alpe <richard@bit42.se>
When stopping a container that is (stuck) in setup state, e.g., fetching
its container image, we need to send SIGTERM to the container wrapper
script rather than forwarding the 'stop' command to podman.
With this fix, and the Finit upgrade, there is no longer a 10 second
timeout before Finit sends SIGKILL to the PID. Instead, the script now
immediately react to the initial SIGTERM.
Related to issue #980
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
For tasks like upgrade we need, e.g., the image URI. Instead of
guessing where on the container command line the image is, use a
meta-data field in the container shell script.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
- More logging when deleting a container
- Update 'exec' and 'shell' commands to support running commands
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
Containers in Infix are read-only, any persistent data is either stored
in volumes or content mounts. To prevent lingering containers, as have
been seen on some test systems, we move cleanup stage from confd to use
the Finit cleanup:script mechanism.
The resulting code is simpler, shorter, and easier to debug and use.
As an added benefit, confd no longer prunes "unused" volumes by itself
on (any!) reconfiguration event, which with the new container life cycle
in Infix was quite fragile. Instead the CLI command 'container prune'
is extended with 'podman volume prune', which coincidentally the help
text already alluded to.
This patch requires Finit v4.10-rc1, or later.
Fixes#902
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
When updating a container configuration we need to rerun the setup stage
of the container. However, a run/task in Finit only runs once for each
runlevel, meaning changes at runtime of a container was broken.
This is a refactor of the container script to allow it to also run as a
pre:script in the Finit setup stage of a sysv/service. Also included is
a fix to the timeout/retry handling in case the container setup fails.
We must call `read -t 60` *before* calling `ip monitor`, otherwise it
will not be called until `ip monitor` returns.
This patch requires Finit v4.9, or later.
Fixes#930
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
- Rename local variables
- Extract image name, using basename, for podman image exists
Fixes the following syslog error message:
root@foo:~$ podman image exists docker://ghcr.io/kernelkit/curios:edge
Error: unsupported transport "docker" for looking up local images
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
Since `conmon` only logs the output from the monitored container's
`stdout`/`stderr`, we might as well get rid of `k8s-logger`, and let
the output pass through to finit, which will then pipe the messages to
syslog.
Fix#836
This is a complete redesign of how the system creates/deletes containers
from the running-config. Containers are now removed synchronously from
confd before any interfaces they may be using are removed, and created
in parallel, using a Finit task, well after confd has finished setting
up interfaces.
The logic previously provided by execd to retry container create on any
route/address changes, or periodically every 60 seconds, is now handled
by a new 'setup' command in the container wrapper script.
Additionally, container create is now split in wget/curl/podman pull of
the image and 'podman create'. This to both consolidate image fetching
and improve user feedback since most of the retry logic (above) revolves
around the image download.
Fixes#835
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
- Refactor logging to simplify code and get proper log level
- Clean up lingering directories from any extracted tarball on error
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
This patch allows running the configure script manually to create and
delete containers. The normal flow via confd has additional handling
to ensure containers are started/stopped on inictl reload.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
Instead of using $HOME, which may be a ramdisk, use /var/tmp which
podman also uses by default. Also, make sure to clean up after
ourselves.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
Disable the default "podman pull" retry value. We use execd to retry
"podman create" on failure. Wihtout this change, a single container
can block start of other containers by 3 * 20 seconds. Now we only
block max 20 seconds before we try starting the next container.
Modern versions of podman (>= 5.0) have this --retry option, but it
does not have CNI, so this is a temporary workaround.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
Should be inverted to a --verbose or --debug flag instead. After this
change we still see the full 'podman create ...' command, with all the
optionas and arguments.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
- Anonymous FTP, or URL encoded ftp://user:hostname@addr/oci.tar.gz
- HTTP/HTTPS fetched with curl, optional credentials support
- Verify download against an optional sha256 checksum
Ensure the unpacked directory name does not contain a ':', it is a
restricted character and cannot be part of the file name. If this
syntax is used we retain it as the name and retag it after load.
Fix#801
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
- Reduce the amount of queues: 3 -> 1
- Simplify post hook
- Refine execd
The resulting simplification of infix_containers_post_hook(), and
touching execd, also ensure container environment variable changes
are propagated.
Fixes#822
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
To be able to handle container restarts, incl. restart policy, at
runtime, most of the container data lives in /var/lib/containers,
which on most systems is backed by a persistent store.
As of issue #823 we no longer keep a writable layer for containers,
nor should we cache container state across reboots, all containers
are recreated at boot. This task cleans up any lingering state.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
Container support in Infix was released with v24.02, so this change may
unfortunately break a few use-cases out there. Regrettable as this is,
the default behavior, including how containers are started after boot,
break other use-cases that were considered more important.
As of this commit:
- all containers in Infix run in read-only mode, use volumes and
mounts for persistence across reboot/stop/start/upgrade
- all containers are now "recreated" at boot or related config changes,
this ensures an OCI image embedded in the Infix image, /lib/oci/, is
always used as the base for a running container
Fixes#823
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
In 0edc2d5 and afbe5ca, just prior to v24.06.0-rc1, support for deleting
containers in the background was added. However, this also broke support
for upgrading containers.
In afbe5ca the start script for containers was renamed from NAME.sh to
S01-NAME.sh, but the container wrapper script's upgrade command was not
updated. Neither was the cleanup and post-hook callbacks in confd! So
when a container had been added to the system, the cleanup callback just
simply deleted the script, preventing it from being recreated at ugprade
This patch fixes the container identification code and also refactors
the execd code to ensure that kill scripts (for deleting in background)
and start scripts are run in the proper order *and* ensuring that execd
also does not accidentally remove the container start script.
Some cosmetic changes to the output at upgrade have also been added.
Fixes#623
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
On any change to a container network interface we should schedule a
restart of the container to activate the changes. This code triggers
also at boot, when applying the whole startup-config, which initctl
handles by queuing any create/touch events for services.
This patch depends on the two previous commtis backporing fixes to
Finit's initctl tool and an upgrade of the k8s-logger.
Fixes#375
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
Also, log the actual `podman create` arguments to the container log so
we can see what the script actually does when customers report bizarre
happenings with containers.
Fixes#370
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
This change adds limited support for container capabilities. It allows
a more fine-grained control than priviliged mode does.
Fixes#365
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
Add exception for local images, the "pull" step must be handled by the
operator, i.e., wget of the latest image to the same location as set up
in the image configuration, e.g., for oci-archive:/tmp/foo.tar.gz that
/tmp/foo.tar.gz exists when issuing the upgrade command.
Fixes#368
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
Sending SIGTERM to conmon is not a safe shutdown of a podman container.
To handle gracefully handle shutdown, restarting and provide an orderly
start of dependencies, we use the Finit sysv trick via container script
wrapper to call 'podman stop foo'.
However, since podman does not support syslog as output for containers
we employ an old FIFO trick with another program, k8s-logger, to allow
logs to reach syslog. Please note that k8s-logger must have properly
started before we call `podman start` -- this makes us fully dependent
on the 'container' wrapper script. Hence the documentation update.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
Because sysrepo callbacks are threaded and factory-default RPC is called
from a separate subscription (to prevent blocking), we cannot prevent
ietf-intefaces.c from being called before infix-containers.c, regardless
of the priority we set for our subscriptions.
When assigning a physical network interface this becomes a bit of a pain
during factory-default RPC since the physical interface is hidden from
the host network namespace.
So, when applying factory to running, we check each interface if it was
a container-network previously, if so we call on the container script in
the exit of the current dagger generation to move the interface back to
the host netns.
This affects all other functions that assume interfaces only live in the
host netns. To that end a set of new helper functions have been added
to wrap iproute2 commands in nsenter when the interface lives elsewhere.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
1. add support for extracting, and finding the directory holding the
index.json metadata file in, OCI images. An archive in Infix is either
a .tar or .tar.gz archive, which we need to unpack for this version of
podman to be able to load them.
2. new RPC '/infix-containers:oci-load' calls 'container load' of an
OCI archive (tarball), optionally gzipped. The resulting image uses
the directory name of the unpacked tarball, so the container script
offers a way to retag the image after loading it.
First class citizens in container transport are docker:// and the OCI
family of URI:s. A docker:// URL, or a local docker-archive:path, is
assumed to be well formed, in which case we leave it up to podman to
handle.
Note: 'podman import' does not fully understand OCI formats. It drops
ENTRYPOINT and COMMAND, while 'podman load' handles things a lot
better. Only letdown is it does not support nameing the image.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
- drop debug logs
- no need to restart upgraded containers, they restart automatically
- fix ordering in volume prune (when containers are removed)
- skip directories in container activation (inbox -> execd queue)
- Fix 'container pull IMG creds USER[:PASS]' bug, extra '=' variable
assignment inside infix.xml
- Fix 'container run IMG CMD ARGS' to actually put the CMD in
ENTRYPOINT and append ARGS to image. The podman run and create
commands are *not* behaving the same way
- Rename 'attach' to 'exec', execute command inside an running container
- Add 'container [shell | connect]' to start shell in -- " -- " -- " --
- Add 'container [stat | cleanup | usage stats]' commands
- Add in="tty" to commands thay may end up being interactive
- Split <ACTION> line into multiple lines for readability
- Fix container shift logic:
root@infix-00-00-00:/> show container
/usr/sbin/container: line 361: shift: shift count out of range
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
In the container configuration context:
edit file ntp.conf
set path /etc/ntp.conf
set content
The last command opens a text editor where you can paste the contents
of the file. This is stored base64 encoded in the datastore as well
as in JSON/XML.
Signed-off-by: Joachim Wiberg <troglobit@gmail.com>
Joachim Wiberg