infix/doc/scripting.md

# Scripting Infix

In some situations a user cannot, or does not want to, use the NETCONF
API for interacting with Infix.  Examples include production tasks and
simpler remote scripting jobs to one or more remote devices.

This document assumes you have the password for the `admin` user, and
that you can connect to the device.  Please see [Finding my Device][1]
for help on locating it.

Furthermore, the example commands shown here that are execute from a PC
to a remote device over SSH, use Linux/UNIX.  With advances lately in
both Windows and macOS, many of the user friendly tools previously only
available in Linux are now available there too.

The shell prompt for the PC laptop side:

```
~$
```

The shell prompt when logged in to an Infix device:

```
admin@example:~$
```

> [!NOTE]
> The shell script commands used here are the raw variants which the CLI
> usually wraps in a warm and snugly blanket.  Meaning they may change
> over time, while the CLI wrappers do *not*.  That being said, please
> let us know if you find any inconsistencies.

## Tips

- Ensure the `admin` user does *not* have `clish` as login shell
- Enable [SSH key authentication](system.md#ssh-authorized-key)
- Deploy same SSH *public* key to many Infix devices
- Secure your *private* SSH key using, e.g., `ssh-agent`

The `ssh-keygen` command, used to create the private/public key pair,
asks for a passphrase and although this is *technically optional* it is
highly recommended to set one.  For ease of use, in particular when
scripting, use `ssh-agent` to avoid retyping the passphrase for every
command.

Useful links on SSH, keys, and using `ssh-agent`:

- <https://en.wikipedia.org/wiki/Ssh-agent>
- <https://www.cyberciti.biz/faq/how-to-use-ssh-agent-for-authentication-on-linux-unix/>
- <https://goteleport.com/blog/how-to-use-ssh-agent-safely/>

## Admin User Not Authorized

All system services and critical configuration files are owned by the
locked-down `root` user.  It is not possible to activate the `root` user
account for remote logins.  Instead, use `admin` user and the `sudo`
command prefix.

Here we are logged in to an example device:

```
admin@example:~$ cp /cfg/startup-config.cfg /cfg/backup-config.cfg
cp: can't create '/cfg/backup-config.cfg': Permission denied
admin@example:~$ sudo cp /cfg/startup-config.cfg /cfg/backup-config.cfg
```

## Examples

The following example commands are run from the PC over SSH.  The
following is a *very brief* introduction.

The notation is `ssh username@address`, where address can be an IPv4 or
IPv6 address, a DNS name, or an mDNS name, e.g. infix.local.  In the
case of IPv6: `address%interface`, where interface differs between
operating systems.  On Linux and macOS the interface name is used, but
on Windows the interface index[^1] is used.

### Logging in to a device

```
~$ ssh admin@fe80::ff:fe00:0%eth0
The authenticity of host 'fe80::ff:fe00:0%eth0 (fe80::ff:fe00:0%eth0)' can't be established.
ED25519 key fingerprint is SHA256:5/9mw64jhmYyD8MD+SwrsG3RXMBbP48pDe2T8bg14RQ.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'fe80::ff:fe00:0%eth0' (ED25519) to the list of known hosts.
admin@fe80::ff:fe00:0%eth0's password: *****
.-------.
|  . .  | Infix OS — Immutable.Friendly.Secure
|-. v .-| https://www.kernelkit.org
'-'---'-'

Run the command 'cli' for interactive OAM

admin@infix-c0-ff-ee:/>
```

### Running a remote command

```
~$ ssh admin@fe80::ff:fe00:0%eth0 echo hej
admin@fe80::ff:fe00:0%eth0's password: *****
hej
```

### Made Easy with mDNS

Connecting to networked devices using IP addresses is the way many
people are used to.  The above example with IPv6 tend to scare off
people, so for the rest of this document we'll use the mDNS name
instead:

```
~$ ssh admin@infix.local%eth0
The authenticity of host 'infix.local%eth0' can't be established.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
...
admin@infix:/>
```

## Factory Reset

The command option `-y` disables any "are you sure?" interaction and
immediately triggers a factory reset and reboot of the device.  It is
when the device boots up it erases all writable storage.

```
~$ ssh admin@infix.local%eth0 factory -y
admin@infix.local%eth0's password: *****
```

## System Reboot

```
~$ ssh admin@infix.local%eth0 reboot
admin@infix.local%eth0's password: *****
```

## Set Date and Time

Devices running Infix may have their system time completely off and this
can cause problems for upgrading and when accessing the web interface
over HTTPS (certificate validation looks at start and end dates).

To set the device's system time, *and* sync that with the RTC:
use the PCs current time as argument:

```
~$ ssh admin@infix.local%eth0 "sudo date -s '2024-03-20 18:14+01:00' && \
                               sudo hwclock -w -u"
admin@infix.local%eth0's password: *****
```

> [!TIP]
> The `-u` option ensures saving system time to the RTC in UTC time.

Verify that the change took:

```
~$ ssh admin@infix.local%eth0 date
admin@infix.local%eth0's password: *****
Wed Mar 20 17:14:47 UTC 2024
```

## Remote Control of Ethernet Ports

There are two ways to do it:

 1. Change the configuration without saving it to `startup-config`
 2. Change the operational state

The first involves sending a NETCONF command/config in XML. The second
we will cover here. We start by querying available interfaces (ports) on
the remote system:

```
~$ ssh admin@infix.local%qtap0 ip -br a
admin@infix.local%qtap0's password:
lo               UP             127.0.0.1/8 ::1/128
e0               UP             fe80::ff:fe00:0/64
e1               UP
e2               UP
e3               UP
e4               UP
e5               UP             fe80::ff:fe00:5/64
e6               UP             fe80::ff:fe00:6/64
e7               UP             fe80::ff:fe00:7/64
e8               UP             fe80::ff:fe00:8/64
e9               UP             192.168.2.200/24 fe80::ff:fe00:9/64
br0              UP
```

Here we see a loopback interface (lo), ten Ethernet ports (e0-e9) and a
bridge (br0).  From this quick glance we can guess that the ports e1-e4
are bridged (you can verify this with the remote command `bridge link`)
because they do not have a link-local IPv6 address.

I know it's port e6 that I want to take down:

```
~$ ssh admin@infix.local%qtap0 ip link set e6 down
admin@infix.local%qtap0's password:
RTNETLINK answers: Operation not permitted
~$ ssh admin@infix.local%qtap0 sudo ip link set e6 down
admin@infix.local%qtap0's password:
```

Changing the operational link state of a port is a privileged command,
so we have to prefix our command with `sudo`.

Inspecting the link state again show the port is now down:

```
~$ ssh admin@infix.local%qtap0 ip -br a
admin@infix.local%qtap0's password:
lo               UP             127.0.0.1/8 ::1/128
e0               UP             fe80::ff:fe00:0/64
e1               UP
e2               UP
e3               UP
e4               UP
e5               UP             fe80::ff:fe00:5/64
e6               DOWN
e7               UP             fe80::ff:fe00:7/64
e8               UP             fe80::ff:fe00:8/64
e9               UP             192.168.2.200/24 fe80::ff:fe00:9/64
br0              UP
```

## Check Network Connectivity

Say you want to perform a [System Upgrade][#system-upgrade] and it just
does not work, a good place to start troubleshooting is ensuring the
device can actually reach the upgrade server.

We run the following `ping` command remotely on the device:

```
~$ ssh admin@infix.local%eth0 ping -c 3 server.local
admin@infix.local%qtap0's password: *****
PING server.local (192.168.2.42) 56(84) bytes of data.
64 bytes from server.local: icmp_seq=1 ttl=64 time=0.201 ms
64 bytes from server.local: icmp_seq=2 ttl=64 time=0.432 ms
64 bytes from server.local: icmp_seq=3 ttl=64 time=0.427 ms

--- server.local ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2050ms
rtt min/avg/max/mdev = 0.201/0.353/0.432/0.107 ms
```

Here we get a reply, so whatever is the issue with the upgrade was
not hiding behind a connectivity issue at least.

## System Upgrade

The underlying software that handles upgrades is called [RAUC][2].  To
trigger an upgrade you (currently) need an FTP/TFTP or HTTP/HTTPS server
where RAUC can fetch the upgrade from.  In this example we use an FTP
server to upgrade the currently inactive "slot":

```
~$ ssh admin@infix.local%eth0 rauc install ftp://server.local/infix-aarch64-24.06.0.pkg
admin@infix.local%eth0's password: *****
installing
  0% Installing
  0% Determining slot states
 20% Determining slot states done.
 20% Checking bundle
 20% Verifying signature
 40% Verifying signature done.
 40% Checking bundle done.
 40% Checking manifest contents
 60% Checking manifest contents done.
 60% Determining target install group
 80% Determining target install group done.
 80% Updating slots
 80% Checking slot rootfs.1
 90% Checking slot rootfs.1 done.
 90% Copying image to rootfs.1
 99% Copying image to rootfs.1 done.
 99% Updating slots done.
100% Installing done.
idle
Installing `ftp://server.local/infix-aarch64-24.06.0.pkg` succeeded
~$
```

The inactive slot is now marked active and will be used on the next
boot.  To upgrade the partition we booted from, we must first reboot.

For more information, see the [Boot Procedure][3] document.

**Alternative:**

If you know your device has sufficient storage, eMMC or RAM disk (check
with the remote command `df -h`), you can also copy the `.pkg` file to
the device instead of having to set up an FTP/TFTP or HTTP/HTTPS server.

Create an upload directory where `admin` has write permission:

```
~$ ssh admin@infix.local%eth0 "sudo mkdir /var/tmp/upload; \
                               sudo chown admin /var/tmp/upload"
admin@infix.local%eth0's password:
```

Copy the file with secure copy, first we show the nasty IPv6 version of
the command:

```
~$ scp infix-aarch64-24.06.0.pkg admin@\[fe80::ff:fe00:0%eth0\]:/var/tmp/upload/
admin@fe80::ff:fe00:0%eth0's password:
infix-aarch64-24.06.0.pkg                              100%  296   601.4KB/s   00:00
```

And the upgrade command itself:

```
~$ ssh admin@infix.local%eth0 rauc install /var/tmp/upload/infix-aarch64-24.06.0.pkg
admin@infix.local%eth0's password: *****
.
.
.
```

Remember to remove the file from the upload directory when you are done,
this can be done before or after the reboot to activate the upgrade.  If
you want to upgrade both "slots", then you can of course keep the file
until you are done (provided the upload directory was created on
persistent storage).

```
~$ ssh admin@infix.local%eth0 rm /var/tmp/upload/infix-aarch64-24.06.0.pkg
admin@infix.local%eth0's password: *****
~$
```

## Controlling LEDs

As part of production testing, LED verification is often expected to be
performed.  Infix uses standard [Linux support for LED management][6],
where LEDs appear in the file system under /sys/class/leds and can be
controlled using *echo* command. `sudo` privileges are required.

When interacting with LEDs this way, first disable the Infix *iitod*
daemon to avoid conflicting LED control.

```
~$ ssh admin@example.local 'initctl stop iitod'
```

Then run the test, e.g., visually control that a red LED labeled 'LAN'
is working.

```
~$ ssh admin@example.local 'echo none | sudo tee /sys/class/leds/red\:lan/trigger'
~$ ssh admin@example.local 'echo 1    | sudo tee /sys/class/leds/red\:lan/brightness'
```

To turn off the same LED, run the following commands.

```
~$ ssh admin@example.local 'echo none | sudo tee /sys/class/leds/red\:lan/trigger'
~$ ssh admin@example.local 'echo 0    | sudo tee /sys/class/leds/red\:lan/brightness'
```

When done with LED testing, enable Infix *iitod* daemon again.

```
~$ ssh admin@example.local 'initctl start iitod'
```

## Reading Power Feed Status

As part of production tests, verification of Power Feed sensors is
often expected to be performed. Infix uses standard [Linux support for
Power management][7], where power sources appear in the file system
under /sys/class/power_supply. The following example reads status of
two power supplies named *pwr1* and *pwr2*.

```
~$ ssh admin@example 'cat /sys/class/power_supply/pwr1/online'
1
~$ ssh admin@example 'cat /sys/class/power_supply/pwr2/online'
0
~$
```

Here, only `pwr1` have power.

[^1]: Press Win-r to bring up the Run dialog, enter `cmd.exe` and press
    enter.  Then type in `ipconfig /all` to list all interfaces, their
    status, as well as interface index.

[1]: discovery.md
[2]: https://rauc.io/
[3]: boot.md#system-upgrade
[6]: https://docs.kernel.org/leds/leds-class.html
[7]: https://docs.kernel.org/power/power_supply_class.html