JaiaBot 3.0.0
JaiaBot micro-AUV software
Loading...
Searching...
No Matches
Major software upgrade

A major software upgrade is defined as updating the Ubuntu release as well as the Jaiabot software release. This is typically done from major Jaiabot versions (e.g., v1 -> v2). It can also be done when a major version spans support for multiple Ubuntu releases (e.g. v1 on Ubuntu "focal" to v1 on Ubuntu "jammy").

Preparing for major upgrade

Ensure all bots have a BNO085 IMU

This section written by Claude

The BNO055 IMU is no longer supported. Any bot still fitted with one must have it replaced with a BNO085 before the fleet can be upgraded. For each of those bots:

  1. Set the imu type to bno085 on the bot itself with jaia admin debconf set imu_type bno085. This is what the major upgrade checks.
  2. Change it in the fleet configuration too with jaia admin fleet edit fleetN.cfg, since the upgraded system is configured from the fleet configuration rather than from the current debconf database. If the fleet configuration is embedded in the upgrade image, re-embed the new one (see the next sections).

The major upgrade playbook checks every bot and aborts the entire fleet upgrade if any bot still reports imu_type as bno055. The fleet configuration is checked as well: bno055 is a retired value with no replacement, so the hub-side check refuses a configuration that still names it, before any bot is touched.

Ensure the Hub has a valid Fleet Configuration

If this fleet was generated prior to fleet configuration files (this includes most 1.y fleets), one will need to be created for the current fleet. This can be done using jaia admin fleet create as described in the Embedded Board Deployment document.

This fleet configuration must then be embedded in the upgrade image (see the next step).

If the fleet was generated using a fleet configuration file, a valid file on the hub should already exist at /etc/jaiabot/fleetN.cfg. In this case that fleet configuration will be reused and no further action is required.

This section written by Claude

The hub's copy is not updated when a bot is added with "Pair new Bot", so before reusing it the upgrade compares the hubs and bots it lists with the hub's inventory (/etc/jaiabot/inventory.yml). If they differ, the upgrade stops on the hub, before any node is touched, and asks for the fleet's current configuration to be embedded in the upgrade image (see the next step). A configuration embedded in the image, or given with -e fleet_cfg=..., is used as it is, and a CloudHub's own upgrade skips the check: its inventory follows the nodes paired with it.

Fleet configurations carry a version (see Fleet configuration versioning below). An older file is migrated automatically by the upgrade when that is possible; when it is not, the upgrade stops on the hub, before any bot is touched, with a message saying what must be changed or that the fleet configuration must be regenerated with jaia admin fleet create. To find out in advance, run jaia admin fleet validate /path/to/fleetN.cfg with the release you are upgrading to.

Major upgrades go one release at a time (1.y to 2.y to 3.y): the upgrade refuses to skip a release, since each release only migrates fleet configurations from the release before it.

Download and flash the upgrade image

Download the desired release CD image (e.g. jaiabot_updates_resolute_3.0.0_arm64.iso).

If you need to embed the fleet configuration (see previous section), do so now with:

jaia admin fleet update_iso /path/to/fleetN.cfg /path/to/jaiabot_updates_resolute_3.0.0_arm64.iso

This section written by Claude

This migrates and validates the fleet config with the fleet config tool of the release on the ISO, embeds it (at major_upgrade/fleetN.cfg within the ISO) and writes a new ISO called /path/to/jaiabot_updates_resolute_3.0.0_arm64_fleetN.iso. If the fleet config cannot be used with that release, this is where you find out.

Now, write the ISO to a USB flash drive (or burn it to a CD-R disc):

Assuming the USB flash drive is /dev/sdb:

# for fleets WITHOUT previous fleet config (most 1.y fleets)
ISO=jaiabot_updates_resolute_3.0.0_arm64_fleetN.iso
# OR for fleets WITH previous fleet config
ISO=jaiabot_updates_resolute_3.0.0_arm64.iso
sudo dd if=$ISO of=/dev/sdb bs=1M status=progress

Insert the flash drive into the fleet Hub or attach a USB CD drive with the CD inserted, then boot or reboot the Hub to mount it.

Running the major upgrade

Use Ansible to run the major upgrade, either on the command line or via the JCU UI.

# from a git checkout of the release being installed - see below
cd jaiabot/config/ansible/major_upgrade
ansible-playbook -i /etc/jaiabot/inventory.yml major-upgrade.yml -e hub_id=1 -e do_backup=yes

where hub_id is the hub in use (the one with the upgrade USB flash key or CD connected) and do_backup is a boolean set to whether the existing (old) rootfs and overlay should be backed up to the /var/log/jaiabot/major_upgrade/vX_codename directory prior to the upgrade.

This section written by Claude

Each node downloads the new images from the hub one at a time, capped at 1.5 MB/s so the upgrade does not saturate the fleet's radio link. Fleets in simulation mode (VirtualBox fleets and VirtualFleets) download uncapped. Pass -e major_upgrade_download_limit_rate=<rate> (a curl --limit-rate value such as 500K, or 0 for no cap) to override either.

Only the hub named by hub_id stages the upgrade (mounts the updates disk and checks the fleet configuration and its SSH key); the other hubs skip staging and download the new images from it like the bots do.

A CloudHub has no USB port: it downloads the update image itself and upgrades only itself, before the rest of the fleet. Its JCU's "Major Upgrade" is limited to the CloudHub and proposes no backup, which a CloudHub's disk is usually too small for.

Nor can a CloudHub be handed the fleet configuration on a USB key or CD, so it takes it from its data bucket. From a workstation with AWS credentials, in this order:

  1. jaia admin fleet migrate fleetN.cfg, with the release being installed.
  2. jaia admin fleet cloudhub prep_major_upgrade fleetN.cfg --region <region>. It refuses a configuration that does not validate, uploads it to s3://<data bucket>/jaia/major_upgrade/fleetN.cfg, gives the CloudHub's IAM role this release's policy, and closes any port 22 rule its creation left open. --dry-run reports what would change without changing it. IAM changes can take a minute or two to apply, so pairing or support access opened straight afterwards may be refused briefly.
  3. The CloudHub's JCU "Major Upgrade". The upgrade prefers the configuration in the data bucket to the one on the CloudHub, and stops before touching anything if it finds neither.

Use the playbook from the release you are installing

This section written by Claude

Run the playbook from a git checkout of the release being installed, not from the copy installed on the hub. A hub still running the old release has the old playbook under /usr/share/jaiabot/config/ansible, and that playbook builds the new boot filesystem with the old release's fleet config tool, which cannot write a preseed the new image can read. Nothing reports an error until after the root filesystem has already been swapped, so the node reboots into the new release unconfigured: no hostname, no network, no jaiabot, reachable only over a serial console.

Upgrading a fleet a few nodes at a time

This section written by Claude

The fleet does not have to be upgraded in one pass. A hub that has already been upgraded can upgrade bots still running the old release, so bots that were out of the water, switched off or otherwise absent can be brought up to the new release whenever they next appear:

# later, with the remaining bots powered on and in the inventory
ansible-playbook -i /etc/jaiabot/inventory.yml major-upgrade.yml -e hub_id=1 -e do_backup=yes

A node is upgraded when it runs the release immediately before the one being installed, and is left alone when it already runs the new one, so re-running is safe and only touches what is still behind. The hub reports No major upgrade to perform: current Ubuntu version is same as new version for itself on these later passes; that is the expected result for a hub that is already current, and the run continues for the bots that are not.

Keep the updates disk attached to the staging hub for as long as the fleet is mixed, since that hub serves the images to every node that has yet to be upgraded.

A failure on one node does not stop the others, which is what makes upgrading in passes possible. It also means a node that genuinely failed is easy to miss in a long run, so check the PLAY RECAP for failed= against each host before treating a fleet as fully upgraded.

An error that stops Ansible from loading the play at all ends the run without printing a PLAY RECAP, so the absence of one is not the same as nothing having happened: the run may already have stopped jaiabot across the fleet. Start it again with sudo systemctl start jaiabot on each node once the error is fixed.

A node held back by a failed check keeps its old filesystems and is left with jaiabot stopped; start it again with sudo systemctl start jaiabot once the cause is understood.

A bot missing from the fleet configuration is held back this way, since the new release's tool will not write a first boot preseed for a bot it has not been told about. Correct the fleet configuration before catching that bot up: the hubs upgraded in the same pass carry the configuration they were given, so a bot left out of it is also absent from the inventory they generate, and a later run would not target it at all.

Major upgrade design

The major upgrade extracts a new filesystem image and configures it, much like a generating a new bot or hub as described in the Embedded Board Deployment document. This means that the state of the previous installation filesystem is largely irrelevant.

The first part of the upgrade is handled by Ansible and the second part is handled by a shell script on each machine.

After the upgrade, the rootfs and overlay partitions are swapped, and the boot partition is overwritten. The data (log) partition is unmodified:

  • Before:
    • /dev/sdX1: [boot (v1)] (fat32)
    • /dev/sdX2: [rootfs (v1)] (ext4 or btrfs)
    • /dev/sdX3: [overlay (v1)] (btrfs)
    • /dev/sdX4: [data] (btrfs)
  • After:
    • /dev/sdX1: [boot (v2)] (fat32)
    • /dev/sdX2: [overlay (v2)] (btrfs)
    • /dev/sdX3: [rootfs (v2)] (btrfs)
    • /dev/sdX4: [data] (btrfs)

This is necessary to allow the running rootfs to bootstrap the new image. This works because both partitions are the same size (8GB) and all references use the filesystem label not the partition ID. Old systems that use ext4 for the rootfs change this to use btrfs going forward.

Ansible steps

See major-upgrade.yml for full details on the steps performed.

This section written by Claude

The playbook ensures that the new Ubuntu version is newer than the existing version and new Jaiabot version is not older than the existing version, to prevent meaningless double-upgrades or accidental major upgrades that should be normal upgrades (e.g. 1.16.0->1.17.0). It also checks that the system runs the release immediately before the one being installed (major_upgrade_previous_major in the ISO's version.txt), so releases are never skipped.

On the hub, before anything else, the staged fleet configuration is migrated and validated with the fleet config tool (jaia-fleet-config.py) taken from the new release's boot tarball on the updates disk (tasks/hub-check-fleet-config.yml). A fleet configuration that the new release cannot use stops the upgrade here, on the hub, with the reason.

The upgrade is staged in /var/log/jaiabot/major_upgrade/v{X}_{ubuntucode} where {X} is the major version to be updated to and {ubuntucode} is the new Ubuntu version. For example: /var/log/jaiabot/major_upgrade/v3_resolute.

The new upgrade image for the rootfs and bootfs are downloaded from the Hub into the staging directory.

A backup of the old rootfs and overlayfs is copied and compressed to /var/log/jaiabot/major_upgrade/v{X}_{ubuntucode}/backup/rootfs.tar.gz and overlay.tar.gz, respectively, as the major upgrade will clear both the old rootfs and overlayfs.

This section written by Claude

Nothing of a node's previous installation is carried over except on the CloudHub, whose /etc/wireguard (the VPN server's keys and peer list, which only its creation makes) is copied into the staging directory and restored on the new image. Every other node comes out of the upgrade as a new node would: with the bootstrap key (id_vpn_tmp) written from the fleet configuration and no CloudHub VPN enrollment. After every major upgrade, re-pair the fleet as for a new fleet: run "Open Fleet Pairing" and then "Pair Fleet to CloudHub" from the JCU on the hub (see CloudHub VPN). Until then nothing reaches the CloudHub over the VPN, including the hub-to-hub link through it.

The CloudHub's upgrade also records the fleet's bootstrap key from the fleet configuration and marks the CloudHub as handed over, on its data partition, so that "Open Fleet Pairing" can admit nodes and open port 22. A CloudHub created by an earlier release can open its security group for pairing only once jaia admin fleet cloudhub prep_major_upgrade has given its IAM role this release's policy (see Running the major upgrade).

The new boot filesystem is prepared from the fleet configuration file by the fleet config tool found inside the new boot tarball (jaiabot/init/fleet_config/jaia-fleet-config.py generate), never by the installed jaia tool: this way the template, schema and migrations always come from the release being installed, whatever release the node runs today (so a hub already on the new release can upgrade bots that are still on the previous one).

Finally the do-major-upgrade.sh is configured to /var/log/jaiabot/major_upgrade/v{X}_{ubuntucode} and run.

Scripted steps

do-major-upgrade.sh finalizes the upgrade:

  • pivot the root filesystem from the normal overlayfs onto just the normally read-only rootfs partition. This frees up the "overlay" partition to use as the new rootfs.
  • extract new rootfs onto old overlay partition
  • restore reused files (Wireguard config)
  • write new boot filesystem
  • swap labels for rootfs/overlay
  • reboot into new filesystem (which runs first-boot cloud-init to finish configuration of new system).

This script logs to /var/log/jaiabot/major_upgrade/v{X}_{ubuntucode}/major_upgrade_final.log

Fleet configuration versioning

This section written by Claude

The fleet configuration is written once when a fleet is created and read again at every major upgrade, possibly years later by a newer release (see the Ansible steps above). This section describes how changes to what a fleet config must contain are tracked, so that a major upgrade either migrates an older file automatically or fails early with a clear message rather than silently producing a misconfigured system.

The proto is the contract

src/lib/messages/fleet_config.proto is the single description of what a fleet config may contain, including the bot/hub settings that debconf asks for on first boot: the NodeSettings message lists every question as a typed field, and its (jaia.field).debconf options say which nodes it applies to (group: ALL|BOT|HUB), whether it is asked before or after another (declaration order), its description, and the few conditions the interactive flow needs (ask_if, unasked_value). Questions are asked at debconf priority MEDIUM, which keeps a question added in a later release from interrupting an interactive apt upgrade; dpkg-reconfigure jaiabot-embedded asks every question regardless. Enum values map to debconf values by dropping the enum name prefix and lowercasing (BOT_TYPE_PAM is pam); a deprecated value says what it maps to ((jaia.ev).debconf.replaced_by).

Two annotations say that an answer does not belong in the fleet-wide settings block, and validation rejects it there:

  • identity: true for what identifies the node itself (type, the ids, mode). It is never stored in a fleet config at all; the tool supplies it per node when it generates the first-boot files.
  • per_node: true for an answer that is different on every node, such as a VIN or a serial number. jaia admin fleet create and edit ask it once per bot or hub and store it in that node's override { } block, so nobody has to build an override set by hand.

From this, scripts/build/fleet-config-debconf-gen.py generates debian/jaiabot-embedded.templates and debian/jaiabot-embedded.config. The build regenerates both whenever the proto or the generator changes, so never edit them by hand. They are committed so that a change to the proto shows its effect on the debconf questions in review; commit the regenerated files with the proto change (the fleet_config unit tests fail if the committed copy is stale). Without a build, regenerate them with:

scripts/build/fleet-config-debconf-gen.py --write

The file itself declares its contract version:

option (jaia.file).fleet_config_version = 2;

PROJECT_FLEET_CONFIG_VERSION in CMake is read from that line. Fleet configs record the version they were written for in their version field; files that predate the field are version 1 (every file created before the 3.y release). A tool refuses a file newer than itself.

The build writes share/jaiabot/fleet_config/fleet_config.desc (a protobuf FileDescriptorSet of the proto and its options). It is all the fleet config tool needs to parse, validate, migrate and render fleet configs, without generated bindings, which is what lets the tool run on a node of the previous release during a major upgrade.

The fleet config tool

jaia-fleet-config.py (src/sh/fleet/) is behind jaia admin fleet:

Command Purpose
jaia admin fleet version the fleet config version this release writes
jaia admin fleet validate fleetN.cfg report the file's version, whether it can be migrated, and every problem
jaia admin fleet migrate fleetN.cfg [-o out.cfg] [--check] rewrite the file at the current version
jaia admin fleet generate fleetN.cfg <bot\|hub> <id> first-boot files for one node; migrates in memory and stores the migrated file on hubs
jaia admin fleet create interactive creation; writes the current version
jaia admin fleet edit fleetN.cfg [-o out.cfg] re-ask every question with the file's answers filled in; whatever a failed migration could not carry over is reported and starts at its default
jaia admin fleet update_iso migrate and validate with the tool of the release on the ISO, then embed
jaia admin fleet cloudhub set_key fleetN.cfg hub30_fleetN.pub record the public key a CloudHub made for itself (cloudhub create runs it)
jaia admin fleet cloudhub prep_major_upgrade fleetN.cfg --region <region> before a CloudHub's major upgrade, upload the fleet configuration to its data bucket, give its AWS role this release's policy and close any port 22 rule its creation left open

It is copied into the boot partition of every image (jaiabot/init/fleet_config/, next to first-boot.preseed.yml.j2) together with fleet_config.desc, so the major upgrade runs the new release's tool from the new boot tarball, on the hub and on every node. It renders the first-boot template with Jinja2's StrictUndefined, so a missing value is an error rather than a blank line, and the template's first line dereferences a variable only this tool defines, so an older generator fails on it instead of writing a broken preseed.

Migrations

A fleet config older than the current version is migrated one version at a time by the steps in MIGRATIONS in jaia-fleet-config.py. Migration happens in memory whenever a file is read (validate, generate, update_iso), and is written back by migrate; hubs always store the migrated form so the next upgrade starts from a known version. A step either produces a file that passes validation or refuses with a list of problems naming the fields and values concerned (never their values, which may be secrets); the operator then fixes the file or regenerates it.

Version 1 to 2 turns the string debconf answers (debconf { key: "jaiabot-embedded/bot_type" value: "echo" }) into the typed settings { bot_type: BOT_TYPE_PAM }, mapping deprecated values through replaced_by, dropping identity answers (type, ids, mode: those are set per node) with a note, refusing unknown questions and values, and writing every unanswered question at its default so the file is a complete record.

The build-time check

fleet_config_contract (built as part of all) compares the current proto against the frozen copy for the version it declares, src/lib/messages/fleet_config/contract/vN/fleet_config.proto (plus, for version 1 whose questions were still a hand-written templates file, jaiabot-embedded.templates). It reruns whenever the proto, a snapshot or the tool changes. Differences are classified and the build fails with instructions unless the two match:

Change Classification What to do
optional/repeated field added compatible refresh the snapshot
enum value added (a new choice) compatible refresh the snapshot
question added with a default compatible refresh the snapshot
a setting's default changed compatible refresh the snapshot (fleet configs carry explicit values)
required field added breaking prefer optional plus a validation rule; otherwise bump and migrate
field or enum value marked [deprecated = true] breaking bump and migrate (the migration clears or maps it)
a FleetConfig field's default changed breaking bump and migrate
question added without a default breaking bump and migrate (the migration must supply a value)
whether a question is set per node changed breaking bump and migrate
field, message or enum value removed or renamed; field number, type or label changed forbidden undo it: mark the old item [deprecated = true] instead

In text format the field name is the wire format, so removing or renaming a field would stop every existing fleet config from parsing. Fields are therefore never deleted, only deprecated, and the check refuses a snapshot history that removes anything - including a removal committed together with a version bump.

Deleting a retired field (planned for release 4)

How old a file has to be readable is not bounded by the upgrade policy. A hub's stored copy is rewritten at every major upgrade, but the copy kept with the fleet's keys is not, and minor upgrades never rewrite either, so a fleet created years ago can still be version 1 on someone's disk when a much newer release is asked to read it.

The plan for bounding the schema is to declare an oldest readable version beside the current one, set to the fleet config version that the first release of the previous major shipped (the first, because a system on the last minor of 3.y may still hold the file 3.0 wrote). A field or enum value deprecated strictly before that floor can then be deleted, so the schema carries at most two majors of history. For release 3 the floor is 1 and nothing is deletable; release 4 would set it to 2 and retire the version 1 scaffolding in one go - debconf, debconf_override, the replaced enum values, migrate_1_to_2 and the v1 snapshot.

Four things make it enforceable, each an extension of something that already exists:

  • the floor is declared as a file option next to fleet_config_version, so the two move together;
  • the contract check already walks the snapshot history pairwise, so it can tell which version first deprecated an item and allow the deletion only when that version is below the floor;
  • the tool reads the version before parsing (it is written first, and absent means 1), so a file below the floor is refused with a message rather than a parse error;
  • the migration chain, and the test that asserts one step per version, start at the floor instead of at 1.

Below the floor the promise changes from "migrated automatically" to "answer the questions again": jaia admin fleet edit reads what it can of the old file and asks the rest, so no one has to find an old release to move forward. Raising the floor therefore has an operational cost - somebody has to touch the oldest fleet configs before their next major upgrade - which is the reason to raise it only when a major release is already being cut.

Compatible change

scripts/build/fleet-config-contract.py write

refreshes the snapshot for the declared version; commit it.

Breaking change

  1. Increment option (jaia.file).fleet_config_version in fleet_config.proto.
  2. Add migrate_N_to_N1 to MIGRATIONS in src/sh/fleet/jaia-fleet-config.py. Deprecated values are mapped by replaced_by automatically; anything without a derivable value must refuse with a message telling the operator what to change.
  3. scripts/build/fleet-config-contract.py write and commit contract/vN+1/ alongside the previous snapshots, which are never modified.
  4. Add a fixture and a migration test under src/test/fleet_config/.

Debugging

scripts/build/fleet-config-contract.py diff 1 2 prints the classified differences between two snapshot versions, show prints the current contract model, and src/test/fleet_config/ (run by ctest -R fleet_config with -Denable_testing=ON) exercises the classification, the check, the generator and the tool's migrations against fixtures.