A major software upgrade is defined as updating the Ubuntu release as well as the Jaiabot software release. This is typically done from major Jaiabot versions (e.g., v1 -> v2). It can also be done when a major version spans support for multiple Ubuntu releases (e.g. v1 on Ubuntu "focal" to v1 on Ubuntu "jammy").
This section written by Claude
The BNO055 IMU is no longer supported. Any bot still fitted with one must have it replaced with a BNO085 before the fleet can be upgraded. For each of those bots:
The major upgrade playbook checks every bot and aborts the entire fleet upgrade if any bot still reports imu_type as bno055. The fleet configuration is checked as well: bno055 is a retired value with no replacement, so the hub-side check refuses a configuration that still names it, before any bot is touched.
If this fleet was generated prior to fleet configuration files (this includes most 1.y fleets), one will need to be created for the current fleet. This can be done using jaia admin fleet create as described in the Embedded Board Deployment document.
This fleet configuration must then be embedded in the upgrade image (see the next step).
If the fleet was generated using a fleet configuration file, a valid file on the hub should already exist at /etc/jaiabot/fleetN.cfg. In this case that fleet configuration will be reused and no further action is required.
This section written by Claude
The hub's copy is not updated when a bot is added with "Pair new Bot", so before reusing it the upgrade compares the hubs and bots it lists with the hub's inventory (/etc/jaiabot/inventory.yml). If they differ, the upgrade stops on the hub, before any node is touched, and asks for the fleet's current configuration to be embedded in the upgrade image (see the next step). A configuration embedded in the image, or given with -e fleet_cfg=..., is used as it is, and a CloudHub's own upgrade skips the check: its inventory follows the nodes paired with it.
Fleet configurations carry a version (see Fleet configuration versioning below). An older file is migrated automatically by the upgrade when that is possible; when it is not, the upgrade stops on the hub, before any bot is touched, with a message saying what must be changed or that the fleet configuration must be regenerated with jaia admin fleet create. To find out in advance, run jaia admin fleet validate /path/to/fleetN.cfg with the release you are upgrading to.
Major upgrades go one release at a time (1.y to 2.y to 3.y): the upgrade refuses to skip a release, since each release only migrates fleet configurations from the release before it.
Download the desired release CD image (e.g. jaiabot_updates_resolute_3.0.0_arm64.iso).
If you need to embed the fleet configuration (see previous section), do so now with:
This section written by Claude
This migrates and validates the fleet config with the fleet config tool of the release on the ISO, embeds it (at major_upgrade/fleetN.cfg within the ISO) and writes a new ISO called /path/to/jaiabot_updates_resolute_3.0.0_arm64_fleetN.iso. If the fleet config cannot be used with that release, this is where you find out.
Now, write the ISO to a USB flash drive (or burn it to a CD-R disc):
Assuming the USB flash drive is /dev/sdb:
Insert the flash drive into the fleet Hub or attach a USB CD drive with the CD inserted, then boot or reboot the Hub to mount it.
Use Ansible to run the major upgrade, either on the command line or via the JCU UI.
where hub_id is the hub in use (the one with the upgrade USB flash key or CD connected) and do_backup is a boolean set to whether the existing (old) rootfs and overlay should be backed up to the /var/log/jaiabot/major_upgrade/vX_codename directory prior to the upgrade.
This section written by Claude
Each node downloads the new images from the hub one at a time, capped at 1.5 MB/s so the upgrade does not saturate the fleet's radio link. Fleets in simulation mode (VirtualBox fleets and VirtualFleets) download uncapped. Pass -e major_upgrade_download_limit_rate=<rate> (a curl --limit-rate value such as 500K, or 0 for no cap) to override either.
Only the hub named by hub_id stages the upgrade (mounts the updates disk and checks the fleet configuration and its SSH key); the other hubs skip staging and download the new images from it like the bots do.
A CloudHub has no USB port: it downloads the update image itself and upgrades only itself, before the rest of the fleet. Its JCU's "Major Upgrade" is limited to the CloudHub and proposes no backup, which a CloudHub's disk is usually too small for.
Nor can a CloudHub be handed the fleet configuration on a USB key or CD, so it takes it from its data bucket. From a workstation with AWS credentials, in this order:
This section written by Claude
Run the playbook from a git checkout of the release being installed, not from the copy installed on the hub. A hub still running the old release has the old playbook under /usr/share/jaiabot/config/ansible, and that playbook builds the new boot filesystem with the old release's fleet config tool, which cannot write a preseed the new image can read. Nothing reports an error until after the root filesystem has already been swapped, so the node reboots into the new release unconfigured: no hostname, no network, no jaiabot, reachable only over a serial console.
This section written by Claude
The fleet does not have to be upgraded in one pass. A hub that has already been upgraded can upgrade bots still running the old release, so bots that were out of the water, switched off or otherwise absent can be brought up to the new release whenever they next appear:
A node is upgraded when it runs the release immediately before the one being installed, and is left alone when it already runs the new one, so re-running is safe and only touches what is still behind. The hub reports No major upgrade to perform: current Ubuntu version is same as new version for itself on these later passes; that is the expected result for a hub that is already current, and the run continues for the bots that are not.
Keep the updates disk attached to the staging hub for as long as the fleet is mixed, since that hub serves the images to every node that has yet to be upgraded.
A failure on one node does not stop the others, which is what makes upgrading in passes possible. It also means a node that genuinely failed is easy to miss in a long run, so check the PLAY RECAP for failed= against each host before treating a fleet as fully upgraded.
An error that stops Ansible from loading the play at all ends the run without printing a PLAY RECAP, so the absence of one is not the same as nothing having happened: the run may already have stopped jaiabot across the fleet. Start it again with sudo systemctl start jaiabot on each node once the error is fixed.
A node held back by a failed check keeps its old filesystems and is left with jaiabot stopped; start it again with sudo systemctl start jaiabot once the cause is understood.
A bot missing from the fleet configuration is held back this way, since the new release's tool will not write a first boot preseed for a bot it has not been told about. Correct the fleet configuration before catching that bot up: the hubs upgraded in the same pass carry the configuration they were given, so a bot left out of it is also absent from the inventory they generate, and a later run would not target it at all.
The major upgrade extracts a new filesystem image and configures it, much like a generating a new bot or hub as described in the Embedded Board Deployment document. This means that the state of the previous installation filesystem is largely irrelevant.
The first part of the upgrade is handled by Ansible and the second part is handled by a shell script on each machine.
After the upgrade, the rootfs and overlay partitions are swapped, and the boot partition is overwritten. The data (log) partition is unmodified:
This is necessary to allow the running rootfs to bootstrap the new image. This works because both partitions are the same size (8GB) and all references use the filesystem label not the partition ID. Old systems that use ext4 for the rootfs change this to use btrfs going forward.
See major-upgrade.yml for full details on the steps performed.
This section written by Claude
The playbook ensures that the new Ubuntu version is newer than the existing version and new Jaiabot version is not older than the existing version, to prevent meaningless double-upgrades or accidental major upgrades that should be normal upgrades (e.g. 1.16.0->1.17.0). It also checks that the system runs the release immediately before the one being installed (major_upgrade_previous_major in the ISO's version.txt), so releases are never skipped.
On the hub, before anything else, the staged fleet configuration is migrated and validated with the fleet config tool (jaia-fleet-config.py) taken from the new release's boot tarball on the updates disk (tasks/hub-check-fleet-config.yml). A fleet configuration that the new release cannot use stops the upgrade here, on the hub, with the reason.
The upgrade is staged in /var/log/jaiabot/major_upgrade/v{X}_{ubuntucode} where {X} is the major version to be updated to and {ubuntucode} is the new Ubuntu version. For example: /var/log/jaiabot/major_upgrade/v3_resolute.
The new upgrade image for the rootfs and bootfs are downloaded from the Hub into the staging directory.
A backup of the old rootfs and overlayfs is copied and compressed to /var/log/jaiabot/major_upgrade/v{X}_{ubuntucode}/backup/rootfs.tar.gz and overlay.tar.gz, respectively, as the major upgrade will clear both the old rootfs and overlayfs.
This section written by Claude
Nothing of a node's previous installation is carried over except on the CloudHub, whose /etc/wireguard (the VPN server's keys and peer list, which only its creation makes) is copied into the staging directory and restored on the new image. Every other node comes out of the upgrade as a new node would: with the bootstrap key (id_vpn_tmp) written from the fleet configuration and no CloudHub VPN enrollment. After every major upgrade, re-pair the fleet as for a new fleet: run "Open Fleet Pairing" and then "Pair Fleet to CloudHub" from the JCU on the hub (see CloudHub VPN). Until then nothing reaches the CloudHub over the VPN, including the hub-to-hub link through it.
The CloudHub's upgrade also records the fleet's bootstrap key from the fleet configuration and marks the CloudHub as handed over, on its data partition, so that "Open Fleet Pairing" can admit nodes and open port 22. A CloudHub created by an earlier release can open its security group for pairing only once jaia admin fleet cloudhub prep_major_upgrade has given its IAM role this release's policy (see Running the major upgrade).
The new boot filesystem is prepared from the fleet configuration file by the fleet config tool found inside the new boot tarball (jaiabot/init/fleet_config/jaia-fleet-config.py generate), never by the installed jaia tool: this way the template, schema and migrations always come from the release being installed, whatever release the node runs today (so a hub already on the new release can upgrade bots that are still on the previous one).
Finally the do-major-upgrade.sh is configured to /var/log/jaiabot/major_upgrade/v{X}_{ubuntucode} and run.
do-major-upgrade.sh finalizes the upgrade:
This script logs to /var/log/jaiabot/major_upgrade/v{X}_{ubuntucode}/major_upgrade_final.log
This section written by Claude
The fleet configuration is written once when a fleet is created and read again at every major upgrade, possibly years later by a newer release (see the Ansible steps above). This section describes how changes to what a fleet config must contain are tracked, so that a major upgrade either migrates an older file automatically or fails early with a clear message rather than silently producing a misconfigured system.
src/lib/messages/fleet_config.proto is the single description of what a fleet config may contain, including the bot/hub settings that debconf asks for on first boot: the NodeSettings message lists every question as a typed field, and its (jaia.field).debconf options say which nodes it applies to (group: ALL|BOT|HUB), whether it is asked before or after another (declaration order), its description, and the few conditions the interactive flow needs (ask_if, unasked_value). Questions are asked at debconf priority MEDIUM, which keeps a question added in a later release from interrupting an interactive apt upgrade; dpkg-reconfigure jaiabot-embedded asks every question regardless. Enum values map to debconf values by dropping the enum name prefix and lowercasing (BOT_TYPE_PAM is pam); a deprecated value says what it maps to ((jaia.ev).debconf.replaced_by).
Two annotations say that an answer does not belong in the fleet-wide settings block, and validation rejects it there:
From this, scripts/build/fleet-config-debconf-gen.py generates debian/jaiabot-embedded.templates and debian/jaiabot-embedded.config. The build regenerates both whenever the proto or the generator changes, so never edit them by hand. They are committed so that a change to the proto shows its effect on the debconf questions in review; commit the regenerated files with the proto change (the fleet_config unit tests fail if the committed copy is stale). Without a build, regenerate them with:
The file itself declares its contract version:
PROJECT_FLEET_CONFIG_VERSION in CMake is read from that line. Fleet configs record the version they were written for in their version field; files that predate the field are version 1 (every file created before the 3.y release). A tool refuses a file newer than itself.
The build writes share/jaiabot/fleet_config/fleet_config.desc (a protobuf FileDescriptorSet of the proto and its options). It is all the fleet config tool needs to parse, validate, migrate and render fleet configs, without generated bindings, which is what lets the tool run on a node of the previous release during a major upgrade.
jaia-fleet-config.py (src/sh/fleet/) is behind jaia admin fleet:
| Command | Purpose |
|---|---|
| jaia admin fleet version | the fleet config version this release writes |
| jaia admin fleet validate fleetN.cfg | report the file's version, whether it can be migrated, and every problem |
| jaia admin fleet migrate fleetN.cfg [-o out.cfg] [--check] | rewrite the file at the current version |
| jaia admin fleet generate fleetN.cfg <bot\|hub> <id> | first-boot files for one node; migrates in memory and stores the migrated file on hubs |
| jaia admin fleet create | interactive creation; writes the current version |
| jaia admin fleet edit fleetN.cfg [-o out.cfg] | re-ask every question with the file's answers filled in; whatever a failed migration could not carry over is reported and starts at its default |
| jaia admin fleet update_iso | migrate and validate with the tool of the release on the ISO, then embed |
| jaia admin fleet cloudhub set_key fleetN.cfg hub30_fleetN.pub | record the public key a CloudHub made for itself (cloudhub create runs it) |
| jaia admin fleet cloudhub prep_major_upgrade fleetN.cfg --region <region> | before a CloudHub's major upgrade, upload the fleet configuration to its data bucket, give its AWS role this release's policy and close any port 22 rule its creation left open |
It is copied into the boot partition of every image (jaiabot/init/fleet_config/, next to first-boot.preseed.yml.j2) together with fleet_config.desc, so the major upgrade runs the new release's tool from the new boot tarball, on the hub and on every node. It renders the first-boot template with Jinja2's StrictUndefined, so a missing value is an error rather than a blank line, and the template's first line dereferences a variable only this tool defines, so an older generator fails on it instead of writing a broken preseed.
A fleet config older than the current version is migrated one version at a time by the steps in MIGRATIONS in jaia-fleet-config.py. Migration happens in memory whenever a file is read (validate, generate, update_iso), and is written back by migrate; hubs always store the migrated form so the next upgrade starts from a known version. A step either produces a file that passes validation or refuses with a list of problems naming the fields and values concerned (never their values, which may be secrets); the operator then fixes the file or regenerates it.
Version 1 to 2 turns the string debconf answers (debconf { key: "jaiabot-embedded/bot_type" value: "echo" }) into the typed settings { bot_type: BOT_TYPE_PAM }, mapping deprecated values through replaced_by, dropping identity answers (type, ids, mode: those are set per node) with a note, refusing unknown questions and values, and writing every unanswered question at its default so the file is a complete record.
fleet_config_contract (built as part of all) compares the current proto against the frozen copy for the version it declares, src/lib/messages/fleet_config/contract/vN/fleet_config.proto (plus, for version 1 whose questions were still a hand-written templates file, jaiabot-embedded.templates). It reruns whenever the proto, a snapshot or the tool changes. Differences are classified and the build fails with instructions unless the two match:
| Change | Classification | What to do |
|---|---|---|
| optional/repeated field added | compatible | refresh the snapshot |
| enum value added (a new choice) | compatible | refresh the snapshot |
| question added with a default | compatible | refresh the snapshot |
| a setting's default changed | compatible | refresh the snapshot (fleet configs carry explicit values) |
| required field added | breaking | prefer optional plus a validation rule; otherwise bump and migrate |
| field or enum value marked [deprecated = true] | breaking | bump and migrate (the migration clears or maps it) |
| a FleetConfig field's default changed | breaking | bump and migrate |
| question added without a default | breaking | bump and migrate (the migration must supply a value) |
| whether a question is set per node changed | breaking | bump and migrate |
| field, message or enum value removed or renamed; field number, type or label changed | forbidden | undo it: mark the old item [deprecated = true] instead |
In text format the field name is the wire format, so removing or renaming a field would stop every existing fleet config from parsing. Fields are therefore never deleted, only deprecated, and the check refuses a snapshot history that removes anything - including a removal committed together with a version bump.
How old a file has to be readable is not bounded by the upgrade policy. A hub's stored copy is rewritten at every major upgrade, but the copy kept with the fleet's keys is not, and minor upgrades never rewrite either, so a fleet created years ago can still be version 1 on someone's disk when a much newer release is asked to read it.
The plan for bounding the schema is to declare an oldest readable version beside the current one, set to the fleet config version that the first release of the previous major shipped (the first, because a system on the last minor of 3.y may still hold the file 3.0 wrote). A field or enum value deprecated strictly before that floor can then be deleted, so the schema carries at most two majors of history. For release 3 the floor is 1 and nothing is deletable; release 4 would set it to 2 and retire the version 1 scaffolding in one go - debconf, debconf_override, the replaced enum values, migrate_1_to_2 and the v1 snapshot.
Four things make it enforceable, each an extension of something that already exists:
Below the floor the promise changes from "migrated automatically" to "answer the questions again": jaia admin fleet edit reads what it can of the old file and asks the rest, so no one has to find an old release to move forward. Raising the floor therefore has an operational cost - somebody has to touch the oldest fleet configs before their next major upgrade - which is the reason to raise it only when a major release is already being cut.
refreshes the snapshot for the declared version; commit it.
scripts/build/fleet-config-contract.py diff 1 2 prints the classified differences between two snapshot versions, show prints the current contract model, and src/test/fleet_config/ (run by ctest -R fleet_config with -Denable_testing=ON) exercises the classification, the check, the generator and the tool's migrations against fixtures.