JaiaBot 3.0.0
JaiaBot micro-AUV software
Loading...
Searching...
No Matches
Embedded Board Deployment

JaiaBot uses the Raspberry Pi (RP) Compute Module 4 (CM4) as the embedded Linux computer. For R&D purposes, it has also been necessary to run the jaiabot software on a Raspberry Pi 3 although this is not ideal due to the port mappings being different from the RP4.

Quick start

Download the appropriate .img.gz from AWS S3 (produced by CircleCI): https://us-east-1.console.aws.amazon.com/s3/buckets/jaia-disk-images?region=us-east-1&bucketType=general&prefix=release/3.y/rpi/&showversions=false

Flash this to as many SD cards as you have bots plus hubs:

# assuming USB thumb drive is on /dev/sdd
gunzip -c jaiabot__rootfs-jammy-v1.16.0__code-v1.16.0.img.gz | sudo dd of=/dev/sdd bs=1M status=progress

This section written by Claude

Create a fleet, generate first-boot configurations, and boot:

# Create the Fleet
jaia admin fleet create private_jaia/fleet_config/fleet5.cfg
# insert flashed SD card for Hub 1
jaia admin fleet generate private_jaia/fleet_config/fleet5.cfg hub 1
# unmount & remove Hub 1 card, insert flashed SD card for Hub 2
jaia admin fleet generate private_jaia/fleet_config/fleet5.cfg hub 2
# unmount & remove Hub 2 card, insert flashed SD card for Bot 1
jaia admin fleet generate private_jaia/fleet_config/fleet5.cfg bot 1
# unmount & remove Bot 1 card, insert flashed SD card for Bot 2
jaia admin fleet generate private_jaia/fleet_config/fleet5.cfg bot 2
# Insert all the SD cards and boot all the bots and hubs and allow to fully first boot configure

Setup steps (new RP CM4)

Create a Fleet Config (do once for all bots/hubs in a fleet)

This new bot or hub must belong to a configured fleet. Fleets are configured using a Fleet Config file which is a Protobuf TextFormat version of the jaiabot.protobuf.FleetConfig message (jaiabot/src/lib/messages/fleet_config.proto).

The simplest way to create a new fleet is to use the jaia tool (e.g. for fleet5):

jaia admin fleet create fleet5.cfg

This section written by Claude

This will prompt you using a whiptail UI to set the various fleet related settings, and write the contents to fleet5.cfg. The bot/hub settings it asks for are the same questions, in the same order, as jaiabot-embedded asks on a node (both come from fleet_config.proto), first for the whole fleet and then, optionally, for sets of bots or hubs whose answers differ. Every dialog has a Back button that returns to the previous question; going back past the first one leaves without writing anything. For a VirtualFleet or VirtualBox fleet, add --simulation; edit keeps the fleet's type.

This tool will also create the hub keys (with a Yubikey per physical hub; the CloudHub makes its own key, see CloudHub SSH key), VPN temporary keys as necessary and include those in the fleet configuration file. It needs ssh-keygen, ykman (yubikey-manager) and whiptail on the machine running it.

The bot/hub settings are stored as typed fields of the settings { } block (with per-node override { } blocks), and the file records the fleet config version it was written for. jaia admin fleet validate fleet5.cfg checks a file, and jaia admin fleet migrate rewrites an older file at the current version; see Fleet configuration versioning.

To change an existing fleet, run

jaia admin fleet edit fleet5.cfg

which asks the same questions with the file's own answers filled in (including the ones that are different on every node, such as each bot's VIN and tail serial number, asked once per bot), so you step through with Back and forward and change only what you need. Hub keys already in the file are kept, so no Yubikey is needed unless you add a hub. An older file is migrated first, and anything a failed migration could not carry over is listed before the questions start and begins at its default.

Once you have a fleet, you can begin flashing the bots/hubs.

Flashing the bootloader configuration

Note: this step should only need to be done once per new CM4.

The RP bootloader on older CM4 boards does not boot from all the USB ports automatically. Thus, we need to update the bootloader configuration once on each CM4 that we receive in order to ensure that the CM4 will boot from the USB thumb drive.

  • Once, to set up your developer machine (e.g. Ubuntu computer), follow the steps to build the usbboot repository. At the time of writing, this repository was at commit 70b14a.
  • For each CM4, follow the steps to
    1. setup the CMIO board J2 jumper and micro USB cable.
    2. update the CM4 bootloader. When editing the boot.conf file, check that the BOOT_ORDER line is set to BOOT_ORDER=0xf25641. For more detail on reading the value, see the BOOT_ORDER documentation. Older CM4s were omitting the 0x5 value (which corresponds to the USB-A port on the CM4 IO board).
      • In the last step (../rpiboot -d . ), I had to use sudo ../rpiboot -d .
      • After running that step (sudo ../rpiboot -d . ), power on the CM4 IO board. The script should detect the board at this point and upload the configuration changes. It will provide some verbose output to this effect.

Generation of the filesystem image

Note: this step should only need to be done once per change to jaiabot/rootfs files (and then reused for any number of USB thumb drives).

Normally you will just download the appropriate image from CircleCI or AWS S3.

The jaiabot/rootfs folder is designed to generate a complete filesystem suitable to boot the RP off a USB thumb drive or other USB disk.

To generate the image, follow the steps in https://github.com/jaiarobotics/jaiabot/blob/3.y/rootfs/README.md.

The result will be something like jaiabot_img-1.0.0~alpha1+5+g90e72a3.img.

Installation of the filesystem image

  • Install the filesystem image via command line (or use something like balenaEtcher https://www.balena.io/etcher/):
    # assuming USB thumb drive is on /dev/sdd
    sudo dd if=jaiabot_img-1.0.0~alpha1+5+g90e72a3.img of=/dev/sdd bs=1M status=progress
    

Set initial first boot configuration

The newly created image needs certain configuration to run properly: is this a bot or a hub, what ID is it, what fleet does it belong to, etc.?

Typically this is automatically provided from the Fleet Config by using jaia admin fleet generate which writes the required first-boot.preseed.yml and SSH keys mentioned, below.

This information is provided via a first-boot.preseed.yml YAML configuration file, conforming to the standards of the cloud-init project's user-data file format.

Additionally, two optional SSH key pairs can be provided (in /boot/firmware/jaiabot/init):

  • For hubs: A key called hub<N>_fleet<F> (private) / hub<N>_fleet<F>.pub (public), where <N> is the hub ID and <F> is the fleet ID. If this is present, it will be installed to /home/jaiabot/.ssh and an SSH config file generated to allow host keys to be automatically added upon first connection to the bots. This key can be generated using:
    cd /path/to/boot/jaiabot/init
    HUB=1; FLEET=5; ssh-keygen -f hub${HUB}_fleet${FLEET} -t ed25519 -N "" -C "hub${HUB}_fleet${FLEET}"
  • For Wireguard setup: A key called id_vpn_tmp (private) / id_vpn_tmp.pub (public) can be provided. jaia admin fleet create generates it, and jaia admin fleet cloudhub create gives the public half to the fleet's CloudHub, where during fleet pairing it may do nothing but enroll one node on the CloudHub VPN (see VPN). A node spends it once it has enrolled. To make one by hand:

    cd /path/to/boot/jaiabot/init
    ssh-keygen -f id_vpn_tmp -t ed25519 -N "" -C "id_vpn_tmp"

    This section written by Claude

    A CloudHub starts with fleet pairing closed, and admits no node until it is opened. To join bots and hubs to it, from the Upgrade GUI (JCU) under Fleet Changes:

    1. Open Fleet Pairing, choosing how long (1 hour to 2 weeks). For a fleet's first pairing do this from the CloudHub's own JCU, at https://run.<base uri>/jcu, since no hub on site can reach the CloudHub yet; once a hub is paired, its own JCU offers it too. While pairing is open the CloudHub's SSH port is open to every address, which also lets Jaia's root keys reach it, and the support page says so.
    2. Pair Fleet to CloudHub, from a hub on site. It runs enrollment on every bot and hub over the fleet WLAN. A node already paired is left as it is, so this is safe to run on a fleet that is partly paired, or to pick up a node that missed an earlier attempt.

    Pairing closes by itself when its time runs out, and Close Fleet Pairing ends it sooner. A node switched on while pairing is open enrolls at its first boot without needing step 2.

The template of the text file first-boot.preseed.yml is provided on the image as /boot/firmware/jaiabot/init/first-boot.preseed.yml.j2 (This is a jinja2 template file used by jaia fleet admin generate).

Configuration will happen automatically without user intervention or the requirement for connecting a monitor or DHCP based SSH session.

To debug the preseed configuration functionality, connect a monitor and/or examine the cloud-init logs in /var/log.

Please note: for security reasons the first-boot script will lock the jaia user password (disallowing password-based logins). Future logins must be done via SSH using key pairs, and sudo is provided without need for a password. You can add any SSH public keys required to login to the appropriate section of the preseed file prior to booting the image.

Further details on first-boot setup

First boot configuration is now handled by cloud-init using the NoCloud data source for the Raspberry Pi image. cloud-init is configured (via the kernel command line parameter ds=nocloud;s=/etc/jaiabot/init in /boot/firmware/cmdline.txt) to use the files in /etc/jaiabot/init (except for first-boot.preseed.yml). The important file is user-data which in our case simply includes two local "user data" YAML files in the same directory:

  • common-first-boot.yml: Common actions that must be undertaken on first boot for all systems. This should only be edited if all the bots/hubs need to be configured in a different way. Only actions that cannot be performed at rootfs generation time (in CircleCI) should be included in this file (e.g., resizing the log partition to fill a disk of unknown size, installing jaiabot-embedded as this flashes the Arduino, etc.). Otherwise it is preferred to use jaiabot/rootfs/customization (either includes.chroot for files to include or hooks for scripts to run).
  • first-boot.preseed.yml: The previously discussed preseed file specific to the bot or hub being configured.

These files are separated to reduce clutter in the first-boot.preseed.yml file. The source for both of these files is in jaiabot/rootfs/customization/includes.chroot/etc/jaiabot/init.

Systemd

We use systemd to launch the jaiabot services on the embedded system, just as any other Ubuntu daemon.

Each application has a service definition, and they are all set to BindTo the jaia.service which exists to provide a common service that can be stopped or started, thereby stopping or starting all the bound services.

Quick start

When using the jaiabot-embedded Debian package, the systemd services are automatically installed to /etc/systemd/system and enabled. No further action is required in this case.

This section written by Claude

All bot/hub configuration is read from the jaiabot-embedded debconf database, so there are no configuration flags to pass. To change any of it, run sudo dpkg-reconfigure jaiabot-embedded, or use jaia admin debconf to list, get and set individual values without stepping through the menus.

When using a built-from-source version of jaiabot, ensure that the local bin directory is on your $PATH (e.g., check that which jaiabot_mission_manager returns the correct binary), then run:

cd jaiabot/config/gen
./systemd-local.sh --enable

This section written by Claude

If the machine has no jaiabot-embedded debconf database — or you want to generate services for a configuration other than the local one — write the answers to a file in debconf-set-selections format and point at it instead:

cd jaiabot/config/gen
./systemd-local.sh --debconf_selections /path/to/bot.selections --enable

You can produce such a file from a configured machine with jaia-debconf.sh selections. This is also how container-build-and-deploy.sh configures a remote target.

See ./systemd-local.sh --help for more options.

Design

The systemd services are all controlled by a single jaiabot.service which doesn't do anything itself, but serves as a single point from which all the other services can be stopped, started, etc.

Then, jaiabot_gobyd and jaiabot_moosdb are bound to jaiabot and start their respective middleware's broker (gobyd and MOOSDB). From there, all the client services are bound to the respective broker's service. This ensures that no Goby apps are started before gobyd and no MOOS apps are started before MOOSDB. The Python apps are also bound to jaiabot_gobyd at the moment since they are used exclusively by the Goby driver applications. If this changes, these services can be updated.

The jaiabot.service waits for the system clock to be synchronized by chrony (from the GPS via gpsd) or until 120 seconds has elapsed.

Generation

The systemd service files are generated via templates much like the application configuration.

This section written by Claude

The generation script lives in: jaiabot/config/gen/systemd.py and is run by the jaiabot-embedded postinst, or by hand for a locally built copy of jaiabot. It reads the bot/hub configuration from debconf and takes only the deployment layout (where the binaries and services live) on the command line.

The generated units carry their configuration in Environment= lines, which the configuration Generators (gen/bot.py, gen/hub.py) read when systemd starts them as children.

To see all the options for configuring this script, run systemd.py --help

This section written by Claude

For a locally built copy, you can use the systemd-local.sh shell script (a thin wrapper around systemd.py that runs it under sudo while preserving the current $PATH).

This script will generally have the correct defaults for the various directories, assuming that the version of the jaiabot apps and the Goby applications (gobyd, etc.) that you wish to run are currently set correctly in the shell $PATH environmental variable at the time of running the gen/systemd-local.sh generation script.

Running ./systemd-local.sh --help will always show the defaults inferred from the $PATH for all the directories (--jaiabot_bin_dir, --jaiabot_share_dir, etc.)

Usage

  • To start, stop, or restart all the JaiaBot applications:
    sudo systemctl start jaiabot
    sudo systemctl stop jaiabot
    sudo systemctl restart jaiabot
  • To start (or stop, restart) a single application:
    sudo systemctl start jaiabot_driver_arduino
  • To get a quick summary of all the applications' status:
    systemctl list-units "jaiabot*"
  • To get the full log from a particular service (stdout/stderr),
    sudo journalctl -u jaiabot_driver_arduino
  • To get a summary and the last few lines of the log,
    sudo systemctl status jaiabot_driver_arduino

Testing with Vagrant

Vagrant is a useful tool for creating and managing full virtual machines (VMs), which allows us to quickly spin up and delete standard amd64 VMs for deployment testing (such as systemd or package testing before deployment to the actual Raspberry Pi hardware).

Install

sudo apt install vagrant

Initialize machine

vagrant init ubuntu/focal64

This generates a Vagrantfile in the current working directory, which can be modified to fit our needs. For example, to create a VM for Virtualbox with 8 CPUs and 1GB memory, we can uncomment and edit this block from the Vagrantfile:

config.vm.provider "virtualbox" do |vb|
# Display the VirtualBox GUI when booting the machine
vb.gui = false
vb.cpus = 8
# Customize the amount of memory on the VM:
vb.memory = "1024"
end

Start machine

To start the machine from the directory with the Vagrantfile, simply run

vagrant up

To ssh in, run

vagrant ssh

Other commands

To destroy the VM, use vagrant destroy.