JaiaBot 2.1.1
JaiaBot micro-AUV software
 
Loading...
Searching...
No Matches
Embedded Board Deployment

JaiaBot uses the Raspberry Pi (RP) Compute Module 4 (CM4) as the embedded Linux computer. For R&D purposes, it has also been necessary to run the jaiabot software on a Raspberry Pi 3 although this is not ideal due to the port mappings being different from the RP4.

Quick start

Download the appropriate .img.gz from AWS S3 (produced by CircleCI): https://us-east-1.console.aws.amazon.com/s3/buckets/jaia-disk-images?region=us-east-1&bucketType=general&prefix=release/2.y/rpi/&showversions=false

Flash this to as many SD cards as you have bots plus hubs:

# assuming USB thumb drive is on /dev/sdd
gunzip -c jaiabot__rootfs-jammy-v1.16.0__code-v1.16.0.img.gz | sudo dd of=/dev/sdd bs=1M status=progress

Create a fleet, authorize the fleet to self-configure the service VPN (vpn.jaia.tech), generate first-boot configurations, and boot:

# Create the Fleet
jaia admin fleet create private_jaia/fleet_config/fleet5.cfg
# Authorize temporary key on vpn.jaia.tech
jaia admin fleet vpn_authorize private_jaia/fleet_config/fleet5.cfg
# insert flashed SD card for Hub 1
jaia admin fleet generate private_jaia/fleet_config/fleet5.cfg hub 1
# unmount & remove Hub 1 card, insert flashed SD card for Hub 2
jaia admin fleet generate private_jaia/fleet_config/fleet5.cfg hub 2
# unmount & remove Hub 2 card, insert flashed SD card for Bot 1
jaia admin fleet generate private_jaia/fleet_config/fleet5.cfg bot 1
# unmount & remove Bot 1 card, insert flashed SD card for Bot 2
jaia admin fleet generate private_jaia/fleet_config/fleet5.cfg bot 2
# Insert all the SD cards and boot all the bots and hubs and allow to fully first boot configure
# Optionally deauthorize temp VPN key
jaia admin fleet vpn_authorize private_jaia/fleet_config/fleet5.cfg --rm

Setup steps (new RP CM4)

Create a Fleet Config (do once for all bots/hubs in a fleet)

This new bot or hub must belong to a configured fleet. Fleets are configured using a Fleet Config file which is a Protobuf TextFormat version of the jaiabot.message.FleetConfig message (jaiabot/src/lib/message/fleet_config.proto).

The simplest way to create a new fleet is to use the jaia tool (e.g. for fleet5):

jaia admin fleet create fleet5.cfg

This will prompt you using a whiptail UI to set the various fleet related settings, and write the contents to fleet5.cfg.

This tool will also create the hub keys, VPN temporary keys as necessary and include those in the fleet configuration file.

Once you have a fleet, you can begin flashing the bots/hubs.

Flashing the bootloader configuration

Note: this step should only need to be done once per new CM4.

The RP bootloader on older CM4 boards does not boot from all the USB ports automatically. Thus, we need to update the bootloader configuration once on each CM4 that we receive in order to ensure that the CM4 will boot from the USB thumb drive.

  • Once, to set up your developer machine (e.g. Ubuntu computer), follow the steps to build the usbboot repository. At the time of writing, this repository was at commit 70b14a.
  • For each CM4, follow the steps to
    1. setup the CMIO board J2 jumper and micro USB cable.
    2. update the CM4 bootloader. When editing the boot.conf file, check that the BOOT_ORDER line is set to BOOT_ORDER=0xf25641. For more detail on reading the value, see the BOOT_ORDER documentation. Older CM4s were omitting the 0x5 value (which corresponds to the USB-A port on the CM4 IO board).
      • In the last step (../rpiboot -d .), I had to use sudo ../rpiboot -d .
      • After running that step (sudo ../rpiboot -d .), power on the CM4 IO board. The script should detect the board at this point and upload the configuration changes. It will provide some verbose output to this effect.

Generation of the filesystem image

Note: this step should only need to be done once per change to jaiabot/rootfs files (and then reused for any number of USB thumb drives).

Normally you will just download the appropriate image from CircleCI or AWS S3.

The jaiabot/rootfs folder is designed to generate a complete filesystem suitable to boot the RP off a USB thumb drive or other USB disk.

To generate the image, follow the steps in https://github.com/jaiarobotics/jaiabot/blob/2.y/rootfs/README.md.

The result will be something like jaiabot_img-1.0.0~alpha1+5+g90e72a3.img.

Installation of the filesystem image

  • Install the filesystem image via command line (or use something like balenaEtcher https://www.balena.io/etcher/): 
    # assuming USB thumb drive is on /dev/sdd
sudo dd if=jaiabot_img-1.0.0~alpha1+5+g90e72a3.img of=/dev/sdd bs=1M status=progress

Set initial first boot configuration

The newly created image needs certain configuration to run properly: is this a bot or a hub, what ID is it, what fleet does it belong to, etc.?

Typically this is automatically provided from the Fleet Config by using jaia admin fleet generate which writes the required first-boot.preseed.yml and SSH keys mentioned, below.

This information is provided via a first-boot.preseed.yml YAML configuration file, conforming to the standards of the cloud-init project's user-data file format.

Additionally, two optional SSH key pairs can be provided (in /boot/firmware/jaiabot/init):

  • For hubs: A key called hub<N>_fleet<F> (private) / hub<N>_fleet<F>.pub (public), where <N> is the hub ID and <F> is the fleet ID. If this is present, it will be installed to /home/jaiabot/.ssh and an SSH config file generated to allow host keys to be automatically added upon first connection to the bots. This key can be generated using:
    cd /path/to/boot/jaiabot/init
    HUB=1; FLEET=5; ssh-keygen -f hub${HUB}_fleet${FLEET} -t ed25519 -N "" -C "hub${HUB}_fleet${FLEET}"
  • For Wireguard setup: A key called id_vpn_tmp (private) / id_vpn_tmp.pub (public) can be provided. This key must first be added to vpn.jaia.tech as a temporary access key (e.g. for 1 day of validity) using jaia admin ssh add. It will then be used by the first-boot setup to configure both the server and client sides of the service (fleet) VPN. This key can be generated and provisioned using:
    cd /path/to/boot/jaiabot/init
    ssh-keygen -f id_vpn_tmp -t ed25519 -N "" -C "id_vpn_tmp"
    jaia admin ssh add --user=ubuntu vpn.jaia.tech "$(cat id_vpn_tmp.pub)" 1d

The template of the text file first-boot.preseed.yml is provided on the image as /boot/firmware/jaiabot/init/first-boot.preseed.yml.j2 (This is a jinja2 template file used by jaia fleet admin generate).

Configuration will happen automatically without user intervention or the requirement for connecting a monitor or DHCP based SSH session.

To debug the preseed configuration functionality, connect a monitor and/or examine the cloud-init logs in /var/log.

Please note: for security reasons the first-boot script will lock the jaia user password (disallowing password-based logins). Future logins must be done via SSH using key pairs, and sudo is provided without need for a password. You can add any SSH public keys required to login to the appropriate section of the preseed file prior to booting the image.

Further details on first-boot setup

First boot configuration is now handled by cloud-init using the NoCloud data source for the Raspberry Pi image. cloud-init is configured (via the kernel command line parameter ds=nocloud;s=/etc/jaiabot/init in /boot/firmware/cmdline.txt) to use the files in /etc/jaiabot/init (except for first-boot.preseed.yml). The important file is user-data which in our case simply includes two local "user data" YAML files in the same directory:

  • common-first-boot.yml: Common actions that must be undertaken on first boot for all systems. This should only be edited if all the bots/hubs need to be configured in a different way. Only actions that cannot be performed at rootfs generation time (in CircleCI) should be included in this file (e.g., resizing the log partition to fill a disk of unknown size, installing jaiabot-embedded as this flashes the Arduino, etc.). Otherwise it is preferred to use jaiabot/rootfs/customization (either includes.chroot for files to include or hooks for scripts to run).
  • first-boot.preseed.yml: The previously discussed preseed file specific to the bot or hub being configured.

These files are separated to reduce clutter in the first-boot.preseed.yml file. The source for both of these files is in jaiabot/rootfs/customization/includes.chroot/etc/jaiabot/init.

Systemd

We use systemd to launch the jaiabot services on the embedded system, just as any other Ubuntu daemon.

Each application has a service definition, and they are all set to BindTo the jaia.service which exists to provide a common service that can be stopped or started, thereby stopping or starting all the bound services.

Quick start

When using the jaiabot-embedded Debian package, the systemd services are automatically installed to /etc/systemd/system and enabled. No further action is required in this case.

When using a built-from-source version of jaiabot, ensure that the local bin directory is on your $PATH (e.g., check that which jaiabot_mission_manager returns the correct binary), then run:

Bot 0 (install and enable):

cd jaiabot/config/gen
./systemd-local.sh bot --bot_index 0 --n_bots 4 --enable

Hub (install and enable):

cd jaiabot/config/gen
./systemd-local.sh hub --n_bots 4 --enable

See ./systemd-local.sh --help for more options.

Design

The systemd services are all controlled by a single jaiabot.service which doesn't do anything itself, but serves as a single point from which all the other services can be stopped, started, etc.

Then, jaiabot_gobyd and jaiabot_moosdb are bound to jaiabot and start their respective middleware's broker (gobyd and MOOSDB). From there, all the client services are bound to the respective broker's service. This ensures that no Goby apps are started before gobyd and no MOOS apps are started before MOOSDB. The Python apps are also bound to jaiabot_gobyd at the moment since they are used exclusively by the Goby driver applications. If this changes, these services can be updated.

The jaiabot.service waits for the system clock to be synchronized via NTP (via the GPS using gpsd) or until 120 seconds has elapsed.

Generation

The systemd service files are generated via templates much like the application configuration.

The generation script lives in: jaiabot/config/gen/systemd.py and can be run to install systemd service jobs for either a locally built copy of jaiabot or used during the Debian package build.

To see all the options for configuring this script, run systemd.py --help

For a locally built copy, you can use the systemd-local.sh shell script (a thin wrapper around systemd.py that executes systemd.py using the current interactive shell settings, such as $PATH).

This script will generally have the correct defaults for the various directories, assuming that the version of the jaiabot apps and the Goby applications (gobyd, etc.) that you wish to run are currently set correctly in the shell $PATH environmental variable at the time of running the gen/systemd-local.sh generation script.

Running ./systemd-local.sh --help will always show the defaults inferred from the $PATH for all the directories (--jaiabot_bin_dir, --jaiabot_share_dir, etc.)

Usage

  • To start, stop, or restart all the JaiaBot applications:
    sudo systemctl start jaiabot
    sudo systemctl stop jaiabot
    sudo systemctl restart jaiabot
  • To start (or stop, restart) a single application:
    sudo systemctl start jaiabot_driver_arduino
  • To get a quick summary of all the applications' status:
    systemctl list-units "jaiabot*"
  • To get the full log from a particular service (stdout/stderr),
    sudo journalctl -u jaiabot_driver_arduino
  • To get a summary and the last few lines of the log,
    sudo systemctl status jaiabot_driver_arduino

Testing with Vagrant

Vagrant is a useful tool for creating and managing full virtual machines (VMs), which allows us to quickly spin up and delete standard amd64 VMs for deployment testing (such as systemd or package testing before deployment to the actual Raspberry Pi hardware).

Install

sudo apt install vagrant

Initialize machine

vagrant init ubuntu/focal64

This generates a Vagrantfile in the current working directory, which can be modified to fit our needs. For example, to create a VM for Virtualbox with 8 CPUs and 1GB memory, we can uncomment and edit this block from the Vagrantfile:

config.vm.provider "virtualbox" do |vb|
# Display the VirtualBox GUI when booting the machine
vb.gui = false
vb.cpus = 8
# Customize the amount of memory on the VM:
vb.memory = "1024"
end

Start machine

To start the machine from the directory with the Vagrantfile, simply run

vagrant up

To ssh in, run

vagrant ssh

Other commands

To destroy the VM, use vagrant destroy.