JaiaBot 3.0.0
JaiaBot micro-AUV software
Loading...
Searching...
No Matches
Cloud Computing

The Jaia Cloud is designed to provide:

  • VirtualFleet: remote access to simulated fleets (intended to be a nearly identical "digital twin" of the corresponding real fleet, if one exists) for user training and scenario planning.
  • CloudHub:
    • remote mission planning and control of real fleets over the internet.
    • remote access to data storage, analysis and visualization.
    • control (start/stop/delete/create) of VirtualFleet

It is possible to have a VirtualFleet without a corresponding real fleet or vice-versa.

Definitions

Jaia Terms

  • Cloud - remote internet-connected on-demand computing provided (in this case) by Amazon Web Services (AWS) on the Elastic Compute Cloud (EC2) virtual machine system.
  • CloudHub - A hub that lives in the Cloud. This machine is always-on. CloudHub always uses the Hub ID 30 with a given fleet. CloudHub is a full featured hub running nearly the same software as a physical hub, built regardless of the communications links in use (Hub2Hub allows CloudHub to get data even without any direct link, such as Iridium, to the fleet).
  • VirtualFleet - a set of VirtualBots and VirtualHubs that run in the cloud on virtual machines.
  • VirtualBot - an amd64 version of the real bot that differs in that all the sensors/actuation are hooked up to simulators rather than the real hardware.
  • VirtualHub - Similar to VirtualBot, just for a hub.

Amazon Web Services (AWS) Terms

  • EC2 - Elastic Compute Cloud: AWS's Virtual Machine infrastructure.
  • S3 - Simple Storage Service: AWS's data storage in the cloud with variable access latencies and corresponding costs.
  • VPC - Virtual Private Cloud: a collection of networking components (subnets, internet gateways, firewall rules, etc) in an isolated environment from other VPCs that might exist in Jaia's or a customer's EC2 account for a given region.

Architecture

Each VPC has one and only one fleet. At a minimum this is a CloudHub machine, which is always on.

VPC

The components of the VPC include:

  • Subnets:
    • jaia__Subnet_CloudHub: A subnet for Cloudhub alone.
    • jaia__Subnet_VirtualFleet_WLAN: A subnet for the VirtualFleet (uses the same private IP addresses as the real fleet's wireless LAN.
  • Route tables:
    • jaia__RouteTable: A simple routing table that sends all non LAN traffic to the internet gateway.
  • Internet gateway:
    • jaia__InternetGateway: A simple internet gateway with default configuration.
  • Elastic IPs:
    • jaia__CloudHub_VM__ElasticIP: A public IPv4 address for access to CloudHub.
  • Security Groups (Firewalls):
    • jaia__SecurityGroup_CloudHub: Allows access to VPNs only running on CloudHub.
    • jaia__SecurityGroup_VirtualFleet: Allows access only within private network (jaia__Subnet_VirtualFleet_WLAN).
  • Instances (Virtual Machines):
    • CloudHub
      • CloudHub VPN
      • VirtualFleet VPN
    • (optionally) VirtualHub1
    • (optionally) VirtualBot1-N

Network addresses

This section written by Copilot

The use of the jaia ip tool (or, in scripts, the standalone jaia_ip binary that it wraps) is recommended for determining IP addresses for a given node, id, fleet, etc.

The network address assignment for the Jaia Cloud is intended to complement the existing fleet specific VPN. This means that a given fleet may have up to three VPN subnets assigned:

  1. Real "fleet VPN": The fleet VPN for remote support (vpn.jaia.tech) - UDP port 51821 + fleet ID. Managed by vpn.jaia.tech.
  2. "VirtualFleet VPN": The virtual fleet VPN for access to the "digital twin" virtual fleet - UDP port 51820. Managed by the fleet's CloudHub.
  3. "cloudhub VPN": The data offload / remote control VPN for access to the real fleet from the CloudHub - UDP port 51821. Managed by the fleet's CloudHub.

These all need to be different because:

  • The customer may choose the host both the virtualfleet and cloudhub VPN on their own (secure) AWS account and not provide Jaia access.
  • Some customers will not have a VirtualFleet at all.
  • Some customers may want to restrict access to the Cloudhub to a different set of users than those who have access to the VirtualFleet (e.g., trainees not able to accidentally command real fleet).

Additionally, each VPC has two subnets assigned as previously mentioned:

  1. CloudHub Subnet: 10.23.255.0/24 and 2001:db8:0:0::/64 where 2001:db8::/56 is replaced by the Amazon EC2 assigned IPv6 block for the VPC.
  2. VirtualFleet WLAN Subnet: 10.23.{flt}.0/24 (10.23.253.0/24 for a fleet above 250, see below) and 2001:db8:0:2::/64 where 2001:db8::/56 is replaced by the Amazon EC2 assigned IPv6 block for the VPC.

Fleet ID ranges

This section written by Claude

The fleet WLAN, the fleet VPN and the VirtualFleet WLAN carry the fleet id in a single IPv4 octet, which is what limits how many fleets those schemes can address. Rather than renumber the fleets already using them, the fleet id range is split:

Fleet ID Fleet WLAN, fleet VPN VirtualFleet WLAN Other networks
0-250 IPv4, as below IPv4 10.23.{flt}.0/24, the same addresses as the fleet WLAN IPv6, as below
251-4000 IPv6, as below IPv4 10.23.253.0/24 IPv6, as below

So a fleet above 250 has no IPv4 address of its own on a real fleet: IPv4 on its WLAN carries internet traffic only, from whatever the access point hands out, and all Jaia traffic is IPv6. The CloudHub and VirtualFleet ethernet subnets of a VPC are the same for every fleet and so stay IPv4 whichever range the fleet is in.

A VirtualFleet is the exception. Its nodes are EC2 instances, and EC2 assigns IPv6 from the block Amazon gives the VPC rather than from a ULA of ours, so a VirtualFleet above 250 cannot mirror its fleet's WLAN the way a lower one does. It stays IPv4 on 10.23.253.0/24 instead, which cannot collide with anything because a VPC belongs to one fleet. Configuration generated on a VirtualFleet node therefore addresses the WIFI link on vfleet_wlan; everywhere else it is wlan, and for a fleet at or below 250 the two give the same addresses.

A VirtualBox fleet has no such limit - its NAT network is local - so it mirrors the fleet WLAN whichever range the fleet is in, taking fddd:7f2e:3258:{fleet}::/64 above 250 with IPv4 from the NAT network's DHCP for internet access only, exactly as a real fleet of that id does. The host SSH ports are forwarded to that IPv4 rather than to the fleet IPv6, because VirtualBox stores an IPv6 port forward but never activates it; import_vms.sh reserves each node's vfleet_wlan address in the NAT network's DHCP so the forward has a fixed address to aim at.

The upper end of the range is 4000 rather than the 65535 the IPv6 scheme has room for, because the fleet id is also the XBee network id (ATID), whose documented range is smaller and which the driver writes in a format that does not currently reach it. Raising it is a matter of that driver and of widening HubStatus.fleet_id, which is a DCCL wire format change; nothing in the addressing itself is in the way.

jaia ip gives the address in the version the fleet actually uses, and jaia admin bounds --ipv4_fleet_id --max reports where the split is. Both come from fleet_id_ipv4_max in src/lib/utils/ip.h, which is the one place it is written down.

IPv4

  1. Fleet VPN: The IPv4 addresses remain as assigned in VPN document: 172.23.flt.bot_or_hub, where flt = Fleet ID, bot_or_hub = 10 + hub ID or 100 + bot ID.
  2. VirtualFleet VPN / CloudHub VPN: No IPv4 addresses are assigned except for the VPN server itself (to allow connections from IPv4 only customers).

IPv6

Prefix

The prefix is fd00::/8 for unique local addresses as defined in RFC4193.

The full 48-bit prefix was (pseudo)-randomly generated as per RFC4193 for each of the three VPN classes.

VPN Class IPv6 Prefix
Fleet VPN fd91:5457:1e5c::/48
VirtualFleet VPN fd6e:cf0d:aefa::/48
CloudHub VPN fd0f:77ac:4fdf::/48
Physical WLAN fddd:7f2e:3258::/48

Subnet

For each VPN class, the Subnet ID is the Fleet ID, so for example, VirtualFleet 3 would have the subnet fd6e:cf0d:aefa:3::/64. This allows up to 2^16 = 65536 fleets to be assigned.

Address

This section written by Claude

For a given node on the network, the 64-bit interface identifier is given as ::1:hub_id for hubs, ::2:bot_id for bots, ::3:customer_id for various customer machines (desktop / laptop / tablet), ::4:rpicam_id for rpicams, and ::5:0 for the gateway, which is the Wireguard server on the VPN networks (.1 on the IPv4 networks). This allows up to 2^16 = 65536 nodes of each type to be assigned per fleet.

Nothing is assigned the gateway address on the fleet WLAN. A node there finds its router by Router Advertisement, and the generated network configuration sets no IPv6 gateway; the address is reserved so that a Wireguard server, which is inside a tunnel where there is no Router Advertisement to answer, has the same identifier on every network.

Group ::0: is left unassigned: it holds the subnet-router anycast address (::) and ::1, which a router sharing the segment conventionally claims for itself. A node given either address fails Duplicate Address Detection and is unreachable.

Some examples include:

Bot (b) or Hub (h)? ID Fleet Fleet VPN Address (s) VirtualFleet VPN (v) Address (fN) CloudHub VPN Address (c)
Bot 5 4 fd91:5457:1e5c:4::2:5 fd6e:cf0d:aefa:4::2:5 fd0f:77ac:4fdf:4::2:5
Bot 6 250 fd91:5457:1e5c:fa::2:6 fd6e:cf0d:aefa:fa::2:6 fd0f:77ac:4fdf:fa::2:6
Hub 20 10 fd91:5457:1e5c:a::1:14 fd6e:cf0d:aefa:a::1:14 fd0f:77ac:4fdf:a::1:14
Hub (CloudHub (ch)) 30 15 fd91:5457:1e5c:f::1:1e fd6e:cf0d:aefa:f::1:1e fd0f:77ac:4fdf:f::1:1e

This section written by Claude

Every fleet in the table is an IPv4 fleet, so jaia ip gives its fleet VPN address as IPv4 (jaia ip b5sf4 is 172.23.4.105); --ip_version ipv6 asks for the IPv6 address listed here instead. The VirtualFleet and CloudHub VPNs are IPv6 for every fleet, so those need no flag.

You can generate the values for the table above yourself using:

jaia ip b5sf4 --ip_version ipv6
jaia ip b5vf4
jaia ip b5cf4
jaia ip b6sf250 --ip_version ipv6
jaia ip b6vf250
jaia ip b6cf250
jaia ip h20sf10 --ip_version ipv6
jaia ip h20vf10
jaia ip h20cf10
jaia ip h30sf15 --ip_version ipv6
jaia ip h30vf15
jaia ip h30cf15
# OR
jaia ip chf15

AWS tags

AMI

  • Name: jaiabot__rootfs-feature_aws-cloud-v3.0.0
  • jaiabot-rootfs-gen_version: 3.0.0
  • jaiabot-rootfs-gen_repository: release
  • jaiabot-rootfs-gen_repository_version: 3.y
  • jaiabot-rootfs-gen_build-date: Fri 08 Dec 2023 02:20:27 UTC
  • jaiabot-rootfs-gen_build-unixtime: 1702002064

This section written by Claude

CircleCI builds the AMI in us-east-1 (and separately in us-gov-east-1 for GovCloud), then copies it, with these tags, to each region listed in the ami_copy_regions parameter of the aws-sync job — currently ca-central-1. Since lookups are by tag rather than by AMI ID, create_vpc.sh and create-virtualfleet.yml resolve the image the same way in every region. To add a region, add it to that parameter; AMIs cannot be copied between the commercial and GovCloud partitions, so each partition needs its own build.

VPC components (including Instances)

  • Name: jaia__COMPONENT__CUSTOMER_NAME: COMPONENT is VPC, Subnet, SecurityGroup, etc.
  • jaia_customer: CUSTOMER_NAME
  • jaiabot-rootfs-gen_repository: same as AMI
  • jaiabot-rootfs-gen_repository_version: same as AMI.
  • jaia_fleet: Fleet ID

CUSTOMER_NAME is the fleet config's customer (default jaia), which jaia admin fleet create and edit ask for when the fleet has a CloudHub.

VM Instances

  • all:
    • Name: jaia__CloudHub_VM__CUSTOMER_NAME, jaia__VirtualHubN_VM__CUSTOMER_NAME, or jaia__VirtualBotN_VM__CUSTOMER_NAME
    • jaia_instance_type: cloudhub, virtualfleet
  • VirtualBot/VirtualHub:
    • jaia_node_id: Bot ID or Hub ID
    • jaia_node_type: "bot" or "hub"

CloudHub settings

This section written by Claude

A CloudHub keeps its creation-time settings in /etc/jaiabot/cloud.env, written by cloud-init on first boot. That file lives on the rootfs, so a major upgrade discards it along with everything else outside the reuse set. Rather than carry it across, the settings are split by where they can be obtained again:

In the fleet config, because AWS cannot be asked for them:

Field Default
includes_cloudhub (top level) true — answer no to the CloudHub question in jaia admin fleet create/edit for a fleet sold without one
customer (top level) jaia — the value of the jaia_customer tag on every AWS resource the fleet owns
cloudhub.base_uri required — the name the authentication front end is served under
cloudhub.admin_email required — address of the fleet_admin user created on first boot
cloudhub.smtp_address required — the relay Authelia sends enrolment and reset mail through
cloudhub.data_bucket jaia--cloudhub-data--fleet<fleet> — the bucket mounted at the bot offload directory
cloudhub.smtp_sender noreply@auth.jaia.tech — the From address, verified with the SMTP provider
cloudhub.smtp_credentials_ssm_parameter /jaia/cloudhub/smtp_credentials — the SSM SecureString holding the SMTP login (see "Required SMTP" below)

customer is a property of the fleet rather than of its CloudHub, so it sits at the top level; the rest are meaningless without hub 30 and validate requires the cloudhub message exactly when hub 30 is in the fleet.

Most fleets have a CloudHub and that is what the questions propose, but a fleet can be built without one. It has to say so — includes_cloudhub: false — rather than simply omitting hub 30, because a fleet that never had one and a fleet that lost it by mistake would otherwise look identical, and only the first should validate. jaia admin fleet has_cloudhub fleetN.cfg prints yes or no, which is what the major upgrade compares against the has_cloudhub answer the operator gives it. jaia admin fleet cloudhub create renders them into vpc.conf for create_vpc.sh. CI gives each run its own customer name by writing it into the fleet config it generates.

Discoverable from AWS, so deliberately not stored: the region, VPC, subnets, security groups, account ID and Elastic IP. Each is available from instance metadata or from a tag lookup within the fleet's VPC.

Carried across the upgrade: /etc/wireguard. The CloudHub's VPN private key cannot be regenerated without every client peer, including the login server, having to be reissued. The same goes for its SSH key, /home/jaia/.ssh/hub30_fleetN, which no fleet config holds (see below).

Accounts and commissioning

This section written by Claude

A new CloudHub's directory has two accounts, neither with a password: set one with Reset password? on the login page, which emails the address shown.

Account Whose Groups Email
jaia_bootstrap Jaia's, for commissioning super_admin, lldap_admin create_cloudhub --bootstrap-email
fleet_admin the customer's super_admin, lldap_admin cloudhub.admin_email

Commissioning, signed in as jaia_bootstrap:

  1. Test the CloudHub's sites.
  2. In the CloudHub's JCU (https://run.<base uri>/jcu), Open Fleet Pairing.
  3. From a hub on site, Pair Fleet to CloudHub, and check every bot and hub reports it is paired.
  4. Close Fleet Pairing.
  5. Before the CloudHub is shipped, delete jaia_bootstrap at https://users.<base uri>.

jaia_bootstrap is created the first time the CloudHub's directory is set up and never again. --bootstrap-email is required, and cannot be support@jaia.tech, which jaia_support already has. A CloudHub made without it, such as one upgraded from 2.y, gets no jaia_bootstrap. A major upgrade sets the directory up a second time, so a marker in the persistent /var/log/jaiabot/auth records that the account has had its turn; deleting it is final. There is also a jaia_support account, in no groups, which the support page moves into the web groups when a customer grants Jaia web access.

CloudHub SSH key

This section written by Claude

A physical hub's SSH key lives on its Yubikey, so the key handle in the fleet config is of no use without the hub itself. A CloudHub has no USB port for one, and its key is an ordinary file instead, so it is made on the CloudHub and never leaves it:

  1. jaia admin fleet create writes no key for hub 30.
  2. On first boot the CloudHub finds no key in its preseed and generates /home/jaia/.ssh/hub30_fleetN, nothing secret having been put in its EC2 user data.
  3. create_cloudhub copies the public half back and records it in the fleet config it was given (jaia admin fleet cloudhub set_key fleetN.cfg hub30_fleetN.pub), with an empty private_key.

Create the CloudHub before generating the boot files of the other nodes. A node configured from a fleet config without the key prints a warning and does not accept the CloudHub until it is paired again (the "Pair new hub" playbook) with the updated config. The same applies to a CloudHub that is deleted and created again, since the new one has a new key.

Every node's /etc/jaiabot/ssh/hub_authorized_keys accepts the CloudHub's key only from its address on the CloudHub VPN (from="fd0f:77ac:4fdf:N::1:1e", where N is the fleet in hex), so a copy of the key is of no use without also being the CloudHub on that VPN.

Neither stored nor carried: which repository the VirtualFleet AMI comes from, and its version. create_cloudhub --repo picks the repository at creation, defaulting to release, and the JCU's "Change JaiaBot repository and update all packages" playbook changes it afterwards. The version is the release branch of whatever jaiabot is installed, read from common-versions.env, so a VirtualFleet raised after a major upgrade matches the upgraded CloudHub rather than the release it came from.

Usage

This section written by Claude

The CloudHub's sites (JCC, JDV, JCU and the rest, see "Cloud Login server" below) are reached through its login page, with no VPN. A tunnel of your own is for CI and development: jaia admin fleet cloudhub create --client-vpn makes the machine running it a peer (desktop1) of the CloudHub's VPNs, installs wg_jaia_ch<fleet>, and adds cloudhub-fleet<fleet> to its /etc/hosts. It is off by default because the peer stays on the CloudHub for as long as it runs, which is a standing way in that a customer's CloudHub should not ship with. In vpc.conf the same switches are ENABLE_CLIENT_VPN and UPDATE_CLIENT_ETC_HOSTS.

Once connected to the appropriate VPN and hosts are configured in /etc/hosts, one can open a web-browser as usual to JCC, etc.

For fleet1, these would be:

# /etc/hosts
fd6e:cf0d:aefa:1::1:1 hub1-virtualfleet1
fd0f:77ac:4fdf:1::1:1e cloudhub-fleet1

Specialty Servers

In addition to the CloudHub and VirtualFleet mentioned above, Jaia runs a number of "speciality" AWS servers, each with a specific primary task:

  • iridium.jaia.tech: Multiplexing support for Iridium satellite messaging.
  • packages.jaia.tech: Hosts Ubuntu (.deb) packages
  • docs.jaia.tech: Documentation
  • vpn.jaia.tech: Service VPN for remote support.
  • `.cloud.jaia.tech: Web access with 2-factor authentication to Jaia's own web services (cc, sandbox.cc, launchpad, data`). Its build, deployment and migration are documented in Confluence: auth.cloud.jaia.tech.

We want to be able to recreate these servers from scratch for consistency of implementation, security, and AWS instance type. We want to be able to upgrade to the latest Ubuntu/Jaiabot release by rebuilding servers, and have the tools in place to recreate them in the event of an instance disk corruption or AWS failure.

This is done using a pair of scripts in jaiabot/rootfs/cloud/aws:

  • create_specialty_server.sh: Creates the new AWS server from a configuration file.
  • delete_specialty_server.sh: Deletes the existing AWS server(s) for a particular server type (matching the SERVER_TYPE given in the configuration file of original invocation of create_specialty_server.sh). Primarily useful for developing a new server type configuration set when you need to create/delete the server multiple times. See "Upgrade an existing specialty server type" below for upgrading a live server.

Graphically, this is the relationship between the various components:

specialty server creation

Create a new specialty server type

To create a new specialty server type (which we'll call "foo") you need to create three files, and by convention these are put in a subdirectory with the same name:

  • jaiabot/rootfs/cloud/aws/foo
    • foo.conf
    • user-data.sh.in or user-data.sh
    • user-data.yaml.in

The docs and iridium server configurations are fairly simple, and one of these may be a useful starting point to copy for a new server configuration.

foo.conf

foo.conf is simply a bash shell script that is sourced by create_specialty_server.sh and must define the following variables:

  • SERVER_TYPE=foo: instance name, used to tag the instance and instance resources. Should match name of file and directory (foo).
  • ENABLE_TERMINATION_PROTECTION=true or false: EC2 termination protection prevents against accidental deletion (termination) of a server unless termination protection is first removed in the console or using the AWS CLI. Useful to set false when testing creating a server (as you likely need to run delete_specialty_server.sh multiple times), but is important to set true once the server is in production to avoid accidental deletion.
  • DEBUG=true or false: If true, outputs more information about what the script is doing.

foo.conf also must define the function function create_security_group_rules() which will be run (with the SECURITY_GROUP_ID as the only parameter) to allow the script to set the appropriate AWS firewall (security group) rules. Instances should restrict the allowed firewall inputs to the minimum necessary to allow functionality.

foo.conf can also optionally include:

  • ELASTIC_IP_ALLOCATION_ID=eipalloc-0b2441c45eff7c158: This is an "elastic IP" address allocation ID for an existing IPv4 address that will be reassigned to this instance. It is typically convenient to reuse the Elastic IP between upgrades to the same server type so that the IP address remains the same as the old machine and thus the DNS record (A record) don't need to be changed. If unset, no Elastic IP will be set, and this should be done manually later in AWS Console. If it's a new server, you'll want to allocate an Elastic IP address manually in the console and assign a domain name A record (e.g., foo.jaia.tech) so that it can be reached conveniently. The domain name settings are done through https://get.tech/, not AWS.
  • IPV6_ADDRESS=2001:db8:ffff:ffff:ffff:ffff:ffff:ffff: Assign this IPv6 address to the instance. If unset a new address will be assigned to the instance. Similarly to the ELASTIC_IP_ALLOCATION_ID, this should be set to upgrade an instance without having to change the domain name (AAAA) record. When creating a new server, you can leave this blank to let AWS allocate an IPv6 address, then just keep track of it for future upgrades.
  • ARCH=arm64: Architecture for the instance (must match INSTANCE_TYPE architecture). Default is arm64 to match t4g.nano
  • INSTANCE_TYPE=t4g.nano: Instance type: default is t4g.nano (very small arm64 instance).
  • REGION=us-west-2: Region, default is us-west-2

This section written by Claude

  • SUBNET_ID=subnet-...: Subnet to launch in. Required for any region but us-west-2. For IPv6, the subnet needs an IPv6 block.
  • SERVER_NAME=foo.example.com: Name tag for the instance and its resources, default ${SERVER_TYPE}.jaia.tech. Set it to run a second server of the same type (a test copy) beside the first; delete_specialty_server.sh given the conf then deletes only that one.
  • IAM_INSTANCE_PROFILE_NAME=...: Instance profile to launch with. It must exist already.
  • DISK_SIZE_GB=8: Size of disk in GB (default 8).
  • Any other variables to be used in preprocessing user-data.*.in (see below).

user-data.sh.in and user-data.yaml.in

These are template files for cloud-init user-data configuration. Both must be included but one may be left empty, except for the header line.

Minimal user-data.sh.in or user-data.sh:

#!/bin/bash

Minimal user-data.yaml.in:

#cloud-config

Both *.in configuration files are preprocessed using bash before being passed to cloud-init so that bash variables (${}) are expanded using the values in foo.conf and jaiabot/scripts/common-versions.env. If user-data.sh is specified instead, it will be used directly without preprocessing.

The contents of the preprocessed files are used by cloud-init to seed the new server instance. See the cloud-init documentation for more details.

Manual steps

Some servers will require manually copying data from the old server, or restoring a backup. For example, when upgrading the packages.jaia.tech server we copy the existing .deb files, etc. from the old server so that all of these .deb packages don't have to be rebuilt.

The user-data.*.in configuration files should install a script that can be run after the server is up and running. See the /home/jaia/sync_from_prior_packages_server.sh script installed by the jaiabot/rootfs/cloud/aws/packages/user-data.sh.in configuration file for an example.

The documentation for that server should describe the use of such a manual synchronization script.

Actually create the server

Run create_specialty_server.sh foo/foo.conf to create the new server.

Delete an existing specialty server type

When testing the creation of a new server type, you may be creating and deleting the server multiple times before you are satisfied with the new configuration. Here it is helpful to set:

ENABLE_TERMINATION_PROTECTION=false

so that you can easily delete the new server until it works correctly. Once you have a working server you can go in to the AWS console and enable termination protection for that instance (or delete and rebuild one last time with ENABLE_TERMINATION_PROTECTION=true).

This section written by Claude

delete_specialty_server.sh also takes the server's .conf in place of a server type: it then works in the conf's REGION and deletes only the server named by its SERVER_NAME. This is how to delete a test copy without touching the production server of the same type.

Upgrade an existing specialty server type

To upgrade an existing server, the following set of steps is recommended:

  1. Remove (or comment out) the ELASTIC_IP_ALLOCATION_ID= and IPV6_ADDRESS= lines in the {server_type}.conf file so that the new instance does not replace the existing (live) old server's IP addresses.
  2. Create the new instance using create_specialty_server.sh.
  3. Test the functionality as much as possible using the new server but with its temporary IPv4/IPv6 addresses.
  4. When this new server is deemed functional, swap the Elastic IP and IPv6 addresses from the old server to the new server in the console under Actions->Networking->Manage IP addresses for the new server. This makes the new server live, so be careful about the timing, depending on the server's usage.
  5. Ensure the new server is fully functional now with these IP addresses (which should map to the correct domain names as well).
  6. Power down (stop) the old server.
  7. After some period (e.g., 1-2 weeks or so) of the new server functioning correctly, disable termination protection for the old server and enable termination protection for the new server (if not already set) under (in the AWS console) Actions->Instance Settings->Change termination protection. Terminate (delete) the old instance.

Cloud Login server (auth.fleetN.jaia.tech / auth.custom_domain)

This section written by Claude

Each CloudHub runs its own authentication server, allowing people to access the CloudHub and VirtualFleet securely using 2-factor [2FA] verification for all resources except the REST_API (which allows machines to use one-factor passwords to be used as API tokens).

This provides a more convenient way to access the JCC and other CloudHub applications without requiring that the client machine have the Wireguard VPN installed, and provide more granular permissions.

cloud server

Implementation

This server is implemented using three open source projects:

  • Authelia for authentication including 2FA (time-based one-time password, TOTP and WebAuthn which supports physical keys like Yubikeys).
  • Caddy for the reverse proxy features.
  • LLDAP: Light LDAP (Lightweight Directory Access Protocol) implementation for user management.

In short, Authelia manages authentication, Caddy manages the reverse proxy (between the insecure HTTP applications and the authenticated HTTPS connection), and LLDAP manages the user information (user names, group, passwords, etc.).

This section written by Claude

An instance of all three of these runs on each CloudHub.

The jaiabot-web-thirdparty package

This section written by Claude

All three come from one Debian package of the jaiabot source, jaiabot-web-thirdparty, which CircleCI builds and publishes alongside the other jaiabot packages for amd64 and arm64:

  • Caddy is built from source by go build in the deb build, from src/web/thirdparty/caddy (a three-line main.go importing Caddy and the replace-response plugin; go.mod/go.sum pin every module). The package Provides, Conflicts with and Replaces Ubuntu's caddy, so it is the only Caddy on a CloudHub.
  • Authelia and LLDAP are the upstream release tarballs, downloaded in the deb build and checked against the SHA-256s recorded in scripts/common-versions.env. LLDAP runs natively from /usr/bin/lldap (web app in /usr/share/lldap/app, bootstrap script in /usr/libexec/lldap), so a CloudHub no longer needs Docker.
  • The AWS CLI comes from Ubuntu's awscli package, a dependency.

The package is in the CloudHub image (rootfs/customization/package-lists/cloud.list.chroot), so a fresh CloudHub's first boot only configures what the image already holds: nothing is fetched from caddyserver.com, apt.authelia.com, Docker Hub or Amazon's CLI downloads, none of which had a bounded timeout before. First boot still needs AWS itself, Let's Encrypt and apt metadata. A Minor Upgrade upgrades the package with the rest of jaiabot (config/ansible/tasks/install-jaiabot.yml); a Major Upgrade's image carries it.

The package enables none of its units: jaia_configure_authelia.sh writes each service's configuration, enables it and starts it (restarting Caddy only when the running process's binary has been replaced by an upgrade, and LLDAP only when its configuration changed, as it already does for Authelia).

Updating the versions. renovate.json at the repository root tells Renovate to open PRs against 3.y: one for Caddy and its modules (the gomod manager, which also moves the untagged replace-response plugin forward by commit), and one per pin in scripts/common-versions.env that carries a # renovate: line (Authelia and LLDAP from their GitHub releases, and the build tools: arduino-cli, nvm, Node, npm, webpack, webpack-cli, the Raspberry Pi firmware). A version PR for Authelia or LLDAP needs its checksums refreshed before it can build: src/web/thirdparty/fetch.sh --update-checksums downloads the new tarballs and rewrites the jaia_sha256_* lines. Authelia migrates its database forward and cannot read it once a newer release has, so never merge a downgrade. Renovate runs only once the Renovate GitHub app is enabled for the repository.

Required DNS entries

This section written by Claude

Each Cloudhub can be supported from either jaia.tech or a custom domain.

For fleet6, assuming the IPv4 address for CloudHub is 203.0.113.42 and the IvP6 address is 2001:db8::42, enter the following in the DNS configuration for jaia.tech:

fleet6 A 203.0.113.42
fleet6 AAAA 2001:db8::42
*.fleet6 CNAME fleet6.jaia.tech.

If using a custom domain, you would change jaia.tech to your domain, and optionally fleet6 to your desired subdomain. For example, to use jaiaf6.gobysoft.org, you would add the DNS entries for gobysoft.org:

jaiaf6 A 203.0.113.42
jaiaf6 AAAA 2001:db8::42
*.jaiaf6 CNAME jaiaf6.gobysoft.org.

Caddy stays stopped until every site's name resolves to the CloudHub's own addresses (jaia-caddy-dns-ready.sh; journalctl -u caddy says which name it is waiting for), and jaia-caddy-start.timer starts it within 30 seconds of that. Let's Encrypt is therefore not asked for certificates before the records exist, so it has no failed attempts to back off from.

Required SMTP

This section written by Claude

Sending email from the Authelia instance is required for registering new 2FA tokens and password resets. By default these emails are sent from noreply@auth.jaia.tech through Postmark, for every fleet.

The relevant cloudhub fields of the fleet configuration are:

Field Default Meaning
smtp_address (required) SMTP server Authelia sends through: submission://smtp.postmarkapp.com:587 for Postmark
smtp_sender noreply@auth.jaia.tech From address. It must be verified with the SMTP provider
smtp_credentials_ssm_parameter /jaia/cloudhub/smtp_credentials AWS SSM Parameter Store SecureString holding the SMTP login as {"username": "...", "password": "..."}. A plain name is looked up in the CloudHub's own account and region; a full ARN can point anywhere the CloudHub's role is allowed to read

jaia admin fleet create asks "Are you using Jaia SMTP?"; yes sets Postmark and the two defaults, and no asks for all three.

For example, a client that hosts their CloudHub in their own AWS account and sends from their own domain:

cloudhub {
base_uri: "jaia.clientdomain.com"
admin_email: "admin@clientdomain.com"
smtp_address: "submission://smtp.postmarkapp.com:587"
smtp_sender: "noreply@auth.clientdomain.com"
smtp_credentials_ssm_parameter: "/client/jaia/smtp_credentials"
}

This section written by Claude

Mail scanners such as Microsoft Defender Safe Links open the links in these emails, and press the buttons on the pages behind them, before the person does. Authelia's reset page spends its one-time token as it loads, and its cancel pages act as they load. So on auth.<base uri>, Caddy:

  • sets a jaia_reset cookie (10 minutes, this host only) on the browser that asks for a reset, and passes GET /reset-password/step2 and /api/reset-password/identity/finish to Authelia only from a browser that carries it. Any other browser gets /_jaia/reset-elsewhere.html, which asks the person to open the link where they asked for it, or to ask again;
  • passes the cancel links (/revoke/reset-password, /revoke/one-time-code) and the calls behind them (DELETE /api/reset-password, DELETE /api/user/session/elevation/<id>) to Authelia only for someone signed in. Anyone else is sent to sign in first.

Authelia's limits on sending reset emails and one-time codes are 20 per 5 minutes and 100 per hour for each address, since a customer's office usually reaches the CloudHub from one address.

create_vpc.sh grants the CloudHub's IAM role ssm:GetParameter on this parameter (and kms:Decrypt through SSM, for parameters encrypted with a customer managed key), and jaia_configure_authelia.sh reads it at configuration time. If the parameter cannot be read, Authelia is configured to send without authenticating, which only works with a relay that allowlists the CloudHub's IP addresses (e.g. Google Workspace SMTP Relay). Postmark will refuse it, and Authelia will not start.

You can use https://www.mail-tester.com/ to check the likelihood that your emails will be caught in spam.

Available services

This section written by Claude

jaia.tech Domains

CloudHub access (https://fleetN.jaia.tech):

VirtualFleet access (https://sim.fleetN.jaia.tech):

The group super_admin gives the same access as run, sim, jdv, and jcu_developer. Note that this group does not automatically have lldap_admin privileges (as this is a special group within LLDAP).

Supporting web pages:

  • https://users.fleetN.jaia.tech: User management by fleet administrators. Add new users, add users to fleet access, and remove old users.
    • Groups: 'lldap_admin'
  • https://auth.fleetN.jaia.tech: Authelia authentication website. Typically the user doesn't need to access this directly unless they want to change their user settings.

Custom Domains

Replace .fleetN.jaia.tech with your custom domain in the examples above, where your custom domain might be jaiafleet6.mybusiness.com or jaiaf3.university.edu, as you prefer.

Navigation menu

Every page on these sites shows a floating Jaia button in the bottom left corner with the fleet number and the login name of the signed-in user. It opens a menu linking to the landing page, Run (JCC, JCU, JDV), Sim (JCC, JCU, JDV), Users and Account, showing only the sites the user's groups allow, plus a Log out button. Sim is shown as "Not running" until the VirtualFleet has been started from JCU, and the sim site itself says so instead of a blank error while it is down.

Caddy injects the menu into every HTML page, including the third-party Authelia and LLDAP pages, using the replace-response plugin. Ubuntu's caddy package doesn't include it, so the CloudHub runs the Caddy that jaiabot-web-thirdparty builds from source with the plugin (see below).