The Jaia Cloud is designed to provide:
It is possible to have a VirtualFleet without a corresponding real fleet or vice-versa.

Each VPC has one and only one fleet. At a minimum this is a CloudHub machine, which is always on.
The components of the VPC include:
This section written by Copilot
The use of the jaia ip tool (or, in scripts, the standalone jaia_ip binary that it wraps) is recommended for determining IP addresses for a given node, id, fleet, etc.
The network address assignment for the Jaia Cloud is intended to complement the existing fleet specific VPN. This means that a given fleet may have up to three VPN subnets assigned:
These all need to be different because:
Additionally, each VPC has two subnets assigned as previously mentioned:
This section written by Claude
The fleet WLAN, the fleet VPN and the VirtualFleet WLAN carry the fleet id in a single IPv4 octet, which is what limits how many fleets those schemes can address. Rather than renumber the fleets already using them, the fleet id range is split:
| Fleet ID | Fleet WLAN, fleet VPN | VirtualFleet WLAN | Other networks |
|---|---|---|---|
| 0-250 | IPv4, as below | IPv4 10.23.{flt}.0/24, the same addresses as the fleet WLAN | IPv6, as below |
| 251-4000 | IPv6, as below | IPv4 10.23.253.0/24 | IPv6, as below |
So a fleet above 250 has no IPv4 address of its own on a real fleet: IPv4 on its WLAN carries internet traffic only, from whatever the access point hands out, and all Jaia traffic is IPv6. The CloudHub and VirtualFleet ethernet subnets of a VPC are the same for every fleet and so stay IPv4 whichever range the fleet is in.
A VirtualFleet is the exception. Its nodes are EC2 instances, and EC2 assigns IPv6 from the block Amazon gives the VPC rather than from a ULA of ours, so a VirtualFleet above 250 cannot mirror its fleet's WLAN the way a lower one does. It stays IPv4 on 10.23.253.0/24 instead, which cannot collide with anything because a VPC belongs to one fleet. Configuration generated on a VirtualFleet node therefore addresses the WIFI link on vfleet_wlan; everywhere else it is wlan, and for a fleet at or below 250 the two give the same addresses.
A VirtualBox fleet has no such limit - its NAT network is local - so it mirrors the fleet WLAN whichever range the fleet is in, taking fddd:7f2e:3258:{fleet}::/64 above 250 with IPv4 from the NAT network's DHCP for internet access only, exactly as a real fleet of that id does. The host SSH ports are forwarded to that IPv4 rather than to the fleet IPv6, because VirtualBox stores an IPv6 port forward but never activates it; import_vms.sh reserves each node's vfleet_wlan address in the NAT network's DHCP so the forward has a fixed address to aim at.
The upper end of the range is 4000 rather than the 65535 the IPv6 scheme has room for, because the fleet id is also the XBee network id (ATID), whose documented range is smaller and which the driver writes in a format that does not currently reach it. Raising it is a matter of that driver and of widening HubStatus.fleet_id, which is a DCCL wire format change; nothing in the addressing itself is in the way.
jaia ip gives the address in the version the fleet actually uses, and jaia admin bounds --ipv4_fleet_id --max reports where the split is. Both come from fleet_id_ipv4_max in src/lib/utils/ip.h, which is the one place it is written down.
The prefix is fd00::/8 for unique local addresses as defined in RFC4193.
The full 48-bit prefix was (pseudo)-randomly generated as per RFC4193 for each of the three VPN classes.
| VPN Class | IPv6 Prefix |
|---|---|
| Fleet VPN | fd91:5457:1e5c::/48 |
| VirtualFleet VPN | fd6e:cf0d:aefa::/48 |
| CloudHub VPN | fd0f:77ac:4fdf::/48 |
| Physical WLAN | fddd:7f2e:3258::/48 |
For each VPN class, the Subnet ID is the Fleet ID, so for example, VirtualFleet 3 would have the subnet fd6e:cf0d:aefa:3::/64. This allows up to 2^16 = 65536 fleets to be assigned.
This section written by Claude
For a given node on the network, the 64-bit interface identifier is given as ::1:hub_id for hubs, ::2:bot_id for bots, ::3:customer_id for various customer machines (desktop / laptop / tablet), ::4:rpicam_id for rpicams, and ::5:0 for the gateway, which is the Wireguard server on the VPN networks (.1 on the IPv4 networks). This allows up to 2^16 = 65536 nodes of each type to be assigned per fleet.
Nothing is assigned the gateway address on the fleet WLAN. A node there finds its router by Router Advertisement, and the generated network configuration sets no IPv6 gateway; the address is reserved so that a Wireguard server, which is inside a tunnel where there is no Router Advertisement to answer, has the same identifier on every network.
Group ::0: is left unassigned: it holds the subnet-router anycast address (::) and ::1, which a router sharing the segment conventionally claims for itself. A node given either address fails Duplicate Address Detection and is unreachable.
Some examples include:
| Bot (b) or Hub (h)? | ID | Fleet | Fleet VPN Address (s) | VirtualFleet VPN (v) Address (fN) | CloudHub VPN Address (c) |
|---|---|---|---|---|---|
| Bot | 5 | 4 | fd91:5457:1e5c:4::2:5 | fd6e:cf0d:aefa:4::2:5 | fd0f:77ac:4fdf:4::2:5 |
| Bot | 6 | 250 | fd91:5457:1e5c:fa::2:6 | fd6e:cf0d:aefa:fa::2:6 | fd0f:77ac:4fdf:fa::2:6 |
| Hub | 20 | 10 | fd91:5457:1e5c:a::1:14 | fd6e:cf0d:aefa:a::1:14 | fd0f:77ac:4fdf:a::1:14 |
| Hub (CloudHub (ch)) | 30 | 15 | fd91:5457:1e5c:f::1:1e | fd6e:cf0d:aefa:f::1:1e | fd0f:77ac:4fdf:f::1:1e |
This section written by Claude
Every fleet in the table is an IPv4 fleet, so jaia ip gives its fleet VPN address as IPv4 (jaia ip b5sf4 is 172.23.4.105); --ip_version ipv6 asks for the IPv6 address listed here instead. The VirtualFleet and CloudHub VPNs are IPv6 for every fleet, so those need no flag.
You can generate the values for the table above yourself using:
This section written by Claude
CircleCI builds the AMI in us-east-1 (and separately in us-gov-east-1 for GovCloud), then copies it, with these tags, to each region listed in the ami_copy_regions parameter of the aws-sync job — currently ca-central-1. Since lookups are by tag rather than by AMI ID, create_vpc.sh and create-virtualfleet.yml resolve the image the same way in every region. To add a region, add it to that parameter; AMIs cannot be copied between the commercial and GovCloud partitions, so each partition needs its own build.
CUSTOMER_NAME is the fleet config's customer (default jaia), which jaia admin fleet create and edit ask for when the fleet has a CloudHub.
This section written by Claude
A CloudHub keeps its creation-time settings in /etc/jaiabot/cloud.env, written by cloud-init on first boot. That file lives on the rootfs, so a major upgrade discards it along with everything else outside the reuse set. Rather than carry it across, the settings are split by where they can be obtained again:
In the fleet config, because AWS cannot be asked for them:
| Field | Default |
|---|---|
| includes_cloudhub (top level) | true — answer no to the CloudHub question in jaia admin fleet create/edit for a fleet sold without one |
| customer (top level) | jaia — the value of the jaia_customer tag on every AWS resource the fleet owns |
| cloudhub.base_uri | required — the name the authentication front end is served under |
| cloudhub.admin_email | required — address of the fleet_admin user created on first boot |
| cloudhub.smtp_address | required — the relay Authelia sends enrolment and reset mail through |
| cloudhub.data_bucket | jaia--cloudhub-data--fleet<fleet> — the bucket mounted at the bot offload directory |
| cloudhub.smtp_sender | noreply@auth.jaia.tech — the From address, verified with the SMTP provider |
| cloudhub.smtp_credentials_ssm_parameter | /jaia/cloudhub/smtp_credentials — the SSM SecureString holding the SMTP login (see "Required SMTP" below) |
customer is a property of the fleet rather than of its CloudHub, so it sits at the top level; the rest are meaningless without hub 30 and validate requires the cloudhub message exactly when hub 30 is in the fleet.
Most fleets have a CloudHub and that is what the questions propose, but a fleet can be built without one. It has to say so — includes_cloudhub: false — rather than simply omitting hub 30, because a fleet that never had one and a fleet that lost it by mistake would otherwise look identical, and only the first should validate. jaia admin fleet has_cloudhub fleetN.cfg prints yes or no, which is what the major upgrade compares against the has_cloudhub answer the operator gives it. jaia admin fleet cloudhub create renders them into vpc.conf for create_vpc.sh. CI gives each run its own customer name by writing it into the fleet config it generates.
Discoverable from AWS, so deliberately not stored: the region, VPC, subnets, security groups, account ID and Elastic IP. Each is available from instance metadata or from a tag lookup within the fleet's VPC.
Carried across the upgrade: /etc/wireguard. The CloudHub's VPN private key cannot be regenerated without every client peer, including the login server, having to be reissued. The same goes for its SSH key, /home/jaia/.ssh/hub30_fleetN, which no fleet config holds (see below).
This section written by Claude
A new CloudHub's directory has two accounts, neither with a password: set one with Reset password? on the login page, which emails the address shown.
| Account | Whose | Groups | |
|---|---|---|---|
| jaia_bootstrap | Jaia's, for commissioning | super_admin, lldap_admin | create_cloudhub --bootstrap-email |
| fleet_admin | the customer's | super_admin, lldap_admin | cloudhub.admin_email |
Commissioning, signed in as jaia_bootstrap:
jaia_bootstrap is created the first time the CloudHub's directory is set up and never again. --bootstrap-email is required, and cannot be support@jaia.tech, which jaia_support already has. A CloudHub made without it, such as one upgraded from 2.y, gets no jaia_bootstrap. A major upgrade sets the directory up a second time, so a marker in the persistent /var/log/jaiabot/auth records that the account has had its turn; deleting it is final. There is also a jaia_support account, in no groups, which the support page moves into the web groups when a customer grants Jaia web access.
This section written by Claude
A physical hub's SSH key lives on its Yubikey, so the key handle in the fleet config is of no use without the hub itself. A CloudHub has no USB port for one, and its key is an ordinary file instead, so it is made on the CloudHub and never leaves it:
Create the CloudHub before generating the boot files of the other nodes. A node configured from a fleet config without the key prints a warning and does not accept the CloudHub until it is paired again (the "Pair new hub" playbook) with the updated config. The same applies to a CloudHub that is deleted and created again, since the new one has a new key.
Every node's /etc/jaiabot/ssh/hub_authorized_keys accepts the CloudHub's key only from its address on the CloudHub VPN (from="fd0f:77ac:4fdf:N::1:1e", where N is the fleet in hex), so a copy of the key is of no use without also being the CloudHub on that VPN.
Neither stored nor carried: which repository the VirtualFleet AMI comes from, and its version. create_cloudhub --repo picks the repository at creation, defaulting to release, and the JCU's "Change JaiaBot repository and update all packages" playbook changes it afterwards. The version is the release branch of whatever jaiabot is installed, read from common-versions.env, so a VirtualFleet raised after a major upgrade matches the upgraded CloudHub rather than the release it came from.
This section written by Claude
The CloudHub's sites (JCC, JDV, JCU and the rest, see "Cloud Login server" below) are reached through its login page, with no VPN. A tunnel of your own is for CI and development: jaia admin fleet cloudhub create --client-vpn makes the machine running it a peer (desktop1) of the CloudHub's VPNs, installs wg_jaia_ch<fleet>, and adds cloudhub-fleet<fleet> to its /etc/hosts. It is off by default because the peer stays on the CloudHub for as long as it runs, which is a standing way in that a customer's CloudHub should not ship with. In vpc.conf the same switches are ENABLE_CLIENT_VPN and UPDATE_CLIENT_ETC_HOSTS.
Once connected to the appropriate VPN and hosts are configured in /etc/hosts, one can open a web-browser as usual to JCC, etc.
For fleet1, these would be:
In addition to the CloudHub and VirtualFleet mentioned above, Jaia runs a number of "speciality" AWS servers, each with a specific primary task:
We want to be able to recreate these servers from scratch for consistency of implementation, security, and AWS instance type. We want to be able to upgrade to the latest Ubuntu/Jaiabot release by rebuilding servers, and have the tools in place to recreate them in the event of an instance disk corruption or AWS failure.
This is done using a pair of scripts in jaiabot/rootfs/cloud/aws:
Graphically, this is the relationship between the various components:

To create a new specialty server type (which we'll call "foo") you need to create three files, and by convention these are put in a subdirectory with the same name:
The docs and iridium server configurations are fairly simple, and one of these may be a useful starting point to copy for a new server configuration.
foo.conf is simply a bash shell script that is sourced by create_specialty_server.sh and must define the following variables:
foo.conf also must define the function function create_security_group_rules() which will be run (with the SECURITY_GROUP_ID as the only parameter) to allow the script to set the appropriate AWS firewall (security group) rules. Instances should restrict the allowed firewall inputs to the minimum necessary to allow functionality.
foo.conf can also optionally include:
This section written by Claude
These are template files for cloud-init user-data configuration. Both must be included but one may be left empty, except for the header line.
Minimal user-data.sh.in or user-data.sh:
Minimal user-data.yaml.in:
Both *.in configuration files are preprocessed using bash before being passed to cloud-init so that bash variables (${}) are expanded using the values in foo.conf and jaiabot/scripts/common-versions.env. If user-data.sh is specified instead, it will be used directly without preprocessing.
The contents of the preprocessed files are used by cloud-init to seed the new server instance. See the cloud-init documentation for more details.
Some servers will require manually copying data from the old server, or restoring a backup. For example, when upgrading the packages.jaia.tech server we copy the existing .deb files, etc. from the old server so that all of these .deb packages don't have to be rebuilt.
The user-data.*.in configuration files should install a script that can be run after the server is up and running. See the /home/jaia/sync_from_prior_packages_server.sh script installed by the jaiabot/rootfs/cloud/aws/packages/user-data.sh.in configuration file for an example.
The documentation for that server should describe the use of such a manual synchronization script.
Run create_specialty_server.sh foo/foo.conf to create the new server.
When testing the creation of a new server type, you may be creating and deleting the server multiple times before you are satisfied with the new configuration. Here it is helpful to set:
ENABLE_TERMINATION_PROTECTION=false
so that you can easily delete the new server until it works correctly. Once you have a working server you can go in to the AWS console and enable termination protection for that instance (or delete and rebuild one last time with ENABLE_TERMINATION_PROTECTION=true).
This section written by Claude
delete_specialty_server.sh also takes the server's .conf in place of a server type: it then works in the conf's REGION and deletes only the server named by its SERVER_NAME. This is how to delete a test copy without touching the production server of the same type.
To upgrade an existing server, the following set of steps is recommended:
This section written by Claude
Each CloudHub runs its own authentication server, allowing people to access the CloudHub and VirtualFleet securely using 2-factor [2FA] verification for all resources except the REST_API (which allows machines to use one-factor passwords to be used as API tokens).
This provides a more convenient way to access the JCC and other CloudHub applications without requiring that the client machine have the Wireguard VPN installed, and provide more granular permissions.

This server is implemented using three open source projects:
In short, Authelia manages authentication, Caddy manages the reverse proxy (between the insecure HTTP applications and the authenticated HTTPS connection), and LLDAP manages the user information (user names, group, passwords, etc.).
This section written by Claude
An instance of all three of these runs on each CloudHub.
This section written by Claude
All three come from one Debian package of the jaiabot source, jaiabot-web-thirdparty, which CircleCI builds and publishes alongside the other jaiabot packages for amd64 and arm64:
The package is in the CloudHub image (rootfs/customization/package-lists/cloud.list.chroot), so a fresh CloudHub's first boot only configures what the image already holds: nothing is fetched from caddyserver.com, apt.authelia.com, Docker Hub or Amazon's CLI downloads, none of which had a bounded timeout before. First boot still needs AWS itself, Let's Encrypt and apt metadata. A Minor Upgrade upgrades the package with the rest of jaiabot (config/ansible/tasks/install-jaiabot.yml); a Major Upgrade's image carries it.
The package enables none of its units: jaia_configure_authelia.sh writes each service's configuration, enables it and starts it (restarting Caddy only when the running process's binary has been replaced by an upgrade, and LLDAP only when its configuration changed, as it already does for Authelia).
Updating the versions. renovate.json at the repository root tells Renovate to open PRs against 3.y: one for Caddy and its modules (the gomod manager, which also moves the untagged replace-response plugin forward by commit), and one per pin in scripts/common-versions.env that carries a # renovate: line (Authelia and LLDAP from their GitHub releases, and the build tools: arduino-cli, nvm, Node, npm, webpack, webpack-cli, the Raspberry Pi firmware). A version PR for Authelia or LLDAP needs its checksums refreshed before it can build: src/web/thirdparty/fetch.sh --update-checksums downloads the new tarballs and rewrites the jaia_sha256_* lines. Authelia migrates its database forward and cannot read it once a newer release has, so never merge a downgrade. Renovate runs only once the Renovate GitHub app is enabled for the repository.
This section written by Claude
Each Cloudhub can be supported from either jaia.tech or a custom domain.
For fleet6, assuming the IPv4 address for CloudHub is 203.0.113.42 and the IvP6 address is 2001:db8::42, enter the following in the DNS configuration for jaia.tech:
If using a custom domain, you would change jaia.tech to your domain, and optionally fleet6 to your desired subdomain. For example, to use jaiaf6.gobysoft.org, you would add the DNS entries for gobysoft.org:
Caddy stays stopped until every site's name resolves to the CloudHub's own addresses (jaia-caddy-dns-ready.sh; journalctl -u caddy says which name it is waiting for), and jaia-caddy-start.timer starts it within 30 seconds of that. Let's Encrypt is therefore not asked for certificates before the records exist, so it has no failed attempts to back off from.
This section written by Claude
Sending email from the Authelia instance is required for registering new 2FA tokens and password resets. By default these emails are sent from noreply@auth.jaia.tech through Postmark, for every fleet.
The relevant cloudhub fields of the fleet configuration are:
| Field | Default | Meaning |
|---|---|---|
| smtp_address | (required) | SMTP server Authelia sends through: submission://smtp.postmarkapp.com:587 for Postmark |
| smtp_sender | noreply@auth.jaia.tech | From address. It must be verified with the SMTP provider |
| smtp_credentials_ssm_parameter | /jaia/cloudhub/smtp_credentials | AWS SSM Parameter Store SecureString holding the SMTP login as {"username": "...", "password": "..."}. A plain name is looked up in the CloudHub's own account and region; a full ARN can point anywhere the CloudHub's role is allowed to read |
jaia admin fleet create asks "Are you using Jaia SMTP?"; yes sets Postmark and the two defaults, and no asks for all three.
For example, a client that hosts their CloudHub in their own AWS account and sends from their own domain:
This section written by Claude
Mail scanners such as Microsoft Defender Safe Links open the links in these emails, and press the buttons on the pages behind them, before the person does. Authelia's reset page spends its one-time token as it loads, and its cancel pages act as they load. So on auth.<base uri>, Caddy:
Authelia's limits on sending reset emails and one-time codes are 20 per 5 minutes and 100 per hour for each address, since a customer's office usually reaches the CloudHub from one address.
create_vpc.sh grants the CloudHub's IAM role ssm:GetParameter on this parameter (and kms:Decrypt through SSM, for parameters encrypted with a customer managed key), and jaia_configure_authelia.sh reads it at configuration time. If the parameter cannot be read, Authelia is configured to send without authenticating, which only works with a relay that allowlists the CloudHub's IP addresses (e.g. Google Workspace SMTP Relay). Postmark will refuse it, and Authelia will not start.
You can use https://www.mail-tester.com/ to check the likelihood that your emails will be caught in spam.
This section written by Claude
CloudHub access (https://fleetN.jaia.tech):
VirtualFleet access (https://sim.fleetN.jaia.tech):
The group super_admin gives the same access as run, sim, jdv, and jcu_developer. Note that this group does not automatically have lldap_admin privileges (as this is a special group within LLDAP).
Supporting web pages:
Replace .fleetN.jaia.tech with your custom domain in the examples above, where your custom domain might be jaiafleet6.mybusiness.com or jaiaf3.university.edu, as you prefer.
Every page on these sites shows a floating Jaia button in the bottom left corner with the fleet number and the login name of the signed-in user. It opens a menu linking to the landing page, Run (JCC, JCU, JDV), Sim (JCC, JCU, JDV), Users and Account, showing only the sites the user's groups allow, plus a Log out button. Sim is shown as "Not running" until the VirtualFleet has been started from JCU, and the sim site itself says so instead of a blank error while it is down.
Caddy injects the menu into every HTML page, including the third-party Authelia and LLDAP pages, using the replace-response plugin. Ubuntu's caddy package doesn't include it, so the CloudHub runs the Caddy that jaiabot-web-thirdparty builds from source with the plugin (see below).