Debian Openstack-Cluster-Installer Notes

March 16, 2022

** WARNING: THESE ARE NOTES. DO NOT USE THIS IN PRODUCTION **

Debian-OCI is pretty excellent. It’s a huge pile of Puppet, Bash, and PHP which ties together logs of annoyingly fiddly sysadmin tasks that you tell yourself that you’ll get around to eventually. It has its traps, though, which I’ve found while hacking on it. Be warned that it also strong opinions (usually the right ones, IMO) about how your network and storage should work. Sticking to the “blessed” path with Openstack in general seems to be a good idea.

PXE boot “Live” Image

The “live” image needs to be built twice if doing a fresh install. Once to create the various subdirs and config files under /etc/openstack-cluster-installer /var/lib/openstack-cluster-installer, and again once you’ve customised anything in /etc/openstack-cluster-installer (for example: SSH keys, local package repos, local repo signing keys, etc.)

Init Cluster

ocicli swift-region-create uk1
ocicli location-create uk1-zone1 uk1
ocicli network-create os-pubnet1 192.168.1.100 32 uk1-zone1 yes
ocicli network-create os-net1 10.66.0.0 24 uk1-zone1 no
ocicli cluster-create os os.tunstall.in

Real TLS certs

More important than the official docs make it seem

ocicli cluster-set os --vip-hostname controller.os.tunstall.in
oci-gen-slave-node-cert controller.os.tunstall.in

Assign machines to the controller role

Must be at least 3 (three) for Corosync/pacemaker to do their job

ocicli machine-add GESY6330001T os controller uk1-zone1
ocicli machine-add GESY631001BJ os controller uk1-zone1
ocicli machine-add GESY631004NT os controller uk1-zone1

At least one compute node must be defined

If this is skipped then the controller nodes will not be able to build because the ENC will fail to look up the list of compute nodes and therefore can’t add firewalls to/from them.

ocicli machine-add GESY633000SU os compute uk1-zone1

ocicli machine-set GESY633000SU --serial-console-device ttyS0 --dc 1 --row 1 --rack 1
ocicli machine-set GESY6330001T --serial-console-device ttyS0 --dc 1 --row 1 --rack 1
ocicli machine-set GESY631001BJ --serial-console-device ttyS0 --dc 1 --row 1 --rack 1
ocicli machine-set GESY631004NT --serial-console-device ttyS0 --dc 1 --row 1 --rack 1

Set the actual ip addresses you want

ocicli machine-ip-remove GESY6330001T 10.66.0.4
ocicli machine-ip-add GESY6330001T os-net1 10.66.0.81

ocicli machine-ip-remove GESY631001BJ 10.66.0.3
ocicli machine-ip-add GESY631001BJ os-net1 10.66.0.82

ocicli machine-ip-remove GESY631004NT 10.66.0.2
ocicli machine-ip-add GESY631004NT os-net1 10.66.0.83

ocicli machine-ip-remove GESY633000SU 10.66.0.2
ocicli machine-ip-add GESY633000SU os-net1 10.66.0.84

This is a mess.

ocicli cluster-set os --use-ovs-ifaces yes

ocicli network-create ceph-osd 10.66.182.0 24 os-zone1 no

ocicli network-add ceph-osd os cephosd 10g1 10g2

ocicli network-set ceph-osd --role ceph-cluster --bridge-name vlan-br-ceph
ocicli network-set os-vmnet --role vm-net --bridge-name br-tun

ocicli machine-ip-remove CZ3924128A 10.66.1.5
ocicli machine-ip-remove CZ3924128A 10.66.100.5
ocicli machine-ip-add CZ3924128A os-net1 10.66.1.11
ocicli machine-ip-add CZ3924128A os-vmnet 10.66.100.11
ocicli machine-ip-add CZ3924128A ceph-osd 10.66.182.11

ocicli machine-ip-remove CZ3924128B 10.66.1.7
ocicli machine-ip-remove CZ3924128B 10.66.100.7
ocicli machine-ip-add CZ3924128B os-net1 10.66.1.21
ocicli machine-ip-add CZ3924128B os-vmnet 10.66.100.21
ocicli machine-ip-add CZ3924128B ceph-osd 10.66.182.21

ocicli machine-ip-remove CZ3924128C 10.66.1.6
ocicli machine-ip-remove CZ3924128C 10.66.100.6
ocicli machine-ip-add CZ3924128C os-net1 10.66.1.31
ocicli machine-ip-add CZ3924128C os-vmnet 10.66.100.31
ocicli machine-ip-add CZ3924128C ceph-osd 10.66.182.31

ocicli machine-ip-remove CZ39241BX7 10.66.1.3
ocicli machine-ip-remove CZ39241BX7 10.66.100.3
ocicli machine-ip-add CZ39241BX7 os-net1 10.66.1.51
ocicli machine-ip-add CZ39241BX7 os-vmnet 10.66.100.51
ocicli machine-ip-add CZ39241BX7 ceph-osd 10.66.182.51

ocicli machine-ip-remove CZ39241BX8 10.66.1.2
ocicli machine-ip-remove CZ39241BX8 10.66.100.2
ocicli machine-ip-add CZ39241BX8 os-net1 10.66.1.52
ocicli machine-ip-add CZ39241BX8 os-vmnet 10.66.100.52
ocicli machine-ip-add CZ39241BX8 ceph-osd 10.66.182.52

ocicli machine-ip-remove CZ39241BX9 10.66.1.4
ocicli machine-ip-remove CZ39241BX9 10.66.100.4
ocicli machine-ip-add CZ39241BX9 os-net1 10.66.1.53
ocicli machine-ip-add CZ39241BX9 os-vmnet 10.66.100.53
ocicli machine-ip-add CZ39241BX9 ceph-osd 10.66.182.53

Both public and private networks must be defined

ocicli network-add os-pubnet1 os all 1g1 1g2
ocicli network-add os-net1 os all 1g1 1g2

Hardcode system serial numbers in the installed OS

echo GESY6330001T > /var/lib/oci/clusters/os/os-controller-1.os.tunstall.in/oci-in-target/etc/oci/system-serial
echo GESY631001BJ > /var/lib/oci/clusters/os/os-controller-2.os.tunstall.in/oci-in-target/etc/oci/system-serial
echo GESY631004NT > /var/lib/oci/clusters/os/os-controller-3.os.tunstall.in/oci-in-target/etc/oci/system-serial
echo GESY633000SU > /var/lib/oci/clusters/os/os-compute-1.os.tunstall.in/oci-in-target/etc/oci/system-serial

ls -l /var/lib/oci/clusters/os/*/oci-in-target/etc/oci/system-serial
cat /var/lib/oci/clusters/os/*/oci-in-target/etc/oci/system-serial

Tell machines to use our repo

for i in {1..3}; do
  mkdir -p /var/lib/oci/clusters/os/os-controller-${i}.os.tunstall.in/oci-in-target/etc/apt/trusted.gpg.d
  mkdir -p /var/lib/oci/clusters/os/os-controller-${i}.os.tunstall.in/oci-in-target/etc/apt/sources.list.d
  cp /etc/apt/trusted.gpg.d/repo.tunstall.in.gpg /var/lib/oci/clusters/os/os-controller-${i}.os.tunstall.in/oci-in-target/etc/apt/trusted.gpg.d/
  cp /etc/apt/sources.list.d/repo.tunstall.in.sources /var/lib/oci/clusters/os/os-controller-${i}.os.tunstall.in/oci-in-target/etc/apt/sources.list.d/
  ls -l /var/lib/oci/clusters/os/os-controller-${i}.os.tunstall.in/oci-in-target/etc/apt/trusted.gpg.d/
  ls -l /var/lib/oci/clusters/os/os-controller-${i}.os.tunstall.in/oci-in-target/etc/apt/sources.list.d/
done

Once installed, machines have the wrong router because they don’t pick it up from DHCP

# ip route
default via 10.66.0.1 dev bond0
10.66.0.0/24 dev bond0 proto kernel scope link src 10.66.0.83

Simple fix:

ip route del default
ip route add default via 10.66.0.1

During initial puppet run, needed to add DB passwords because puppet couldn’t get past it. Possibly due to them not being able to reach the VIP.

Hyperconverged Ceph

Also known as “Ceph OSDs on Compute Nodes”. This is completely undocumented in OCI.

root@os-pxe:/etc/openstack-cluster-installer# ocicli machine-list
| serial     | hostname                        | Cur_ip     | status     |
| ---------- | ------------------------------- | ---------- | ---------- |
| CZ3924128A | os-compute-1.os.tunstall.in  | 10.66.1.11 | installing |
| CZ3924128B | os-compute-2.os.tunstall.in  | 10.66.1.21 | installing |
| CZ3924128C | os-compute-3.os.tunstall.in  | 10.66.1.31 | installing |
| CZ39241BX7 | os-controller-1.os.tunstall.in | 10.66.1.51 | installed  |
| CZ39241BX8 | os-controller-2.os.tunstall.in | 10.66.1.52 | installed  |
| CZ39241BX9 | os-controller-3.os.tunstall.in | 10.66.1.53 | installed  |
root@os-pxe:/etc/openstack-cluster-installer# ocicli machine-set CZ3924128C --compute-is-cephosd yes
| Ok. |
| --- |
root@os-pxe:/etc/openstack-cluster-installer# ocicli machine-set CZ3924128B --compute-is-cephosd yes
| Ok. |
| --- |

Customised Config

OCI will copy anything from:

/var/lib/oci/clusters/<CLUSTER>/<HOSTNAME>/oci-in-target/

to systems it builds. This can be used to override anything OCI has set up already.

Trusting Extra Repos

for i in {1..3}; do
  mkdir -p /var/lib/oci/clusters/os/os-controller-${i}.os.tunstall.in/oci-in-target/etc/apt/trusted.gpg.d
  mkdir -p /var/lib/oci/clusters/os/os-controller-${i}.os.tunstall.in/oci-in-target/etc/apt/sources.list.d
  cp /etc/apt/trusted.gpg.d/repo.tunstall.in.gpg /var/lib/oci/clusters/os/os-controller-${i}.os.tunstall.in/oci-in-target/etc/apt/trusted.gpg.d/
  cp /etc/apt/sources.list.d/repo.tunstall.in.sources /var/lib/oci/clusters/os/os-controller-${i}.os.tunstall.in/oci-in-target/etc/apt/sources.list.d/
  cp /root/.vimrc /var/lib/oci/clusters/os/os-controller-${i}.os.tunstall.in/oci-in-target/root/
  ls -l /var/lib/oci/clusters/os/os-controller-${i}.os.tunstall.in/oci-in-target/etc/apt/*
done

Non-Standard Network Configuration

While the normal OCI network configuration is fairly flexible, there are situations where this isn’t quite enough. Dropping standard Debian network config into

/var/lib/oci/clusters/<CLUSTER>/<HOSTNAME>/oci-in-target/etc/network

does the job nicely:

# cp -r os-controller-1.os.tunstall.in/oci-in-target/etc/network os-controller-2.os.tunstall.in/oci-in-target/

# sed -i 's/81/82/g' os-controller-2.os.tunstall.in/oci-in-target/etc/network/interfaces.d/*

# cat os-controller-2.os.tunstall.in/oci-in-target/etc/network/interfaces.d/*
# Interface bond-mgmt
auto bond-mgmt
allow-br-ex bond-mgmt

iface bond-mgmt inet manual
  # Address for Openstack API internal communication. PVID 1 on the NICs so needs to be on the bond interface.
  ovs_bridge br-ex
  ovs_type OVSBond
  ovs_bonds eno1 eno2
  ovs_options bond_mode=balance-tcp lacp=active other_config:lacp-time=fast
  up ifup eno1
  up ifup eno2
 
# MTU 9000 in os
#  up ovs-vsctl set Interface ${IFACE} mtu_request=9000
#  up /bin/sleep 1 ; /bin/ip link set mtu 9000 dev bond-mgmt

# br-ex Interface

auto br-ex
iface br-ex inet static
  # Address for Openstack API internal communication. PVID 1 on the NICs so needs to be on the bridge
  address 10.66.0.82/24
  gateway 10.66.0.1
  ovs_type OVSBridge
  ovs_ports vlan-internet
  #ovs_ports bond-openstack vlan-internet vlan-vmnet os
  #ovs_mtu 9000

# NUCs
allow-bond-mgmt eno1
iface eno1 inet manual
  ovs_type OVSPort
  ovs_bonds bond-mgmt
#  up /bin/ip link set mtu 9000 dev eno1

# NUCs have a single interface, OVS needs two for a bond :-)
allow-bond-mgmt eno2
iface eno2 inet manual
  pre-up /sbin/ip link add eno2 type dummy
  ovs_type OVSPort
  ovs_bonds bond-mgmt
#  up /bin/ip link set mtu 9000 dev eno2

# Interface vlan-internet
auto vlan-internet
allow-br-ex vlan-internet
iface vlan-internet inet manual
  ovs_type OVSIntPort
  ovs_bridge br-ex
  ovs_options tag=4093
  ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif
  #ovs_mtu 9000

#  address 10.66.100.82/24
#  network 10.66.100.0/24

  # NUCs
#  vlan-raw-device bond-mgmt
  # os
#  vlan-raw-device bond-openstack

# Interface vlan-vmnet
# Openstack VM networks live here

auto vlan-vmnet
allow-br-ex vlan-vmnet
iface vlan-vmnet inet static
  ovs_type OVSIntPort
  ovs_bridge br-ex
  ovs_options tag=100
  ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif
  ovs_mtu 9000 # MTU 9000 in os
  address 10.66.100.82/24
  network 10.66.100.0
  broadcast 10.66.100.255

TLS Cert Config

API hostname: os-api.os.tunstall.in

ocicli cluster-set os --self-signed-api-cert no

Weird Things

High load during initial puppet run

During the initial (first) puppet run on NUCs (4 core, 16 GiB) I found that the primary host got into a high-load (~25) I/O bound state where many services were writing lots of errors to the logs which caused package installations to time out. Stopping the following services helped a lot:

barbican-api
neutron-api
neutron-l3-agent
neutron-openvswitch-agent
neutron-rpc-server
aodh-notifier

To install on a non-`sda` device

ocicli machine-display-install-cmd GESY6330001T

oci-install-with-report \
--static-iface type=bond,iface0=eno1,iface1=Array,addr=10.66.0.81/24:10.66.0.1,extra_ovs=yes,ovsbr=br-ex \
--release bullseye \
--debootstrap-url http://10.66.0.98:9999/debian \
--sources.list-mirror http://10.66.0.98:9999/debian \
--security-mirror http://10.66.0.98:9999/security \
--dest-hdd sda \
--no-cloud-init \
--extra-packages gnupg2,haveged,uuid-runtime,iotop,iftop,man-db,curl,less,lsb-release,joe,ssl-cert,most,screen,vim,vim-tiny,tcpd,xfsdump,unzip,tcpdump,ntpstat,ca-certificates,rpcbind,lftp,at,tree,lsof,bind9-host,dnsutils,strace,tmux,nano,bash-completion,openssl,file,net-tools,iproute2,ipmitool,ca-certificates,xfsprogs,e2fsprogs,parted,nmap,mtr-tiny \
--hook-script /usr/bin/openstack-cluster-installer-bodi-hook-script \
--root-ssh-key /root/.ssh/authorized_keys \
--install-nonfree-repo \
--postinstall-packages q-text-as-data,firmware-bnx2,firmware-bnx2x,plymouth,puppet,bridge-utils,grc,ccze,ncdu,lvm2,intel-microcode,smartmontools,kexec-tools,iptables-persistent,lshw,jq,apg,dmsetup,dnsutils,eperl,gettext-base,lynx,quota,acct,netcat,psmisc,chrony,openvswitch-switch --hostname os-controller-1.os.tunstall.in \
--add-serial-getty ttyS1 \
--tty-autologin yes \
--reboot-after-install

Install to RAID

tl;dr Linux disk detection is non-deterministic, especially when HP RAID controllers are involved.

ocicli machine-set CZ3924128B --install-on-raid yes --raid-type 1 --raid-dev0 sdf --raid-dev1 sdj
ocicli machine-install-os CZ3924128B
ocicli machine-set CZ3924128A --install-on-raid yes --raid-type 1 --raid-dev0 sda --raid-dev1 sdb
ocicli machine-install-os CZ3924128A
ocicli machine-set CZ3924128C --install-on-raid yes --raid-type 1 --raid-dev0 sda --raid-dev1 sdc
ocicli machine-install-os CZ3924128C

Old fashioned network interface names

OCI likes to use the ancient eth0 style interface naming. This is inconvenient. Until we patch the image generator not to do this, it can be accomplished post-boot with:

Management Nodes

sed -i 's/bond-slaves.*$/bond-slaves enp196s0f0 enp196s0f1/' /etc/network/interfaces

Compute Nodes

sed -i 's/bond-slaves.*$/bond-slaves enp100s0f0 enp100s0f1/' /etc/network/interfaces

…and then followed up with a reboot

OpenVSwitch Patching between bridges

ovs-vsctl add-port vlan-br-ex patch-vlan-br-ex
ovs-vsctl set interface patch-vlan-br-ex type=patch options:peer=patch-br-ex
ovs-vsctl set interface patch-br-ex type=patch options:peer=patch-vlan-br-ex

Enable LLDP on OpenVSwitch

OpenVSwitch doesn’t automatically enable LLDP, which can be annoying in datacentre environments

TODO: make this happen using /etc/network/interfaces

for i in eno5 eno6 eno7 eno8 ens1f0 ens1f1; do
 ip link set dev ${i} up
 ovs-vsctl set interface $i lldp:enable=true
done

ovs-vsctl add-aa-mapping br-ex 0 0
ovs-vsctl set AutoAttach . system_name=${HOSTNAME}

Rename the frontend controller API

Openstack cluster DB:

UPDATE endpoint SET url = REPLACE(url, 'os-api.os.tunstall.in', 'controller.os.tunstall.in');

fix /etc/keystone/puppet.conf on controller nodes:

[keystone_authtoken]
region_name=RegionOne
auth_url=auth_url=https://controller.os.tunstall.in/identity # THIS LINE WILL HAVE THE PREVIOUS NAME
username=admin
password=solarwinds123
project_name=admin
interface=public

…and then restart EVERYTHING

systemctl stop placement-api.service octavia-worker.service octavia-housekeeping.service octavia-health-manager.service octavia-api.service nova-spicehtml5proxy.service nova-serialproxy.service nova-scheduler.service nova-novncproxy.service nova-consoleauth.service nova-conductor.service nova-api.service nova-api-metadata.service neutron-rpc-server.service neutron-openvswitch-agent.service neutron-l3-agent.service neutron-api.service heat-engine.service heat-api.service heat-api-cfn.service barbican-api.service aodh-notifier.service aodh-listener.service aodh-evaluator.service aodh-api.service cinder-api.service cinder-scheduler.service

systemctl start placement-api.service octavia-worker.service octavia-housekeeping.service octavia-health-manager.service octavia-api.service nova-spicehtml5proxy.service nova-serialproxy.service nova-scheduler.service nova-novncproxy.service nova-consoleauth.service nova-conductor.service nova-api.service nova-api-metadata.service neutron-rpc-server.service neutron-openvswitch-agent.service neutron-l3-agent.service neutron-api.service heat-engine.service heat-api.service heat-api-cfn.service barbican-api.service aodh-notifier.service aodh-listener.service aodh-evaluator.service aodh-api.service cinder-api.service cinder-scheduler.service

Django fills up apache logs

Django will sometimes spazz out and need you to do:

/usr/share/openstack-dashboard/manage.py collectstatic --noinput --clear; /usr/share/openstack-dashboard/manage.py compress --force

…after apache is restarted. This bug isn’t supposed to exist any more, but whatevs: https://bugzilla.redhat.com/show_bug.cgi?id=1220070

Reset a machine’ state for reinstallation

Sometimes it is necessary to “reset” the state of an already-installed machine. Due to the way the OCI Puppet ENC works you can get into situations where an installation has “succeeded” incorrectly.

UPDATE machines SET status='live' WHERE hostname LIKE 'os-compute-1.%';
UPDATE machines SET puppet_status='notrun' WHERE hostname LIKE 'os-compute-1.%;