Debian Openstack-Cluster-Installer Notes
** WARNING: THESE ARE NOTES. DO NOT USE THIS IN PRODUCTION **
Debian-OCI is pretty excellent. It’s a huge pile of Puppet, Bash, and PHP which ties together logs of annoyingly fiddly sysadmin tasks that you tell yourself that you’ll get around to eventually. It has its traps, though, which I’ve found while hacking on it. Be warned that it also strong opinions (usually the right ones, IMO) about how your network and storage should work. Sticking to the “blessed” path with Openstack in general seems to be a good idea.
PXE boot “Live” Image
The “live” image needs to be built twice if doing a fresh install. Once to create the various subdirs and config files under /etc/openstack-cluster-installer /var/lib/openstack-cluster-installer, and again once you’ve customised anything in /etc/openstack-cluster-installer (for example: SSH keys, local package repos, local repo signing keys, etc.)
Init Cluster
ocicli swift-region-create uk1
ocicli location-create uk1-zone1 uk1
ocicli network-create os-pubnet1 192.168.1.100 32 uk1-zone1 yes
ocicli network-create os-net1 10.66.0.0 24 uk1-zone1 no
ocicli cluster-create os os.tunstall.in
Real TLS certs
More important than the official docs make it seem
ocicli cluster-set os --vip-hostname controller.os.tunstall.in
oci-gen-slave-node-cert controller.os.tunstall.in
Assign machines to the controller role
Must be at least 3 (three) for Corosync/pacemaker to do their job
ocicli machine-add GESY6330001T os controller uk1-zone1
ocicli machine-add GESY631001BJ os controller uk1-zone1
ocicli machine-add GESY631004NT os controller uk1-zone1
At least one compute node must be defined
If this is skipped then the controller nodes will not be able to build because the ENC will fail to look up the list of compute nodes and therefore can’t add firewalls to/from them.
ocicli machine-add GESY633000SU os compute uk1-zone1
ocicli machine-set GESY633000SU --serial-console-device ttyS0 --dc 1 --row 1 --rack 1
ocicli machine-set GESY6330001T --serial-console-device ttyS0 --dc 1 --row 1 --rack 1
ocicli machine-set GESY631001BJ --serial-console-device ttyS0 --dc 1 --row 1 --rack 1
ocicli machine-set GESY631004NT --serial-console-device ttyS0 --dc 1 --row 1 --rack 1
Set the actual ip addresses you want
ocicli machine-ip-remove GESY6330001T 10.66.0.4
ocicli machine-ip-add GESY6330001T os-net1 10.66.0.81
ocicli machine-ip-remove GESY631001BJ 10.66.0.3
ocicli machine-ip-add GESY631001BJ os-net1 10.66.0.82
ocicli machine-ip-remove GESY631004NT 10.66.0.2
ocicli machine-ip-add GESY631004NT os-net1 10.66.0.83
ocicli machine-ip-remove GESY633000SU 10.66.0.2
ocicli machine-ip-add GESY633000SU os-net1 10.66.0.84
This is a mess.
ocicli cluster-set os --use-ovs-ifaces yes
ocicli network-create ceph-osd 10.66.182.0 24 os-zone1 no
ocicli network-add ceph-osd os cephosd 10g1 10g2
ocicli network-set ceph-osd --role ceph-cluster --bridge-name vlan-br-ceph
ocicli network-set os-vmnet --role vm-net --bridge-name br-tun
ocicli machine-ip-remove CZ3924128A 10.66.1.5
ocicli machine-ip-remove CZ3924128A 10.66.100.5
ocicli machine-ip-add CZ3924128A os-net1 10.66.1.11
ocicli machine-ip-add CZ3924128A os-vmnet 10.66.100.11
ocicli machine-ip-add CZ3924128A ceph-osd 10.66.182.11
ocicli machine-ip-remove CZ3924128B 10.66.1.7
ocicli machine-ip-remove CZ3924128B 10.66.100.7
ocicli machine-ip-add CZ3924128B os-net1 10.66.1.21
ocicli machine-ip-add CZ3924128B os-vmnet 10.66.100.21
ocicli machine-ip-add CZ3924128B ceph-osd 10.66.182.21
ocicli machine-ip-remove CZ3924128C 10.66.1.6
ocicli machine-ip-remove CZ3924128C 10.66.100.6
ocicli machine-ip-add CZ3924128C os-net1 10.66.1.31
ocicli machine-ip-add CZ3924128C os-vmnet 10.66.100.31
ocicli machine-ip-add CZ3924128C ceph-osd 10.66.182.31
ocicli machine-ip-remove CZ39241BX7 10.66.1.3
ocicli machine-ip-remove CZ39241BX7 10.66.100.3
ocicli machine-ip-add CZ39241BX7 os-net1 10.66.1.51
ocicli machine-ip-add CZ39241BX7 os-vmnet 10.66.100.51
ocicli machine-ip-add CZ39241BX7 ceph-osd 10.66.182.51
ocicli machine-ip-remove CZ39241BX8 10.66.1.2
ocicli machine-ip-remove CZ39241BX8 10.66.100.2
ocicli machine-ip-add CZ39241BX8 os-net1 10.66.1.52
ocicli machine-ip-add CZ39241BX8 os-vmnet 10.66.100.52
ocicli machine-ip-add CZ39241BX8 ceph-osd 10.66.182.52
ocicli machine-ip-remove CZ39241BX9 10.66.1.4
ocicli machine-ip-remove CZ39241BX9 10.66.100.4
ocicli machine-ip-add CZ39241BX9 os-net1 10.66.1.53
ocicli machine-ip-add CZ39241BX9 os-vmnet 10.66.100.53
ocicli machine-ip-add CZ39241BX9 ceph-osd 10.66.182.53
Both public and private networks must be defined
ocicli network-add os-pubnet1 os all 1g1 1g2
ocicli network-add os-net1 os all 1g1 1g2
Hardcode system serial numbers in the installed OS
echo GESY6330001T > /var/lib/oci/clusters/os/os-controller-1.os.tunstall.in/oci-in-target/etc/oci/system-serial
echo GESY631001BJ > /var/lib/oci/clusters/os/os-controller-2.os.tunstall.in/oci-in-target/etc/oci/system-serial
echo GESY631004NT > /var/lib/oci/clusters/os/os-controller-3.os.tunstall.in/oci-in-target/etc/oci/system-serial
echo GESY633000SU > /var/lib/oci/clusters/os/os-compute-1.os.tunstall.in/oci-in-target/etc/oci/system-serial
ls -l /var/lib/oci/clusters/os/*/oci-in-target/etc/oci/system-serial
cat /var/lib/oci/clusters/os/*/oci-in-target/etc/oci/system-serial
Tell machines to use our repo
for i in {1..3}; do
mkdir -p /var/lib/oci/clusters/os/os-controller-${i}.os.tunstall.in/oci-in-target/etc/apt/trusted.gpg.d
mkdir -p /var/lib/oci/clusters/os/os-controller-${i}.os.tunstall.in/oci-in-target/etc/apt/sources.list.d
cp /etc/apt/trusted.gpg.d/repo.tunstall.in.gpg /var/lib/oci/clusters/os/os-controller-${i}.os.tunstall.in/oci-in-target/etc/apt/trusted.gpg.d/
cp /etc/apt/sources.list.d/repo.tunstall.in.sources /var/lib/oci/clusters/os/os-controller-${i}.os.tunstall.in/oci-in-target/etc/apt/sources.list.d/
ls -l /var/lib/oci/clusters/os/os-controller-${i}.os.tunstall.in/oci-in-target/etc/apt/trusted.gpg.d/
ls -l /var/lib/oci/clusters/os/os-controller-${i}.os.tunstall.in/oci-in-target/etc/apt/sources.list.d/
done
Once installed, machines have the wrong router because they don’t pick it up from DHCP
# ip route
default via 10.66.0.1 dev bond0
10.66.0.0/24 dev bond0 proto kernel scope link src 10.66.0.83
Simple fix:
ip route del default
ip route add default via 10.66.0.1
During initial puppet run, needed to add DB passwords because puppet couldn’t get past it. Possibly due to them not being able to reach the VIP.
Hyperconverged Ceph
Also known as “Ceph OSDs on Compute Nodes”. This is completely undocumented in OCI.
root@os-pxe:/etc/openstack-cluster-installer# ocicli machine-list
| serial | hostname | Cur_ip | status |
| ---------- | ------------------------------- | ---------- | ---------- |
| CZ3924128A | os-compute-1.os.tunstall.in | 10.66.1.11 | installing |
| CZ3924128B | os-compute-2.os.tunstall.in | 10.66.1.21 | installing |
| CZ3924128C | os-compute-3.os.tunstall.in | 10.66.1.31 | installing |
| CZ39241BX7 | os-controller-1.os.tunstall.in | 10.66.1.51 | installed |
| CZ39241BX8 | os-controller-2.os.tunstall.in | 10.66.1.52 | installed |
| CZ39241BX9 | os-controller-3.os.tunstall.in | 10.66.1.53 | installed |
root@os-pxe:/etc/openstack-cluster-installer# ocicli machine-set CZ3924128C --compute-is-cephosd yes
| Ok. |
| --- |
root@os-pxe:/etc/openstack-cluster-installer# ocicli machine-set CZ3924128B --compute-is-cephosd yes
| Ok. |
| --- |
Customised Config
OCI will copy anything from:
/var/lib/oci/clusters/<CLUSTER>/<HOSTNAME>/oci-in-target/
to systems it builds. This can be used to override anything OCI has set up already.
Trusting Extra Repos
for i in {1..3}; do
mkdir -p /var/lib/oci/clusters/os/os-controller-${i}.os.tunstall.in/oci-in-target/etc/apt/trusted.gpg.d
mkdir -p /var/lib/oci/clusters/os/os-controller-${i}.os.tunstall.in/oci-in-target/etc/apt/sources.list.d
cp /etc/apt/trusted.gpg.d/repo.tunstall.in.gpg /var/lib/oci/clusters/os/os-controller-${i}.os.tunstall.in/oci-in-target/etc/apt/trusted.gpg.d/
cp /etc/apt/sources.list.d/repo.tunstall.in.sources /var/lib/oci/clusters/os/os-controller-${i}.os.tunstall.in/oci-in-target/etc/apt/sources.list.d/
cp /root/.vimrc /var/lib/oci/clusters/os/os-controller-${i}.os.tunstall.in/oci-in-target/root/
ls -l /var/lib/oci/clusters/os/os-controller-${i}.os.tunstall.in/oci-in-target/etc/apt/*
done
Non-Standard Network Configuration
While the normal OCI network configuration is fairly flexible, there are situations where this isn’t quite enough. Dropping standard Debian network config into
/var/lib/oci/clusters/<CLUSTER>/<HOSTNAME>/oci-in-target/etc/network
does the job nicely:
# cp -r os-controller-1.os.tunstall.in/oci-in-target/etc/network os-controller-2.os.tunstall.in/oci-in-target/
# sed -i 's/81/82/g' os-controller-2.os.tunstall.in/oci-in-target/etc/network/interfaces.d/*
# cat os-controller-2.os.tunstall.in/oci-in-target/etc/network/interfaces.d/*
# Interface bond-mgmt
auto bond-mgmt
allow-br-ex bond-mgmt
iface bond-mgmt inet manual
# Address for Openstack API internal communication. PVID 1 on the NICs so needs to be on the bond interface.
ovs_bridge br-ex
ovs_type OVSBond
ovs_bonds eno1 eno2
ovs_options bond_mode=balance-tcp lacp=active other_config:lacp-time=fast
up ifup eno1
up ifup eno2
# MTU 9000 in os
# up ovs-vsctl set Interface ${IFACE} mtu_request=9000
# up /bin/sleep 1 ; /bin/ip link set mtu 9000 dev bond-mgmt
# br-ex Interface
auto br-ex
iface br-ex inet static
# Address for Openstack API internal communication. PVID 1 on the NICs so needs to be on the bridge
address 10.66.0.82/24
gateway 10.66.0.1
ovs_type OVSBridge
ovs_ports vlan-internet
#ovs_ports bond-openstack vlan-internet vlan-vmnet os
#ovs_mtu 9000
# NUCs
allow-bond-mgmt eno1
iface eno1 inet manual
ovs_type OVSPort
ovs_bonds bond-mgmt
# up /bin/ip link set mtu 9000 dev eno1
# NUCs have a single interface, OVS needs two for a bond :-)
allow-bond-mgmt eno2
iface eno2 inet manual
pre-up /sbin/ip link add eno2 type dummy
ovs_type OVSPort
ovs_bonds bond-mgmt
# up /bin/ip link set mtu 9000 dev eno2
# Interface vlan-internet
auto vlan-internet
allow-br-ex vlan-internet
iface vlan-internet inet manual
ovs_type OVSIntPort
ovs_bridge br-ex
ovs_options tag=4093
ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif
#ovs_mtu 9000
# address 10.66.100.82/24
# network 10.66.100.0/24
# NUCs
# vlan-raw-device bond-mgmt
# os
# vlan-raw-device bond-openstack
# Interface vlan-vmnet
# Openstack VM networks live here
auto vlan-vmnet
allow-br-ex vlan-vmnet
iface vlan-vmnet inet static
ovs_type OVSIntPort
ovs_bridge br-ex
ovs_options tag=100
ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif
ovs_mtu 9000 # MTU 9000 in os
address 10.66.100.82/24
network 10.66.100.0
broadcast 10.66.100.255
TLS Cert Config
API hostname: os-api.os.tunstall.in
ocicli cluster-set os --self-signed-api-cert no
Weird Things
High load during initial puppet run
During the initial (first) puppet run on NUCs (4 core, 16 GiB) I found that the primary host got into a high-load (~25) I/O bound state where many services were writing lots of errors to the logs which caused package installations to time out. Stopping the following services helped a lot:
barbican-apineutron-apineutron-l3-agentneutron-openvswitch-agentneutron-rpc-serveraodh-notifier
To install on a non-sda device
ocicli machine-display-install-cmd GESY6330001T
oci-install-with-report \
--static-iface type=bond,iface0=eno1,iface1=Array,addr=10.66.0.81/24:10.66.0.1,extra_ovs=yes,ovsbr=br-ex \
--release bullseye \
--debootstrap-url http://10.66.0.98:9999/debian \
--sources.list-mirror http://10.66.0.98:9999/debian \
--security-mirror http://10.66.0.98:9999/security \
--dest-hdd sda \
--no-cloud-init \
--extra-packages gnupg2,haveged,uuid-runtime,iotop,iftop,man-db,curl,less,lsb-release,joe,ssl-cert,most,screen,vim,vim-tiny,tcpd,xfsdump,unzip,tcpdump,ntpstat,ca-certificates,rpcbind,lftp,at,tree,lsof,bind9-host,dnsutils,strace,tmux,nano,bash-completion,openssl,file,net-tools,iproute2,ipmitool,ca-certificates,xfsprogs,e2fsprogs,parted,nmap,mtr-tiny \
--hook-script /usr/bin/openstack-cluster-installer-bodi-hook-script \
--root-ssh-key /root/.ssh/authorized_keys \
--install-nonfree-repo \
--postinstall-packages q-text-as-data,firmware-bnx2,firmware-bnx2x,plymouth,puppet,bridge-utils,grc,ccze,ncdu,lvm2,intel-microcode,smartmontools,kexec-tools,iptables-persistent,lshw,jq,apg,dmsetup,dnsutils,eperl,gettext-base,lynx,quota,acct,netcat,psmisc,chrony,openvswitch-switch --hostname os-controller-1.os.tunstall.in \
--add-serial-getty ttyS1 \
--tty-autologin yes \
--reboot-after-install
Install to RAID
tl;dr Linux disk detection is non-deterministic, especially when HP RAID controllers are involved.
ocicli machine-set CZ3924128B --install-on-raid yes --raid-type 1 --raid-dev0 sdf --raid-dev1 sdj
ocicli machine-install-os CZ3924128B
ocicli machine-set CZ3924128A --install-on-raid yes --raid-type 1 --raid-dev0 sda --raid-dev1 sdb
ocicli machine-install-os CZ3924128A
ocicli machine-set CZ3924128C --install-on-raid yes --raid-type 1 --raid-dev0 sda --raid-dev1 sdc
ocicli machine-install-os CZ3924128C
Old fashioned network interface names
OCI likes to use the ancient eth0 style interface naming. This is inconvenient. Until we patch the image generator not to do this, it can be accomplished post-boot with:
Management Nodes
sed -i 's/bond-slaves.*$/bond-slaves enp196s0f0 enp196s0f1/' /etc/network/interfaces
Compute Nodes
sed -i 's/bond-slaves.*$/bond-slaves enp100s0f0 enp100s0f1/' /etc/network/interfaces
…and then followed up with a reboot
OpenVSwitch Patching between bridges
ovs-vsctl add-port vlan-br-ex patch-vlan-br-ex
ovs-vsctl set interface patch-vlan-br-ex type=patch options:peer=patch-br-ex
ovs-vsctl set interface patch-br-ex type=patch options:peer=patch-vlan-br-ex
Enable LLDP on OpenVSwitch
OpenVSwitch doesn’t automatically enable LLDP, which can be annoying in datacentre environments
TODO: make this happen using /etc/network/interfaces
for i in eno5 eno6 eno7 eno8 ens1f0 ens1f1; do
ip link set dev ${i} up
ovs-vsctl set interface $i lldp:enable=true
done
ovs-vsctl add-aa-mapping br-ex 0 0
ovs-vsctl set AutoAttach . system_name=${HOSTNAME}
Rename the frontend controller API
Openstack cluster DB:
UPDATE endpoint SET url = REPLACE(url, 'os-api.os.tunstall.in', 'controller.os.tunstall.in');
fix /etc/keystone/puppet.conf on controller nodes:
[keystone_authtoken]
region_name=RegionOne
auth_url=auth_url=https://controller.os.tunstall.in/identity # THIS LINE WILL HAVE THE PREVIOUS NAME
username=admin
password=solarwinds123
project_name=admin
interface=public
…and then restart EVERYTHING
systemctl stop placement-api.service octavia-worker.service octavia-housekeeping.service octavia-health-manager.service octavia-api.service nova-spicehtml5proxy.service nova-serialproxy.service nova-scheduler.service nova-novncproxy.service nova-consoleauth.service nova-conductor.service nova-api.service nova-api-metadata.service neutron-rpc-server.service neutron-openvswitch-agent.service neutron-l3-agent.service neutron-api.service heat-engine.service heat-api.service heat-api-cfn.service barbican-api.service aodh-notifier.service aodh-listener.service aodh-evaluator.service aodh-api.service cinder-api.service cinder-scheduler.service
systemctl start placement-api.service octavia-worker.service octavia-housekeeping.service octavia-health-manager.service octavia-api.service nova-spicehtml5proxy.service nova-serialproxy.service nova-scheduler.service nova-novncproxy.service nova-consoleauth.service nova-conductor.service nova-api.service nova-api-metadata.service neutron-rpc-server.service neutron-openvswitch-agent.service neutron-l3-agent.service neutron-api.service heat-engine.service heat-api.service heat-api-cfn.service barbican-api.service aodh-notifier.service aodh-listener.service aodh-evaluator.service aodh-api.service cinder-api.service cinder-scheduler.service
Django fills up apache logs
Django will sometimes spazz out and need you to do:
/usr/share/openstack-dashboard/manage.py collectstatic --noinput --clear; /usr/share/openstack-dashboard/manage.py compress --force
…after apache is restarted. This bug isn’t supposed to exist any more, but whatevs: https://bugzilla.redhat.com/show_bug.cgi?id=1220070
Reset a machine’ state for reinstallation
Sometimes it is necessary to “reset” the state of an already-installed machine. Due to the way the OCI Puppet ENC works you can get into situations where an installation has “succeeded” incorrectly.
UPDATE machines SET status='live' WHERE hostname LIKE 'os-compute-1.%';
UPDATE machines SET puppet_status='notrun' WHERE hostname LIKE 'os-compute-1.%;