Powering Up a POD
This guide describes how to power up a previously installed CORD POD that has been powered down (cleanly or otherwise). The end goal of the power up procedure is a fully functioning CORD POD.
Boot the Head Node
- Physical POD: Power on the head node
- CiaB: Bring up the head1 VM:
cd ~/cord/build; VAGRANT_CWD=~/cord/build/scenarios/cord vagrant up head1 --provider libvirt
Check the Head Node Services
fabricinterfaces are up and have IP addresses
Verify that MAAS UI is running and accessible:
- Physical POD:
Troubleshooting: MAAS UI not available on CiaB. If you are running a CiaB and there is no webserver on port 8080, it might be necessary to refresh port forwarding to the prod VM. Run
ps ax|grep 8080and look for an SSH command (will look something like this):
31353 pts/5 S 0:00 ssh -o User=vagrant -o Port=22 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ForwardX11=no -o IdentityFile="/users/acb/cord/build/scenarios/cord/.vagrant/machines/head1/libvirt/private_key" -L *:8080:192.168.121.14:80 -N 192.168.121.14
A workaround is to kill this process, and then copy and paste the command above into another window on the CiaB server to set up a new SSH port forwarding connection.
- Physical POD:
Verify that the following Docker containers are running: mavenrepo, switchq, automation, provisioner, generator, harvester, storage, allocator, registry
sudo lxc listto ensure that juju lxc containers are running. If any are stopped, use
sudo lxc start <name>to restart them.
Verify that the following OpenStack commands work:
$ keystone user-list
$ nova list --all-tenants
$ neutron net-list
Troubleshooting: OpenStack commands give SSL error. Sometimes Keystone starts up in a strange state and OpenStack commands will fail with various SSL errors. To fix this, it is often sufficient to run:
ssh ubuntu@keystone sudo service apache2 restart
Power on Leaf and Spine Switches
- Physical POD: power on the switches.
- CiaB: nothing to do; CiaB using OvS switches that come up when the CiaB server is booted.
Check the Switches
- Physical POD: On the head node:
- Get switch IPs by running: cord prov list
- Verify that ping works for all switch IPs
- CiaB: run
sudo ovs-vsctl showon the CiaB server; you should see
Boot the Compute Nodes
- Physical POD: Log into the MAAS UI and power on the compute node.
- CiaB: Log into the MAAS UI and power on the compute node.
Check the Compute Nodes
Once the compute nodes are up:
- Login to the head node
Verify that nova service-list shows the compute node as “up”.
It may take a few minutes until the node's status is updated in Nova.
Verify that you can log into the compute nodes from the head node as the ubuntu user
Verify that XOS UI is running and accessible:
- Physical POD:
If it's not working, try restarting XOS (replace
rcord with the name of your
cd /opt/cord_profile; docker-compose -p rcord restart
Verify that VTN is initialized correctly:
- Make sure the compute nodes have COMPLETE status.
- Prior to rebooting existing OpenStack VMs:
- Make sure some ports show up
- If not, try this:
onos> cordvtn-sync-neutron-states <keystone-url> admin admin <password>
onos> cordvtn-sync-xos-states <xos-url> firstname.lastname@example.org <password>
Boot OpenStack VMs
To bring up OpenStack VMs that were running before the POD was shut down:
- Get list of VM IDs:
nova list --all-tenants
- For each VM:
$ nova start <vm-id>
$ nova console-log <vm-id>
- Inspect the console log to make sure that the network interfaces get IP addresses.
To restart a vSG inside the vSG VM:
- SSH to the vSG VM
sudo rm /root/network_is_setup
- Save the vSG Tenant in the XOS UI
- Once the synchronizer has re-run, make sure you can ping 188.8.131.52 from inside the vSG container
sudo docker exec -ti vcpe-222-111 ping 184.108.40.206