Installing OpenStack Ironic on Ubuntu

At CBWS we use OpenStack as our Undercloud on top of which we built our public cloud platform. We already offer virtual machines on a number of generations of hardware.

To ensure we can offer dedicated machines and to make sure they act like their virtual counterparts we want these machines to exist alongside them in a single API.

This is where OpenStack Ironic comes in, Ironic is a bare metal service which integrates with the OpenStack Nova compute. OpenStack Nova builds (virtual) machines on top of a hypervisor, normally libvirt, but it can also call into Ironic to machine dedicated machines.

To get familiar with OpenStack Ironic we are in this blog going to install it on a Ubuntu server.

Preparation

Package installation

To get started we will first install the needed packages using apt:

apt install ironic-api ironic-conductor python3-ironicclient genisoimage

You might notice it will automatically install ipmitool which is used to talk the IPMI protocol to control the hardware of the bare metal machines we will add to Ironic.

Create a database

Now we need to create a database for Ironic, as I already use MySQL I will create a user and database. Make sure to change the password from Sup6r!Pa$s to something else:

CREATE DATABASE `ironic`;
GRANT USAGE ON *.* TO `ironic`@`%` IDENTIFIED BY 'Sup6r!Pa$s';
GRANT ALL PRIVILEGES ON `ironic`.* TO `ironic`@`%`;

Create messaging user

rabbitmqctl add_user ironic Sup6r!Pa$s
rabbitmqctl set_permissions -p / ironic ".*" ".*" ".*"

Create Keystone user

Now, we create a Keystone user for Ironic and grant it the admin role in the service project:

openstack user create --domain Default --project service --password "Sup6r!Pa$s" ironic
openstack role add --project service --user ironic admin

Ironic API and Conductor Configuration

API

Next up open /etc/ironic/ironic.conf, we have to change a few sections, firstly for the database and the messaging system:

[database]
connection=mysql+pymysql://ironic:Sup6r!Pa$s@localhost/ironic?charset=utf8

[DEFAULT]
transport_url = rabbit://ironic:Sup6r!Pa$s@localhost:5672/

As I want to integrate the OpenStack Ironic API with OpenStack Keystone (the identity API) we need to set the following:

[DEFAULT]
auth_strategy=keystone

[keystone_authtoken]
www_authenticate_uri = http://localhost:5000
auth_url = http://localhost:5000
memcached_servers = localhost:11211
auth_type = password
project_domain_name = Default
user_domain_name = Default
project_name = service
username = ironic
password = Sup6r!Pa$s

Now that the Ironic API has configured we can restart it using systemd:

systemctl restart ironic-api

The API should now be running on port 6385:

curl localhost:6385 | jq

The response should look something like:

{
  "name": "OpenStack Ironic API",
  "description": "Ironic is an OpenStack project which enables the provision and management of baremetal machines.",
  "default_version": {
    "id": "v1",
    "links": [
      {
        "href": "http://localhost:6385/v1/",
        "rel": "self"
      }
    ],
    "status": "CURRENT",
    "min_version": "1.1",
    "version": "1.82"
  },
  "versions": [
    {
      "id": "v1",
      "links": [
        {
          "href": "http://localhost:6385/v1/",
          "rel": "self"
        }
      ],
      "status": "CURRENT",
      "min_version": "1.1",
      "version": "1.82"
    }
  ]
}

Conductor

Now we need to configure the conductor service (you may choose to run it on a different machine), open the same config file /etc/ironic/ironic.conf again.

[neutron]
auth_url = http://localhost:5000
memcached_servers = localhost:11211
auth_type = password
project_domain_name = Default
user_domain_name = Default
project_name = service
username = ironic
password = Sup6r!Pa$s

[glance]
auth_url = http://localhost:5000
memcached_servers = localhost:11211
auth_type = password
project_domain_name = Default
user_domain_name = Default
project_name = service
username = ironic
password = Sup6r!Pa$s

[cinder]
auth_url = http://localhost:5000
memcached_servers = localhost:11211
auth_type = password
project_domain_name = Default
user_domain_name = Default
project_name = service
username = ironic
password = Sup6r!Pa$s

[nova]
auth_url = http://localhost:5000
memcached_servers = localhost:11211
auth_type = password
project_domain_name = Default
user_domain_name = Default
project_name = service
username = ironic
password = Sup6r!Pa$s

Hardware types

OpenStack Ironic has interfaces to configure the different parts of the hardware of the bare metal machines it manages.

Firstly there are hardware types, these are at the time of writing:

These hardware types in turn need specific interfaces to:

As IPMI and Redfish are industry standards most of these interfaces have IPMI and Redfish options which should be enough to configure most machines.

In this post I will be using Redfish, to avoid surprises and to know exactly which interfaces are enabled I will set specifically which ones to enable:

[DEFAULT]
enabled_hardware_types = redfish
enabled_boot_interfaces = redfish-virtual-media
enabled_console_interfaces = ipmitool-socat,no-console
enabled_deploy_interfaces = direct
enabled_inspect_interfaces = redfish
enabled_management_interfaces = redfish
enabled_network_interfaces = flat
enabled_power_interfaces = redfish
enabled_raid_interfaces = redfish
enabled_storage_interfaces = cinder,noop
enabled_vendor_interfaces = ipmitool,no-vendor

Uploading ramdisk images

Download the images from https://tarballs.opendev.org/openstack/ironic-python-agent/dib/files/

openstack image create deploy-vmlinuz --public \
  --disk-format raw --container-format bare \
  --file ipa-centos9-stable-2023.1.kernel
openstack image create deploy-initrd --public \
  --disk-format raw --container-format bare \
  --file ipa-centos9-stable-2023.1.initramfs

initrd 9b57c2c7-f783-4e0d-aeb4-3862b0ce4e58 kernel 0b146f4e-add6-4486-b3cd-e2a2cefeed43

[conductor]
deploy_kernel_by_arch = x86_64:0b146f4e-add6-4486-b3cd-e2a2cefeed43
deploy_ramdisk_by_arch = x86_64:9b57c2c7-f783-4e0d-aeb4-3862b0ce4e58

Creating an ESP image

For UEFI based machines it is necessary to create an ESP system image which will be embedded into the boot ISO created by Ironic alongside the ramdisk and kernel.

You can use the following script to create the ESP image on a Ubuntu machine. This will take the signed shim and GRUB EFI from Ubuntu and build a esp.img out of it in the working directory.

#!/bin/bash

DEST=esp.img
GRUB2=/usr/lib/grub/x86_64-efi-signed/grubx64.efi.signed
SHIM=/usr/lib/shim/shimx64.efi.signed

dd if=/dev/zero of=$DEST bs=4096 count=1024
mkfs.msdos -F 12 -n ESP_IMAGE $DEST

# The following commands require mtools to be installed
mmd -i $DEST EFI EFI/BOOT
mcopy -i $DEST -v $SHIM ::EFI/BOOT/BOOTX64.efi
mcopy -i $DEST -v $GRUB2 ::EFI/BOOT/GRUBX64.efi
mdir -i $DEST ::EFI/BOOT

If you get dropped to the GRUB prompt you can also (while testing if it can boot) load the default config file with configfile (cd0)/EFI/BOOT/grub.cfg.

As we used the Ubuntu GRUB EFI in this example it will be compiled to read the GRUB configuration file from /EFI/ubuntu/grub.cfg, as such we have to configure Ironic to write the config to that path in the generated boot ISO. We can do that by setting the grub_config_path option (this way we don’t need to manually load the correct GRUB config like done above):

[DEFAULT]
grub_config_path = EFI/ubuntu/grub.cfg

Next restart the Ironic conductor:

systemctl restart ironic-conductor

Upload the newly created ESP image to the OpenStack Glance API

openstack image create deploy-esp --public \
  --disk-format raw --container-format bare \
  --file esp.img

Then configure it in /etc/ironic/ironic.conf:

[conductor]
bootloader = f424a946-f91a-40a3-b84a-e44b1f1d43dc

Making images accessible

add-user www-data ironic

Configuring OpenStack

To make Ironic accessible from the OpenStack cloud we need to add it to the Keystone database. We can do this with the following commands:

openstack service create --name ironic --description "Ironic baremetal provisioning service" baremetal
openstack endpoint create --region nl-ein \
    baremetal admin http://$IRONIC_NODE:6385
openstack endpoint create --region nl-ein \
    baremetal public http://$IRONIC_NODE:6385
openstack endpoint create --region nl-ein \
    baremetal internal http://$IRONIC_NODE:6385

With this you should now be able to talk to Ironic with the OpenStack CLI client:

openstack baremetal node list

Of course as we haven’t added any nodes (machines) we won’t see anything, but you shouldn’t get an error.

Horizon dashboard plugin

Beside the CLI it is also nice to be able to manage Ironic through the OpenStack Horizon dashboard, for this you can install the plugin with the python3-ironic-ui package:

apt install python3-ironic-ui

To use it restart the dashboard by restarting Apache:

systemctl restart apache2

To find the plugin go to the dashboard and go to an admin project, then go to Admin -> System -> Ironic Bare Metal Provisioning. Here you can now add Ironic nodes (bare metal machines).

Adding a machine

Now OpenStack Ironic has been configured we can try to add a machine to it. As we only enabled the Redfish driver we can double check that with:

baremetal driver list

It should only return redfish and it should show it as being active on the server you installed the Ironic conductor on.

To see what properties we can set on the nodes (machine) we add to Ironic we can run the following command:

baremetal driver property list redfish

To start most important properties will be redfish_address, redfish_username and redfish_password. As I will be using a Dell PowerEdge R630 and Dell by default uses a self signed certificate I will also set redfish_verify_ca to False. The Redfish API on Dell is available at https://<iDRAC IP>/redfish/v1, this is what we will be using as redfish_address. I also add a user for my OpenStack installation to the iDRAC.

Creating a node

Depending on the hardware type we will have to use a specific driver, with the Dell R630 I have to use the iDRAC specific Redfish driver.

iDRAC specific

For the Sushy Redfish library to work with Dell servers we also need to install the python3-sushy-oem-idrac package which will add support for Dell OEM resources. This will resolve the following error:

sushy.exceptions.ExtensionError: Sushy Extension Error: No extensions found for "manager" under namespace "sushy.resources.manager.oems"

To add the node we will run:

export IRONIC_API_VERSION=1.11
export OS_BAREMETAL_API_VERSION=1.11

baremetal node create --driver idrac \
  --bios-interface idrac-redfish \
  --inspect-interface idrac-redfish \  
  --management-interface idrac-redfish \  
  --power-interface idrac-redfish \  
  --raid-interface no-raid \  
  --vendor-interface no-vendor \
  --name <node name> \
  --resource-class r630 \
  --driver-info redfish_username=openstack \
  --driver-info redfish_password=<iDRAC password> \
  --driver-info redfish_address=https://<iDRAC IP>/redfish/v1 \
  --driver-info redfish_verify_ca=False
  
baremetal port create <NIC MAC address> \
  --node e3360e98-84ec-49e7-87f0-f2291a873e3a

Generic Redfish

export IRONIC_API_VERSION=1.11
export OS_BAREMETAL_API_VERSION=1.11

baremetal node create --driver redfish --name <node name> --resource-class r630 --driver-info redfish_username=openstack --driver-info redfish_password=<iDRAC password> --driver-info redfish_address=https://<iDRAC IP>/redfish/v1 --driver-info redfish_verify_ca=False
baremetal port create <NIC MAC address> --node e3360e98-84ec-49e7-87f0-f2291a873e3a

Validating the Ironic driver options

To check if the provided options are valid you can run the validate command:

baremetal node validate <uuid>

It is normal for boot and deploy to say they are missing deploy_kernel and deploy_ramdisk, these will be provided by OpenStack Nova later. Should check console, network and rescue errors

Next step is to mark the node as manageable so OpenStack Ironic knows it is ready to control it, to do this run:

baremetal node manage <uuid>

To see the status you can check the logs:

journalctl -f -u ironic-conductor.service

You should see something like:

ironic-conductor[3801960]: 2025-02-01 11:34:41.654 3801960 INFO ironic.conductor.task_manager [None req-b59e8cc1-c687-4b01-9f0c-a1444e77d339 4e2d92156f184de58aa03e9b29a453a9 bd32ddd8f16243ba9dbeb4befdc1b762 - - default default] Node e3360e98-84ec-49e7-87f0-f2291a873e3a moved to provision state "verifying" from state "enroll"; target provision state is "manageable"
ironic-conductor[3801960]: 2025-02-01 11:35:03.193 3801960 INFO ironic.conductor.utils [None req-b59e8cc1-c687-4b01-9f0c-a1444e77d339 4e2d92156f184de58aa03e9b29a453a9 bd32ddd8f16243ba9dbeb4befdc1b762 - - default default] Detected vendor Dell Inc. for node e3360e98-84ec-49e7-87f0-f2291a873e3a
ironic-conductor[3801960]: 2025-02-01 11:35:11.965 3801960 INFO ironic.conductor.utils [None req-b59e8cc1-c687-4b01-9f0c-a1444e77d339 4e2d92156f184de58aa03e9b29a453a9 bd32ddd8f16243ba9dbeb4befdc1b762 - - default default] Updated boot_mode uefi, secure_boot Falsefor node e3360e98-84ec-49e7-87f0-f2291a873e3a
ironic-conductor[3801960]: 2025-02-01 11:35:11.988 3801960 INFO ironic.conductor.task_manager [None req-b59e8cc1-c687-4b01-9f0c-a1444e77d339 4e2d92156f184de58aa03e9b29a453a9 bd32ddd8f16243ba9dbeb4befdc1b762 - - default default] Node e3360e98-84ec-49e7-87f0-f2291a873e3a moved to provision state "manageable" from state "verifying"; target provision state is "None"
ironic-conductor[3801960]: 2025-02-01 11:35:11.990 3801960 INFO ironic.conductor.verify [None req-b59e8cc1-c687-4b01-9f0c-a1444e77d339 4e2d92156f184de58aa03e9b29a453a9 bd32ddd8f16243ba9dbeb4befdc1b762 - - default default] Successfully verified node e3360e98-84ec-49e7-87f0-f2291a873e3a

Here you can see it did detect the machine vendor as Dell Inc and marked it as manageable. This can also be seen with baremetal node show under the properties field.

Inspecting the hardware

OpenStack Ironic can inspect nodes, this will lookup the hardware that the machine has and store it in the database. It can do either a basic *out-of-band inspection via iDRAC using Redfish or IPMI, or it can do in-band inspection by booting up the machine using the iDRAC’s virtual media and running inspection scripts.

In Configuring Ironic we have set the inspect interface to Redfish which will do an out-of-band inspection using the iDRAC. To trigger inspection run:

baremetal node inspect <uuid>

Once it is done you should see at least memory_mb, cpus, cpu_arch, local_gb as properties on the node.

Making the machine available

To make the node available for allocation run (note: this will wipe all disks on the machine):

baremetal node provide <uuid>

This will first turn on the machine, it will boot it using Redfish virtual media with an ISO image generated by Ironic based on the above made ramdisk images. It will then start wiping the machine, making it ready for the next end-user to pick it up.

You can see the status of the cleaning process with:

baremetal node show <uuid>

Here the provisioning_state of clean wait means the node is cleaning and will report back to Ironic one it is ready. Once cleaning has finished the provisioning_state changes to available. At this point it will be available for allocation in the next steps.

Integrating with OpenStack

Until this point we have only used Ironic directly, not together with any of the other OpenStack services like Nova to create new machines using the same API as virtual machines. This is where the next section comes in.

First we need to configure the integration Neutron which will ensure physical ports in Ironic machines get synchronized with Neutron. After that we will configure the Nova compute service to create new machines based on Ironic hardware.

Integrating with Neutron

To integrate the flat network interface of Ironic with OpenStack Neutron we need to install the Neutron Ironic agent, otherwise known as networking-baremetal. We can install it on Ubuntu with:

apt install ironic-neutron-agent

And then configure the Neutron server ML2 configuration /etc/neutron/plugins/ml2/ml2_conf.ini by adding the baremetal mechanism driver:

[ml2]
mechanism_drivers = ovs,baremetal

Next configure Neutron to use it’s credentials to talk to Ironic by changing /etc/neutron/neutron.conf:

[ironic]
# Ironic authentication type
auth_type=password

# Keystone API endpoint
auth_url=http://IDENTITY_IP:5000/v3

# Ironic keystone project name
project_name=service

# Ironic keystone admin name
username=ironic

# Ironic keystone admin password
password=IRONIC_PASSWORD

# Ironic keystone project domain
# or set project_domain_id
project_domain_name=Default

# Ironic keystone user domain
# or set user_domain_id
user_domain_name=Default

Then restart the Neutron agent’s systemd service and Neutron server:

systemctl restart ironic-neutron-agent neutron-server

Integrating with Nova

To integrate OpenStack Ironic with OpenStack Nova we have to first install a few packages:

apt install nova-compute-ironic nova-compute

The first being the Ironic integration with Nova Compute, the second being Nova compute itself. OpenStack Nova compute normally talks to the hypervisor but in this case talks to the OpenStack Ironic API to manage instances.

Next we need to set the following options in the /etc/nova/nova.conf file:

[default]

# Defines which driver to use for controlling virtualization.
# Enable the ironic virt driver for this compute instance.
compute_driver=ironic.IronicDriver

# Amount of memory in MB to reserve for the host so that it is always
# available to host processes.
# It is impossible to reserve any memory on bare metal nodes, so set
# this to zero.
reserved_host_memory_mb=0

[filter_scheduler]

# Enables querying of individual hosts for instance information.
# Not possible for bare metal nodes, so set it to False.
track_instance_changes=False

[scheduler]

# This value controls how often (in seconds) the scheduler should
# attempt to discover new hosts that have been added to cells.
# If negative (the default), no automatic discovery will occur.
# As each bare metal node is represented by a separate host, it has
# to be discovered before the Compute service can deploy on it.
# The value here has to be carefully chosen based on a compromise
# between the enrollment speed and the load on the Compute scheduler.
# The recommended value of 2 minutes matches how often the Compute
# service polls the Bare Metal service for node information.
discover_hosts_in_cells_interval=120

You also need to configure the Nova credentials so it can talk to the Ironic API using it’s Keystone user:

[ironic]

# Ironic authentication type
auth_type=password

# Keystone API endpoint
auth_url=http://IDENTITY_IP:5000/v3

# Ironic keystone project name
project_name=service

# Ironic keystone admin name
username=ironic

# Ironic keystone admin password
password=IRONIC_PASSWORD

# Ironic keystone project domain
# or set project_domain_id
project_domain_name=Default

# Ironic keystone user domain
# or set user_domain_id
user_domain_name=Default

To now discover the new Nova compute service run the following command:

nova-manage cell_v2 discover_hosts --by-service

With this it should now be visible in the Nova compute service list:

openstack compute service list --long

Creating a flavor

Now we need to create a OpenStack Nova flavor which will match the r630 resource class we assigned to the Ironic node. In Nova this resource class will be expressed as the CUSTOM_R630 resource. To create a flavor use the following:

openstack flavor create --ram 196608 --disk 892 --vcpus 32 --description "Dedicated machine R630" g2-dedicated --property resources:CUSTOM_R630=1

>> Home