Installing a user-provisioned cluster on bare metal - Installing on bare metal | Installing (2024)

Note that RHCOS is based on Red Hat Enterprise Linux (RHEL) 9.2 and inherits all of its hardware certifications and requirements.See Red Hat Enterprise Linux technology capabilities and limits.

Additional resources

• Optimizing storage

Additional resources

• See Configuring a three-node cluster for details about deploying three-node clusters in bare metal environments.

• See Approving the certificate signing requests for your machines for more information about approving cluster certificate signing requests after installation.

Additional resources

• See Installing RHCOS and starting the OpenShift Container Platform bootstrap process for details on setting the disk.EnableUUID parameter’s value to TRUE on VMware vSphere for user-provisioned infrastructure.

Setting the cluster node hostnames through DHCP

On Red Hat Enterprise Linux CoreOS (RHCOS) machines, the hostname is set through NetworkManager. By default, the machines obtain their hostname through DHCP. If the hostname is not provided by DHCP, set statically through kernel arguments, or another method, it is obtained through a reverse DNS lookup. Reverse DNS lookup occurs after the network has been initialized on a node and can take time to resolve. Other system services can start prior to this and detect the hostname as localhost or similar. You can avoid this by using DHCP to provide the hostname for each cluster node.

Additionally, setting the hostnames through DHCP can bypass any manual DNS record name configuration errors in environments that have a DNS split-horizon implementation.

Network connectivity requirements

You must configure the network connectivity between machines to allow OpenShift Container Platform clustercomponents to communicate. Each machine must be able to resolve the hostnamesof all other machines in the cluster.

This section provides details about the ports that are required.

NTP configuration for user-provisioned infrastructure

OpenShift Container Platform clusters are configured to use a public Network Time Protocol (NTP) server by default. If you want to use a local enterprise NTP server, or if your cluster is being deployed in a disconnected network, you can configure the cluster to use a specific time server. For more information, see the documentation for Configuring chrony time service.

If a DHCP server provides NTP server information, the chrony time service on the Red Hat Enterprise Linux CoreOS (RHCOS) machines read the information and can sync the clock with the NTP servers.

• The Kubernetes API

• The OpenShift Container Platform application wildcard

• The bootstrap, control plane, and compute machines

Example DNS configuration for user-provisioned clusters

This section provides A and PTR record configuration samples that meet the DNS requirements for deploying OpenShift Container Platform on user-provisioned infrastructure. The samples are not meant to provide advice for choosing one DNS solution over another.

In the examples, the cluster name is ocp4 and the base domain is example.com.

$TTL 1W@INSOAns1.example.com.root (2019070700; serial3H; refresh (3 hours)30M; retry (30 minutes)2W; expiry (2 weeks)1W ); minimum (1 week)INNSns1.example.com.INMX 10smtp.example.com.;;ns1.example.com.INA192.168.1.5smtp.example.com.INA192.168.1.5;helper.example.com.INA192.168.1.5helper.ocp4.example.com.INA192.168.1.5;api.ocp4.example.com.INA192.168.1.5 (1)api-int.ocp4.example.com.INA192.168.1.5 (2);*.apps.ocp4.example.com.INA192.168.1.5 (3);bootstrap.ocp4.example.com.INA192.168.1.96 (4);control-plane0.ocp4.example.com.INA192.168.1.97 (5)control-plane1.ocp4.example.com.INA192.168.1.98 (5)control-plane2.ocp4.example.com.INA192.168.1.99 (5);compute0.ocp4.example.com.INA192.168.1.11 (6)compute1.ocp4.example.com.INA192.168.1.7 (6);;EOF

$TTL 1W@INSOAns1.example.com.root (2019070700; serial3H; refresh (3 hours)30M; retry (30 minutes)2W; expiry (2 weeks)1W ); minimum (1 week)INNSns1.example.com.;5.1.168.192.in-addr.arpa.INPTRapi.ocp4.example.com. (1)5.1.168.192.in-addr.arpa.INPTRapi-int.ocp4.example.com. (2);96.1.168.192.in-addr.arpa.INPTRbootstrap.ocp4.example.com. (3);97.1.168.192.in-addr.arpa.INPTRcontrol-plane0.ocp4.example.com. (4)98.1.168.192.in-addr.arpa.INPTRcontrol-plane1.ocp4.example.com. (4)99.1.168.192.in-addr.arpa.INPTRcontrol-plane2.ocp4.example.com. (4);11.1.168.192.in-addr.arpa.INPTRcompute0.ocp4.example.com. (5)7.1.168.192.in-addr.arpa.INPTRcompute1.ocp4.example.com. (5);;EOF

1. API load balancer: Provides a common endpoint for users, both human and machine, to interact with and configure the platform. Configure the following conditions:

   • Layer 4 load balancing only. This can be referred to as Raw TCP or SSL Passthrough mode.

   • A stateless load balancing algorithm. The options vary based on the load balancer implementation.

   Do not configure session persistence for an API load balancer. Configuring session persistence for a Kubernetes API server might cause performance issues from excess application traffic for your OpenShift Container Platform cluster and the Kubernetes API that runs inside the cluster.

   Configure the following ports on both the front and back of the load balancers:

   Table 7. API load balancer

   Port	Back-end machines (pool members)	Internal	External	Description
   6443	Bootstrap and control plane. You remove the bootstrap machine from the loadbalancer after the bootstrap machine initializes the cluster control plane. Youmust configure the /readyz endpoint for the API server health check probe.	X	X	Kubernetes API server
   22623	Bootstrap and control plane. You remove the bootstrap machine from the loadbalancer after the bootstrap machine initializes the cluster control plane.	X		Machine config server

   The load balancer must be configured to take a maximum of 30 seconds from thetime the API server turns off the /readyz endpoint to the removal of the APIserver instance from the pool. Within the time frame after /readyz returns anerror or becomes healthy, the endpoint must have been removed or added. Probingevery 5 or 10 seconds, with two successful requests to become healthy and threeto become unhealthy, are well-tested values.

2. Application Ingress load balancer: Provides an ingress point for application traffic flowing in from outside the cluster. A working configuration for the Ingress router is required for an OpenShift Container Platform cluster.

   Configure the following conditions:

   • Layer 4 load balancing only. This can be referred to as Raw TCP or SSL Passthrough mode.

   • A connection-based or session-based persistence is recommended, based on the options available and types of applications that will be hosted on the platform.

   If the true IP address of the client can be seen by the application Ingress load balancer, enabling source IP-based session persistence can improve performance for applications that use end-to-end TLS encryption.

   Configure the following ports on both the front and back of the load balancers:

   Table 8. Application Ingress load balancer

   Port	Back-end machines (pool members)	Internal	External	Description
   443	The machines that run the Ingress Controller pods, compute, or worker, by default.	X	X	HTTPS traffic
   80	The machines that run the Ingress Controller pods, compute, or worker, by default.	X	X	HTTP traffic

   If you are deploying a three-node cluster with zero compute nodes, the Ingress Controller pods run on the control plane nodes. In three-node cluster deployments, you must configure your application Ingress load balancer to route HTTP and HTTPS traffic to the control plane nodes.

Example load balancer configuration for user-provisioned clusters

This section provides an example API and application Ingress load balancer configuration that meets the load balancing requirements for user-provisioned clusters. The sample is an /etc/haproxy/haproxy.cfg configuration for an HAProxy load balancer. The example is not meant to provide advice for choosing one load balancing solution over another.

In the example, the same load balancer is used for the Kubernetes API and application ingress traffic. In production scenarios, you can deploy the API and application ingress load balancers separately so that you can scale the load balancer infrastructure for each in isolation.

global log 127.0.0.1 local2 pidfile /var/run/haproxy.pid maxconn 4000 daemondefaults mode http log global option dontlognull option http-server-close option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000listen api-server-6443 (1) bind *:6443 mode tcp option httpchk GET /readyz HTTP/1.0 option log-health-checks balance roundrobin server bootstrap bootstrap.ocp4.example.com:6443 verify none check check-ssl inter 10s fall 2 rise 3 backup (2) server master0 master0.ocp4.example.com:6443 weight 1 verify none check check-ssl inter 10s fall 2 rise 3 server master1 master1.ocp4.example.com:6443 weight 1 verify none check check-ssl inter 10s fall 2 rise 3 server master2 master2.ocp4.example.com:6443 weight 1 verify none check check-ssl inter 10s fall 2 rise 3listen machine-config-server-22623 (3) bind *:22623 mode tcp server bootstrap bootstrap.ocp4.example.com:22623 check inter 1s backup (2) server master0 master0.ocp4.example.com:22623 check inter 1s server master1 master1.ocp4.example.com:22623 check inter 1s server master2 master2.ocp4.example.com:22623 check inter 1slisten ingress-router-443 (4) bind *:443 mode tcp balance source server compute0 compute0.ocp4.example.com:443 check inter 1s server compute1 compute1.ocp4.example.com:443 check inter 1slisten ingress-router-80 (5) bind *:80 mode tcp balance source server compute0 compute0.ocp4.example.com:80 check inter 1s server compute1 compute1.ocp4.example.com:80 check inter 1s

• You want to make postinstallation changes to the bridge, such as changing the Open vSwitch (OVS) or OVN-Kubernetes br-ex bridge network. The configure-ovs.sh shell script does not support making postinstallation changes to the bridge.

• You want to deploy the bridge on a different interface than the interface available on a host or server IP address.

• You want to make advanced configurations to the bridge that are not possible with the configure-ovs.sh shell script. Using the script for these configurations might result in the bridge failing to connect multiple network interfaces and facilitating data forwarding between the interfaces.

Prerequisites

• Optional: You have installed the nmstate API so that you can validate the NMState configuration.

Procedure

1. Create a NMState configuration file that has decoded base64 information for your customized br-ex bridge network:

   Example of an NMState configuration for a customized br-ex bridge network

   interfaces:- name: enp2s0 (1) type: ethernet (2) state: up (3) ipv4: enabled: false (4) ipv6: enabled: false- name: br-ex type: ovs-bridge state: up ipv4: enabled: false dhcp: false ipv6: enabled: false dhcp: false bridge: port: - name: enp2s0 (5) - name: br-ex- name: br-ex type: ovs-interface state: up copy-mac-from: enp2s0 ipv4: enabled: true dhcp: true ipv6: enabled: false dhcp: false# ...

   1	Name of the interface.
   2	The type of ethernet.
   3	The requested state for the interface after creation.
   4	Disables IPv4 and IPv6 in this example.
   5	The node NIC to which the bridge attaches.

2. Use the cat command to base64-encode the contents of the NMState configuration:

   $ cat <nmstate_configuration>.yaml | base64 (1)

   1	Replace <nmstate_configuration> with the name of your NMState resource YAML file.

3. Create a MachineConfig manifest file and define a customized br-ex bridge network configuration analogous to the following example:

   apiVersion: machineconfiguration.openshift.io/v1kind: MachineConfigmetadata: labels: machineconfiguration.openshift.io/role: worker (1) name: 10-br-ex-worker (2)spec: config: ignition: version: 3.2.0 storage: files: - contents: source: data:text/plain;charset=utf-8;base64,<base64_encoded_nmstate_configuration> (3) mode: 0644 overwrite: true path: /etc/nmstate/openshift/cluster.yml# ...

   1	For each node in your cluster, specify the hostname path to your node and the base-64 encoded Ignition configuration file data for the machine type. If you have a single global configuration specified in an /etc/nmstate/openshift/cluster.yml configuration file that you want to apply to all nodes in your cluster, you do not need to specify the hostname path for each node. The worker role is the default role for nodes in your cluster. The .yaml extension does not work when specifying the hostname path for each node or all nodes in the MachineConfig manifest file.
   2	The name of the policy.
   3	Writes the encoded base64 information to the specified path.

Prerequisites

• You created a MachineConfig manifest object that includes a customized br-ex bridge configuration.

Procedure

1. Edit the MachineConfig CR by entering the following command:

   $ oc edit mc <machineconfig_custom_resource_name>

2. Add each compute node configuration to the CR, so that the CR can manage roles for each defined compute node in your cluster.

3. Create a Secret object named extraworker-secret that has a minimal static IP configuration.

4. Apply the extraworker-secret secret to each node in your cluster by entering the following command. This step provides each compute node access to the Ignition config file.

   $ oc apply -f ./extraworker-secret.yaml

5. Create a BareMetalHost resource and specify the network secret in the preprovisioningNetworkDataName parameter:

   Example BareMetalHost resource with an attached network secret

   apiVersion: metal3.io/v1alpha1kind: BareMetalHostspec:# ... preprovisioningNetworkDataName: ostest-extraworker-0-network-config-secret# ...

6. To manage the BareMetalHost object within the openshift-machine-api namespace of your cluster, change to the namespace by entering the following command:

   $ oc project openshift-machine-api

7. Get the machine sets:

   $ oc get machinesets

8. Scale each machine set by entering the following command. You must run this command for each machine set.

   $ oc scale machineset <machineset_name> --replicas=<n> (1)

   1	Where <machineset_name> is the name of the machine set and <n> is the number of compute nodes.

Additional resources

• See Load balancing requirements for user-provisioned infrastructure for more information on the API and application ingress load balancing requirements.

• See Cluster capabilities for more information on enabling cluster capabilities that were disabled before installation.

• See Optional cluster capabilities in OpenShift Container Platform 4.18 for more information about the features provided by each capability.

Prerequisites

• You have an existing install-config.yaml file.

• You reviewed the sites that your cluster requires access to and determined whether any of them need to bypass the proxy. By default, all cluster egress traffic is proxied, including calls to hosting cloud provider APIs. You added sites to the Proxy object’s spec.noProxy field to bypass the proxy if necessary.

  The Proxy object status.noProxy field is populated with the values of the networking.machineNetwork[].cidr, networking.clusterNetwork[].cidr, and networking.serviceNetwork[] fields from your installation configuration. For installations on Amazon Web Services (AWS), Google Cloud Platform (GCP), Microsoft Azure, and Red Hat OpenStack Platform (RHOSP), the Proxy object status.noProxy field is also populated with the instance metadata endpoint (169.254.169.254).

Procedure

1. Edit your install-config.yaml file and add the proxy settings. For example:

   apiVersion: v1baseDomain: my.domain.comproxy: httpProxy: http://<username>:<pswd>@<ip>:<port> (1) httpsProxy: https://<username>:<pswd>@<ip>:<port> (2) noProxy: example.com (3)additionalTrustBundle: | (4) -----BEGIN CERTIFICATE----- <MY_TRUSTED_CA_CERT> -----END CERTIFICATE-----additionalTrustBundlePolicy: <policy_to_add_additionalTrustBundle> (5)

   1	A proxy URL to use for creating HTTP connections outside the cluster. TheURL scheme must be http.
   2	A proxy URL to use for creating HTTPS connections outside the cluster.
   3	A comma-separated list of destination domain names, IP addresses, or other network CIDRs to exclude from proxying. Preface a domain with . to match subdomains only. For example, .y.com matches x.y.com, but not y.com. Use * to bypass the proxy for all destinations.
   4	If provided, the installation program generates a config map that is named user-ca-bundle inthe openshift-config namespace that contains one or more additional CAcertificates that are required for proxying HTTPS connections. The Cluster NetworkOperator then creates a trusted-ca-bundle config map that merges these contentswith the Red Hat Enterprise Linux CoreOS (RHCOS) trust bundle, and this config map is referenced in the trustedCA field of the Proxy object. The additionalTrustBundle field is required unlessthe proxy’s identity certificate is signed by an authority from the RHCOS trustbundle.
   5	Optional: The policy to determine the configuration of the Proxy object to reference the user-ca-bundle config map in the trustedCA field. The allowed values are Proxyonly and Always. Use Proxyonly to reference the user-ca-bundle config map only when http/https proxy is configured. Use Always to always reference the user-ca-bundle config map. The default value is Proxyonly.

   The installation program does not support the proxy readinessEndpoints field.

   If the installer times out, restart and then complete the deployment by using the wait-for command of the installer. For example: $ ./openshift-install wait-for install-complete --log-level debug

2. Save the file and reference it when installing OpenShift Container Platform.

Prerequisites

• You have an existing install-config.yaml file.

Procedure

• Ensure that the number of compute replicas is set to 0 in your install-config.yaml file, as shown in the following compute stanza:

  compute:- name: worker platform: {} replicas: 0

  You must set the value of the replicas parameter for the compute machines to 0 when you install OpenShift Container Platform on user-provisioned infrastructure, regardless of the number of compute machines you are deploying. In installer-provisioned installations, the parameter controls the number of compute machines that the cluster creates and manages for you. This does not apply to user-provisioned installations, where the compute machines are deployed manually.

• If you are deploying a three-node cluster with zero compute nodes, the Ingress Controller pods run on the control plane nodes. In three-node cluster deployments, you must configure your application ingress load balancer to route HTTP and HTTPS traffic to the control plane nodes. See the Load balancing requirements for user-provisioned infrastructure section for more information.

• When you create the Kubernetes manifest files in the following procedure, ensure that the mastersSchedulable parameter in the <installation_directory>/manifests/cluster-scheduler-02-config.yml file is set to true. This enables your application workloads to run on the control plane nodes.

• Do not deploy any compute nodes when you create the Red Hat Enterprise Linux CoreOS (RHCOS) machines.

Prerequisites

• You have created the Ignition config files for your cluster.

• You have configured suitable network, DNS and load balancing infrastructure.

• You have an HTTP server that can be accessed from your computer, and from the machines that you create.

• You have reviewed the Advanced RHCOS installation configuration section for different ways to configure features, such as networking and disk partitioning.

Procedure

1. Obtain the SHA512 digest for each of your Ignition config files. For example, you can use the following on a system running Linux to get the SHA512 digest for your bootstrap.ign Ignition config file:

   $ sha512sum <installation_directory>/bootstrap.ign

   The digests are provided to the coreos-installer in a later step to validate the authenticity of the Ignition config files on the cluster nodes.

2. Upload the bootstrap, control plane, and compute node Ignition config files that the installation program created to your HTTP server. Note the URLs of these files.

   You can add or change configuration settings in your Ignition configs before saving them to your HTTP server. If you plan to add more compute machines to your cluster after you finish installation, do not delete these files.

3. From the installation host, validate that the Ignition config files are available on the URLs. The following example gets the Ignition config file for the bootstrap node:

   $ curl -k http://<HTTP_server>/bootstrap.ign (1)

   Example output

    % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0{"ignition":{"version":"3.2.0"},"passwd":{"users":[{"name":"core","sshAuthorizedKeys":["ssh-rsa...

   Replace bootstrap.ign with master.ign or worker.ign in the command to validate that the Ignition config files for the control plane and compute nodes are also available.

4. Although it is possible to obtain the RHCOS images that are required for your preferred method of installing operating system instances from theRHCOS image mirrorpage, the recommended way to obtain the correct version of your RHCOS images are from the output of openshift-install command:

   $ openshift-install coreos print-stream-json | grep '\.iso[^.]'

   Example output

   "location": "<url>/art/storage/releases/rhcos-4.17-aarch64/<release>/aarch64/rhcos-<release>-live.aarch64.iso","location": "<url>/art/storage/releases/rhcos-4.17-ppc64le/<release>/ppc64le/rhcos-<release>-live.ppc64le.iso","location": "<url>/art/storage/releases/rhcos-4.17-s390x/<release>/s390x/rhcos-<release>-live.s390x.iso","location": "<url>/art/storage/releases/rhcos-4.17/<release>/x86_64/rhcos-<release>-live.x86_64.iso",

   The RHCOS images might not change with every release of OpenShift Container Platform. You must download images with the highest version that is less than or equal to the OpenShift Container Platform version that you install. Use the image versions that match your OpenShift Container Platform version if they are available. Use only ISO images for this procedure. RHCOS qcow2 images are not supported for this installation type.

   ISO file names resemble the following example:

   rhcos-<version>-live.<architecture>.iso

5. Use the ISO to start the RHCOS installation. Use one of the following installation options:

   • Burn the ISO image to a disk and boot it directly.

   • Use ISO redirection by using a lights-out management (LOM) interface.

6. Boot the RHCOS ISO image without specifying any options or interrupting the live boot sequence. Wait for the installer to boot into a shell prompt in the RHCOS live environment.

   It is possible to interrupt the RHCOS installation boot process to add kernel arguments. However, for this ISO procedure you should use the coreos-installer command as outlined in the following steps, instead of adding kernel arguments.

7. Run the coreos-installer command and specify the options that meet your installation requirements. At a minimum, you must specify the URL that points to the Ignition config file for the node type, and the device that you are installing to:

   $ sudo coreos-installer install --ignition-url=http://<HTTP_server>/<node_type>.ign <device> --ignition-hash=sha512-<digest> (1) (2)

   1	You must run the coreos-installer command by using sudo, because the core user does not have the required root privileges to perform the installation.
   2	The --ignition-hash option is required when the Ignition config file is obtained through an HTTP URL to validate the authenticity of the Ignition config file on the cluster node. <digest> is the Ignition config file SHA512 digest obtained in a preceding step.

   If you want to provide your Ignition config files through an HTTPS server that uses TLS, you can add the internal certificate authority (CA) to the system trust store before running coreos-installer.

   The following example initializes a bootstrap node installation to the /dev/sda device. The Ignition config file for the bootstrap node is obtained from an HTTP web server with the IP address 192.168.1.2:

   $ sudo coreos-installer install --ignition-url=http://192.168.1.2:80/installation_directory/bootstrap.ign /dev/sda --ignition-hash=sha512-a5a2d43879223273c9b60af66b44202a1d1248fc01cf156c46d4a79f552b6bad47bc8cc78ddf0116e80c59d2ea9e32ba53bc807afbca581aa059311def2c3e3b

8. Monitor the progress of the RHCOS installation on the console of the machine.

   Be sure that the installation is successful on each node before commencing with the OpenShift Container Platform installation. Observing the installation process can also help to determine the cause of RHCOS installation issues that might arise.

9. After RHCOS installs, you must reboot the system. During the system reboot, it applies the Ignition config file that you specified.

10. Check the console output to verify that Ignition ran.

    Example command

    Ignition: ran on 2022/03/14 14:48:33 UTC (this boot)Ignition: user-provided config was applied

11. Continue to create the other machines for your cluster.

    You must create the bootstrap and control plane machines at this time. If the control plane machines are not made schedulable, also create at least two compute machines before you install OpenShift Container Platform.

    If the required network, DNS, and load balancer infrastructure are in place, the OpenShift Container Platform bootstrap process begins automatically after the RHCOS nodes have rebooted.

    RHCOS nodes do not include a default password for the core user. You can access the nodes by running ssh core@<node>.<cluster_name>.<base_domain> as a user with access to the SSH private key that is paired to the public key that you specified in your install_config.yaml file. OpenShift Container Platform 4 cluster nodes running RHCOS are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes by using SSH is not recommended. However, when investigating installation issues, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on a target node, SSH access might be required for debugging or disaster recovery.

Prerequisites

• You have created the Ignition config files for your cluster.

• You have configured suitable network, DNS and load balancing infrastructure.

• You have configured suitable PXE or iPXE infrastructure.

• You have an HTTP server that can be accessed from your computer, and from the machines that you create.

• You have reviewed the Advanced RHCOS installation configuration section for different ways to configure features, such as networking and disk partitioning.

Procedure

1. Upload the bootstrap, control plane, and compute node Ignition config files that theinstallation program created to your HTTP server. Note the URLs of these files.

   You can add or change configuration settings in your Ignition configsbefore saving them to your HTTP server.If you plan to add more compute machines to your cluster after you finishinstallation, do not delete these files.

2. From the installation host, validate that the Ignition config files are available on the URLs. The following example gets the Ignition config file for the bootstrap node:

   $ curl -k http://<HTTP_server>/bootstrap.ign (1)

   Example output

    % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0{"ignition":{"version":"3.2.0"},"passwd":{"users":[{"name":"core","sshAuthorizedKeys":["ssh-rsa...

   Replace bootstrap.ign with master.ign or worker.ign in the command to validatethat the Ignition config files for the control plane and compute nodes are also available.

3. Although it is possible to obtain the RHCOS kernel, initramfs and rootfsfiles that are required for your preferred method of installing operating system instances from theRHCOS image mirrorpage, the recommended way to obtain the correct version of your RHCOS files arefrom the output of openshift-install command:

   $ openshift-install coreos print-stream-json | grep -Eo '"https.*(kernel-|initramfs.|rootfs.)\w+(\.img)?"'

   Example output

   "<url>/art/storage/releases/rhcos-4.17-aarch64/<release>/aarch64/rhcos-<release>-live-kernel-aarch64""<url>/art/storage/releases/rhcos-4.17-aarch64/<release>/aarch64/rhcos-<release>-live-initramfs.aarch64.img""<url>/art/storage/releases/rhcos-4.17-aarch64/<release>/aarch64/rhcos-<release>-live-rootfs.aarch64.img""<url>/art/storage/releases/rhcos-4.17-ppc64le/49.84.202110081256-0/ppc64le/rhcos-<release>-live-kernel-ppc64le""<url>/art/storage/releases/rhcos-4.17-ppc64le/<release>/ppc64le/rhcos-<release>-live-initramfs.ppc64le.img""<url>/art/storage/releases/rhcos-4.17-ppc64le/<release>/ppc64le/rhcos-<release>-live-rootfs.ppc64le.img""<url>/art/storage/releases/rhcos-4.17-s390x/<release>/s390x/rhcos-<release>-live-kernel-s390x""<url>/art/storage/releases/rhcos-4.17-s390x/<release>/s390x/rhcos-<release>-live-initramfs.s390x.img""<url>/art/storage/releases/rhcos-4.17-s390x/<release>/s390x/rhcos-<release>-live-rootfs.s390x.img""<url>/art/storage/releases/rhcos-4.17/<release>/x86_64/rhcos-<release>-live-kernel-x86_64""<url>/art/storage/releases/rhcos-4.17/<release>/x86_64/rhcos-<release>-live-initramfs.x86_64.img""<url>/art/storage/releases/rhcos-4.17/<release>/x86_64/rhcos-<release>-live-rootfs.x86_64.img"

   The RHCOS artifacts might not change with every release of OpenShift Container Platform.You must download images with the highest version that is less than or equalto the OpenShift Container Platform version that you install. Only usethe appropriate kernel, initramfs, and rootfs artifacts described belowfor this procedure.RHCOS QCOW2 images are not supported for this installation type.

   The file names contain the OpenShift Container Platform version number.They resemble the following examples:

   • kernel: rhcos-<version>-live-kernel-<architecture>

   • initramfs: rhcos-<version>-live-initramfs.<architecture>.img

   • rootfs: rhcos-<version>-live-rootfs.<architecture>.img

4. Upload the rootfs, kernel, and initramfs filesto your HTTP server.

   If you plan to add more compute machines to your cluster after you finishinstallation, do not delete these files.

5. Configure the network boot infrastructure so that the machines boot from theirlocal disks after RHCOS is installed on them.

6. Configure PXE or iPXE installation for the RHCOS images and begin the installation.

   Modify one of the following example menu entries for your environment and verifythat the image and Ignition files are properly accessible:

   • For PXE (x86_64):

     DEFAULT pxebootTIMEOUT 20PROMPT 0LABEL pxeboot KERNEL http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> (1) APPEND initrd=http://<HTTP_server>/rhcos-<version>-live-initramfs.<architecture>.img coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/bootstrap.ign (2) (3)

     1	Specify the location of the live kernel file that you uploaded to your HTTPserver.The URL must be HTTP, TFTP, or FTP; HTTPS and NFS are not supported.
     2	If you use multiple NICs, specify a single interface in the ip option.For example, to use DHCP on a NIC that is named eno1, set ip=eno1:dhcp.
     3	Specify the locations of the RHCOS files that you uploaded to yourHTTP server. The initrd parameter value is the location of the initramfs file,the coreos.live.rootfs_url parameter value is the location of therootfs file, and the coreos.inst.ignition_url parameter value is thelocation of the bootstrap Ignition config file.You can also add more kernel arguments to the APPEND line to configure networkingor other boot options.

     This configuration does not enable serial console access on machines with a graphical console. To configure a different console, add one or more console= arguments to the APPEND line. For example, add console=tty0 console=ttyS0 to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information, see How does one set up a serial terminal and/or console in Red Hat Enterprise Linux? and "Enabling the serial console for PXE and ISO installation" in the "Advanced RHCOS installation configuration" section.

   • For iPXE (x86_64+ aarch64):

     kernel http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> initrd=main coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/bootstrap.ign (1) (2)initrd --name main http://<HTTP_server>/rhcos-<version>-live-initramfs.<architecture>.img (3)boot

     1	Specify the locations of the RHCOS files that you uploaded to yourHTTP server. The kernel parameter value is the location of the kernel file,the initrd=main argument is needed for booting on UEFI systems,the coreos.live.rootfs_url parameter value is the location of the rootfs file,and the coreos.inst.ignition_url parameter value is thelocation of the bootstrap Ignition config file.
     2	If you use multiple NICs, specify a single interface in the ip option.For example, to use DHCP on a NIC that is named eno1, set ip=eno1:dhcp.
     3	Specify the location of the initramfs file that you uploaded to your HTTP server.

     This configuration does not enable serial console access on machines with a graphical console. To configure a different console, add one or more console= arguments to the kernel line. For example, add console=tty0 console=ttyS0 to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information, see How does one set up a serial terminal and/or console in Red Hat Enterprise Linux? and "Enabling the serial console for PXE and ISO installation" in the "Advanced RHCOS installation configuration" section.

     To network boot the CoreOS kernel on aarch64 architecture, you need to use a version of iPXE build with the IMAGE_GZIP option enabled. See IMAGE_GZIP option in iPXE.

   • For PXE (with UEFI and Grub as second stage) on aarch64:

     menuentry 'Install CoreOS' { linux rhcos-<version>-live-kernel-<architecture> coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/bootstrap.ign (1) (2) initrd rhcos-<version>-live-initramfs.<architecture>.img (3)}

     1	Specify the locations of the RHCOS files that you uploaded to yourHTTP/TFTP server. The kernel parameter value is the location of the kernel file on your TFTP server.The coreos.live.rootfs_url parameter value is the location of the rootfs file, and the coreos.inst.ignition_url parameter value is the location of the bootstrap Ignition config file on your HTTP Server.
     2	If you use multiple NICs, specify a single interface in the ip option.For example, to use DHCP on a NIC that is named eno1, set ip=eno1:dhcp.
     3	Specify the location of the initramfs file that you uploaded to your TFTP server.

7. Monitor the progress of the RHCOS installation on the console of the machine.

   Be sure that the installation is successful on each node before commencing with the OpenShift Container Platform installation. Observing the installation process can also help to determine the cause of RHCOS installation issues that might arise.

8. After RHCOS installs, the system reboots. During reboot, the system applies the Ignition config file that you specified.

9. Check the console output to verify that Ignition ran.

   Example command

   Ignition: ran on 2022/03/14 14:48:33 UTC (this boot)Ignition: user-provided config was applied

10. Continue to create the machines for your cluster.

    You must create the bootstrap and control plane machines at this time. If thecontrol plane machines are not made schedulable, alsocreate at least two compute machines before you install the cluster.

    If the required network, DNS, and load balancer infrastructure are in place, the OpenShift Container Platform bootstrap process begins automatically after the RHCOS nodes have rebooted.

    RHCOS nodes do not include a default password for the core user. You can access the nodes by running ssh core@<node>.<cluster_name>.<base_domain> as a user with access to the SSH private key that is paired to the public key that you specified in your install_config.yaml file. OpenShift Container Platform 4 cluster nodes running RHCOS are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes by using SSH is not recommended. However, when investigating installation issues, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on a target node, SSH access might be required for debugging or disaster recovery.

• Passing kernel arguments to the live installer

• Running coreos-installer manually from the live system

• Customizing a live ISO or PXE boot image

Using advanced networking options for PXE and ISO installations

Networking for OpenShift Container Platform nodes uses DHCP by default to gather allnecessary configuration settings. To set up static IP addresses or configure special settings, such as bonding, you can do one of the following:

To configure a PXE or iPXE installation, use one of the following options:

To configure an ISO installation, use the following procedure.

Disk partitioning

Disk partitions are created on OpenShift Container Platform cluster nodes during the Red Hat Enterprise Linux CoreOS (RHCOS) installation. Each RHCOS node of a particular architecture uses the same partition layout, unless you override the default partitioning configuration. During the RHCOS installation, the size of the root file system is increased to use any remaining available space on the target device.

OpenShift Container Platform monitors the following two filesystem identifiers:

For the default partition scheme, nodefs and imagefs monitor the same root filesystem, /.

To override the default partitioning when installing RHCOS on an OpenShift Container Platform cluster node, you must create separate partitions. Consider a situation where you want to add a separate storage partition for your containers and container images. For example, by mounting /var/lib/containers in a separate partition, the kubelet separately monitors /var/lib/containers as the imagefs directory and the root file system as the nodefs directory.

# coreos-installer install --ignition-url http://10.0.2.2:8080/user.ign \ --save-partlabel 'data*' /dev/disk/by-id/scsi-<serial_number>

# coreos-installer install --ignition-url http://10.0.2.2:8080/user.ign \ --save-partindex 6 /dev/disk/by-id/scsi-<serial_number>

# coreos-installer install --ignition-url http://10.0.2.2:8080/user.ign --save-partindex 5- /dev/disk/by-id/scsi-<serial_number>

coreos.inst.save_partlabel=data*

coreos.inst.save_partindex=5-

coreos.inst.save_partindex=6

Identifying Ignition configs

When doing an RHCOS manual installation, there are two types of Ignition configs that you can provide, with different reasons for providing each one:

Default console configuration

Red Hat Enterprise Linux CoreOS (RHCOS) nodes installed from an OpenShift Container Platform 4.18 boot image use a default console that is meant to accomodate most virtualized and bare metal setups. Different cloud and virtualization platforms may use different default settings depending on the chosen architecture. Bare metal installations use the kernel default settings which typically means the graphical console is the primary console and the serial console is disabled.

The default consoles may not match your specific hardware configuration or you might have specific needs that require you to adjust the default console. For example:

Console configuration is inherited from the boot image. This means that new nodes in existing clusters are unaffected by changes to the default console.

You can configure the console for bare metal installations in the following ways:

Enabling the serial console for PXE and ISO installations

By default, the Red Hat Enterprise Linux CoreOS (RHCOS) serial console is disabled and all output is written to the graphical console. You can enable the serial console for an ISO installation and reconfigure the bootloader so that output is sent to both the serial console and the graphical console.

To configure a PXE installation, make sure the coreos.inst.install_dev kernel command line option is omitted, and use the shell prompt to run coreos-installer manually using the above ISO installation procedure.

Customizing a live RHCOS ISO or PXE install

You can use the live ISO image or PXE environment to install RHCOS by injecting an Ignition config file directly into the image. This creates a customized image that you can use to provision your system.

For an ISO image, the mechanism to do this is the coreos-installer iso customize subcommand, which modifies the .iso file with your configuration. Similarly, the mechanism for a PXE environment is the coreos-installer pxe customize subcommand, which creates a new initramfs file that includes your customizations.

The customize subcommand is a general purpose tool that can embed other types of customizations as well. The following tasks are examples of some of the more common customizations:

Customizing a live RHCOS ISO image

You can customize a live RHCOS ISO image directly with thecoreos-installer iso customizesubcommand. When you boot the ISO image, the customizations are applied automatically.

You can use this feature to configure the ISO image to automatically install RHCOS.

Applying your customizations affects every subsequent boot of RHCOS.

Customizing a live RHCOS PXE environment

You can customize a live RHCOS PXE environment directly with thecoreos-installer pxe customizesubcommand. When you boot the PXE environment, the customizations are applied automatically.

You can use this feature to configure the PXE environment to automatically install RHCOS.

Applying your customizations affects every subsequent boot of RHCOS.

Advanced RHCOS installation reference

This section illustrates the networking configuration and other advanced options that allow you to modify the Red Hat Enterprise Linux CoreOS (RHCOS) manual installation process. The following tables describe the kernel arguments and command-line options you can use with the RHCOS live installer and the coreos-installer command.

ip=10.10.10.2::10.10.10.254:255.255.255.0:core0.example.com:enp1s0:nonenameserver=4.4.4.41

ip=10.10.10.2::10.10.10.254:255.255.255.0::enp1s0:nonenameserver=4.4.4.41

ip=10.10.10.2::10.10.10.254:255.255.255.0:core0.example.com:enp1s0:noneip=10.10.10.3::10.10.10.254:255.255.255.0:core0.example.com:enp2s0:none

ip=10.10.10.2::10.10.10.254:255.255.255.0:core0.example.com:enp1s0:noneip=::::core0.example.com:enp2s0:none

ip=enp1s0:dhcpip=10.10.10.2::10.10.10.254:255.255.255.0:core0.example.com:enp2s0:none

nameserver=1.1.1.1nameserver=8.8.8.8

team=team0:em1,em2ip=team0:dhcp

Procedure

1. To enable multipath and start the multipathd daemon, run the following command on the installation host:

   $ mpathconf --enable && systemctl start multipathd.service

   • Optional: If booting the PXE or ISO, you can instead enable multipath by adding rd.multipath=default from the kernel command line.

2. Append the kernel arguments by invoking the coreos-installer program:

   • If there is only one multipath device connected to the machine, it should be available at path /dev/mapper/mpatha. For example:

     $ coreos-installer install /dev/mapper/mpatha \ (1)--append-karg rd.multipath=default \--append-karg root=/dev/disk/by-label/dm-mpath-root \--append-karg rw

     1	Indicates the path of the single multipathed device.

   • If there are multiple multipath devices connected to the machine, or to be more explicit, instead of using /dev/mapper/mpatha, it is recommended to use the World Wide Name (WWN) symlink available in /dev/disk/by-id. For example:

     $ coreos-installer install /dev/disk/by-id/wwn-<wwn_ID> \ (1)--append-karg rd.multipath=default \--append-karg root=/dev/disk/by-label/dm-mpath-root \--append-karg rw

     1	Indicates the WWN ID of the target multipathed device. For example, 0xx194e957fcedb4841.

     This symlink can also be used as the coreos.inst.install_dev kernel argument when using special coreos.inst.* arguments to direct the live installer. For more information, see "Installing RHCOS and starting the OpenShift Container Platform bootstrap process".

3. Check that the kernel arguments worked by going to one of the worker nodes and listing the kernel command line arguments (in /proc/cmdline on the host):

   $ oc debug node/ip-10-0-141-105.ec2.internal

   Example output

   Starting pod/ip-10-0-141-105ec2internal-debug ...To use host binaries, run `chroot /host`sh-4.2# cat /host/proc/cmdline...rd.multipath=default root=/dev/disk/by-label/dm-mpath-root...sh-4.2# exit

   You should see the added kernel arguments.

Prerequisites

1. You are in the RHCOS live environment.

2. You have an iSCSI target that you want to install RHCOS on.

Procedure

1. Mount the iSCSI target from the live environment by running the following command:

   $ iscsiadm \ --mode discovery \ --type sendtargets --portal <IP_address> \ (1) --login

   1	The IP address of the target portal.

2. Install RHCOS onto the iSCSI target by running the following command and using the necessary kernel arguments, for example:

   $ coreos-installer install \ /dev/disk/by-path/ip-<IP_address>:<port>-iscsi-<target_iqn>-lun-<lun> \ (1) --append-karg rd.iscsi.initiator=<initiator_iqn> \ (2) --append.karg netroot=<target_iqn> \ (3) --console ttyS0,115200n8 --ignition-file <path_to_file>

   1	The location you are installing to. You must provide the IP address of the target portal, the associated port number, the target iSCSI node in IQN format, and the iSCSI logical unit number (LUN).
   2	The iSCSI initiator, or client, name in IQN format. The initiator forms a session to connect to the iSCSI target.
   3	The the iSCSI target, or server, name in IQN format.

   For more information about the iSCSI options supported by dracut, see the dracut.cmdline manual page.

3. Unmount the iSCSI disk with the following command:

   $ iscsiadm --mode node --logoutall=all

Prerequisites

1. You are in the RHCOS live environment.

2. You have an iSCSI target you want to install RHCOS on.

3. Optional: you have multipathed your iSCSI target.

Procedure

1. Mount the iSCSI target from the live environment by running the following command:

   $ iscsiadm \ --mode discovery \ --type sendtargets --portal <IP_address> \ (1) --login

   1	The IP address of the target portal.

2. Optional: enable multipathing and start the daemon with the following command:

   $ mpathconf --enable && systemctl start multipathd.service

3. Install RHCOS onto the iSCSI target by running the following command and using the necessary kernel arguments, for example:

   $ coreos-installer install \ /dev/mapper/mpatha \ (1) --append-karg rd.iscsi.firmware=1 \ (2) --append-karg rd.multipath=default \ (3) --console ttyS0 \ --ignition-file <path_to_file>

   1	The path of a single multipathed device. If there are multiple multipath devices connected, or to be explicit, you can use the World Wide Name (WWN) symlink available in /dev/disk/by-path.
   2	The iSCSI parameter is read from the BIOS firmware.
   3	Optional: include this parameter if you are enabling multipathing.

   For more information about the iSCSI options supported by dracut, see the dracut.cmdline manual page.

4. Unmount the iSCSI disk:

   $ iscsiadm --mode node --logout=all

Additional resources

• See Installing RHCOS and starting the OpenShift Container Platform bootstrap process for more information on using special coreos.inst.* arguments to direct the live installer.

Configuring registry storage for bare metal and other manual installations

As a cluster administrator, following installation you must configure yourregistry to use storage.

Configuring storage for the image registry in non-production clusters

You must configure storage for the Image Registry Operator. For non-productionclusters, you can set the image registry to an empty directory. If you do so,all images are lost if you restart the registry.

Configuring block registry storage for bare metal

To allow the image registry to use block storage types during upgrades as a cluster administrator, you can use the Recreate rollout strategy.