Migrated RHEL 7.0 Server Fails to Boot after yum Update to RHEL 7.3
Reference Number: AA-00406 Views: 4840 Created: 01-31-2017 14:57 Last Updated: 02-08-2017 12:30

Symptoms: 

After successfully migrating an RHEL 7.0 workload to an Amazon environment and then yum updating to RHEL 7.3, the updated instance fails to boot.

Applies To:

DynaCenter 6.4.0 and later

Background: 

Running yum update on a migrated RHEL 7.0 workload to update it to RHEL 7.3 can result in an improperly configured /boot/grub2/grub.cfg file if you update the entire system, including the kernel, at the same time. The misconfigured /boot/grub2/grub.cfg file prevents the OS from functioning normally after the updated system is rebooted.  

Resolution: 

To ensure that the RHEL instance can boot after it is upgraded to RHEL 7.3, do one of the following:


Workaround 1

1. Update the kernel on the source server and then reboot the system:

yum -y update kernel

2. After the kernel update completes, update the rest of the packages and then reboot the system:

yum -y update


reboot

3. Remigrate the server workload.


Workaround 2

1. Remigrate the server workload.

2. Update the kernel on the migrated server and then reboot the system:

yum -y update kernel

3. After the kernel update completes, update the rest of the packages and then reboot the system:

yum -y update


reboot


 Workaround 3

1. Stop the migrated instance that is failing to boot and detach its root device, making note of the volume ID.

2. From the AWS Management Console, create a new temporary RHEL 7 instance in the same region and availability zone as the server that failed to boot.

Note: A small server size will do for this purpose. 

3. Attach the root device volume of the failed instance to the temporary instance.

4. Log in to the temporary instance and mount the volume, for example):

mkdir /tmp/mnt

mount /dev/xvdf1 /tmp/mnt

5. Edit the /boot/grub2/grub.cfg to comment out the first menuentry in the file, for example:

#menuentry 'Red Hat Enterprise Linux Server (3.10.0-514.6.1.el7.x86_64) 7.0 (Maipo)' --class red --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-123.el7.x86_64-advanced-13ab9a80-a360-446b-bf1b-3fc65d3aa7fb' {

#       load_video

#       set gfxpayload=keep

#       insmod gzio

#       insmod part_msdos

#       insmod xfs

#       if [ x$feature_platform_search_hint = xy ]; then

#         search --no-floppy --fs-uuid --set=root 7f3cd3c1-e1ef-468d-9c9c-a4e3dc046d4e

#       else

#         search --no-floppy --fs-uuid --set=root 7f3cd3c1-e1ef-468d-9c9c-a4e3dc046d4e

#       fi
#       linux16 /vmlinuz-3.10.0-514.6.1.el7.x86_64 root=/dev/rhel/root ro net.ifnames=0 crashkernel=auto vconsole.font=latarcyrheb-sun16 vconsole.keymap=us rhgb quiet LANG=en_US.UTF-8

#       initrd16 /initramfs-3.10.0-514.6.1.el7.x86_64.img

#}

6. Unmount and detach the volume from the temporary instance.

7. Attach the volume to the original failed instance as root device /dev/sda1 and start the instance.

8. Log in to the original instance and rebuild /boot/grub2/grub.cfg:

grub2-mkconfig > /boot/grub2/grub.cfg

9. Reboot the original instance and it should now be running the latest kernel.