How to resolve missing dependency on exadata-sun-computenode-minimum

I've been really busy last few months - except spending a lot of time on M25 I've been doing a lot of Exadata installations and consolidations. I haven't posted for some time now but the good news is that I got many drafts and presentations ideas.

This is a quick post about an issue I had recently. I had to integrate AD authentication over Kerberos on the compute nodes (blog post to follow) but had to do compute node upgrade before that. This was Exadata X5-2 QR running 12.1.2.1.1 which had to be upgraded to 12.1.2.3.1 but I was surprised when dbnodeupdate failed with 'Minimum' dependency check failed. You'll also notice the following in the logs:

exa01db01a: Exadata capabilities missing (capabilities required but not supplied by any package)
exa01db01a NOTE: Unexpected configuration - Contact Oracle Support

Starting with 11.2.3.3.0 the exadata-*computenode-exact and exadata-*computenode-minimum rpms were introduced. An update to 11.2.3.3.0 or later by default assumes the 'exact' rpm will be used to 'update to' with yum hence before running the upgrade dbnodeupdate will check if there are missing packages/dependencies.

Best way to check what is mnissing is to run yum check:

[root@exa01db01a ~]# yum check
Loaded plugins: downloadonly
exadata-sun-computenode-minimum-12.1.2.1.1.150316.2-1.x86_64 has missing requires of elfutils-libelf-devel >= ('0', '0.158', '3.2.el6')
exadata-sun-computenode-minimum-12.1.2.1.1.150316.2-1.x86_64 has missing requires of elfutils-libelf-devel(x86-64) >= ('0', '0.158', '3.2.el6')
exadata-sun-computenode-minimum-12.1.2.1.1.150316.2-1.x86_64 has missing requires of glibc-devel(x86-32) >= ('0', '2.12', '1.149.el6_6.5')
exadata-sun-computenode-minimum-12.1.2.1.1.150316.2-1.x86_64 has missing requires of libsepol(x86-32) >= ('0', '2.0.41', '4.el6')
exadata-sun-computenode-minimum-12.1.2.1.1.150316.2-1.x86_64 has missing requires of libselinux(x86-32) >= ('0', '2.0.94', '5.8.el6')
exadata-sun-computenode-minimum-12.1.2.1.1.150316.2-1.x86_64 has missing requires of elfutils-libelf(x86-32) >= ('0', '0.158', '3.2.el6')
exadata-sun-computenode-minimum-12.1.2.1.1.150316.2-1.x86_64 has missing requires of libcom_err(x86-32) >= ('0', '1.42.8', '1.0.2.el6')
exadata-sun-computenode-minimum-12.1.2.1.1.150316.2-1.x86_64 has missing requires of e2fsprogs-libs(x86-32) >= ('0', '1.42.8', '1.0.2.el6')
exadata-sun-computenode-minimum-12.1.2.1.1.150316.2-1.x86_64 has missing requires of libaio(x86-32) >= ('0', '0.3.107', '10.el6')
exadata-sun-computenode-minimum-12.1.2.1.1.150316.2-1.x86_64 has missing requires of libaio-devel(x86-32) >= ('0', '0.3.107', '10.el6')
exadata-sun-computenode-minimum-12.1.2.1.1.150316.2-1.x86_64 has missing requires of libstdc++-devel(x86-32) >= ('0', '4.4.7', '11.el6')
exadata-sun-computenode-minimum-12.1.2.1.1.150316.2-1.x86_64 has missing requires of compat-libstdc++-33(x86-32) >= ('0', '3.2.3', '69.el6')
exadata-sun-computenode-minimum-12.1.2.1.1.150316.2-1.x86_64 has missing requires of zlib(x86-32) >= ('0', '1.2.3', '29.el6')
exadata-sun-computenode-minimum-12.1.2.1.1.150316.2-1.x86_64 has missing requires of libxml2(x86-32) >= ('0', '2.7.6', '17.0.1.el6_6.1')
exadata-sun-computenode-minimum-12.1.2.1.1.150316.2-1.x86_64 has missing requires of elfutils >= ('0', '0.158', '3.2.el6')
exadata-sun-computenode-minimum-12.1.2.1.1.150316.2-1.x86_64 has missing requires of elfutils(x86-64) >= ('0', '0.158', '3.2.el6')
exadata-sun-computenode-minimum-12.1.2.1.1.150316.2-1.x86_64 has missing requires of ntsysv >= ('0', '1.3.49.3', '2.el6_4.1')
exadata-sun-computenode-minimum-12.1.2.1.1.150316.2-1.x86_64 has missing requires of ntsysv(x86-64) >= ('0', '1.3.49.3', '2.el6_4.1')
exadata-sun-computenode-minimum-12.1.2.1.1.150316.2-1.x86_64 has missing requires of glibc(x86-32) >= ('0', '2.12', '1.149.el6_6.5')
exadata-sun-computenode-minimum-12.1.2.1.1.150316.2-1.x86_64 has missing requires of nss-softokn-freebl(x86-32) >= ('0', '3.14.3', '18.el6_6')
exadata-sun-computenode-minimum-12.1.2.1.1.150316.2-1.x86_64 has missing requires of libgcc(x86-32) >= ('0', '4.4.7', '11.el6')
exadata-sun-computenode-minimum-12.1.2.1.1.150316.2-1.x86_64 has missing requires of libstdc++(x86-32) >= ('0', '4.4.7', '11.el6')
exadata-sun-computenode-minimum-12.1.2.1.1.150316.2-1.x86_64 has missing requires of compat-libstdc++-296 >= ('0', '2.96', '144.el6')
exadata-sun-computenode-minimum-12.1.2.1.1.150316.2-1.x86_64 has missing requires of compat-libstdc++-296(x86-32) >= ('0', '2.96', '144.el6')
Error: check all

Somehow all x86-32 packages and three x86-64 packages were removed. The x86-32 packages will be removed during as part of upgrade anyway, they were not present after the upgrade. I didn't spend too much do understand why or how the packages were removed. I was told additional packages were installed before and then removed. Perhaps one had few dependencies and all got messed up when it was removed.

Anyway to solve this you need to download the patch for the same version (12.1.2.1.1). The p20746761_121211_Linux-x86-64.zip patch is still available from MOS 888828.1. So after that you unzip it, mount the iso, test install all the package to make sure nothing is missing and there are no conflicts and then finally install the packages:

[root@exa01db01a x86_64]# rpm -ivh --test zlib-1.2.3-29.el6.i686.rpm glibc-2.12-1.149.el6_6.5.i686.rpm nss-softokn-freebl-3.14.3-18.el6_6.i686.rpm libaio-devel-0.3.107-10.el6.i686.rpm libaio-0.3.107-10.el6.i686.rpm e2fsprogs-libs-1.42.8-1.0.2.el6.i686.rpm libgcc-4.4.7-11.el6.i686.rpm libcom_err-1.42.8-1.0.2.el6.i686.rpm elfutils-libelf-0.158-3.2.el6.i686.rpm libselinux-2.0.94-5.8.el6.i686.rpm libsepol-2.0.41-4.el6.i686.rpm glibc-devel-2.12-1.149.el6_6.5.i686.rpm elfutils-libelf-devel-0.158-3.2.el6.x86_64.rpm libstdc++-devel-4.4.7-11.el6.i686.rpm libstdc++-4.4.7-11.el6.i686.rpm compat-libstdc++-296-2.96-144.el6.i686.rpm compat-libstdc++-33-3.2.3-69.el6.i686.rpm libxml2-2.7.6-17.0.1.el6_6.1.i686.rpm elfutils-0.158-3.2.el6.x86_64.rpm ntsysv-1.3.49.3-2.el6_4.1.x86_64.rpm
Preparing...                ########################################### [100%]

[root@exa01db01a x86_64]# rpm -ivh zlib-1.2.3-29.el6.i686.rpm glibc-2.12-1.149.el6_6.5.i686.rpm nss-softokn-freebl-3.14.3-18.el6_6.i686.rpm libaio-devel-0.3.107-10.el6.i686.rpm libaio-0.3.107-10.el6.i686.rpm e2fsprogs-libs-1.42.8-1.0.2.el6.i686.rpm libgcc-4.4.7-11.el6.i686.rpm libcom_err-1.42.8-1.0.2.el6.i686.rpm elfutils-libelf-0.158-3.2.el6.i686.rpm libselinux-2.0.94-5.8.el6.i686.rpm libsepol-2.0.41-4.el6.i686.rpm glibc-devel-2.12-1.149.el6_6.5.i686.rpm elfutils-libelf-devel-0.158-3.2.el6.x86_64.rpm libstdc++-devel-4.4.7-11.el6.i686.rpm libstdc++-4.4.7-11.el6.i686.rpm compat-libstdc++-296-2.96-144.el6.i686.rpm compat-libstdc++-33-3.2.3-69.el6.i686.rpm libxml2-2.7.6-17.0.1.el6_6.1.i686.rpm elfutils-0.158-3.2.el6.x86_64.rpm ntsysv-1.3.49.3-2.el6_4.1.x86_64.rpm
Preparing...              ########################################### [100%]
1:libgcc                  ########################################### [  5%]
2:elfutils-libelf-devel   ########################################### [ 10%]
3:nss-softokn-freebl      ########################################### [ 15%]
4:glibc                   ########################################### [ 20%]
5:glibc-devel             ########################################### [ 25%]
6:elfutils                ########################################### [ 30%]
7:zlib                    ########################################### [ 35%]
8:libaio                  ########################################### [ 40%]
9:libcom_err              ########################################### [ 45%]
10:libsepol               ########################################### [ 50%]
11:libstdc++              ########################################### [ 55%]
12:libstdc++-devel        ########################################### [ 60%]
13:libaio-devel           ########################################### [ 65%]
14:libselinux             ########################################### [ 70%]
15:e2fsprogs-libs         ########################################### [ 75%]
16:libxml2                ########################################### [ 80%]
17:elfutils-libelf        ########################################### [ 85%]
18:compat-libstdc++-296   ########################################### [ 90%]
19:compat-libstdc++-33    ########################################### [ 95%]
20:ntsysv                 ########################################### [100%]

[root@exa01db01a x86_64]# yum check
Loaded plugins: downloadonly
check all

After that dbnodeupdate check completed successfully I upgraded the node to 12.1.3.2.1 in no time.

With Exadata you are allowed to install packages on the compute nodes as long as they don't break any dependencies but you cannot install anything on the storage cells. Here's oracle official statement:
Is it acceptable / supported to install additional or 3rd party software on Exadata machines and how to check for conflicts? (Doc ID 1541428.1)

Update 23.08.2016:

You might also get errors for two more packages in case you have updated to from OEL5 to OEL6 and now you try to patch the compute node:

fuse-2.8.3-4.0.2.el6.x86_64 has missing requires of kernel >= ('0', '2.6.14', None)
2:irqbalance-1.0.7-5.0.1.el6.x86_64 has missing requires of kernel >= ('0','2.6.32', '358.2.1')

Refer to the following note for more information and how to fix it:
FUSE & IRQBALANCE rpm fails during yum update with "requires of kernel >=" (Doc ID 2086184.1)