Oracle have greatly improved the automation and patching processes and it generally goes smooth. Gone are the days where cells patching would take 3hrs to patch. I usually stage and re-run the prechecks prior the patching window. I actually spend more time in preparation than actual execution, that is to find the required patches, check the release notes, check for conflicts and check the patch known issues since we were applying not the latest but the patch from previous quarter.

Now this time I didn't run all prechecks as they usually come up ok, funny that prechecks only take few minutes to complete and confirm system is ok. Of course the one time I didn't run precheck prior patching and I run into issues. Upon running cell patching prechecks I got the follow errors:

[root@exa01db01-c patch_18.1.30.0.0.200713.1]# ./patchmgr -cells cell_group -patch_check_prereq

2020-09-18 14:46:04 +0000        :Working: DO: Check cells have ssh equivalence for root user. Up to 10 seconds per cell ...
2020-09-18 14:46:05 +0000        :SUCCESS: DONE: Check cells have ssh equivalence for root user.
2020-09-18 14:46:09 +0000        :Working: DO: Initialize files. Up to 1 minute ...
2020-09-18 14:46:10 +0000        :Working: DO: Setup work directory
2020-09-18 14:46:33 +0000        :SUCCESS: DONE: Setup work directory
2020-09-18 14:46:35 +0000        :SUCCESS: DONE: Initialize files.
2020-09-18 14:46:35 +0000        :Working: DO: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction. Up to 40 minutes ...
2020-09-18 14:46:48 +0000        :INFO   : Wait correction of degraded md11 due to md partner size mismatch. Up to 30 minutes.
2020-09-18 14:46:49 +0000        :SUCCESS: DONE: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction.
2020-09-18 14:46:49 +0000        :Working: DO: Check space and state of cell services. Up to 20 minutes ...
FAILED for following cells

exa01cel04:  exa01cel04 172.16.0.92 2020-09-18 14:47:21 +0000: Directory /boot either not mounted properly or does not have enough space
exa01cel05:  exa01cel05 172.16.0.93 2020-09-18 14:47:21 +0000: Directory /boot either not mounted properly or does not have enough space
exa01cel06:  exa01cel06 172.16.0.94 2020-09-18 14:47:22 +0000: Directory /boot either not mounted properly or does not have enough space
FAILED With Exception for following cells

exa01cel02:  exa01cel02 172.16.0.90 2020-09-18 14:47:24 +0000: Errors were encountered when verifying state of CELLBOOT USB.
exa01cel03:  exa01cel03 172.16.0.91 2020-09-18 14:47:25 +0000: Errors were encountered when verifying state of CELLBOOT USB.
2020-09-18 14:47:29 +0000        :FAILED : For details, check the following files in the /u01/stage/2020Q3/patch_18.1.30.0.0.200713.1:
2020-09-18 14:47:29 +0000        :FAILED :  - <cell_name>.log
2020-09-18 14:47:29 +0000        :FAILED :  - patchmgr.stdout
2020-09-18 14:47:29 +0000        :FAILED :  - patchmgr.stderr
2020-09-18 14:47:29 +0000        :FAILED :  - patchmgr.log
2020-09-18 14:47:29 +0000        :FAILED : DONE: Check space and state of cell services.

Ok, I've seen storage cells pre-check to fail before but nothing like this. As you can guess there was nothing wrong with the boot volume of these storage cells.

I always check the image status and server profile, before and after patching to make sure they patching has completed successfully.  However running it again I noticed there is something wrong and it this issue spans across compute nodes as well. Hand on heart they were fine during last patching, how come they were corrupted now is still unknown to me. I decided to check the hardware profile again across all servers and here's what I got:

[root@exa01db01-c ~]# dcli -g all_group -l root /opt/oracle.SupportTools/CheckHWnFWProfile
exa01cel01: Current server model ORACLE_SERVER_X5-2L_ORACLE_SERVER_X5-2L is not found in supported server list
exa01cel02: Current server model ORACLE_SERVER_X5-2L_ORACLE_SERVER_X5-2L is not found in supported server list
exa01cel03: Current server model ORACLE_SERVER_X5-2L_ORACLE_SERVER_X5-2L is not found in supported server list
exa01cel04: Current server model ORACLE_SERVER_X7-2L_ORACLE_SERVER_X7-2L is not found in supported server list
exa01cel05: Current server model ORACLE_SERVER_X7-2L_ORACLE_SERVER_X7-2L is not found in supported server list
exa01cel06: Current server model ORACLE_SERVER_X7-2L_ORACLE_SERVER_X7-2L is not found in supported server list
exa01db01-c: Current server model ORACLE_SERVER_X5-2_ORACLE_SERVER_X5-2 is not found in supported server list
exa01db02-c: [SUCCESS] The hardware and firmware matches supported profile for server=ORACLE_SERVER_X5-2
exa01db03-c: [SUCCESS] The hardware and firmware matches supported profile for server=ORACLE_SERVER_X7-2
exa01db04-c: [SUCCESS] The hardware and firmware matches supported profile for server=ORACLE_SERVER_X7-2

It's obvious that servers are not supported but I couldn't understand where is this information coming from. Running another script that runs during system boot suggests that's not production hardware:

[root@exa01db01-c ~]# /opt/oracle.cellos/validations/init.d/checkdeveachboot
2020-09-18 13:58:15 +0000
2020-09-18 13:58:15 +0000 /opt/oracle.cellos/validations/init.d/checkdeveachboot started at 2020-09-18 13:58:15 +0000
2020-09-18 13:58:15 +0000 #^#^# [WARNING] [COMMON] 5 This is non production hardware
2020-09-18 13:58:15 +0000 34:0:0:0

MOS doesn't suggest anything except some old bugs. Following the scripts and functions I got to the exadata.img.hw.cache file:

[root@exa01db01-c ~]# cat /etc/exadata/config/hardware/exadata.img.hw.cache
System Model=ORACLE SERVER X5-2
System Manufacturer=Oracle Corporation
System Serial Number=1509NM10H3
Chassis Serial Number=1509NM10H3
BIOS Vendor=American Megatrends Inc.
BIOS Version=30300200
BIOS Release Date=07/10/2019
System Model=ORACLE SERVER X5-2
ORACLE SERVER X5-2
System Manufacturer=Oracle Corporation
Oracle Corporation
System Architecture=x86_64
Cellboot USB Serial=

One could not notice the redundant line below the System model. The file is generated during system boot and is used by CheckHWnFWProfile to confirm firmware matches the hardware, exadata.img.hw command to determine the model of the cell or database server and other scripts. How come the cache got corrupted is still a puzzle. Fortunately the fix was really simple - removing the redundant lines off the exadata.img.hw.cache resolved the issue and I was able to carry on with the patching.

[root@exa01db01-c ~]# /opt/oracle.SupportTools/CheckHWnFWProfile
[SUCCESS] The hardware and firmware matches supported profile for server=ORACLE_SERVER_X5-2

It's an easy fix. I'm posting this so that if anyone runs into this issue could quickly resolve the problem and not stress or spend too much time in troubleshooting. Fortunately this was pre-prod and I had enough time to troubleshoot the issue.

(Technical) I decided to look further and understand what the sequence is:

• the script cellFirstboot.sh is invoked during system boot
• then the function cellFirstboot_checkAndUpdateExadataImgHwCache () is invoked
• the function then calls exadata.img.hw --update
• the exadata.img.hw on the other hand calls dmidecode and ipmitool to gather all the information like system model, manufacturer, serial number and BIOS information.

So it looks like it's dmidecode or ipmitool to blame. Unfortunately I don't have access to this system anymore. I suspect that it could run into this issue again upon next patching. The system might need a power off as one of Oracle notes suggests:

The dmidecode command queries the SMBIOS tables, dumping the various data contained therein, such as hardware components, serial numbers and BIOS version. A missing value for the system product name is an indication that the tables do not contain the correct information or that there's an issue in retrieving the data. This can happen if the BIOS flashing has not completed or initialized successfully. It typically follows a recent replacement of the system board.

Hope this helps someone.

The Exadata Cloud Service X8M is using the new resource model so that you always provision 2 compute nodes and 3 storage cells (equivalent to a quarter rack) and then you scale out as much as you needed. If you haven't read my previous posts on provisioning and scaling I suggest you do that now.

Respectively I provisioned X8M and then added two more compute nodes, the operation was successful and I have database instances running across all nodes already. Then I decided to give the new 19c upgrade feature a try and that's where I ran into the error. Apparently the scale operation (or GI) did not set the correct permissions of the file above:

test-exacs-sfe4w1: -rw-rw---- 1 grid oinstall 174 Dec  7 15:49 /u01/app/oraInventory/ContentsXML/oui-patch.xml
test-exacs-sfe4w2: -rw-rw---- 1 grid oinstall 174 Dec  7 12:45 /u01/app/oraInventory/ContentsXML/oui-patch.xml
test-exacs-sfe4w3: -rw-r--r-- 1 grid oinstall 174 Nov 20 03:25 /u01/app/oraInventory/ContentsXML/oui-patch.xml
test-exacs-sfe4w4: -rw-r--r-- 1 grid oinstall 174 Nov 20 03:25 /u01/app/oraInventory/ContentsXML/oui-patch.xml

After setting the correct permissions of oui-patch.xml file the provisioning of database home 19c was successful. Thus, if you are provisioning 19c home on your Exadata Cloud Service make sure you check the permissions first. Again, that's not a rocket science but good to be available in the search engines should someone run into this problem and don't want to go the SR route.

You can find more information on home provisioning in these two files:

/var/opt/oracle/log/dummydb19000/clone/clone.log
/var/opt/oracle/log/dummydb19000/clone/clone-cmd.log

The clone-cmd.log file has the actual the error:

You can find the log of this install session at:
 /u01/app/oraInventory/logs/InstallActions2020-12-04_07-50-55PM/installActions2020-12-04_07-50-55PM.log
SEVERE:OUI-10094:Problem in saving inventory. Installation cannot continue. oracle.sysman.oui.patch.PatchException: java.io.FileNotFoundException: /u01/app/oraInventory/ContentsXML/oui-patch.xml (Permission denied)
[FATAL] OUI-10094:Problem in saving inventory. Installation cannot continue. oracle.sysman.oui.patch.PatchException: java.io.FileNotFoundException: /u01/app/oraInventory/ContentsXML/oui-patch.xml (Permission denied)

Later on I found a note in MOS on (almost) the same issue, for reference:

opatchauto apply Results java.io.FileNotFoundException: /ContentsXML/oui-patch.xml (Permission denied) Error in Non-OUI Nodes (Doc ID 2582139.1)

I raised SR asking the support to raise a bug but they will be updating the above note to include Exadata Cloud Service as well.

To reiterate, if you plan an upgrade to a new release at the moment, this should be 19c. If you run 12.1, 12.2 or 18c you should plan to upgrade to 19c soon if you don't want to pay extended fees or be supported at all. Note 742060.1 is your friend:

Back to the 19c upgrade, there is nothing complex in this upgrade but the devil is in the details. To be able to use this feature there are number of prerequisites:

• This upgrade feature is only available for the the new Exadata Cloud Service Resource Model. If your Exadata Cloud Service X8M then you are already using the new resource model, if not consider switching your Exadata Cloud service to the new resource model (blog post to follow)
• The Exadata Cloud Service system must be running OEL 7 i.e. you must be running Exadata System Software release 19.x
• The Grid Infrastructure version must be 19c
• The must be Oracle Database home provisioned on the system using one of the latest two releases. At the time of writing this post, 19c homes on X8M are provisioned under 19.8 and the latest available version is 19.9
• The upgraded database must be in archivelog mode and having flashback enabled. Unless you have explicitly disabled these features they are turned on by default when you create a database using the cloud console.

Here are my software versions for reference:

• Exadata Cloud Service X8M + 2 compute nodes + 2 storage servers
• Exadata System Software version 20.1.2.0.0.200930
• Grid Infrastructure version 19.9.0.0.201020
• Oracle Database 12c version 12.2.0.1.201020
• Oracle Home version 19.8.0.0.200714

Before upgrading a database it's recommended to run the prechecks and make sure your database is ready for an upgrade. The precheck is very quick, non intrusive and it only takes few minutes to complete. Also, make sure you take a full backup before running the upgrade.

I did the upgrade of my 12.2 test database and it completed successfully! For an empty 4GB database it took 1hr 40 minutes to complete.

Further to that I decided to apply latest available patch 19.9 on my newly upgraded database. Similarly with the upgrade to 19c the precheck takes 10 minutes to complete. Then to apply the patch it takes another 30 minutes. I could have upgraded database home to 19.9 prior the database upgrade but decided to give it a try. Here are the timings for an empty 12.2 database upgrade to 19.8 and patch to 19.9:

Reference to the documentation:

[2020-08-19T16:19:51.077+01:00] [as] [NOTIFICATION] [] [oracle.as.install.engine.modules.validation.oracle.as.install.engine.modules.validation.handler.homeQueries.verifyAllDependencies] [host: soahost1] [nwaddr: 192.168.0.100] [tid: 34] [userId: appldevr1] [ecid: 0000NG7C62ZAPPO5yjc9yW1VFIsO000002,0] Cause:Not all dependent featuresets for install type "BI Platform Distribution with Samples" could be found. The following prerequisites were found to be missing:[[
jrf_wlsFmw - 12.2.1.3.0glcm_opatchauto_fmw - 13.9.2.0.0
]]

As the note below suggests OBIEE 12.2.1.4 is not supported on FMW Infrastructure 12.2.1.4 and version 12.2.1.3 should be used instead:

Oracle Business Intelligence 12.2.1.4 is only certified and supported to be installed using Fusion Middleware infrastructure 12.2.1.3 (this includes Weblogic 12.2.1.3).
There are no plans to certify OBI 12.2.1.4 with Fusion Middleware Infrastructure 12.2.1.4.

Is Oracle Business Intelligence 12.2.1.4 Certified or Supported With Fusion Middleware Infrastructure 12.2.1.4? (Doc ID 2606828.1)

This can be confirmed in the compatibility matrix for OBIEE:

I then successfully installed FMW Infrastructure 12.2.1.3 and OBIEE 12.2.1.4 but got another error while running the RCU:

Here's the text of the error so that search engines can index it:

ORA-12899: value too large for column
COMPONENT_SCHEMA_INFO"."DB_PORTNUMBER" (actual: 15, maximum: 10)

I couldn't find what the issue is. There are many notes on MOS with similar symptoms but none seem to be close to my setup. One however caught my attention suggesting that I should enter the connection details as connection parameters and not as a connection string.

I dropped the repository and after trying again, it worked out! Just something to consider if someone ever runs into this issue.

Perhaps I should have mentioned in the earlier post but due to cloud and on-premises terminology the following are used interchangeably:

• Database server, compute nodes, virtual machines - are the servers where Oracle database is running.
• Storage servers, storage cells - as the name suggest these are the servers holding all disks drives and storing your data.

When you provision X8M cloud service you will always have 2 compute nodes and 3 storage servers. Additional compute nodes can be added online through the console with no downtime. Then you can scale the compute node processing power within the provisioned system, adding or subtracting CPU cores as needed. The increment of CPU cores is in multiples of the number of database servers currently provisioned for the cloud VM cluster. For example, if you have 4 database servers provisioned, you can add CPU cores in multiples of 4.

Currently, the system can scale up to 32 compute nodes or about 1600 OCPUs. That's a lot, I'm finding it hard sometimes to manage a full rack with 8 compute nodes let alone 32. On the other hand, if you want to easily scale up and down your services and/or move them to different nodes that would be very easy. One thing makes me wonder - how long will Grid Infrastructure patching take. The same is performed in a rolling fashion, with only one node being upgraded at a time. It takes nearly one hour for GI patching to complete on one node and arranging 24-32 hours of downtime might be hard if not impossible for some customers.

Having this option to easily scale up to 32 compute nodes is great but mind the following note from Oracle:

The Exadata X8M shape does not support removing storage or database servers from an existing X8M instance.

It's a shame this feature is not supported but I think it's only a matter of time to see it coming. With X8M elasticity in the cloud it makes easier for customers to start small and grow big and not spending fortune in the beginning. What would make this offer more appealing is the support of removing of compute node and storage servers. There could be different reasons for that, say for example business need to handle the peak sales of Black Friday or there is end of year reports that need to complete within a given time frame. It would be great to scale out the infrastructure to handle the demand and then scale down again. Well that's not supported today but hopefully we will see soon as well as having more than one VM Cluster on the same infrastructure.

Back in days at E-DBA (Red Stack) I had to do many different upgrades over the years. Adding compute nodes or storage cells was something I had to do many times. That would usually take two days - one at the data center and one remote. I had to do all stacking and racking, arrange and connect all the cables, re-image the nodes on the first day and then add them to the cluster or ASM on the second day. That's only my time but there is also procurement, delivery, storage and paperwork like arranging access. Here is how you do it in the cloud.

I will be adding two more compute nodes to my X8M cloud service which looks like this now:

To add more compute nodes or storage you go to Exadata Infrastructure resource and at the top left corner there is Scale Infrastructure button which brings the following screen. Here you can choose whether you want to add compute or storage server and how many, you can add only one type of resource at a time:

The operation completes immediately on the Exadata Infrastructure resource and the VM Cluster resource moves to Updating state. Like any other cloud resource, you can't run any other operations during this state and you need to wait for its completion:

The process will add the two compute nodes to the cluster, one at the time. It takes around 2 hours per compute nodes and it's fully automated. By the time it completes you will have all four compute nodes under Virtual Machines and their details:

And that's all, quick and painless, complete in less than 5 hours.

Next week I'll be scaling the infrastructure with more storage nodes. Stay tuned!

Back to cloud, till now Oracle offered quarter, half and full racks shapes only. That wasn't a great choice for many customers, including for the one I'm working at the moment. Customer already had an X7 half rack in OCI and wanted to provision a new Exadata environment for performance testing in OCI. Problem was that we needed about 220TB of usable space in DATA diskgroup and with half rack we get around 240TB. That would have put us in very difficult position in the future when databases grow and was a tough decision to make. Fortunately, at that time Oracle announced the release of X8M in the OCI that provides so much desired elastic configurations. We can now provision the same number of compute nodes and few more storage cells to fulfil the storage requirements.

Comparing the Exadata Cloud service X7 and X8M shapes one could not notice the difference in CPU and usable disk space:

• Exadata.Quarter2.92 (x7): Provides a 2-node Exadata DB system with up to 92 CPU cores, and 106 TB of usable storage.
• Exadata.Quarter3.100 (X8): Provides a 2-node Exadata DB system with up to 100 CPU cores, and 149 TB of usable storage.

That's expected, the X8 has newer CPUs with more cores and bigger disks on the storage cells (10TB compared to 14TB). However, what is not expected is the price difference between these two systems:

That's good saving of $7000 per month. The price is a reason good enough to go for X8M, let alone the more CPUs, more space and features like PMEM and RoCE available on X8M.

Now, to the more interesting stuff. The new X8M shape is using the new Resource Model and the system is not available under DB System anymore. Instead, the Exadata system is now provisioned under Exadata Infrastructure, this works as a top-level resource where one manages the infrastructure, that is the number of compute nodes and storage cells. Then there is the VM Cluster resource which is child resource of the infrastructure one and that's where database homes, databases, networking and cluster details reside. In short, under Exadata Infrastructure you scale out the physical resources (compute and storage) and schedule the upgrade of the Exadata Storage Software. Then under VM Clusters you scale up the existing compute nodes and schedule the upgrade of the Grid Infrastructure. It will become more clearer in the screenshots below.

Provisioning Exadata Infrastructure

The new resource model has two resource types - Exadata Infrastructure and VM Clusters. Respectively one needs to provision each separately, starting with Exadata Infrastructure:

The initial Exadata Cloud Service instance will have 2 database servers and 3 storage servers by default, that's the equivalents of an X8 quarter rack. After provisioning, additional storage servers, compute servers can be dynamically added to the configuration.

The provisioning is instantaneous as there are no physical resources yet but this new resource works as a parent for the VM Cluster that will be provisioned next. The only option available here is setting a preference of the maintenance schedule but the same can be changed later.

Provisioning VM Cluster

Moving on to the VM Cluster resource, that's where actual provisioning happens. The process is similar to the previous generation Exadata provisioning - you need to specify the cluster name, how many of the CPU cores you want to assign to this cluster, whether you want sparse group and/or local backups. There's an option to choose the Grid Infrastructure version but the only available at the moment is 19c. The other options you specify are client and backup subnets, hostname prefix, time zone and type of licenses you want to use.

The process takes few hours to complete and then you get the cluster up and running ready to host databases:

The databases/compute nodes are referred as virtual machines now, the tab looks exactly the same as before:

Interestingly however, if you try provisioning another VM Cluster you get this message:

This suggests that we might get the option to provision more clusters under the same infrastructure. That already available for on-premises Exadata for few years now and it is natural to come to cloud at some point, similarly to how elastic configurations are now available.

The newly provisioned system runs September release of Exadata Storage Software:

Kernel version: 4.14.35-1902.304.6.4.el7uek.x86_64 #2 SMP Tue Aug 4 12:36:50 PDT 2020 x86_64
Image kernel version: 4.14.35-1902.304.6.4.el7uek
Image version: 20.1.2.0.0.200930
Image activated: 2020-11-17 15:04:40 +0000
Image status: success
Node type: GUEST
System partition on device: /dev/mapper/VGExaDb-LVDbSys1

While I was writing this post I noticed a new version 20.1.4.0.0 was released to fix a critical bug in 20.1.3.0.0. We'll see how long before we get this update available in the cloud and will blog about it.

The other thing to note, which is also a significant change for Exadata platform is the change in the directory structure:

[root@test-exacs-sfe4w1 ~]# df -h
Filesystem                                                Size  Used Avail Use% Mounted on
devtmpfs                                                  684G     0  684G   0% /dev
tmpfs                                                     1.4T  209M  1.4T   1% /dev/shm
tmpfs                                                     684G  5.3M  684G   1% /run
tmpfs                                                     684G     0  684G   0% /sys/fs/cgroup
/dev/mapper/VGExaDb-LVDbSys1                               61G  6.0G   56G  10% /
/dev/mapper/VGExaDb-LVDbTmp                               3.0G  104M  2.9G   4% /tmp
/dev/mapper/VGExaDb-LVDbHome                              4.0G   58M  4.0G   2% /home
/dev/mapper/VGExaDb-LVDbOra1                              250G  5.1G  245G   3% /u01
/dev/mapper/VGExaDbDisk.grid19.0.0.0.200714.img-LVDBDisk   20G   14G  6.4G  69% /u01/app/19.0.0.0/grid
/dev/mapper/VGExaDb-LVDbVar1                               10G  1.2G  8.9G  12% /var
/dev/mapper/VGExaDbDisk.u02_extra.img-LVDBDisk            763G  163G  562G  23% /u02
/dev/mapper/VGExaDb-LVDbVarLog                             30G  409M   30G   2% /var/log
/dev/mapper/VGExaDb-LVDbVarLogAudit                        10G  150M  9.9G   2% /var/log/audit
/dev/sda1                                                 509M   84M  425M  17% /boot
tmpfs                                                     137G     0  137G   0% /run/user/0
tmpfs                                                     137G     0  137G   0% /run/user/2000
/dev/asm/acfsvol01-292                                    1.1T  125G  996G  12% /acfs01
tmpfs                                                     137G     0  137G   0% /run/user/12138

In next posts I will go through infrastructure scale (compute and storage), Infrastructure and GI patching, cloning databases off object storage on to the new system and all the required steps to get them in cloud UI.

Here's the cloud service data sheet:
Oracle Exadata Cloud Service X8M data sheet

If you are interested of what's behind the scenes here's the X8M-2 data sheet:
Oracle Exadata X8M-2 data sheet

Link to Exadata Cloud Service documentation

The following page describes the differences between X8-2 and X8M-2.

I'm currently working on a Supercluster project where I provide database support. To help my colleagues and the whole project I decided to step out of my comfort zone and install the latest 12.2.1.4 version of Oracle SOA, BPM, OSB and MFT in high availability.

It's not my usual area of expertise, that is any database or system administration and it's been a while since I've seen FMW installation but the documentation is very detailed and clear on how to install it in cluster. For this purpose I followed the Enterprise Deployment Guide (EDG) for SOA. Unfortunately at the time of writing this the EDG for 12.2.1.4 was not available so I decided to follow the one for 12.2.1.3.

Turns out there is slight difference in the internal product stack, in particular when expanding the domain with Business Process Management (BPM) and I had to start from scratch.

I'll go through the installation sequence here with some details. The way to run all products (SOA, BPM, OSB and MFT) in cluster is as follows:

• First, consider using the Enterprise Deployment workbook to capture all the details of the installation.
• Provision two servers: I'm running these on Supercluster so that's Solaris 11.4.
• Provision shared storage: That is to install all products on the shared storage (/u01) provided by the ZFS over IB. The domain configuration is kept local (/u02)
• The FMW/WLS configuration consists of provisioning two domains - one for SOA/OSB/BPM and one for MFT.
• Provision 2x Virtual IPs: That is to allow the Admin servers to move to second server should the first one fail. You need to manually failover the VIP. One VIP is for the SOA Admin server and one for MFT Admin server.
• Download the software stack from edelivery: Each product is distributed separately: Infra, SOA/BPM, OSB and MFT.
• Download the latest certified JDK - at the time of writing I used JDK version 1.8.0_261.
• You first install the FMW Infrastructure and then each product within the same directory.
• Install all the products using silent mode.
• Run Repository Creation Utility (RCU) to create all the schemas - select all schemas at the components page and pick up an unique prefix for your schemas.
• Create the initial infrastructure domain for SOA: That is the main part of the installation where you configure the managed servers, clusters and machines.
• There are few post steps after that: configure node manager so that it can start admin and managed servers, create boot.properties so that managed server can start automatically and finally move the MSERVER_HOME to a local storage using the pack and unpack commands.
• Then, you extend the domain for with OSB and BPM by running config.sh and propagating the changes to second host.

Following the guide was very easy and provides extensive information on design, preparation and execution. However, when I got to the point to extend the domain with BPM I got the following error:

It turns out that in 12.2.1.4 there was an internal stack change and one cannot extend SOA domain with BPM when Reference Configuration domain was used.

I found this in the SOA 12.2.1.4 installation guide, remember I'm using the EDG 12.2.1.3:

Note: A Reference Configuration domain cannot be extended to BPM. You need to create a Classic domain to extend to BPM.

I had to start from scratch and create classic domain instead of reference configuration domain. You do this on the Configuration Template for Oracle SOA Suite step of domain configuration. You need to do the same for OSB and BPM - use the classic components.

Now that you have an operational SOA domain you need to create a new one for MFT. To do so, follow the same steps as for creating the initial infrastructure domain. That's well documented in the EDG. Keep in mind that you need a new prefix for MFT schemas and bind the admin server on its own VIP address.

You already have node manager so all you need to do is add the MFT domain directory to node manager configuration to allow it to manage the domain.

Here are the links to the EDG guide and compatibility matrix:
Enterprise Deployment Guide for Oracle SOA Suite
Oracle Fusion Middleware System Requirements and Specifications
Oracle Fusion Middleware Supported System Configurations
Planning Workbook for Business Intelligence [XLS]

Here are few references to useful MOS notes:
How Many Nodemanagers are Required to Run a SOA Suite Domain Based on the Enterprise Deployment Guide (Doc ID 1605739.1)
Step by step demo configuration of a simple clustered domain with Oracle SOA, BPM, and OSB on 12.2.1.4.0 (Doc ID 2684414.1)

Hope it helps.

I patched Exadata recently, it was X5-2 running ESS 18.1.21 and 12.1.0.2.191015 BP. I applied latest available at that time - ESS 18.1.24 and 12.1.0.2.200114. And while everything was fine with storage cells, compute nodes and GI patching, the DB post-patching failed during the datapatch, shortly after running datapatch I got the following error:

/u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_221343_2020_03_06_18_49_03/sqlpatch_invocation.log:
Patch 30372179 rollback (pdb CDB$ROOT): WITH ERRORS
  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/30372179/23142433/30372179_rollback_PDB_CDBROOT_2020Mar06_18_51_42.log (errors)
    Error at line 5480: ORA-46385: Message 46385 not found;  product=RDBMS; facility=ORA

So the patch 30372179 failed to rollback. This patch was a merge request actually:
Patch 30372179: MERGE REQUEST ON TOP OF DATABASE BP 12.1.0.2.191015 FOR BUGS 29990674

The patch will fix two bugs 23604553 and 25985768. The first one is related to pre-12.1 database upgrades and can be ignored but the second is related to unified auditing:
Patch 25985768: TRANSFER_UNIFIED_AUDIT_RECORDS PROCEDURE FAILS WITH ORA-46385 ON RAC

Now, when you rollback this patch it will try to drop the AUD$ table and that's where it fails during rollback:

Naturally checking the logfile shows what the issue is:

vi /u01/app/oracle/cfgtoollogs/sqlpatch/30372179/23142433/30372179_rollback_ELHUBT2_CDBROOT_2020Mar06_18_51_42.log
....
SQL> -- Drop the unified audit internal table
SQL> drop table audsys.aud$unified;
drop table audsys.aud$unified
                  *
ERROR at line 1:
ORA-46385: Message 46385 not found;  product=RDBMS; facility=ORA
; arguments: [AUDSYS] [AUD$UNIFIED]
....

The solution is very simple and that is to comment the drop command on line 42 in the rollback SQL script located under the ORACLE_HOME:

$ORACLE_HOME/sqlpatch/**/rollback_files/rdbms/admin/cataudit_bug****_rollback.sql

You are probably reading this because datapatch failed already ? Don't worry just re-run datapatch and this time it will rollback the patch successfully. If you have happen to be reading this prior datapatch then make sure you comment the drop command in advance.

The reason for the rollback of this patch was because of the conflict with 12.1.0.2.200114. If you still want to apply the patch afterwards please note that the bug 25985768 has been replaced by new bug 28186466 and you will need the following patch which is already available for 12.1.0.2.200114:
Patch 28186466: DBMS_AUDIT_MGMT.TRANSFER_UNIFIED_AUDIT_RECORDS NOT FUNCTIONING

For more information refer to the MOS note:
Rollback Datapatch 25985768 Fails on Drop Table audsys.aud$unified (Doc ID 2573394.1)

Also, the following MOS note is very useful to track this bug as it's been superseded four times now:
Performance Issues While Monitoring the Unified Audit Trail of an Oracle12c Database (Doc ID 2063340.1)

I used the database cloud backup module to take a backup of a local database and restore it in the cloud.

A good start would be this link to the documentation.

The installation is really simple, download the installation from here, unzip the file and install the module. I've cut most of the output to keep it simple and focus on the actual steps.

Now here's the fun bit, it took me almost than an hour to install it and that's because of the values I had to pass in the host parameter:

java -jar opc_install.jar -host https://em2.storage.oraclecloud.com/v1/Storage-a468785 -opcId 'svetoslav.gyurov@redstk.com' -opcPass 'welcome1' -walletDir $ORACLE_HOME/opc -libDir $ORACLE_HOME/lib

where em2 is EMEA Commecial 2 data centre, a468785 is your indentity domain and then you supply your crentials as well.

That's it, once installed you can now run backups of your database to the cloud:

RMAN> set encryption on identified by welcome1 only;
RMAN> run {
allocate CHANNEL c1 DEVICE TYPE sbt PARMS='SBT_LIBRARY=libopc.so, SBT_PARMS=OPC_PFILE=/u01/app/oracle/product/11.2.0.4/db_1/dbs/opcorcl.ora)';
backup database;
}

Keep in mind that the cloud backup module requires an encryption hence you need to set a password on the backup at least or you'll get the following error:

ORA-27030: skgfwrt: sbtwrite2 returned error
ORA-19511: non RMAN, but media manager or vendor specific failure, error text:
KBHS-01602: backup piece 0qkqgtua_1_1 is not encrypted

Now, on the other host, start a dummy instance and restore the spfile file first:

RMAN> set dencryption on identified by welcome1;
RMAN> set dbid=1000458402;
RMAN> run {
allocate CHANNEL c1 DEVICE TYPE sbt PARMS='SBT_LIBRARY=libopc.so, SBT_PARMS=OPC_PFILE=/u01/app/oracle/product/11.2.0.4/db_1/dbs/opcorcl.ora)';
restore spfile to pfile '/u01/app/oracle/product/11.2.0.4/db_1/dbs/initorcl.ora' from autobackup;
}

Then restart the instance in nomount state with the new spfile and restore the database, as simple as that:

RMAN> set dencryption on identified by welcome1;
RMAN> set dbid=1000458402;
RMAN> run {
allocate CHANNEL c1 DEVICE TYPE sbt PARMS='SBT_LIBRARY=libopc.so, SBT_PARMS=(OPC_PFILE=/u01/app/oracle/product/11.2.0.4/db_1/dbs/opcorcl.ora)';
restore controlfile from autobackup;
alter database mount;
restore database;
}

If you got a relatively small database estate that's a good way to backup your database to offside location. It my case it was a bit slow, it took about 30 minutes to backup 4GB database but the restore within cloud was quicker.

We live in exciting times, Oracle Database 12.2 for Exadata was released earlier today.

The 12.2 database was already available on the Exadata Express Cloud Service and Database as a service for a few months now.

Today, it has been released for Exadata on-premises, five days earlier than the initial Oracle announcement of 15th Feb.

The documentation suggests that to run 12.2 database you need to run at least 12.1.2.2.1 Exadata storage software but better go with the recommended version of 12.2.1.1.0 which was released just a few days ago. Here are few notes on running 12.2 database on Exadata:

Recommended Exadata storage software: 12.2.1.1.0 or higher
Supported Exadata storage software: 12.1.2.2.1 or higher

Full Exadata offload functionality for Database 12.2, and IORM support for Database 12.2 container databases and pluggable databases requires Exadata 12.2.1.1.0 or higher.

Exadata Storage Server version 12.2.1.1.0 will be required for full Exadata functionality including  'Smart Scan offloaded filtering', 'storage indexes' and'

Current Oracle Database and Grid Infrastructure version must be 11.2.0.3, 11.2.0.4, 12.1.0.1 or 12.1.0.2.  Upgrades from 11.2.0.1 or 11.2.0.2 directly to 12.2.0.1 are not supported.

There is a completely new note on how to upgrade to 12.2 GI and RDBMS on Exadata:

12.2 Grid Infrastructure and Database Upgrade steps for Exadata Database Machine running 11.2.0.3 and later on Oracle Linux (Doc ID 2111010.1)

The 12.2 GI and RDBMS binaries are available from MOS as well as edelivery:

Patch 25528839: Grid Software clone version 12.2.0.1.0
Patch 25528830: Database Software clone version 12.2.0.1.0

The recommended Exadata storage software for running 12.2 RDBMS:

Exadata 12.2.1.1.0 release and patch (21052028) (Doc ID 2207148.1)

For more details about 12.2.1.1.0 Exadata storage software refer to this slide deck and of course a link to the 12.2 documentation here as we all need to start getting familiar with it.

Also, last week Oracle released the February version of OEDA to support the new Exadata SL6 hardware. It does not support 12.2 database yet and I guess we'll see another release in February to support the 12.2 GI and RDBMS.

Happy upgrading! :)

This might not be relevant anymore, shame on me for keeping it draft for few months. However, there are still people running an older versions of Exadata storage software and it might still help someone out there.

With the release of Exadata storage software 12.1.2.2.0, Oracle announced that some 32bit (i686) packages will be removed from the OS as part of the upgrade.

This happened to me in the summer last year (blog post) and I thought back then that someone messed up the dependencies. After seeing it again for another customer a month later I thought it might be something else. So after checking the release notes for all the recent patches, I found this for the 12.1.2.2.0 release:

Note that several i686 packages may be removed from the database note when being updated. Run dbnodeupdate.sh with -N flag at prereq check time to see exactly what rpms will be removed from your system.

Now, this will be all ok if you haven't installed any additional packages. If you, however, like many other customers has packages like LDAP or Kerberos then your dbnodeupdate pre-check will fail with “Minimum' dependency check failed." and broken dependencies since all i686 package will be removed as part of dbnodeupdate pre-check.

The way around that is to run dbnodeupdate with -N flag and check the logs of what packages will be removed and what will be impacted. Then manually remove any packages you installed manually. After the Exadata storage software update, you'd need to install the relevant version of the packages again.

Having said that I need to mention the below note on what software is allowed to install on Exadata:

Is it acceptable / supported to install additional or 3rd party software on Exadata machines and how to check for conflicts? (Doc ID 1541428.1)

It's been a busy month - five Exadata deployments in the past three weeks and new personal best - 2x Exadata X6-2 Eighth Racks with CoD and storage upgrade deployed in only 6hrs!

An issue I encountered with the first deployment was that onecommand wouldn't change the storage cells names. The default cell names (not hostnames!) are based on where they are mounted within the rack and they are assigned by the elastic configuration script. The first cell name is ru02 (rack unit 02), the second cell is ru04, third is ru06 and so on.

Now, if you are familiar with the cell and grid disks you would know that their names are based on the cell name. In other words, I got my cell, grid and ASM disks with the wrong names. Exachk would report the following failures for every grid disk:

Grid Disk name DATA01_CD_00_ru02 does not have cell name (exa01cel01) suffix
Naming convention not used. Cannot proceed further with
automating checks and repair for bug 12433293

Apart from exachk complaining, I wouldn't feel comfortable with similar names on my Exadata.

Fortunately cell, grid and ASM disk names can be changed and here is how to do it:

Stop the cluster and CRS on each compute node:

/u01/app/12.1.0.2/grid/bin/crsctl stop cluster -all
/u01/app/12.1.0.2/grid/bin/crsctl stop crs

Login to each storage server and rename cell name, cell and grid disks, use the following to build the alter commands:

You don't need cell services shut but the grid disks shouldn't be in use i.e. make sure to stop the cluster first!

cell -e alter cell name=exa01cel01
for i in `cellcli -e list celldisk | awk '{print $1}'`; do echo "cellcli -e alter celldisk $i name=$i"; done | sed -e "s/ru02/exa01cel01/2"
for i in `cellcli -e list griddisk | awk '{print $1}'`; do echo "cellcli -e alter griddisk $i name=$i"; done | sed -e "s/ru02/exa01cel01/2"

If you get the following error restart the cell services and try again:

GridDisk DATA01_CD_00_ru02 alter failed for reason: CELL-02548: Grid disk is in use.

Start the cluster on each compute node:

/u01/app/12.1.0.2/grid/bin/crsctl start crs

We've got all cell and grid disks fixed, now we need to rename the ASM disks. To rename ASM disk you need to mount the diskgroup in restricted mode i.e. running on one node only and no one using it. If the diskgroup is not in restricted mode you'll get:

ORA-31020: The operation is not allowed, Reason: disk group is NOT mounted in RESTRICTED state.

Stop the second compute node, default dbm01 database and the MGMTDB database:

srvctl stop database -d dbm01
srvctl stop mgmtdb

Mount diskgroups in restricted mode:

If you are running 12.1.2.3.0+ and high redundancy DATA diskgroup, it is VERY likely that the voting disks are in the DATA diskgroup. Because of that, you wouldn't be able to dismount the diskgroup. The only way I found around that was to force stop ASM and start it manually in a restricted mode:

srvctl stop asm -n exa01db01 -f
sqlplus / as sysasm
startup mount restricted

alter diskgroup all dismount;
alter diskgroup data01 mount restricted;
alter diskgroup reco01 mount restricted;
alter diskgroup dbfs_dg mount restricted;

Rename the ASM disks, use the following build the alter commands:

select 'alter diskgroup ' || g.name || ' rename disk ''' || d.name || ''' to ''' || REPLACE(d.name,'RU02','exa01cel01')  || ''';' from v$asm_disk d, v$asm_diskgroup g where d.group_number=g.group_number and d.name like '%RU02%';

select 'alter diskgroup ' || g.name || ' rename disk ''' || d.name || ''' to ''' || REPLACE(d.name,'RU04','exa01cel03') || ''';' from v$asm_disk d, v$asm_diskgroup g where d.group_number=g.group_number and d.name like '%RU04%';

select 'alter diskgroup ' || g.name || ' rename disk ''' || d.name || ''' to ''' || REPLACE(d.name,'RU06','exa01cel03') || ''';' from v$asm_disk d, v$asm_diskgroup g where d.group_number=g.group_number and d.name like '%RU06%';

Finally stop and start CRS on both nodes.

It's only when I thought everything was ok I discovered one more reference to those pesky names. These were the fail group names which again are based on the storage cell name. Following will make it more clear:

select group_number,failgroup,mode_status,count(*) from v$asm_disk where group_number > 0 group by group_number,failgroup,mode_status;

GROUP_NUMBER FAILGROUP                      MODE_ST   COUNT(*)
------------ ------------------------------ ------- ----------
           1 RU02                           ONLINE          12
           1 RU04                           ONLINE          12
           1 RU06                           ONLINE          12
           1 EXA01DB01                  ONLINE           1
           1 EXA01DB02                  ONLINE           1
           2 RU02                           ONLINE          10
           2 RU04                           ONLINE          10
           2 RU06                           ONLINE          10
           3 RU02                           ONLINE          12
           3 RU04                           ONLINE          12
           3 RU06                           ONLINE          12

For each diskgroup we've got three fail groups (three storage cells). The other two fail groups EXA01DB01 and EXA01DB02 are the quorum disks.

Unfortunately, you cannot rename failgroups in ASM. My immediate thought was to drop each failgroup and add it back with the intention that it will resolve the problem. Unfortunately, since this was a quarter rack I couldn't do it, here's an excerpt from the documentation:

If a disk group is configured as high redundancy, then you can do this procedure on a Half Rack or greater. You will not be able to do this procedure on a Quarter Rack or smaller with high redundancy disk groups because ASM will not allow you to drop a failure group such that only one copy of the data remains (you'll get an ORA-15067 error).

The last option was to recreate the diskgroups. I've done this many times before when the compatible.rdbms parameter was set to too high and I had to install some earlier version of 11.2. However, since oracle decided to move the voting disks to DATA this became a bit harder. I couldn't drop DBFS_DG because that's where the MGMTDB was created, I couldn't drop DATA01 either because of the voting disks and some parameter files. I could have renamed RECO01 diskgroup but decided to keep it "consistently wrong" across all three diskgroups.

Fortunately, this behvaiour might change with the January 2017 release of OEDA. The following bug fix suggests that DBFS_DG will always be configured as high redundancy and host the voting disks:

24329542: oeda should make dbfs_dg as high redundancy and locate ocr/vote into dbfs_dg

There is also a feature request to support failgroup rename but it's not very popular, to be honest. Until we get this feature, exachk will report the following failure:

failgroup name (RU02) for grid disk DATA01_CD_00_exa01cel01 is not cell name
Naming convention not used. Cannot proceed further with
automating checks and repair for bug 12433293

I've deployed five Exadata X6-2 machines so far and had this issue on all of them.

This issue seems to be caused a bug in OEDA. The storage cell names should have been changed as part of step "Create Cell Disks" of onecommand. I keep the logs from some older deployments where it's very clear that each cell was renamed as part of this step:

Initializing cells...

EXEC## |cellcli -e alter cell name = exa01cel01|exa01cel01.local.net|root|

I couldn't find that command in the logs of the deployements I did. Obviously, the solution for now, is to manually rename the cell before you run step "Create Cell Disks" of onecommand.

Update 04.02.2017:

This problem has been logged by someone else a month earlier under the following bug:

Bug 25317550 : OEDA FAILS TO SET CELL NAME RESULTING IN GRID DISK NAMES NOT HAVING RIGHT SUFFIX

I had the pleasure to deploy another Exadata in the first week of 2017 and got my first issue this year.

As we know starting with Exadata X5, Oracle introduced the concept of Elastic Configuration. Apart from allowing you to mix and match the number of compute nodes and storage cells they have also changed how the IP addresses are assigned on the admin (eth0) interface. Prior X5, Exadata had default IP addresses set at the factory in the range of IP addresses was 192.168.1.1 to 192.168.1.203 but since this could collide with the customer's network they changed the way those IPs are assigned. In short - the IP address on eth0 on the compute nodes and storage cells is assigned within 172.16.2.1 to 172.16.7.254 range. The first time node boots it will assign its hostname and IP address based on the IB ports its connected to.

Now to the real problem, I was doing the usual stuff - changing ILOMs, setting cisco and IB switches and was about to perform the initial elastic configuration (applyElasticConfig.sh) so I had upload all the files I need for the deployment on the first compute node. I've changed my laptop address to an IP within the same range and was surprised when I got connection timed out when I tried to ssh to the first compute node (172.16.2.44). I thought this was an unfortunate coincidence since I rebooted the IB switches almost at the time I powered on the compute nodes but I was wrong. For some reason, ALL servers did not get their eth0 IP addresses assigned hence they were not accessible.

I was very surprised to what's causing this issue and I've spent the afternoon troubleshooting it. I thought Oracle changed the way they assign the IP addresses but the scripts haven't been changed for a long time. It didn't take long before I find out what was causing it. Three lines in /sbin/ifup script were the reason eth0 interface wasn't up with the 172.2.16.X IP address:

if ip link show ${DEVICE} | grep -q "UP"; then
exit 0
fi

This line will check if the interface is UP before proceeding further and bring the interface up. Actually, the eth0 interface is brought UP already by the elastic configuration script to check if there is a link on the interface. Then at the end of the script when ifup script is invoked to bring the interface up it will stop the execution since the interface is already UP.

The solution is really simple - comment out the three lines (line 73-75) in /sbin/ifup script and reboot each node.

This wasn't the first X6 I deploy and I never had this problem before so I did some further investigation. The /sbin/ifup scripts is part of initscripts package. It turns out that the check for the interface being UP was introduced in one minor version of the package and then removed in the latest package. Unfortunately, the last entry in the Changelog is from Apr 12 2016 so that's not very helpful but here's a summary:

initscripts-9.03.53-1.0.1.el6.x86_64.rpm 11-May-2016 19:49 947.9 K <-- not affected
initscripts-9.03.53-1.0.1.el6_8.1.x86_64.rpm 12-Jul-2016 16:42 948.0 K <-- affected
initscripts-9.03.53-1.0.2.el6_8.1.x86_64.rpm 13-Jul-2016 08:26 948.1 K <-- affected
initscripts-9.03.53-1.0.3.el6_8.2.x86_64.rpm 23-Nov-2016 05:06 948.3 K <-- latest version, not affected

I had this problem on three Exadata machine so far. So, if you are doing deployment of new Exadata in the next few days or weeks it's very likely that you will be affected, unless your Exadata has been factory deployed after 23rd Nov 2016. That's the day when the latest initscripts package was released.

Update 24.01.2017:

This problem has been fixed in 12.1.2.3.3.161208:

25143049 - ADD NEW INITSCRIPTS RPM TO EXADATA REPOSITORY TO FIX IFUP ISSUE

Where the latest package has been added to the patch (initscripts-9.03.53-1.0.3.el6_8.2.x86_64.rpm)

For many customers I work with SSH won't be available between the OEM and monitoring hosts. Therefore you cannot push the management agent on to the host from the OEM console. Customer might have to raise CR to allow SSH between the hosts but this might take a while and it's really unnecessary.

In that case the management agent has to be installed in silent mode. That is when the agent won't be pushed from OEM to the host but pulled by the host and installed. There are thrее ways to do that - using AgentPull script, agentDeploy script or using RPM file.

When using AgentPull script, you download a script from the OMS and then run it. It will download the agent package and install it on the local host. Using the agentDeploy script is very similar but to obtain it you use EMCLI. The third method of using RPM file is similar - using EMCLI you download RPM file and install it on the system. These methods require HTTP/HTTPS access to the OMS, AgentDeploy and RPM file also require ELICLI to be installed. For that reason I always use AgentPull method since it's quicker and really straight forward. Another benefit of using AgentPull method is that if you don't have HTTP/HTTPS access to the OEM you can simply copy and paste the script.

Download a script from the OEM first, use curl or wget. The monitoring hosts usually don't have HTTP access but many of them do so by using HTTP proxy. Download the script and make it executable:

curl "https://oem13c.local.net:7802/em/install/getAgentImage" --insecure -o AgentPull.sh
chmod +x AgentPull.sh

If a proxy server is not available then you can simply copy and paste the script, the location of the script on the OEM server is:

/opt/app/oracle/em13c/middleware/oms/install/unix/scripts/AgentPull.sh.template

Make sure you edit the file change the oms host and oms port parameters.

Check what are the available platforms:

[oracle@exa01db01 ~]$ ./AgentPull.sh -showPlatforms

Platforms    Version
Linux x86-64    13.1.0.0.0

Run the script by suppling the sysman password, platform you want to install, agent registration password and the agent base directory:

[oracle@exa01db01 ~]$ ./AgentPull.sh LOGIN_USER=sysman LOGIN_PASSWORD=welcome1 PLATFORM="Linux x86-64" AGENT_REGISTRATION_PASSWORD=welcome1 AGENT_BASE_DIR=/u01/app/oracle/agent13c

It takes less than two minutes to install the agent and then at the end you'll see the following messages:

Agent Configuration completed successfully
The following configuration scripts need to be executed as the "root" user. Root script to run : /u01/app/oracle/agent13c/agent_13.1.0.0.0/root.sh
Waiting for agent targets to get promoted...
Successfully Promoted agent and its related targets to Management Agent

Now login as root and run the root.sh file.

To be honest, I use this method regardless of the circumstances, it's so much easier and faster.

Read this post if your Exadata compute nodes have 512/768GB of RAM or you plan to upgrade to the same.

There has been a lot of information about hugepages and I wouldn’t go into too much details. For efficiency, the (x86) CPU allocates RAM by chunks (pages) of 4K bytes and those pages can be swapped to disk. For example, if your SGA allocates 32GB this will take 8388608 pages and given that Page Table Entry consume 8bytes that’s 64MB to look-up. Hugepages, on the other hand, are 2M. Pages that are used as huge pages are reserved inside the kernel and cannot be used for other purposes. Huge pages cannot be swapped out under memory pressure, obviously there is decreased page table overhead and page lookups are not required since the pages are not subject to replacement. The bottom line is that you need to use them, especially now with the amount of RAM we get nowadays.

For every new Exadata deployment I usually set the amount of hugepages to 60% of the physical RAM:
256GB RAM = 150 GB (75k pages)
512GB RAM = 300 GB (150k pages)
768GB RAM = 460 GB (230k pages)

This allows databases to allocate SGA from the hugepages. If you want to allocate the exact number of hugepages that you need, Oracle has a script which will walk through all instances and give you the number of hugepages you need to set on the system, you can find the Doc ID in the reference below.

This also brings important point – to make sure your databases don’t allocate from both 4K and 2M pages make sure the parameter use_large_pages is set to ONLY for all databases. Starting with 11.2.0.3 (I think) you’ll find hugepages information in the alertlog when database starts:

************************ Large Pages Information *******************
Per process system memlock (soft) limit = 681 GB

Total Shared Global Region in Large Pages = 2050 MB (100%)

Large Pages used by this instance: 1025 (2050 MB)
Large Pages unused system wide = 202863 (396 GB)
Large Pages configured system wide = 230000 (449 GB)
Large Page size = 2048 KB

*******************************************************************

Now there is one more parameter you need to change if you deploy or upgrade Exadata with 512/768GB of RAM. That is the total amount of shared memory, in pages, that the system can use at one time or kernel.shmall. On Exadata, this parameter is set to 214G by default which is enough if your compute nodes have only 256GB of RAM. If the sum of all databases SGA memory is less than 214GB that’s ok but the moment you try to start another database you’ll get the following error:

Linux-x86_64 Error: 28: No space left on device

For that reason, if you deploy or upgrade Exadata with 512G/768GB of physical RAM make sure you upgrade kernel.shmall too!

Some Oracle docs suggest this parameter should be set to the half of the physical memory, other suggest it should be set to the all available memory. Here’s how to calculate it:

kernel.shmall = physical RAM size / pagesize

To get the pagesize run getconf PAGE_SIZE on the command prompt. You need to set shmall to at least match the size of the hugepages – because that’s where we’d allocate SGA memory from. So if you run Exadata with 768G of RAM and have 460 GB of hugepages you’ll set shmall to 120586240 (460GB / 4K pagesize).

Using HUGEPAGES does not alter the calculation for configuring shmall!

Reference:

HugePages on Linux: What It Is… and What It Is Not… (Doc ID 361323.1)
Upon startup of Linux database get ORA-27102: out of memory Linux-X86_64 Error: 28: No space left on device (Doc ID 301830.1)
Oracle Linux: Shell Script to Calculate Values Recommended Linux HugePages / HugeTLB Configuration (Doc ID 401749.1)