Then six months later, problem showed up again. The new version of the application was calling another web service, which obviously was missing from the hosts file and this time I decided to investigate the problem and find out what’s really happening. After I received the email I immediately logged in to the server and fired several nslookups and ping requests to the host that was causing problems, both were successful and returned correct result.  I’ve double checked hosts file, nsswitch.conf file and all the network settings, everything was correct. Meanwhile the Weblogic server log kept getting java.net.UnknownHostException for the very same host.

Obviously the problem required different approach. I’ve found useful Java procedure to call the function getByName and in some way simulate the application behavior, webservice hostname was intentionally changed,  this is the procedure:

[root@srv tmp]# cd /tmp/
cat > DomainResolutionTest.java
java.net.InetAddress;
import java.net.UnknownHostException;
import java.io.PrintWriter;
import java.io.StringWriter;

public class DomainResolutionTest {

public static void main(String[] args) {
if (args.length == 0) args = new String[] { "sve.to" };

try {
InetAddress ip = InetAddress.getByName(args[0]);
System.out.println(ip.toString());
}catch (UnknownHostException uhx) {
System.out.println("ERROR: " + uhx.getMessage() + "\n" + getStackTrace(uhx));
Throwable cause = uhx.getCause();
if (cause != null) System.out.println("CAUSE: " + cause.getMessage());
}

}

public static String getStackTrace(Throwable t)
{
StringWriter sw = new StringWriter();
PrintWriter pw = new PrintWriter(sw, true);
t.printStackTrace(pw);
pw.flush();
sw.flush();
return sw.toString();
}

}

Then just compile the procedure and execute it:

[root@srv tmp]# javac DomainResolutionTest.java
[root@srv tmp]# java DomainResolutionTest
sve.to/95.154.250.125
[root@srv tmp]# java DomainResolutionTest
sve.to/95.154.250.125

Running the procedure several times returned the correct address and no error occurred, but looping the procedure for some time returned the exception I was looking for:

while 1>0; do java DomainResolutionTest; done > 1
^C
[root@srv tmp]# wc -l 2
2648 2
[root@srv tmp]# grep Unknown 2
java.net.UnknownHostException: sve.to
[root@srv tmp]# less 2
......
sve.to/95.154.250.125
ERROR: sve.to
java.net.UnknownHostException: sve.to
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:849)
at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1202)
at java.net.InetAddress.getAllByName0(InetAddress.java:1153)
at java.net.InetAddress.getAllByName(InetAddress.java:1083)
at java.net.InetAddress.getAllByName(InetAddress.java:1019)
at java.net.InetAddress.getByName(InetAddress.java:969)
at DomainResolutionTest.main(DomainResolutionTest.java:12)

sve.to/95.154.250.125
....

From what I understand is that Java process is trying to lookup the IP address of the requested host, but first it takes all IP address of the local interfaces (eth0 and lo) including their default IPV6 address and then try to resolve these IP addresses to hostnames. Although I didn’t configured IPV6 addresses for the interfaces, they already had default ones. This is because the OS had IPV6 enabled by default and respectively the interfaces got default IPV6 addresses. During the installation I removed localhost6 (::1) record from hosts file, which later caused this error and also missing record for eth0 IP address.

The problem may be that the JVM performs both IPv6 and IPv4 queries and if the DNS server is not configured to handle IPv6 queries, the application might issue an unknown host exception. If the DNS is not configured to handle IPv6 queries properly, the application must wait for the IPv6 query to time out.  The workaround for this is to make Java use only IPV4 stack and run Java process with -Djava.net.preferIPv4Stack=true parameter. This will make Java process to prefer running in IPV4, thus avoiding the error for IPV6 look up.  Unfortunately running the above Java procedure with this parameter again returned UnknownHostException.

It looks like genuine bug with IPv6 in Java, I also saw few bugs opened at Sun regarding this Java behavior, but there was no solution. Finally after adding hostname for local interface’s IPV6 addresses in host file, the exceptions disappeared:

::1                                          localhost6
fe80::20c:29ff:fe36:4144                     srv6

So for the future installations I’ll be explicitly disabling IPV6 on the installed systems. The easiest way to do that is like this:

cat >> /etc/sysctl.conf
#disable all ipv6 capabilities on a kernel level
sysctl net.ipv6.conf.all.disable_ipv6 = 1

Regards,
Sve

When using APEX Listener with Oracle XE, APEX won’t work properly and white screen appears when APEX is open. This is because the APEX images are stored in the XML DB repository, but APEX Listener have to be run with parameter –apex-images pointing to directory containing the images at the filesystem. To solve this I downloaded the latest patch of APEX and copied the images from the patch.

If you have another database running on the same machine, keep in mind this.

Install Oracle 11g XE and update Oracle APEX to latest version:

1. Download Oracle Database Express Edition 11g Release 2 for Linux x64

2. Install Oracle 11g XE:
rpm -ivh oracle-xe-11.2.0-1.0.x86_64.rpm

3. Configure Express Edition:
/etc/init.d/oracle-xe configure
Port: 1522
Password: secret

4. Update APEX to 4.1

Download APEX 4.1
cd /tmp
unzip -q apex_4.1.zip
cd apex
sqlplus / as sysdba
@apexins SYSAUX SYSAUX TEMP /i/
@apxldimg.sql /tmp/apex

5. Update APEX to version 4.1.1
Download patch set 13331096 from MOS

Disable Oracle XML DB HTTP server:

SQL> EXEC DBMS_XDB.SETHTTPPORT(0);
PL/SQL procedure successfully completed.

SQL> COMMIT;
Commit complete.

SQL> SELECT DBMS_XDB.GETHTTPPORT FROM DUAL;
GETHTTPPORT
———–
0

Run apxpatch.sql to patch the system:

SQL> @apxpatch.sql

Update the Images Directory When Running the Embedded PL/SQL Gateway:

@apxldimg.sql /tmp/patch

Commit complete.

Once the update finished do not enable Oracle XML DB HTTP server, because we’ll be using Oracle APEX Listener, which will setup next.

Install APEX Listener 2.0.0

1. Download Oracle APEX Listener 2.0.0 beta

2. Download and install latest JRE 1.6 version, currently latest version is 1.6.34

Unpack to /opt/jre1.6.0_34

3. Unlock and set password for apex_public_user at the Oracle XE database:
alter user APEX_PUBLIC_USER account unlock;
alter user APEX_PUBLIC_USER identified by secret;

4. Patch Oracle APEX to support RESTful  Services:
cd /oracle/apxlsnr/apex_patch/
sqlplus / as sysdba @catpatch.sql

Set passwords for both users APEX_LISTENER and APEX_REST_PUBLIC_USER.

5. Install Oracle APEX Listener:
mkdir /oracle/apxlsnr/
cd /oracle/apxlsnr/
unzip apex_listener.2.0.0.215.16.35.zip

Now this is tricky, for XE edition the images are kept in the XML DB repository, so images have to be copied from the patch to the listener home:
cp /tmp/patch/images .

6. Configure Oracle APEX Listener:
export JAVA_HOME=/opt/jre1.6.0_34
export PATH=$JAVA_HOME/bin:$PATH

Set APEX listener config dir:
java -jar apex.war configdir $PWD/config

Configure the listener:
java -jar apex.war

Once configuration is complete, listener is started. It has to be stopped and run with appropriate parameters, use Ctrl-C to stop it.

7. Finally start the listener:
java -jar apex.war standalone –apex-images /oracle/apxlsnr/images

In case you want to run it in background here’s how to do it:
nohup java -jar apex.war standalone –apex-images /oracle/apxlsnr/images > apxlsnr.log &

Periodically I was seeing exceptions like these:
ConnectionPoolException [error=BAD_CONFIGURATION]
…
Caused by: oracle.ucp.UniversalConnectionPoolException: Universal Connection Pool already exists in the Universal Connection Pool Manager. Universal Connection Pool cannot be added to the Universal Connection Pool Manager

I found that if APEX Listener is not configured with RESTful Services then these messages appeared in the log and could be safety ignored.

Regards,
Sve

Just a little technical notes – Oracle VM (xen) has three main setup configurations within /etc/xen/xend-config.sxp:

Bridge Networking – this configuration is configured by default and it’s simplest to configure. Using this type of networking means that the VM guest should have IP from the same network as the VM host. Another thing is that the VM guest could take advantage of DHCP, if any. The following lines should be uncommented in /etc/xen/xend-config.sxp:
(network-script network-bridge)
(vif-script vif-bridge)

Routed Networking with NAT – this configuration is most common where a private LAN must be used, for example you have a VM host running  on your notebook and you can’t get another IP from corporate or lab network. For this you have to setup private LAN and NAT the VM guests so they can access the rest of the network. The following lines should be uncommented in /etc/xen/xend-config.sxp:
(network-script network-nat)
(vif-script vif-nat)

Two-way Routed Network – this configuration requires more manual steps, but offers greater flexibility. This one is exactly the same at the second one, except the fact that VM guests are exposed on the external network. For example when VM guest make connection to external machine, its original IP is seen. The following lines should be uncommented in /etc/xen/xend-config.sxp:
(network-script network-route)
(vif-script vif-route)

Typically only one of the above can be used at one time and selection and choice depends on the network setup. For second and third configurations to work, a “route” must be added to the Default Gateway. For example if my Oracle VM host has an IP address 192.168.143.10, then on the default gateway (192.168.143.1) a route has to be added to explicitly route all connection requests to my VM guests through my VM host. Something like that:
route add -net 10.0.1.0 netmask 255.255.255.0 gw 192.168.143.10

Now back to the case itself. Each of the RAC nodes had two NICs – one for the public connections and one for the private, which is used by GI an RAC. The public network was 10.0.1.X and private 192.168.1.X. What I wanted was to run the VM guests at my lab and access them directly with IP addresses from the lab network, which was 192.168.143.X. As we know the default network configuration is to use bridged networking so I went with this one. Having the vm guests config files all I had to do was to change the first address of every guest:

From:
vif = ['mac=00:16:3e:22:0d:04, ip=10.0.1.11, bridge=xenbr0', 'mac=00:16:3e:22:0d:14, ip=192.168.1.11',]

To:
vif = ['mac=00:16:3e:22:0d:04, ip=192.168.143.151, bridge=xenbr0', 'mac=00:16:3e:22:0d:14, ip=192.168.1.11',]

This turned to be real nightmare, I’ve spent half a day looking why my VM gusts doesn’t have access to the lab network. They had access to VM host, but not to the outside world. Maybe because I’m running Oracle VM on top of VMWare, but finally I  gave up this configuration.

Thus I had to use one of the other two network configurations – Routed Networking with NAT OR Two-way Routed Network. Either case I didn’t have access to the default gateway and would not be able to put static route to my VM guests.

Here is how I solved this – to run three nodes RAC on Oracle VM Server 2.2.1, keep their original network configuration and access them with IP address from my lab network (192.168.143.X). I’ve put logical IP’s of the VM guests on the VM host using ip (ifconfig could also be used) and then using iptables change packet destination to the VM guests themselves (10.0.1.X).

1. Change Oracle VM configuration to Two-way Routed Network, comment the lines for default bridge configuration and remove comments for routed networking:
(network-script network-route)
(vif-script vif-route)

2. Configure VM host itself for forwarding:
echo 1 > /proc/sys/net/ipv4/conf/all/proxy_arp
iptables -t nat -A POSTROUTING -s 10.0.1.0 -j MASQUERADE

3. Set network alias with the IP address that you want to use for the VM guests:
ip addr add 192.168.143.151/32 dev eth0:1
ip addr add 192.168.143.152/32 dev eth0:2
ip addr add 192.168.143.153/32 dev eth0:3

4. Create iptables rules in PREROUTING chain that will redirect the request to VM guests original IPs once it receive it on the lab network IP:
iptables -t nat -A PREROUTING -d 192.168.143.151 -i eth0 -j DNAT –to-destination 10.0.1.11
iptables -t nat -A PREROUTING -d 192.168.143.152 -i eth0 -j DNAT –to-destination 10.0.1.12
iptables -t nat -A PREROUTING -d 192.168.143.153 -i eth0 -j DNAT –to-destination 10.0.1.13

5. Just untar the VM guest in /OVS/running_pool/

[root@ovm22 running_pool]# ls -al /OVS/running_pool/dbnode1/
total 26358330
drwxr-xr-x 2 root root        3896 Aug  6 17:27 .
drwxrwxrwx 6 root root        3896 Aug  3 11:18 ..
-rw-r–r– 1 root root  2294367596 May 16  17:27 swap.img
-rw-r–r– 1 root root  4589434792 May 16  17:27 system.img
-rw-r–r– 1 root root 20107128360 May 16  17:27 u01.img
-rw-r–r– 1 root root         436 Aug 6 11:20 vm.cfg

6. Run the guest:
xm create /OVS/running_pool/dbnode1/vm.cfg

Now I have a three node RAC, nodes have their original public IPs and I can access them using my lab network IPs. The mapping is like this:

Request to 192.168.143.151 –> the IP address is up on the VM host –> on the VM host iptables takes action –> packet destination IP address is changed to 10.0.1.11 –> static route is already in place at VM host routing packet to the vif interface of the VM guest.

Now I can access my dbnode1 (10.0.1.11) directly with its lab network IP 192.168.143.151.

Useful links:
http://wiki.kartbuilding.net/index.php/Xen_Networking
http://wiki.xensource.com/xenwiki/XenNetworking

Regards,
Sve

Unfortunately two weeks passed and by the time (this morning) I had to start with the DG setup the FRA (2TB big) got exhausted, because of too many archivelogs. When I saw this the first thing it came to my mind was to delete the backup as it was too big and I already had it at the disaster site!?. Deleting archivelogs was not an option as I need these archivelogs so the standby could catch up with the primary. So what I’ve done was to delete the backup without thinking and then moved on to duplicate the primary database for standby.

I’ve setup the standby instance and when issued “DUPLICATE TARGET DATABASE FOR STANDBY NOFILENAMECHECK DORECOVER;” I got the following errors:

RMAN-03002: failure of Duplicate Db command at 06/05/2012 11:12:04
RMAN-03015: error occurred in stored script Memory Script
RMAN-06136: ORACLE error from auxiliary database: ORA-01180: can not create datafile 1
ORA-01110: data file 1: '+DATA/orcl/datafile/system.347.666704609'

I was surprised by this error and started thinking if there wasn’t a problem with the storage (it happened twice before to have read only disks), but this was not the case. Once the diskgroup is mounted it means that ASM can read/write to its disks.

It’s obvious what my mistake is, but it took me a while until I realize what I’ve done. Deleting backup at the primary database means that it no longer knows for my backup and for that reason I could not clone the primary database. That’s why I got error that datafile 1 could not be created, simply it has no backups to restore it from. Thinking now what if the backup was located at NFS share, then I would definitely not delete it, but maybe move the archivelogs and manually register them later on the standby.

So now I started a new backup, waiting for it to finish and got replicated to the disaster site.

Regards,
Sve

UDE-00008: operation generated ORACLE error 31623
ORA-31623: a job is not attached to this session via the specified handle
ORA-06512: at "SYS.DBMS_DATAPUMP", line 2315
ORA-06512: at "SYS.DBMS_DATAPUMP", line 3185
ORA-06512: at line 1

These gave me no sign of what could be the problem, but when I tried to export only the metadata I got ORA-00600 which helped me finding the solution:

Tue May 29 12:21:13 2012
Errors in file /oracle/admin/brego/udump/brego_ora_21785.trc:
ORA-00600: internal error code, arguments: [kwqbgqc: bad state], [1], [1], [], [], [], [], []
Tue May 29 12:22:01 2012
kwqiclode Warning : 600 happened during Queue                         load

The error was caused by invalid DataPump queue, refer to the following MOS notes to resolve the error:
DataPump Import (IMPDP) Fails With Errors UDI-8 OCI-22303 ORA-600 [kwqbgqc: bad state] [ID 1086334.1]
Errors ORA-31623 And ORA-600 [kwqbgqc: bad state] During DataPump Export Or Import [ID 754401.1]

Regards,
Sve

In file /etc/init.d/oracle-xe, at line 556 the code is failing:

+ status='oracle 2889 1 0 May13 ? 00:00:05 /oracle/ora112se/bin/tnslsnr LISTENER -inherit'
+ '[' 'oracle 2889 1 0 May13 ? 00:00:05 /oracle/ora112se/bin/tnslsnr LISTENER -inherit' == '' ']'

As you can see I have another database running and because of the XE startup script found the keyword LISTENER it’s supposing that the XE LISTENER is already running.

Simply changing line 556 from:

status=`ps -ef | grep tns | grep oracle`

To:

status=`ps -ef | grep tns | grep oracle | grep xe`

is fixing the error and now XE listener is started automatically during boot.

Regards,
Sve

The first step was to create form and report by using the web service. At this point web service request process is created where authentication and input parameter are described. The problem appears if you try to change any parameter and update the web service process. This is the error:

 Error error updating web service parameters
ORA-01403: no data found

To fix this, one option is to apply patch 12934733 on top of APEX 4.1. The other option is to apply latest patch set for APEX version 4.1.1, patch number 13331096.

At the time I got the error patch set wasn’t released yet and I went with the patch only to fix this issue. Later I’ve decided to update the APEX to latest version 4.1.1 and I’ll review the update process at glance.

To upgrade to APEX 4.1.1 make sure first to review the release notes here. The process is really simple and takes few minutes.

Before applying the patch make sure to prevent access to the APEX. In my case I’m using Oracle Database 11g Express Edition and I’m using Embedded PL/SQL gateway. Then apply the patch using apxpatch.sql and update the images directory. Because I’m using Express Edition, my images are stored in the XML DB repository and script apxldimg.sql has to be used to upload the new images within the repository.

Disabling Oracle XML DB HTTP Server:

SQL> SELECT DBMS_XDB.GETHTTPPORT FROM DUAL;

GETHTTPPORT
-----------
 0

SQL> EXEC DBMS_XDB.SETHTTPPORT(0);

PL/SQL procedure successfully completed.

SQL> COMMIT;

Commit complete.

SQL> SELECT DBMS_XDB.GETHTTPPORT FROM DUAL;

GETHTTPPORT
-----------
 0

Run apxpatch.sql to patch the system:

SQL> @apxpatch.sql

.......

timing for: Complete Patch
Elapsed: 00:06:25.48

Updating the Images Directory When Running the Embedded PL/SQL Gateway:

@apxldimg.sql /tmp/patch

.......

Commit complete.

timing for: Load Images
Elapsed: 00:04:12.56

Directory dropped.

Enabling Oracle XML DB HTTP Server:

SQL> EXEC DBMS_XDB.SETHTTPPORT(8080);

PL/SQL procedure successfully completed.

SQL> COMMIT;

Commit complete.

APEX is now updated to version 4.1.1

Regards,
Sve

I’m using Oracle Express Edition 11.2, which by default doesn’t have the APEXExport utility. That’s why you need first to create the directory holding it and then copy the utility there.

Create destination directory holding the utility and copy APEXExport.class there:

mkdir -p $ORACLE_HOME/utilities/oracle/apex

Create backup directory in the SVN project for holding the backup files:

[oracle@dbsrv ~]$ svn mkdir -m "Making a backup dir." http://192.168.8.34/svn/oracle/Corporate_Portal/trunk/Apex/Backup

Committed revision 33.

At this point you’ll be asked for password which must be cached in order the automated backup to work correctly. The password itself is written (I assume you’re logged in with oracle user) in file in this directory: /home/oracle/.subversion/auth/svn.simple/

Finally you need to decide where the backup directory will be checked out. This will be used to store all the APEX backups, which will be committed to the SVN repository:

cd /oracle
svn checkout http://192.168.8.34/svn/oracle/Corporate_Portal/trunk/Apex/Backup

This is the script for automating the backup of APEX applications, because the formatting is not correct here is download link for the script:

#!/bin/bash

### Automated APEX applications backup to Subversion (SVN), v1
### 2012-03-07
### Svetoslav Gyurov, http://sve.to

# Environment settings for Oracle database and Java
export ORACLE_HOME=/u01/app/oracle/product/11.2.0/xe
export JAVA_HOME=/oracle/ora112ee/jdk

export PATH=$ORACLE_HOME/bin:$PATH
export CLASSPATH=$ORACLE_HOME/jdbc/lib/ojdbc5.jar:$ORACLE_HOME/utilities

# How many days/backups should be kept back
export RETENTION=7
export COMPRESS=0

# SVN directory for Oracle APEX
SVN_DIR=/oracle/Backup

# Get current date and oldest backup
DATE=`date +%Y%m%d`
OLDEST_BACKUP=`date --date="-${RETENTION} days" +%Y%m%d`


# Change to SVN directory and export data
cd $SVN_DIR
$JAVA_HOME/bin/java oracle.apex.APEXExport -db 192.168.8.34:1522:XE -user sve -password secret -applicationid 100

exitValue=$?

# If APEXExport is successfully continue, otherwise fail with message
if [ $exitValue -eq 0 ];
then
svn update

for i in `ls -1 f*.sql`
do
if [ $COMPRESS -eq 1 ];
then
gzip $i
mv $i.gz ${DATE}_$i.gz
svn add ${DATE}_$i.gz

if [ -f ${OLDEST_BACKUP}_$i.gz ];
then
rm ${OLDEST_BACKUP}_$i.gz
svn remove ${OLDEST_BACKUP}_$i.gz
fi
else
mv $i ${DATE}_$i
svn add ${DATE}_$i

if [ -f ${OLDEST_BACKUP}_$i ];
then
rm ${OLDEST_BACKUP}_$i
svn remove ${OLDEST_BACKUP}_$i
fi
fi
done

svn commit -m "Daily APEX backup completed and committed."

echo Daily APEX backup completed.

else
echo "APEX applications could not be exported"
fi

Of course there are few parameters which need to be set. These are ORACLE_HOME, JAVA_HOME, RETENTION period defines how many backups to be kept back and COMPRESS defines whether the exported applications be compressed with gzip. Also don’t forget to change the application id, you could also put the whole workspace to be backed up.

Finally put the script at the crontab:

31 1 * * * /home/oracle/apex_backup.sh > /dev/null 2>&1

To check whether the backup was successful run svn info or svn log within the backup directory or just browse the repository:

cd /oracle/Backup
[oracle@dbsvn Backup]$ svn info
Path: .
URL: http://192.168.8.34/svn/oracle/Corporate_Portal/trunk/Apex/Backup
Repository Root: http://192.168.8.34/svn/oracle
Repository UUID: 40fb9d41-50a5-4b10-b803-b133fb623816
Revision: 59
Node Kind: directory
Schedule: normal
Last Changed Author: oracle
Last Changed Rev: 59
Last Changed Date: 2012-03-07 01:31:07 +0200 (Wed, 07 Mar 2012)

[oracle@dbsrv Backup]$ svn log
------------------------------------------------------------------------
r59 | oracle | 2012-03-07 01:31:07 +0200 (Wed, 07 Mar 2012) | 1 line

Daily APEX backup completed and commited.
------------------------------------------------------------------------

Feel free to modify the script for your own needs.

Regards,
Sve

This happen when I had to shutdown one server for maintenance and then boot it again. Once the OS started I notice that the database didn’t ran automatically. When I try to run it manually I got the following error:

SQL> startup
 ORA-01078: failure in processing system parameters
 ORA-01565: error in identifying file '+DATA/orcl/spfileorcl.ora'
 ORA-17503: ksfdopn:2 Failed to open file +DATA/orcl/spfileorcl.ora
 ORA-15077: could not locate ASM instance serving a required diskgroup

The reason for this was that the ASM diskgroup wasn’t mounted, again I tried to mount it manually, when I got this error:

SQL> alter diskgroup data mount;
 alter diskgroup data mount
 *
 ERROR at line 1:
 ORA-15032: not all alterations performed
 ORA-15024: discovered duplicately numbered ASM disk 29

So how could this happen as I didn’t change any of the parameter and configuration, the database was started for just three months. Running quick script with kfed:

ls -1 /dev/oracle/*for i in `ls -1 /dev/oracle/*`; do echo /dev/oracle/$i; kfed read /dec/oracle/$i | grep kfdhdb.dskname; done

showed me that I there are really two duplicate disks. ASM does not support disks with the same disk names, even if this is the same disk, but accessed by different path. The duplicate disk was an old disk, which wasn’t removed at the time of the change, only its filename was changed so once ASM discovers disks in the future this one would be excluded.

Looking at the asm_diskstring I found that it has changed to ‘/dev/oracle/*’ and should be ‘/dev/oracle/orcl*’ and that’s why ASM was scanning all the disks. This made me think what could happen for the time database was running and why this happen.

After looking at the logs and made few tests I had a conclusion. At some point when the DBA added new disks to ASM he done it through the dbca and he changed the discovery string for unknown reason. What happens in background is that dbca, implicitly is recreating the ASM parameter file with the new discovery string entered by the DBA. This is why ASM started with wrong discovery string when I rebooted the server and it discovered more disks.

The database ran at 26 March and a month later at 29 April new disks were added to the ASM and its parameter was recreated, almost two months later at 7 July I turned it back with correct values:

-rw-r----- 1 oracle oinstall 1336 Jul 7 18:42 init.ora
-rw-r----- 1 oracle oinstall 1327 Apr 29 11:32 init.ora.662011221038

Regards,
Sve

This is a short note for installing Oracle Database with ASM under Oracle Enterprise Linux 6.x, which is not certified YET!

If you are running ASM with ASMLib you may get the following error when try to create ASM disk:

[root@db-app2 ~]# oracleasm createdisk DISK01 /dev/mapper/data01
Writing disk header: done
Instantiating disk: failed
Clearing disk header: done

Writing the header on the disk fails without obvious reason. Well, it turned out that Security Linux was blocking the access to the disk. Because configuration of SELinux is not part of the startup configuration any more I just forgot it.

To disable the SELinux edit file /etc/selinux/config and configure SELINUX=disabled. Then reboot your system and run getenforce to confirm that SELinux is disabled:

[root@db-app2 ~]# /usr/sbin/getenforce
Disabled

The ASM disk is created successfully this time:

[root@db-app2 ~]# oracleasm createdisk DISK01 /dev/mapper/data01
Writing disk header: done
Instantiating disk: done

Regards,
Sve

 WARNING: failed to read mirror side 1 of virtual extent 2917 logical extent 0 of file 271 in group [1.2242406296] from disk DATA_0000 allocation unit 24394 reason error; if possible,will try another mirror side
Errors in file /oracle/app/oracle/diag/rdbms/baandb/baandb/trace/baandb_ora_17785.trc:
WARNING: Read Failed. group:1 disk:0 AU:24394 offset:1007616 size:8192
WARNING: failed to read mirror side 1 of virtual extent 2917 logical extent 0 of file 271 in group [1.2242406296] from disk DATA_0000 allocation unit 24394 reason error; if possible,will try another mirror side
Errors in file /oracle/app/oracle/diag/rdbms/baandb/baandb/trace/baandb_ora_17785.trc: 

At first I though there is a storage problem, but looking at the ASM views everything seemed to be all right and these seemed to be false messages. I deleted all the trace files, but then few minutes later the file system became again full. It turned out that generated log per minute were more than 60MBor around 7GB for two hours, because of this huge number of messages the machine was already loaded.

Then after quick MOS search I found that this is a Bug 10422126: FAILED TO READ MORROR SIDE 1 and there is a 70KB patch for 11.2.0.2.

The following MOS notes are also useful:
WARNING: ‘Failed To Read Mirror Side 1′ continuously reported in the alert log [ID 1289905.1]
Huge number of alert log entries: ‘WARNING: IO Failed…’ ‘WARNING: failed to read mirror side 1 of virtual extent …’ [ID 1274852.1]

After applying the patch everything became normal and no more false messages appeared in the logs. The bug is fixed in 11.2.0.3.

Regards,
Sve

The system was running latest HP-UX 11.31 and we were going to install Oracle GI 11.2.0.2, the LUN was presented from HP EVA storage.

I couldn’t believe what they are saying and wanted them to show me what exactly they are doing. Unfortunately they were correct, after installing GI 11.2.0.2 software only, we tried to create an asm instance with asmca, but no disks were discovered although everything looked correct.

While I was looking around I remembered that the disk owner patch in HP-UX is a mandatory and it should be installed as the installation guide says this explicitly. I asked the customer and he said that all the required patches are installed, but when I checked the patch wasn’t  installed. The patch number as per installation guide is PHCO_41479, but the latest version is PHCO_41903. Also running kfed against disk on system on which the patch is not installed shows following:

KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

I installed the patch and double checked everything and thought that this could be the reason why we are not seeing the disk, so I try to discover the disk, but again without success. The disk couldn’t be seen at ASM so I had to go deeper and see what asmca was actually doing. For the purpose I had to trace the system calls and for HP-UX the utility capable of doing this was tusc. There is MOS note describing how to trace systems call and what utilities should be used with different unix distributions [ ID 110888.1].

I run asmca and then using tusc got attached to its process, then changed the discovery string, pointing exactly to the disk I would like to use (in my case /dev/rdisk/disk3). So this is the paragraph which makes sense to me:

access("/dev/rdisk/disk3", W_OK|R_OK) ........................................................................... = 0
.......
open("/dev/rdisk/disk3", O_RDONLY|O_NDELAY|0x800, 0) ............................................................ = 7
lseek(7, 8192, SEEK_SET) ........................................................................................ = 8192
read(7, "L V M R E C 0 1 \r/ % aeN e2\va0".., 1024) ............................................................. = 1024
lseek(7, 73728, SEEK_SET) ....................................................................................... = 73728
read(7, "L V M R E C 0 1 \r/ % aeN e2\va0".., 1024) ............................................................. = 1024
close(7) ........................................................................................................ = 0

The disk is first successfully tested for read and write access and it’s opened for read-only in non-blocking mode. Then first 1024 bytes are read from offset 8192 from /dev/rdisk/disk3. This looked like a LVM header, AHA! So it seems that the disk was once used as LVM Physical Volume. Although the disk is not part of any volume group it has a LVM header and that’s why asmca it not showing this disk as CANDIDATE. It turned out that storage admins did not recreate the virtual disk on the storage, but the LUN was once used for LVM on another server.

After doing dd on the disk now the header looks better and disk could be seen as CANDIDATE:

oracle@vm:/$ dd if=/dev/zero of=/dev/rdisk/disk3 bs=1024k count=10

Now tusc output shows that header is filled with zeros:

read(7, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0".., 1024) ........................................................ = 1024

Just for troubleshooting purpose I try to read the disk header with kfed, before and after showed the same error:

KFED-00322: file not found; arguments: [kfbtTraverseBlock] [Invalid OSM block type] [] [0]

If you are not sure whether the disk contains valuable information you could import the physical volume and activate the volume group. In my case I was sure that the disk should be deleted and simple dd do the job.

Regards,
Sve

I’m looking forward to meet and have a chat with these great people, of course with a lot of friends and have a nice and relaxing weekend.

This is the link to the conference event.

See you there

First you need to download the source of latest version of oci8 driver, here. Also you need to download and install the latest version of Oracle Instant Client Packages – Basic and SDK, here.

The next step is to install both packages on the system:

rpm -ivh oracle-instantclient11.2-basic-11.2.0.3.0-1.x86_64.rpm
rpm -ivh oracle-instantclient11.2-devel-11.2.0.3.0-1.x86_64.rpm

After installing the instant client, you have to build the oci8:

tar xfzv oci8-1.4.6.tgz
cd oci8-1.4.6
phpize
./configure -with-oci8=shared,instantclient,/usr/lib/oracle/11.2/client64/lib
make install

Module will be installed in PHP modules directory, which is /usr/lib64/php/modules/.

Edit php.init and add the following line to make oci8 driver to be loaded by PHP:
extension=oci8.so

After restarting the web server,  you now have Oracle support enabled in PHP. Oracle support could be tested from command line without restarting the web server, with this simple script:

cat > /tmp/oracle.php

<?php
$username = "scott";
$passwd = "tiger";
$db="(DESCRIPTION=
           (ADDRESS_LIST=
             (ADDRESS=(PROTOCOL=TCP)
               (HOST=192.168.8.36)(PORT=1521)
             )
           )
             (CONNECT_DATA=(SERVICE_NAME=orcl))
      )";
$conn = OCILogon($username,$passwd,$db);
if (!$conn)
{
    echo "Connection failed";
    echo "Error Message: [" . OCIError($conn) . "]";
    exit;
}
else
{
    echo "Connected!";
}

Running the script shows that it is connecting successfully to the database:

[root@app tmp]# php oracle.php
Connected!

There is also manual at PHP for installation and configuration of OCI8, which could be found here.

Regards,
Sve

After you successfully setup Subversion (here) it’s time to start integrating tools with SVN. Integration of Oracle SQL Data Modeler is really easy and it’s done in few simple steps:

1. Integrated Oracle SQL Data Modeler with SVN:
Start Oracle SQL Data Modeler
Go to View -> Team -> Versioning Navigator
At the Versioning Navigator window, right click on Subversion and then New Repository Connection.
Populate the fileds at new window Create Subversion Connection with repository URL, username, password and name for the repository.

2. Import files to the repository:
Once configured you have to import all the files on which you are working, including the database design to the repository. For this purpose you could use Data Modeler itself, but I prefer TortoiseSVN to create the repository structure and then import all the files in it. As I mention in preventions post, the best practice is to create three directories trunk, branch and tags and then import your working directories to trunk.

When these files are imported you should delete them from your hard drive and checkout the same files from the repository so you have a working copy.

3. Start using Oracle SQL Data Modeler and SVN:
Checking out from the repository could be done by TortoiseSVN or by Oracle SQL Data Modeler -> Versioning menu -> Checkout. Once you have a working copy you could open the design and start working on it.

Next time you make a change and save the design you’ll be able to commit these changes to the repository, again by going to Versioning menu and then Commit. Using comments is also good idea so you could easily track when and what changes you’ve made to the design.

Regards,
Sve

After running opatch apply I got that the following files are still active:
d:\app\11.2.0\grid\bin\oraclient11.dll
d:\app\11.2.0\grid\bin\orageneric11.dll
d:\app\11.2.0\grid\bin\orapls11.dll
d:\app\11.2.0\grid\bin\oracommon11.dll
d:\app\11.2.0\grid\bin\oci.dll
d:\app\11.2.0\grid\bin\orahasgen11.dll
d:\app\11.2.0\grid\bin\oraocr11.dll
d:\app\11.2.0\grid\bin\oraocrb11.dll
d:\app\11.2.0\grid\bin\oraocrutl11.dll
d:\app\11.2.0\grid\bin\mDNSResponder.exe
d:\app\11.2.0\grid\bin\ocssd.exe
d:\app\11.2.0\grid\bin\cssdagent.exe
d:\app\11.2.0\grid\bin\cssdmonitor.exe
d:\app\11.2.0\grid\bin\evmd.exe
d:\app\11.2.0\grid\bin\evmlogger.exe
d:\app\11.2.0\grid\bin\gipcd.exe
d:\app\11.2.0\grid\bin\gpnpd.exe
d:\app\11.2.0\grid\bin\octssd.exe

It’s unlikely to have something running, because I have stopped all GI processes. Again to find out which is the process holding the dll’s I’ve used ProcessExplorer. It seemed that process WmiPrvSE.exe had the dlls open:

Description of WMI:
The wmiprvse.exe file is otherwise known as Windows Management Instrumentation. It is a Microsoft Windows-based component that provides control and information about management in an enterprise environment. Developers use the wmiprvse.exe file in order to develop applications used for monitoring purposes.

For some reason WMI is holding the CRS dlls. Stop the WMI service or kill the process and this should release the lock on the drivers and allow the opatch to proceed.

Regards,
Sve

• All my files are kept at one place. At any time I could checkout the whole application and deploy it on different server.
• I have a history (revisions) of my changes. If I know something worked yesterday I could just checkout the file from yesterday and get it working.
• Using APEXExport I’m doing daily backups of my applications.
• Oracle SQL Developer and SQL Data Modeler are easy to integrate with SVN.
• When the time comes, I’ll create a branch, stable version of my application, by doing so I’ll have at any time a working and stable copy of my application.
• Having branches I could still develop and improve the application itself, if a bug appears I’ll be fixing it in the branch not in the main version (trunk).
• At any time more people could join the project and this would not stop the development process.

Just saw today, that Eddie Award retweeted Subversion Best Practices: Repository Structure, so I recommend you to get familiar with SVN first, before start using it.

Usually I’m using Debian for CVS systems, but in this case I’m installing Subversion on OEL 5.6, the procedure is the same for OEL 6.x.

First of all you need to install web server and svn packages. Assuming you have a configured repository, this is how to installed the packages:
yum install httpd mod_dav_svn subversion

If you want to change specific parameters you could edit Apache configuration file /etc/httpd/conf.d/httpd.conf, otherwise it’s not necessary.

Next you configure the SVN repository and authentication, the repository itself will be created next.

Edit file vi /etc/httpd/conf.d/subversion.conf and paste following:
<Location /svn>
DAV svn
SVNParentPath /var/www/svn
SVNListParentPath on
SVNIndexXSLT “/repos-web/view/repos.xsl”
AuthType Basic
AuthName “Oracle Repository authentication required”
AuthUserFile /etc/httpd/conf.d/svn.users
Require valid-user
</Location>

Create users who can access the repository:
htpasswd -cm /etc/httpd/conf.d/svn.users oracle
New password:
Re-type new password:
Adding password for user oracle
[root@db ~]# cat /etc/httpd/conf.d/svn.users
oracle:$apr1$9t19J…$hCF2GJTlizZfnPjKyk9rk/

Create the SVN repository:
mkdir /var/www/svn/
cd /var/www/svn/
svnadmin create oracle
chown -R apache:apache oracle

Finally restart apache and make sure it starts after reboot:
/etc/init.d/httpd restart
chkconfig httpd on

Troubleshooting:
In case you see the following error then most probably you’ve omitted the SVNListParentPath parameter:
Could not fetch resource information.  [403, #0]
(2)No such file or directory: The URI does not contain the name of a repository.  [403, #190001]

These is also an issue if you use SVNListParentPath and AuthzSVNAccessFile, bug description and workaround could be found here.

Conclusion:
Now you are ready to create you’re first project and start using SVN to maintain the source code of your applications. Given the example, the access URL would be http://hostname/svn/oracle or locally svn info file:///var/www/svn/oracle/. For maintaining the code you could use TortoiseSVN, which is excellent client for Windows.

The parameter I specified early SVNIndexXSLT will define the repository style once opened in a web browser. In my case I used a package called repo-style, which could be found here.

This is how my repository looks like:

plus the repository history looks really cool

It’s much better than the default one:

Regards,
Sve

Now I’m an Oracle DBA, although I still do some SA and apart from my daily duties, I wanted to feel the same way. To create something that will be useful and make me satisfied, to learn new things. During the six years within this company, I saw a lot of things and found the company is missing a centralised tool for managing company assets, car park, people contacts details, certificates and more. I felt that this could be a chance to be useful and create let’s say corporate portal, integrate it with AD and have people use it.  I thought that Oracle APEX is the perfect fit for this case and perfect for me to learn APEX.

I’m pretty sure there are a lot of brilliant developers out there and I do not tend to call myself developer, but personally I feel like APEX is the development tool for me as an Oracle DBA. The very first moment I created and run my first application in APEX, I felt the same was as I was feeling back when I was writing PHP/C applications, I wanted to learn more. I got few books, read forums and online docs, but I think that the main progress I made with APEX was because I had an idea to follow.

Now, few months later I feel prepared to start developing corporate portal for the company needs. I’ve created SVN repo for this project, integrated SQL Data Modeler with SVN, created backup script with APEXExport and SVN, integrated APEX with Microsoft AD (both auth and load users from it), integrated it with Jasper for PDF reports, put some fancy jQuery for pulling data and at after all APEX is using Oracle Database. As I said, I do not tend to call myself developer, but as an Oracle DBA, I feel great and thankful that I can use APEX to create and support company applications.

That’s why I like Oracle APEX.

Regards,
Sve

For the purpose I have setup a VMWare virtual machine with 2 CPUs, 4GB RAM and 32 GB HDD, one network interface. Installed Oracle Enterprise Linux 6.1 (64 bit) with following parameters:

• Perform custom disk layout, I dedicated 4GB for swap and the rest for the root (/) file system and formatted it with ext4.
• Perform default installation, needed packages will be installed later.
• Set hostname, timezone and root password.
• After installation disable the firewall and most of some of the services, like IPV6.

After installation the network adapter won’t be available that why you have to install several packages and then compile the VMWare tools. Insert the installation DVD/ISO and install the following packages:

mount /dev/cdrom /mnt
rpm -ivh -ivh gcc-4.4.5-6.el6.x86_64.rpm cloog-ppl-0.15.7-1.2.el6.x86_64.rpm cpp-4.4.5-6.el6.x86_64.rpm glibc-devel-2.12-1.25.el6.x86_64.rpm
glibc-headers-2.12-1.25.el6.x86_64.rpm kernel-uek-headers-2.6.32-100.34.1.el6uek.x86_64.rpm ppl-0.10.2-11.el6.x86_64.rpm
mpfr-2.4.1-6.el6.x86_64.rpm kernel-uek-devel-2.6.32-100.34.1.el6uek.x86_64.rpm
umount /dev/cdrom

Then disconnect the drive and from the console go to VM->Guest->Install/Upgrade VMWare Tools, then install the guest additions:

cp /mnt/VMwareTools-8.3.2-257589.tar.gz /tmp
umount /mnt
cd /tmp
tar xfz VMwareTools-8.3.2-257589.tar.gz
cd vmware-tools-distrib
./vmware-install.pl

At this point you should be able to configure the network interfaces.

Before starting the installation, download the packages from OTN and transfer them to the server. The installation consist of two zip packages, which are 5.5GB total, but this includes Oracle Weblogic Server 10.3.5, which is installed by default from the wizard.

Oracle Enterprise Manager Cloud Control 12c installation prerequisites

For the installation of Enterprise Manager Cloud Control I’m following the documentation:
Oracle® Enterprise Manager Cloud Control Basic Installation Guide 12c Release 1 (12.1.0.1)

1. From Oracle Database 10.2.0.5 onwards, all versions are certified for Management Repository. The last two releases 11.2.0.2 and 11.2.0.3 do not need additional patches for it to be configured successfully. For the rest of the version additional patches are needed, refer to MOS for more information.

Except the support for Management Repository few more parameters are needed to be set. They could be set before or after the installation. For setting database initialization parameters refer to Table-6 or Table-7 from Appendix A at the documentation.

Once you are ready you could run the EM Prerequisite Kit which is run by the wizard during the installation.

2. According to the Oracle documentation for small environment, you need following servers parameters:
For the OMS: 2 CPUs, 4 GB RAM and 7 GB space excluding the installation which is 5.5 GB.
For the Management Repository: 2 CPUs, 2 GB RAM and 50 GB space.

3. Packages and kernel parameters required for OMS:
The following packages should be installer, either from ISO or from public yum server:

yum install make.x86_64 binutils.x86_64 libaio.x86_64 glibc-common.x86_64 libstdc++.x86_64 sysstat.x86_64 glibc-devel.i686 glibc-devel.x86_64

The shmmax kernel parameter should be set to value bigger than 4GB. In OEL 6.1 this parameter is far beyond and it’s set to 64GB. Its current value could be retrieved by following command:

cat /proc/sys/kernel/shmmax

4. Create group and user for the installation of Enterprise Manager 12c
The installation could not be done by root and oracle user has to be created. I’m using the same group id and user id as they would be created by oracle-validated package (which is not yet available for OEL 6.x).

groupadd -g 54321 oinstall
useradd -u 54321 -g oinstall -s /bin/bash -d /home/oracle -m oracle
passwd oracle

5. Configure limits.conf file:
The following two parameter has to be set in /etc/security/limits.conf file
oracle soft nofile 4096
oracle hard nofile 4096

Oracle Enterprise Manager Cloud Control 12c installation

Proceed with default installation and following the installation wizard.

If the wizard gives you warning at “Checking whether required GLIBC installed on system” although you have installed all the prerequisites you could ignore the warning. The installer is checking whether the package glibc-devel.i386 is installed, but you have already installed glibc-devel.i686.

Supply the repository details and please be sure to check whether the database control doesn’t exists. Otherwise you’ll get an error after supplying the database credentials to drop the database control of the repository database:
$ORACLE_HOME/bin/emca -deconfig dbcontrol db -repos drop -SYS_PWD <sys_password> -SYSMAN_PWD <sysman_password>

Once the installation is complete you’ll get a screen with installation summary and details how to access the console.

Then you could login and select default home page. This is how the console look like:

Meanwhile I just saw two useful installation guides:

• Installing Oracle Enterprise Manager 12c on OL 5.7 by Martin Bach
• How to Install Oracle Enterprise Manager Cloud Control 12c by Gokhan Atil

Regards,
Sve

A lot of organisations deploy private cloud environments to offer great flexibility to business users and meet SLAs and security requirements. The Oracle response to these demands is Oracle Enterprise Manager Cloud Control 12c which offers complete life cycle cloud management – setup, deliver and manage clouds.

Once deployed IT must first setup the cloud infrastructure by defining share storage pools and compute resources. Administrators could also design a catalogue of virtual machines, assemblies, databases and applications and publish it.

It allows administrators to monitor usage, setup metering and chargeback for users, manage deployed applications, manage assets, link applications to MOS. It also allows quick to identify, diagnose and resolve incidents from a single console.

 Keynotes:

• The press release, Oracle Unveils Oracle Enterprise Manager 12c.
• Short flash presentation of Cloud Management with Oracle Enterprise Manager.
• Oracle Enterprise Manager Cloud Control 12c Video Series are available here.
• Meanwhile Oracle Learning is adding more videos to YouTube and they could be found here.
• The full installer (OMS, Agent and Repository) is available for Linux x86-64 (64-bit) and could be download from here.
• The Oracle Management Agent software can be downloaded using the Self Update feature within the Enterprise Manager Cloud Control console or use Self Update feature in offline mode then manually download the sar files available here.
• Documentation is available here, thanks to Gokhan Atil for the link.
• Installation requires Oracle Weblogic Server 10.3.5, which is already included in the package and installed by default.
• Installation requires certified  Oracle Database on which Oracle Management Repository will be configured. The certified versions of Oracle Database are from 10.2.0.5 onwards. The EM12c itself could manage databases from 9.2.0.8 onwards.
• You could upgrade your EM 10g GC Release 5 (10.2.0.5.0) or EM 11g GC Release 1 (11.1.0.1.0) to Enterprise Manager Cloud Control 12c (12.1.0.1).
• I’m still unable to find any MOS notes regarding EM12c – few notes already appeared at MOS, refer to last update for more information.

UPDATE 1:

• What’s New in Enterprise Manager 12c Install, here.
• Documentation library is now available here.

 UPDATE 2:

• Enterprise Manager Grid Control and Database Control Certification with 11g R2 Database [ID 1266977.1]
• If repository database is less than 11.2.0.2, additional packages has to be applied. Refer at MOS Certification for more information.

UPDATE 3:

• How to Install Enterprise Manager Cloud Control 12.1.0.1 (12c) [ID 1359176.1]
• EM12c: How to install Enterprise Manager Cloud Control 12c Agent [ID 1360183.1]
• Master Index for Cloud Control Agent Installation, Upgrade and Patching [ID 1363767.1]
• FAQ: Enterprise Manager Cloud Control 12c Install / Upgrade Frequently Asked Questions [ID 1363863.1]
• Master Index for Cloud Control Oracle Management Service (OMS) and Repository Installation, Upgrade and Patching [ID 1363769.1]

Regards,
Sve

Master Index for Cloud Control Oracle Management Service (OMS) and Repository Installation, Upgrade and Patching [ID 1363769.1]

I’ve been many times in Slovenia, but It’s my first time being at the SIOUG and I really liked it. There were a lot of people, the hotel was at the sea side and Croatia was just on the other side of the sea, just a few kilometres away. Speaking about Croatia I met Vladimir and Davor from HROUG.

I also met Doug Burns and Debra Lilley and they are really cool people.

Oracle Open World starts in two days. Although I won’t be able to attend it this year I’m pretty excited about the new products and announcements.

Regards,
Sve

p.s. I met Stanley in a cave where The Human Fish lives, wonder what was he doing there.

So during the weekend I received mail from my colleagues above error messages they received from the database: connect error, Socket read timed out. It wasn’t a rush as the database is not yet in production, but it’s ahead and this was the first task for the Monday. Next day I looked around and everything was up and running, except that I wasn’t able to login through the listener and I also wasn’t able to stop or relocate it. Looking at the logs I found at some point the following message: TNS-12531: TNS:cannot allocate memory which explains the previous message.

That was weird, the server on which error appeared was the first one and had only GI running and SCAN LISTENER. This really looked like a memory leak, it’s a Windows so maybe that was obvious. I decided to look around the processes using the Resource Monitor when I found a lot of many cmd.exe processes. To confirm the problem I used Process Explorer which is a very nice tool for Windows. As could be seen below I’ve got plenty of cmd processes which were spawned, but not (obviously) closed after completion:

It turned out that this is a bug for 11.2.0.2 and Windows (64 bit). The Oracle CVU resource (ora.cvu), which by default is started on the first node in the cluster (this makes sense now) it’s doing checks on every six hours (CHECK_INTERVAL=21600) and leaves process open. Because of this the heap memory is exhausted and that’s the reason why the SCAN LISTENER is failing and giving the error message TNS-12531: TNS:cannot allocate memory

The following errors could be seen in Windows Eventlog, once the patch is applied the errors disappeared:
Faulting application lsnrctl.exe, version 11.2.0.2, time stamp 0x4cea8f55, faulting module kernel32.dll, version 6.0.6001.18538, time stamp 0x4cb73957, exception code 0xc0000142, fault offset 0x00000000000b1b48, process id 0x1eac, application start time 0x01cc6ab588f992c0.

Faulting application cmd.exe, version 6.0.6001.18000, time stamp 0x47918bde, faulting module kernel32.dll, version 6.0.6001.18538, time stamp 0x4cb733e1, exception code 0xc0000142, fault offset 0x0006f1e7, process id 0×1004, application start time 0x01cc6af0fa982500.

Faulting application sclsspawn.exe, version 0.0.0.0, time stamp 0x4ce622a7, faulting module kernel32.dll, version 6.0.6001.18538, time stamp 0x4cb73957, exception code 0xc0000142, fault offset 0x00000000000b1b48, process id 0x1ca0, application start time 0x01cc6c0e5efd5380.

This is the bug at MOS:
Bug 12529945: CVU HEALTH CHECKS EXHAUST WINDOWS HEAP MEMORY

The bug should have been fixed in BP8, but I applied the latest one BP10:
Patch 12849789: ORACLE 11G 11.2.0.2 PATCH 10 BUG FOR WINDOWS (64-BIT AMD64 AND INTEL EM64)

Regards,
Sve

Looking at the guest status (hpvmstatus -p) the following is observed:
[Remote Console]
Remote Console not configured

This is how to configure the IP address of the virtual iLO Remote Console:
root@vmhost1:/# hpvmmodify -p 1 -K 192.168.11.193 -L 255.255.255.0

What the command does is configuring IP alias as could be seen in netstat:
lan0:110  1500 192.168.11.0    192.168.11.193  0                  0     0                  0     0

Running again hpvmstatus we see that the Remote Console is now configured:
[Remote Console]
Remote Console Ip Address:      192.168.11.193
Remote Console Net Mask:        255.255.255.0

Now about the users, by default the root user of the host is able login to all virtual consoles, although it’s not listed in following section:
[Authorized Administrators]
Oper Groups             :
Admin Groups            :
Oper Users              :
Admin Users             :

Let’s create a dedicated user for the first guest and then login:
root@vmhost1:/root# useradd -d /var/opt/hpvm/guests/vm01 -c ‘vm01 console’ adminvm1
root@vmhost1:/root# passwd adminvm1
root@vmhost1:/root# hpvmmodify -p 1 -u adminvm1:admin

Running status again we can see that the user is added to the administrators:
[Authorized Administrators]
Oper Groups             :
Admin Groups            :
Oper Users              :
Admin Users             : adminvm1

Let’s try now to login with our newly created user:
sve@angmar:~$ ssh adminvm1@192.168.11.193
Password:

vMP MAIN MENU

CO: Console
CM: Command Menu
CL: Console Log
SL: Show Event Logs
VM: Virtual Machine Menu
HE: Main Help Menu
X: Exit Connection

[vm01] vMP>

Voila, it’s working!

To enable access to the virtual iLO Remote Console through telnet (which by default is not supported) these two patches has to be applied on the host: PHCO_41595 and PHNE_41452. Two more considerations should be taken into consideration: The virtual iLO Remote Console’s SSH server host keys can change as it uses the host system’s SSH server host keys. If the guest is migrated to another system then these host keys will change. Another one is that guest administrator accounts are not migrated during Online VM Migration.

Regards,
Sve

Previously I had to create a logical volume, copy the content of the ISO on in and then mount the logical volume, like this one:

root@vmhost1:/# ll /tmp/test.iso
-rw-r--r--   1 root       sys        105906176 Sep  8 11:57 /tmp/test.iso
root@vmhost1:/# lvcreate -L 102 -n lvtest vg00
Warning: rounding up logical volume size to extent boundary at size "128" MB.
Logical volume "/dev/vg00/lvtest" has been successfully created with
character device "/dev/vg00/rlvtest".
Logical volume "/dev/vg00/lvtest" has been successfully extended.
Volume Group configuration for /dev/vg00 has been saved in /etc/lvmconf/vg00.conf
root@vmhost1:/# dd if=/tmp/test.iso of=/dev/vg00/rlvtest bs=1024k
101+0 records in
101+0 records out
root@vmhost1:/# mount /dev/vg00/lvtest /mnt
root@vmhost1:/# bdf /mnt
/dev/vg00/lvtest    102888  102888       0  100% /mnt

With the enhancement of ISOIMAGE-ENH, now you’re able to mount and umount iso images (like in Linux):

root@vmhost1:/# kcmodule fspd=loaded
root@vmhost1:/# mount -F cdfs /tmp/test.iso /mnt
root@vmhost1:/# bdf /mnt
/dev/fspd1          102888  102888       0  100% /mnt

If you get an error like this one:
mount: /tmp/test.iso is an invalid operand
then most probably you’ve forgotten to load the fspd module.

The package could be installed during installation or downloaded from Software Depot home, here and it’s available only for HP-UX 11iv3.

Regards,
Sve

• Thin provisioning – It’s something the other vendors had for long time, it was about time to have such a feature on the EVA. This feature dynamically increases the space allocated to a virtual disk.
• Large LUN support – Because this feature was missing, last year I’ve spent one month for migrating data from several big disks to another diskgroup with LVM. This will support snapshot replication of LUNs greater than 2 TB, expanding and shrinking of the large LUNs is also supported.
• Online virtual disk migration (change Vraid or disk group) – Another ‘must have’ feature, change a virtual disk’s redundancy level or disk group membership without impacting host I/O. Previously one has to create another virtual disk in required level and move data from the other.

There are also a lot of fixes included in this release and some of them sounds really scary. For more information refer to the release notes.

Regards,
Sve