Posts Tagged ‘dba’

RHEL6 udev and EMC PowerPath

January 26th, 2015 No comments

I’m working on Oracle database migration project where customer have chosen commodity x86 hardware with RHEL6 and EMC storage.

I’ve done many similar installations in the past and I always used the native MPIO in Linux (DM-Multipath) to load balance and failover I/O paths. This time however I’ve got EMC PowerPath doing the load balance and failover and got the native MPIO disabled. From my point of view it’s the same, whether I’ll be using /dev/emcpower* or /dev/mapper/* it’s the same. Obviously PowerPath has some advantages over the native MPIO which I really can’t tell yet. That’s a good paper from EMC giving a comparison between the native MPIO in different operating systems.

As mentioned before the aggregated logical names (pseudo names) with EMC PowerPath could be found under /dev/emcpowerX. I partitioned the disks with GPT tables and aligned the first partition to match the storage sector size. Also added to following line to udev rules to make sure my devices will get the proper permissions:

ACTION=="add", KERNEL=="emcpowerr1", OWNER:="oracle", GROUP:="dba", MODE="0600"

I restarted the server and then later udev to make sure ownership and permissions were picked up correctly. Upon running asmca to create ASM with the first disk group I got the following errors:

Configuring ASM failed with the following message:
One or more disk group(s) creation failed as below:
Disk Group DATA01 creation failed with the following message:
ORA-15018: diskgroup cannot be created
ORA-15031: disk specification '/dev/emcpowerr1' matches no disks
ORA-15025: could not open disk "/dev/emcpowerr1"
ORA-15056: additional error message

Well that’s strange, I’m sure the file had to correct permissions. However listing the file proved that it didn’t have the correct permissions. I repeated the process several times and always got the same result, you can use simple touch command to get the same result:

[root@testdb ~]# ls -al /dev/emcpowerr1
brw-rw---- 1 oracle dba 120, 241 Jan 23 12:35 /dev/emcpowerr1
[root@testdb ~]# touch /dev/emcpowerr1
[root@testdb ~]# ls -al /dev/emcpowerr1
brw-rw---- 1 root root 120, 241 Jan 23 12:35 /dev/emcpowerr1

Something was changing the ownership of the file and I didn’t know what. Well you’ll be no less surprised than I was to find that linux has a similar auditing framework as the Oracle database.

Auditctl will allow you to audit any file for any syscall run against it. In my case I would like to know which process is changing the ownership of my device file. Another helpful command is ausyscall whic allows you to map syscall names and numbers. In other words I would like to know what is the chmod syscall number on a 64bit platform (it does matter):

[root@testdb ~]# ausyscall x86_64 chmod --exact

Then I would like to set up auditing for all chmod calls against my device file:

[root@testdb ~]# auditctl -a exit,always -F path=/dev/emcpowerr1 -F arch=b64 -S chmod
[root@testdb ~]# touch /dev/emcpowerr1
[root@testdb ~]# tail -f /var/log/audit/audit.log
type=SYSCALL msg=audit(1422016631.416:4208): arch=c000003e syscall=90 success=yes exit=0 a0=7f3cfbd36960 a1=61b0 a2=7fff5c59b830 a3=0 items=1 ppid=60056 pid=63212 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="udevd" exe="/sbin/udevd" key=(null)
type=CWD msg=audit(1422016631.416:4208):  cwd="/"
type=PATH msg=audit(1422016631.416:4208): item=0 name="/dev/emcpowerr1" inode=28418 dev=00:05 mode=060660 ouid=54321 ogid=54322 rdev=78:f1 nametype=NORMAL
[root@testdb ~]# auditctl -D
No rules

Gotcha! So it was udev changing the permissions but why ?

I spent half day going through logs and tracing udev but couldn’t find anything.

At the end of the day I found an article by RHEL on which they had exactly the same problem. The solution was to have “add|change” into the ACTION directive instead of only “add”.

So here is the rule you need to have in order for UDEV to set a persistent ownership/permission on EMC PowerPath device files in RHEL 6:

[root@testdb ~]# cat /etc/udev/rules.d/99-oracle-asm.rules
ACTION=="add|change", KERNEL=="emcpowerr1", OWNER:="oracle", GROUP:="dba", MODE="0600"

Hope it helps and you don’t have to spent half day as I did.


Categories: linux, oracle Tags:

Upgrade to Oracle Enterprise Manager Cloud Control 12c Release 3 (

August 15th, 2013 4 comments

Just a quick wrap up on EM12cR3 upgrade. I have to say that I was pleasantly surprised that everything went so smooth. I didn’t expected anything else, but with so how many products and components we have there I got few things in mind. The version I got was already upgraded to so it was really easy for me to run the upgrade.

Things to watch out for:
– You need to be already running OEM version to be able to upgrade to If not you must apply BP1 to your installation and then patch to Remember to patch the agents as well.
– The upgrade to is out-of-place upgrade, so you need to point out to new middleware home and you’ll need additional 15 GB for the installation.
– The installation takes between 1-2 hours to complete depending on you machine power.
– I didn’t stopped any of the agents during the upgrade.
– After the upgrade all OMS components were started automatically.

Here is what I’ve done:
1. Definitely take backup of the middleware home and database as well. You don’t want to end up removing the agents and reinstalling the OMS. I had 400+ targets and failure wasn’t an option. For the middleware home I used simple tar and RMAN for the repository database.

2. Stop the OMS and other components:

cd $ORACLE_HOME/bin/
cd bin
./emctl stop oms -all

3. It’s required that the EMKey be copied to the repository prior upgrade, if you miss that the installer will kindly remind you. There is also note  in the documentation:

$OMS_HOME/bin/emctl config emkey -copy_to_repos_from_file -repos_host [repository_host] -repos_port [port] -repos_sid [sid] -repos_user [username] -emkey_file $OMS_HOME/sysman/config/emkey.ora

4. The only command I run during the upgrade was a simple grant, everything else was done by the installer:

 SQL> grant execute on dbms_random to dbsnmp;

Grant succeeded.

5. Once the upgrade is complete:
– upgrade all the agents from the console:

Setup -> Manage Cloud Control -> Upgrade agents

– as a post upgrade step the old agent homes should be deleted:

Setup -> Manage Cloud Control -> Upgrade agents -> Post Agent Upgrade Tasks

– and secure the EMKey:

emctl config emkey -remove_from_repos

The upgrade guide is very useful, consider it before doing the upgrade.

Also have a look on New Features In Oracle Enterprise Manager Cloud Control where the most notable thing is support of Oracle Database 12c and its new features, but there are plenty of other new features and improvements as well.

Categories: oracle Tags: , ,

Failed creating physical standby database

June 5th, 2012 No comments

These days I’m implementing Oracle Dataguard for two Oracle databases 10.2 as part of disaster recovery project, one of them is around 1.7TB, not yet production. As part of the DG setup backups have to be available for both primary and standby. I preferred to use ASM and was able to negotiate with the storage admin to run storage replication for the FRA disks during the backup. This way I would have the same structure and files, locally at the disaster site immediately after the backup of the primary database is completed.

Unfortunately two weeks passed and by the time (this morning) I had to start with the DG setup the FRA (2TB big) got exhausted, because of too many archivelogs. When I saw this the first thing it came to my mind was to delete the backup as it was too big and I already had it at the disaster site!?. Deleting archivelogs was not an option as I need these archivelogs so the standby could catch up with the primary. So what I’ve done was to delete the backup without thinking and then moved on to duplicate the primary database for standby.

I’ve setup the standby instance and when issued “DUPLICATE TARGET DATABASE FOR STANDBY NOFILENAMECHECK DORECOVER;” I got the following errors:

RMAN-03002: failure of Duplicate Db command at 06/05/2012 11:12:04
RMAN-03015: error occurred in stored script Memory Script
RMAN-06136: ORACLE error from auxiliary database: ORA-01180: can not create datafile 1
ORA-01110: data file 1: '+DATA/orcl/datafile/system.347.666704609'

I was surprised by this error and started thinking if there wasn’t a problem with the storage (it happened twice before to have read only disks), but this was not the case. Once the diskgroup is mounted it means that ASM can read/write to its disks.

It’s obvious what my mistake is, but it took me a while until I realize what I’ve done. Deleting backup at the primary database means that it no longer knows for my backup and for that reason I could not clone the primary database. That’s why I got error that datafile 1 could not be created, simply it has no backups to restore it from. Thinking now what if the backup was located at NFS share, then I would definitely not delete it, but maybe move the archivelogs and manually register them later on the standby.

So now I started a new backup, waiting for it to finish and got replicated to the disaster site.


Categories: oracle Tags: , ,

ASM diskgroup unable to mount because of duplicate disk

March 6th, 2012 2 comments

This is something I would like to share for a long time, but for one or another reason I didn’t. The error is nothing special, but then I run into very interesting case.

This happen when I had to shutdown one server for maintenance and then boot it again. Once the OS started I notice that the database didn’t ran automatically. When I try to run it manually I got the following error:

SQL> startup
 ORA-01078: failure in processing system parameters
 ORA-01565: error in identifying file '+DATA/orcl/spfileorcl.ora'
 ORA-17503: ksfdopn:2 Failed to open file +DATA/orcl/spfileorcl.ora
 ORA-15077: could not locate ASM instance serving a required diskgroup

The reason for this was that the ASM diskgroup wasn’t mounted, again I tried to mount it manually, when I got this error:

SQL> alter diskgroup data mount;
 alter diskgroup data mount
 ERROR at line 1:
 ORA-15032: not all alterations performed
 ORA-15024: discovered duplicately numbered ASM disk 29

So how could this happen as I didn’t change any of the parameter and configuration, the database was started for just three months. Running quick script with kfed:

ls -1 /dev/oracle/*for i in `ls -1 /dev/oracle/*`; do echo /dev/oracle/$i; kfed read /dec/oracle/$i | grep kfdhdb.dskname; done

showed me that I there are really two duplicate disks. ASM does not support disks with the same disk names, even if this is the same disk, but accessed by different path. The duplicate disk was an old disk, which wasn’t removed at the time of the change, only its filename was changed so once ASM discovers disks in the future this one would be excluded.

Looking at the asm_diskstring I found that it has changed to ‘/dev/oracle/*’ and should be ‘/dev/oracle/orcl*’ and that’s why ASM was scanning all the disks. This made me think what could happen for the time database was running and why this happen.

After looking at the logs and made few tests I had a conclusion. At some point when the DBA added new disks to ASM he done it through the dbca and he changed the discovery string for unknown reason. What happens in background is that dbca, implicitly is recreating the ASM parameter file with the new discovery string entered by the DBA. This is why ASM started with wrong discovery string when I rebooted the server and it discovered more disks.

The database ran at 26 March and a month later at 29 April new disks were added to the ASM and its parameter was recreated, almost two months later at 7 July I turned it back with correct values:

-rw-r----- 1 oracle oinstall 1336 Jul 7 18:42 init.ora
-rw-r----- 1 oracle oinstall 1327 Apr 29 11:32 init.ora.662011221038


Categories: oracle Tags: , ,

Setup Subversion on Oracle Enterprise Linux

November 1st, 2011 5 comments

As I mentioned in my previous post I’m using Subversion to keep my SQL design, APEX application and all application serving files. Concurrent Versions System are not something new and they are must for big projects. Although I’m single person, I’m finding SVN very useful for many reasons:

  • All my files are kept at one place. At any time I could checkout the whole application and deploy it on different server.
  • I have a history (revisions) of my changes. If I know something worked yesterday I could just checkout the file from yesterday and get it working.
  • Using APEXExport I’m doing daily backups of my applications.
  • Oracle SQL Developer and SQL Data Modeler are easy to integrate with SVN.
  • When the time comes, I’ll create a branch, stable version of my application, by doing so I’ll have at any time a working and stable copy of my application.
  • Having branches I could still develop and improve the application itself, if a bug appears I’ll be fixing it in the branch not in the main version (trunk).
  • At any time more people could join the project and this would not stop the development process.

Just saw today, that Eddie Award retweeted Subversion Best Practices: Repository Structure, so I recommend you to get familiar with SVN first, before start using it.

Usually I’m using Debian for CVS systems, but in this case I’m installing Subversion on OEL 5.6, the procedure is the same for OEL 6.x.

First of all you need to install web server and svn packages. Assuming you have a configured repository, this is how to installed the packages:
yum install httpd mod_dav_svn subversion

If you want to change specific parameters you could edit Apache configuration file /etc/httpd/conf.d/httpd.conf, otherwise it’s not necessary.

Next you configure the SVN repository and authentication, the repository itself will be created next.

Edit file vi /etc/httpd/conf.d/subversion.conf and paste following:
<Location /svn>
DAV svn
SVNParentPath /var/www/svn
SVNListParentPath on
SVNIndexXSLT “/repos-web/view/repos.xsl”
AuthType Basic
AuthName “Oracle Repository authentication required”
AuthUserFile /etc/httpd/conf.d/svn.users
Require valid-user

Create users who can access the repository:
htpasswd -cm /etc/httpd/conf.d/svn.users oracle
New password:
Re-type new password:
Adding password for user oracle
[root@db ~]# cat /etc/httpd/conf.d/svn.users

Create the SVN repository:
mkdir /var/www/svn/
cd /var/www/svn/
svnadmin create oracle
chown -R apache:apache oracle

Finally restart apache and make sure it starts after reboot:
/etc/init.d/httpd restart
chkconfig httpd on

In case you see the following error then most probably you’ve omitted the SVNListParentPath parameter:
Could not fetch resource information.  [403, #0]
(2)No such file or directory: The URI does not contain the name of a repository.  [403, #190001]

These is also an issue if you use SVNListParentPath and AuthzSVNAccessFile, bug description and workaround could be found here.

Now you are ready to create you’re first project and start using SVN to maintain the source code of your applications. Given the example, the access URL would be http://hostname/svn/oracle or locally svn info file:///var/www/svn/oracle/. For maintaining the code you could use TortoiseSVN, which is excellent client for Windows.

The parameter I specified early SVNIndexXSLT will define the repository style once opened in a web browser. In my case I used a package called repo-style, which could be found here.

This is how my repository looks like:

plus the repository history looks really cool 🙂

It’s much better than the default one:



Categories: linux, oracle Tags: , ,

Why I like Oracle Application Express

October 27th, 2011 4 comments

Few years ago when I was Linux system administrator I already knew PHP and I used it to create monitoring scripts for the servers and services that I was responsible for. They were really simple application which I kept open in a browser window and integrated javascript to alert me in case some service goes down. The web design wasn’t my strong side, I guess people are born with it, but for my applications these were enough, although they were not as fancy and beautiful as I wanted to be. Then I start coding ASNI C (as a SA this is a must), created some client-server applications, loading data in MySQL and displaying it with PHP. Both PHP/C are much similar and programming in one of them clearly sets your mind of thinking in a way that it’s very easy to start the other. As system administrator I really liked using Bash, PHP, C for my daily duties. Writing applications, automating few process and protectively monitoring the systems, these kept my brain busy, made me think for new ideas and approaches and especially learned new things, I felt really positive and useful.

Now I’m an Oracle DBA, although I still do some SA and apart from my daily duties, I wanted to feel the same way. To create something that will be useful and make me satisfied, to learn new things. During the six years within this company, I saw a lot of things and found the company is missing a centralised tool for managing company assets, car park, people contacts details, certificates and more. I felt that this could be a chance to be useful and create let’s say corporate portal, integrate it with AD and have people use it.  I thought that Oracle APEX is the perfect fit for this case and perfect for me to learn APEX.

I’m pretty sure there are a lot of brilliant developers out there and I do not tend to call myself developer, but personally I feel like APEX is the development tool for me as an Oracle DBA. The very first moment I created and run my first application in APEX, I felt the same was as I was feeling back when I was writing PHP/C applications, I wanted to learn more. I got few books, read forums and online docs, but I think that the main progress I made with APEX was because I had an idea to follow.

Now, few months later I feel prepared to start developing corporate portal for the company needs. I’ve created SVN repo for this project, integrated SQL Data Modeler with SVN, created backup script with APEXExport and SVN, integrated APEX with Microsoft AD (both auth and load users from it), integrated it with Jasper for PDF reports, put some fancy jQuery for pulling data and at after all APEX is using Oracle Database. As I said, I do not tend to call myself developer, but as an Oracle DBA, I feel great and thankful that I can use APEX to create and support company applications.

That’s why I like Oracle APEX.


Categories: oracle Tags: ,