HP Data Protector 6.11 post upgrade database cleanup and defrag

I recently upgraded our Data Protector backup software to 6.11 to get the latest and the greatest. Everything seemed to work just fine, but after a few backup runs I noticed that backing up the Data Protector Database did not work as it should. The log said:

[Normal] From: DBBDA@dpserv.mydomain.intern "[Database]: dpserv.mydomain.intern" Time: 18.12.2009 14:35:25
 STARTING Database HotBackup Disk Agent on dpserv.mydomain.intern "[Database]: dpserv.mydomain.intern".

[Major] From: DBBDA@dpserv.mydomain.intern "[Database]: dpserv.mydomain.intern" Time: 18.12.2009 14:35:25
[81:522] Cannot back up internal database because another database check in progress.

[Warning] From: DBBDA@dpserv.mydomain.intern "[Database]: dpserv.mydomain.intern" Time: 18.12.2009 14:35:26
Cannot un-lock the keystore.

[Major] From: DBBDA@dpserv.mydomain.intern "[Database]: dpserv.mydomain.intern" Time: 18.12.2009 14:35:26
ABORTED Database HotBackup Disk Agent on dpserv.mydomain.intern"[Database]: dpserv.mydomain.intern"

A manual run of omnidbcheck revealed the same thing. I could not find any signs of any other database checks, and a reboot did not help.

Deleting the lock-file (dbcheck.lk) from the {omniback}/tmp folder did the trick though. The backup would go through, but for some reason the backup was unbelievably slow. In fact it would hang for almost 3 hours before even beginning to backup the database. This was not the case on DP 6.0 only one week earlier. omnidbcheck would not report any errors though. Apparently the database needed some cleaning and defragmentation.

I started off running a "omnisv -stop" (to stop all services) and did a cold backup of the whole {omniback}-foldertree. Then I started the services again with "omnisv -start".

I then ran "omnidbutil -info" which showed the following:

Media Management database space usage:

Space used by Diskspace usage Records used Records total
============================================================================
Devices 96 17 20
Libraries 96 2 5
Cartridges 1184 296 852
Compounds 32 0 12
Pools 64 26 30
Media 160 288 297

Catalog database space usage:

Space used by Diskspace usage Records used Records total
============================================================================
Sessions 352 197 284
Objects 256 517 552
Object versions 10080 5236 7810
Positions 12896 77086 97960
Filenames 3175520 31876226 31876278

Detail catalog binary files usage:

Diskspace usage Size limit DC Directory
============================================================================
2068 4096 C:/Program Files/OmniBack/db40/dcbf
----------------------------------------------------------------------------
2068 4096

Session messages binary files usage:

Diskspace usage Num. of files SMBFs Directory
============================================================================
16 191 C:/Program Files/OmniBack/db40/msg

Serverless integrations binary files usage:

Diskspace usage Num. of files SIBFs Directory
============================================================================
0 0 C:/Program Files/OmniBack/db40/meta

I decided to try and purge unused and obsolete filenames from the database:

"omnidbutil -purge -filenames -force" (took a couple of hours) and another "omnidbutil -info" showed this:

Media Management database space usage:

Space used by Diskspace usage Records used Records total
============================================================================
Devices 96 17 20
Libraries 96 2 5
Cartridges 1184 296 852
Compounds 32 0 12
Pools 64 26 30
Media 160 288 297

Catalog database space usage:

Space used by Diskspace usage Records used Records total
============================================================================
Sessions 352 203 284
Objects 256 517 552
Object versions 10080 5354 7810
Positions 12896 77366 97960
Filenames 3175520 13472221 31876336

Detail catalog binary files usage:

Diskspace usage Size limit DC Directory
============================================================================
2099 4096 C:/Program Files/OmniBack/db40/dcbf
----------------------------------------------------------------------------
2099 4096

Session messages binary files usage:

Diskspace usage Num. of files SMBFs Directory
============================================================================
17 197 C:/Program Files/OmniBack/db40/msg

Serverless integrations binary files usage:

Diskspace usage Num. of files SIBFs Directory
============================================================================
0 0 C:/Program Files/OmniBack/db40/meta

Right, apparently there was a lot of unused filename records.

According to HP the best way to clean up and defrag the database would be to export all records and import them right back in again.

I created two empty folders (f:\dump\dp\cdb and f:\dumå\dp\mmdb) and ran

omnidbutil -writedb -mmdb f:\dumå\dp\mmdb -cdb f:\dump\dp\cdb

followed by:

omnidbutil -readdb -mmdb f:\dumå\dp\mmdb -cdb f:\dump\dp\cdb

I then followed the on-screen instructions and took a backup of {omniback}\dcbf and {omniback}\msg
Again I ran "omnidbutil -info"

Media Management database space usage:

Space used by Diskspace usage Records used Records total
============================================================================
Devices 96 17 20
Libraries 96 2 5
Cartridges 448 296 297
Compounds 32 0 12
Pools 64 26 30
Media 160 288 297

Catalog database space usage:

Space used by Diskspace usage Records used Records total
============================================================================
Sessions 288 209 212
Objects 256 517 528
Object versions 6688 5450 5460
Positions 10208 77217 77221
Filenames 1319392 13474587 13474618

Detail catalog binary files usage:

Diskspace usage Size limit DC Directory
============================================================================
2135 4096 C:/Program Files/OmniBack/db40/dcbf
----------------------------------------------------------------------------
2135 4096

Session messages binary files usage:

Diskspace usage Num. of files SMBFs Directory
============================================================================
17 203 C:/Program Files/OmniBack/db40/msg

Serverless integrations binary files usage:

Diskspace usage Num. of files SIBFs Directory
============================================================================
0 0 C:/Program Files/OmniBack/db40/meta

Notice that space usage for the catalog datbase/filenames has decreased by 1.8 GB. Nothing seems to be missing when using the DP Cell Manager GUI and running a omnidbcheck now takes 11 minutes! The slight increase in disk space usage is because I had to run a quite large backup before doing the writedb/readdb operation.

XenServer on HP bl460c servers - nx and sep CPU flags

We just installed two XenServers to check out they hype. People are praising XenServer left and right, so we figured what the heck.

We installed on two HP BL460c G1 blade servers. Installation was a breeze, but when we tried to create a pool with XenCenter and add the two servers we got the following message:

08.12.2009 12:57:22 Error: Adding server 'dnxen2.mydomain.intern' to pool 'DN XenPool' - The hosts in this pool are not homogeneous. cpus differ

A quick look reveals the following difference between the cpus :

[root@dnxen1 ~]# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5440 @ 2.83GHz
stepping : 6
cpu MHz : 2833.454
cache size : 6144 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu de tsc msr pae cx8 apic sep mtrr cmov pat clflush acpi mmx fxsr sse sse2 ss ht nx constant_tsc up pni vmx est
bogomips : 5668.59


[root@dnxen2 ~]# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5440 @ 2.83GHz
stepping : 6
cpu MHz : 2833.456
cache size : 6144 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu de tsc msr pae cx8 apic mtrr cmov pat clflush acpi mmx fxsr sse sse2 ss ht constant_tsc up pni vmx est
bogomips : 5668.07

The flags "sep" and "nx" are enabled on dnxen1, but not on dnxen2. According to cpufeature.h "nx" is is "Execute Disable" and is a AMD-defined CPU feature (Strane since the CPU is Intel) and "sep" is something called "Sysenter/sysexit". Doesnt tell me much - but this page has a better description of NX: http://blog.incase.de/index.php/cpu-feature-flags-and-their-meanings/

"NX No eXecute, a flag that can be set on memory pages to disable execution of code in these pages"

And it seems like "sysenter/sysexit" are cpu instructions.

So let's have a look at the BIOS:

Advanced Options > Processor options > No-Execute memory protection (Changed from disabled to enabled on dnxen2).

The "Sysenter/sysexit" setting is nowhere to be found in the bios. In fact - there were no other differences between the two servers at all.

However after rebooting both servers list the same flags, so apparently enabling "nx" will enable the "sep"-flag as well. Well, XenServer is happy and and dnxen2 is was able to join the pool.

Off to do more testing.