Netapp FAS 2040 CIFS backup performance benchmark

We recently invested in a new filer from Netapp. Unfortunately we can't do agent based backups of the large CIFS-filesystem on the filer, and NDMP dump to tape is also problematic. Actually the NDMP backup itself is not a problem, but restore is. NDMP backups initiated from Data Protector will do a raw block copy, not a file copy of the individual files on the filesystem and thus making restore quite interesting.

We still chose to go for NetApp though, because we trust that the snapshot functionality will allow us to use less backups to tape (hopefully as little as once or twice a month) and thus making high speed a less of a critical issue.

The obvious solution is to do a normal Network share backup from DP (in production this will probably be done on a snapshot, but for now this will do). Basically I'm going to run a series of tests to determine the number of data streams and virtual tape devices which will give the best performance. Note that there are no other users attached to the NetApp filer or the Ethernet switch.

The results are considerably slower than what we can expect from a larger backup due to the relatively large overhead with small jobs.

I noticed that this type of backup is very CPU intensive on the Cell Manager. It would pretty much max out at 100% constantly during. Memory was not a problem though.

Setup

Server: HP Proliant DL380 G3 / Windows 2003 x86 / 2 GB ram / Dual Intel Xeon 2.8 ghz
Backup software: Data Protector 6.0
Network: HP Procurve 2824 switch (single gigabit connection)
Storage system: NetApp FAS 2040
Tape library: HP 6636 VLS (Virtual tape library) connected to the DP Cell Manager by FC.
Test data: A collection of user home folders on 7 shares containg 39.108 files (19.932 MB)


Test Results

Test 1
1 data stream / 1 drive (Load balancing Min:1 / Max:1 and Concurrency: 1
Total 1427 seconds = 13,97 MB/sec

Test 2
2 data streams / 2 drives (Load balancing Min:2 / Max:2 and Concurrency: 1 per drive)
Total 1128 seconds = 17,67 MB/sec

Test 3
4 data streams / 2 drives (Load balancing Min:2 / Max:2 and Concurrency: 2 per drive)
Total 1002 seconds = 19,9 MB/sec

Test 4
2 data streams / 1 drives (Load balancing Min:1 / Max:1 and Concurrency: 2 per drive)
Total 1135 seconds = 17,56 MB/sec

Test 5
3 data streams / 1 drives (Load balancing Min:1 / Max:1 and Concurrency: 3 per drive)
Total 1015 seconds = 19.64 MB/sec

Test 6
4 data streams / 1 drives (Load balancing Min:1 / Max:1 and Concurrency: 4 per drive)
Total 983 seconds = 20,28 MB/sec

Test 7
5 data streams / 1 drives (Load balancing Min:1 / Max:1 and Concurrency: 5 per drive)
Total 994 seconds = 20,05 MB/sec

Test 8
7 data streams / 1 drives (Load balancing Min:1 / Max:1 and Concurrency: 7 per drive)
Total 964 seconds = 20,67 MB/sec

Test 9
8 (7) data streams / 2 drives (Load balancing Min:2 / Max:2 and Concurrency: 4 per drive)
Total 960 seconds = 20,76 MB/sec

Test 10
I decided to increase the amount of data to 37508 mb and rerun the backup with settings from Test 8 to see if the results would be any better.
Total 1597 seconds = 23,49 MB/sec

I was a little disappointed about the results. They are considerably slower than what I experienced with the old HP MSA1000 SAN using a backup agent. On the other hand - I knew that this type of backup is slow.

It also seems that you wont gain much in terms of speed from using > 3 data streams, but you will stress the Netapp filer a little more (I noticed CPU would be higher the more streams you use. One stream would average 9% on the Netapp and 8 streams would average 14%).

Perhaps I can achieve slightly better results by using a faster CPU on the Data Protector Cell Manager server, but I doubt we're talking about anything higher that 30MB/sec at best.

On the other hand - it really does not matter much on a day to day basis if I'm only going to do a full backup to tape only a couple of times a month. A 3 TB volume should take apx 36 hours to finish.

Microsoft Security Essentials Performance test

Today I grabbed the release version (v1.0) of Microsofts new Security Essentials package (aka Morro) and decided to run a quick and dirty performance benchmark to compare it to a couple of other AntiVirus/Malware solutions.

I tested on an old Celeron 2.2ghz laptop with 512mb ram and Windows XP SP3.

I did the following measurements:

- Boot-up time (from switching on until notepad.exe in the startup folder was running).

- Avatilable physical memory (after bootup has finished)

- Total memory usage

- Deep and Standard scan of a folder containing apx 8000 files /10 gb.

- CPU usage during scan.

First out was Avast version 4.8:

- Boot time: 76 seconds

- Available physical memory: 261 mb

- Total memory usage: 249 mb

- Full deep scan: 1140 seconds

- Standard scan: 260 seconds

- Resource usage during scan is shown below:

Next was ESET NOD32 Business 4.0:

- Boot time: 98 seconds

- Available physical memory: 296 mb

- Total memory usage: 182 mb

- Full deep scan: 290 seconds

- Standard scan: 175 seconds

- Resource usage during scan is shown below:

And finally Microsoft Security Essentials :

- Boot time: 80 seconds

- Available physical memory: 251 mb

- Total memory usage: 261 mb

- Full deep scan: 757 seconds (when selecting a foldertree you're not able to select scanning method)

- Standard scan: n/a

- Resource usage during scan is shown below:

Conclusions:

NOD32 has the lowest memory footprint and the fastest full scanning (although I suspect that it didn't scan inside all archives), but boot-up time is slightly slower than the Avast and MSE. CPU usage during scan is a little lower with Avast than MSE and NOD32, but for all practical purposes you probably won't notice any difference. In fact all these solutions seems to do fairly well on my old, slow laptop.

All three solutions found the malware and viruses I had put in there for the test.

So which one would I choose?

Probably NOD32 if I had the money to spend (especially for business purposes since the business version integrates with a central policy server).

If I had to go for a free solution I would choose MSE. It seems to perform as well as Avast, but has a better and more intuitive GUI (although I miss some advanced options that Avast have). In addition I won't have to go to Avasts webpage to register every year (I have previously installed Avast for friends and family, and this renewal process that Avast requires is really confusing for a lot of people). MSE is basically set-it-and-forget-it.

My only worry is that I have a feeling that Malware/virus-makers are going to target MSE directly if too many people start using it :-)

iTunes 9 - slow startup

After upgrading to iTunes version 9, iTunes is incredibly slow at starting. I have a library of about 2300 songs and it went from loading in about 5 seconds on the last 8.x relase to apx 30 seconds on version 9.0.1.8.

I tried a few tweaks like turning off automatic update on playlists, disabling automatic discovery etc but nothing really made much difference.

When searching through the library file (usually c:\users\{username}\music\itunes\iTunes Music Library.xml) i noticed it had a few links to a network share from which I imported a few files rather than to the local library folder on my c-drive.

These remained from before I checked the "copy files to the iTunes Media folder..."-option. Apparently iTunes 9.x now tries to verify these files upon startup (and fails when the network share is not available). Removing the files from the library and importing them again (creating copies to my local itunes music folder) did the trick. iTunes launching time is now back to what it used to be.

Exchange 2007 - old routing group connectors

The following error recently started showing up on our Exchange 2007 server:

Event Type: Warning

Event Source: MSExchange ADAccess

Event Category: Validation

Event ID: 2159

Date: 20.09.2009

Time: 14:15:30

User: N/A

Computer: EXSRV1

Description:

Process edgetransport.exe (Transport) (PID=4900). Configuration object CN=EXSRV1-EXSRV20,CN=Connections,CN=Exchange Routing Group (DWBGZMFD01QNBJR),CN=Routing Groups,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=DN,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=mycompany,DC=intern read from dc1.mycompany.intern failed validation and will be excluded from the result set. Set event logging level for Validation category to Maximum to get additional events about each failure.

EXSRV20 is our old Exchange 2000 server, and looking at the object with adsiedit shows me that I indeed seem to have forgotten to remove the routing group connectors used when upgrading to Exchange 2007 last year. Brainfart! I guess I was lucky that the object is not valid anymore so the error is being logged.

Powershell does the trick:

Get-RoutingGroupConnector

Name SourceRoutingGroup TargetRoutingGroup

---- ------------------ ------------------

EXSRV1-EXSRV20 Exchange Routing Group ... TRONDHEIM

WARNING: Object DNSRV1-DNSRV20 has been corrupted and it is in an inconsistent

state. The following validation errors have occurred:

WARNING: TargetTransportServerVsis is mandatory.

WARNING: TargetTransportServerVsis is mandatory.

EXSRV20-EXSRV1 TRONDHEIM Exchange Routing Group ...

The following commands:

Remove-RoutingGroupConnector EXSRV1-EXSRV20

and

Remove-RoutingGroupConnector EXSRV20-EXSRV1

disposed of the two connectors in a clean manner, and the eventlog is clean again.

Windows 7 hibernate problems

Theres been one problem with my installation of Windows 7 so far. Ever so often when I start up from hibernate the computer will just omit the hibernate procedure and boot normally instead. It will only happen maybe one out of ten times, but it always seem to happen when I have a lot of stuff running so it's annoying!

According to Microsoft (http://support.microsoft.com/default.aspx/kb/974772) this could be due to a an inconcistency between the physical amount of ram, and the size of the hibernate file. I have 4 gigs and the hiberfil.sys appears to be too small:

C:\Windows\system32>dir /a c:\hiberfil.sys

Volume in drive C has no label.

Volume Serial Number is F08F-0F2D

Directory of c:\

17.09.2009 21:32 2 817 048 576 hiberfil.sys

Microsoft recommends increasing the size (run from cmd.exe as administrator):

C:\Windows\system32> powercfg /hibernate /size 100

The hiberfile size has been set to: 3756064768 bytes.

C:\Windows\system32>dir /a c:\hiberfil.sys

Volume in drive C has no label.

Volume Serial Number is F08F-0F2D

Directory of c:\

18.09.2009 00:25 3 756 064 768 hiberfil.sys

Now I guess we'll just wait and see if it had any effect.

Locale error message in MRBS 1.2.1

We are currently using this neat little php web application for booking resources (meeting rooms, video projectors, video conference equipment etc) called mrbs (Meeting room booking system, http://mrbs.sourceforge.net/). It's really a perfect (free) tool for our use - we tried to use Outlook/Exchange for this earlier but just had to give up - it was too complex for our users. Anyway, now especially our secretaries love mrbs, and as we all know - happy secretaries means happy sysadmins.

It's currently running on an old RedHat 9 server so it's time to move it to a new server.

After installing mrbs 1.2.1 on my new Ubuntu 8.04 server and importing the database from my old Redhat 9 server things worked fine - except one thing. On top of each screen it would always say:

[Warning: Server failed to set locale to "no_NO.utf-8" (Unix)] (in Internet Explorer)

or

[Warning: Server failed to set locale to "en_GB.utf-8" (Unix)] (in Chrome)

On the server my locale is en_US.UTF-8 (LANG=en_US.UTF-8) and after changing the parameter:

$override_locale = "";

to

$override_locale = "en_US-UTF8";

in config.inc.php the annoying error disappeared.

Windows 2003 - dcdiag.exe fails on systemlog test

During Active Directory cleanup I found that dcdiag.exe (see below) gives me a failure warning on the systemlog. I thought this was weird on a otherwise well functioning domain so I decided to check it out.

C:\Program Files\Support Tools>dcdiag

Domain Controller Diagnosis

Performing initial setup:

Done gathering initial info.

Doing initial required tests

Testing server: Default-First-Site-Name\dc1

Starting test: Connectivity

......................... dc1 passed test Connectivity

Doing primary tests

Testing server: Default-First-Site-Name\dc1

Starting test: Replications

......................... dc1 passed test Replications

Starting test: NCSecDesc

......................... dc1 passed test NCSecDesc

Starting test: NetLogons

......................... dc1 passed test NetLogons

Starting test: Advertising

......................... dc1 passed test Advertising

Starting test: KnowsOfRoleHolders

......................... dc1 passed test KnowsOfRoleHolders

Starting test: RidManager

......................... dc1 passed test RidManager

Starting test: MachineAccount

......................... dc1 passed test MachineAccount

Starting test: Services

......................... dc1 passed test Services

Starting test: ObjectsReplicated

......................... dc1 passed test ObjectsReplicated

Starting test: frssysvol

......................... dc1 passed test frssysvol

Starting test: frsevent

......................... dc1 passed test frsevent

Starting test: kccevent

......................... dc1 passed test kccevent

Starting test: systemlog

An Error Event occured. EventID: 0x00000457

Time Generated: 03/20/2009 09:50:04

(Event String could not be retrieved)

An Error Event occured. EventID: 0x00000457

Time Generated: 03/20/2009 09:50:05

(Event String could not be retrieved)

An Error Event occured. EventID: 0x00000457

Time Generated: 03/20/2009 09:50:05

(Event String could not be retrieved)

......................... dc1 failed test systemlog

Starting test: VerifyReferences

......................... dc1 passed test VerifyReferences

Running partition tests on : ForestDnsZones

Starting test: CrossRefValidation

......................... ForestDnsZones passed test CrossRefValidation

Starting test: CheckSDRefDom

......................... ForestDnsZones passed test CheckSDRefDom

Running partition tests on : DomainDnsZones

Starting test: CrossRefValidation

......................... DomainDnsZones passed test CrossRefValidation

Starting test: CheckSDRefDom

......................... DomainDnsZones passed test CheckSDRefDom

Running partition tests on : Schema

Starting test: CrossRefValidation

......................... Schema passed test CrossRefValidation

Starting test: CheckSDRefDom

......................... Schema passed test CheckSDRefDom

Running partition tests on : Configuration

Starting test: CrossRefValidation

......................... Configuration passed test CrossRefValidation

Starting test: CheckSDRefDom

......................... Configuration passed test CheckSDRefDom

Running partition tests on : dirnat

Starting test: CrossRefValidation

......................... dirnat passed test CrossRefValidation

Starting test: CheckSDRefDom

......................... dirnat passed test CheckSDRefDom

Running enterprise tests on : dirnat.intern

Starting test: Intersite

......................... dirnat.intern passed test Intersite

Starting test: FsmoCheck

......................... dirnat.intern passed test FsmoCheck

First thing I did was to check the system log on the local server, and I found the culprit:

Event Type: Error

Event Source: TermServDevices

Event Category: None

Event ID: 1111

Date: 17.08.2009

Time: 09:50:04

User: N/A

Computer: dc1

Description:

Driver Microsoft Shared Fax Driver required for printer Fax is unknown. Contact the administrator to install the driver before you log in again.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Data:

0000: 00 00 00 00 12 0d 00 00 ........

I also found two more log posts from 09:50:05 saying exactly the same (with different printer names). Hex 457 (0x00000457) = 1111, so basically it seems that dcdiag.exe gives me this warning due to the fact I was running from within a terminal services client session and local printer mapping had failed during connect (which it always does since I don't have the drivers for these printers installed on the servers). I suppose I could always disable printer mapping in the mstsc client prior to connecting, but since it's harmless I think I'll just live with it.