Windows 2008 terminal server licenses not being assigned to users

We have a Windows 2003 domain and have just set up some new terminal servers using a Windows 2008 terminal server licensing manager server in the domain (per user licensing). This license server is not a DC.

The problem was that most users would not be assigned licenses from the license server and the event viewer on the licensing manager server said the following:

"The Terminal Services license server cannot update the license attributes for user "XXX" in the Active Directory Domain "mydomain.intern". Ensure that the computer account for the license server is a member of Terminal Server License Servers group in Active Directory domain "mydomain.intern".
If the license server is installed on a domain controller, the Network Service account also needs to be a member of the Terminal Server License Servers group.
If the license server is installed on a domain controller, after you have added the appropriate accounts to the Terminal Server License Servers group, you must restart the Terminal Services Licensing service to track or report the usage of TS Per User CALs."

Well, sure enough the server in question was not member of the "Terminal Server License Servers" group at first but was later added. Then we restarted both the terminal servers and licensing servers but the problem remained.

A little further investigation showed that this problem occured for apx 2/3 of our users. Checking users permissions with powershell:

get-user jdoe |get-adpermission -user s-1-5-32-561 |fl

User : S-1-5-32-561
Identity : mydomain.intern/Users/John Doe
Deny : False
AccessRights : {ReadProperty, WriteProperty}
ExtendedRights :
IsInherited : False
Properties : {Terminal-Server}
ChildObjectTypes :
InheritedObjectType :
InheritanceType : None

revealed that the group "Terminal Server License Servers" is present with some special permissions on the accounts who works. Checking an account that does not work did not list this permission. By the way S-1-5-32-561 refers to the "well known SID" for the group we're looking for:

SID: S-1-5-32-561
Name: BUILTIN\Terminal Server License Servers
Description: An alias. A group for Terminal Server License Servers.

My first thought was that it was inheritance related, but the users OU showed no trace of the "Terminal Server Licensing Servers"-group and looking at the output above clearly says "inheritancetype: none". Interestingly enough all newly created users gets the correct permissions which indicates that the permissions comes as a part of default settings from the AD-Schema. When checking the schema I saw that the "Terminal Server Licensing Servers"-group indeed is present with some kind of permissions on the users object, but the AD Schema MMC-snapin doesnt seem to be able to list which particular permissions this is.

Anyway - at one point a job must have been triggered that tried to set these permissions for all user accounts (?) in my domain, but it must have stopped at one point. I can't really think of any other possibility than that this must have happened during the TS licensing server installation. Can this somehow be trigged manually? Or is there another way to get this done by the book? Google brought me no further.

The obvious and simple workaround was to simply add the permissions manually through powershell and hope for the best, but it still leaves me with a feeling that something is not quite right.

Update:
Thanks to powershell I succeeded in finding a pattern. The following script lists all users where "Terminal Server License Servers" have access rights:

$users = get-user | get-adpermission -user s-1-5-32-561
write-host $users.count
$count = 0
while ($count -lt $users.count)
{
   write-host $count $users[$count].identity
   $count++
}

The users listed had the following in common; they either:
a) were created after we upgraded the domain from windows 2000 to Windows 2003 (SP1)
  or
b) were members of domain admins, account operators or printer operators.

We already know that new accounts receive the correct access right upon creation and admins/operators accounts automatically have their access rights adjusted automatically by Active Directory mechanisms running hourly on the PDC-emulator according to permissions set on System/adminSDholder container in AD (just try adding a user to account operators and notice that account operators access rights to this user account will disappear after a while by itself).

In other words - these permissions have never been applied to all user accounts in the domain. I cannot imagine that this is by design, if it was there would be a lot of MS customers scratching their heads. I suppose our domain is an exception, so I guess I'll just stick to applying the necessary access rights settings by powershell (with the following command).

get-user * | Add-ADPermission -user S-1-5-32-561 -accessrights ReadProperty,Writeproperty -properties Terminal-Server -inheritancetype none 

HP VLS6636 deduplication real world performance

A few months we installed a new virtual tape library - a HP VLS6636. The library and our Data Protector 6.11 backup server are connected through a FC switch, and most server backups are done via regular gigabit ethernet.

Currently the library has 24 x 476(500) GB SATA disks. A total of apx 11 TB raw capacity which after initialization leaves 8.8 TB physical capacity. 1.04 TB is reserved by the system (temp-space for deduplication etc) so the space left for backup data is 7.76 TB.

The total size of all our backup data is apx 3,04 TB with an average of 0,26 TB differential data daily. It's all kinds of data - but mainly Winows Server OS. My goal was to hold 5 weeks of full an ddifferential (Mon-Thu) backups, some manual backups and in addition leave room for some future growth. The rough calculation for our current needs is like this:

Full backup: 5 x 3.04 TB
Differential backup: 4 x 0.26 TB x 5 (4 days a week for 5 weeks)
A total of 15.2 + 5.2 = 20,4 TB
+ 10 TB growth and manual backups

In other words - we require 4 times the available space on the VLS6636. Time to start filling it up :)

The first reading (7.1) is done after two full backups and a few manually defined backups.

The increase in Logical data usage after 4.2 is due to a couple of full backups outside the regular backup schedules, so you're in fact looking at 7 full backup sets.Notice that even though the Logical data increases sharply, there's hardly any increase in used capacity.

After 11.3 you'll notice a sharp increase in the logical data and a decrease in available capacity. This is due to changes in backup procedures for Virtual Servers (VRanger Pro upgrade). Basically the same VM's are being dumped to file, but the dump - and metadata files look different and the VLS6636 naturally interprets it as new data and thus ruining the nice stats. I expect the data usage to go back to what it was prior to 11.3 once backup data from the old VM backup procedures have expired.

This illustrates the importance of remembering things like this when using deduplication technology. Seemingly small changes can have a significant impact on storage needs. This also goes for the backup jobs. Creating two similar jobs with different names will be considered completely different sets and thus they will not deduplicate at all.

All in all; as long as the backup definitions and data are fairly static in terms structure and contents - the deduplication works very well. Even the differential backups deduplicate quite well, I would estimate an average of 5:1.

When it comes to speed the VLS6636 really performs decent. We experience speeds up to 60 MB/sec when using 4 simultanous datastreams. On average mixed backup to VLS will happen at apx 40/MB sec. The backup server is fairly old, so I would not be surprised if the results will be even better with a new backup server.

We have also restored files for verification purposes, and haven't noticed any problems. Yet ;-)

HP BladeSystem C7000 Onboard Administrator lost contact with ILO2 on a Proliant bl460c blade

A while ago we upgraded our HP BladeSystem C7000 Onboard Administrator (OA) firmware to the latest 2.60. A few weeks later we upgraded the ILO2 firmware on the blades to 1.80. All but one worked fine. The last blade would not communicate properly with OA. Nothing serious I suppose, but I'm always suspicious - thinking that small issues could lead to something horrible if not dealt with properly. Plus - it's annoying.

Symptoms:

• I could not initiate a ILO remote control session from the OA web interface (it would time out and say "Unable to perform autologon").
• The only way I could connect to ILO was adding a local user with HP Lights-Out online configuration utility (hponcfg) from Windows running locally on the blade.
• The configured ILO DNS-name would not register with OA.
• OA / Device Bay Summary would report "iLO name: [Acquiring]" and "iLO DVD Status: unknown".
• IML Log for the blade was not available from OA.

I tried all tricks I could think of:

• Ran a emulated eject/insert of the blade ("reset server bay XX" from OA CLI).
• Reset all settings on the ILO card to default from hponcfg.
• Rebooted the ILO card from hponcfg.
• Upgraded ILO2 firmware to 1.81
• Restarted OA.

However - what finally solved it was simple enough. I simply reapplied the 2.60 firmware upgrade again. Fair enough.

HP Data Protector 6.11 post upgrade database cleanup and defrag

I recently upgraded our Data Protector backup software to 6.11 to get the latest and the greatest. Everything seemed to work just fine, but after a few backup runs I noticed that backing up the Data Protector Database did not work as it should. The log said:

[Normal] From: DBBDA@dpserv.mydomain.intern "[Database]: dpserv.mydomain.intern" Time: 18.12.2009 14:35:25
 STARTING Database HotBackup Disk Agent on dpserv.mydomain.intern "[Database]: dpserv.mydomain.intern".

[Major] From: DBBDA@dpserv.mydomain.intern "[Database]: dpserv.mydomain.intern" Time: 18.12.2009 14:35:25
[81:522] Cannot back up internal database because another database check in progress.

[Warning] From: DBBDA@dpserv.mydomain.intern "[Database]: dpserv.mydomain.intern" Time: 18.12.2009 14:35:26
Cannot un-lock the keystore.

[Major] From: DBBDA@dpserv.mydomain.intern "[Database]: dpserv.mydomain.intern" Time: 18.12.2009 14:35:26
ABORTED Database HotBackup Disk Agent on dpserv.mydomain.intern"[Database]: dpserv.mydomain.intern"

A manual run of omnidbcheck revealed the same thing. I could not find any signs of any other database checks, and a reboot did not help.

Deleting the lock-file (dbcheck.lk) from the {omniback}/tmp folder did the trick though. The backup would go through, but for some reason the backup was unbelievably slow. In fact it would hang for almost 3 hours before even beginning to backup the database. This was not the case on DP 6.0 only one week earlier. omnidbcheck would not report any errors though. Apparently the database needed some cleaning and defragmentation.

I started off running a "omnisv -stop" (to stop all services) and did a cold backup of the whole {omniback}-foldertree. Then I started the services again with "omnisv -start".

I then ran "omnidbutil -info" which showed the following:

Media Management database space usage:

Space used by Diskspace usage Records used Records total
============================================================================
Devices 96 17 20
Libraries 96 2 5
Cartridges 1184 296 852
Compounds 32 0 12
Pools 64 26 30
Media 160 288 297

Catalog database space usage:

Space used by Diskspace usage Records used Records total
============================================================================
Sessions 352 197 284
Objects 256 517 552
Object versions 10080 5236 7810
Positions 12896 77086 97960
Filenames 3175520 31876226 31876278

Detail catalog binary files usage:

Diskspace usage Size limit DC Directory
============================================================================
2068 4096 C:/Program Files/OmniBack/db40/dcbf
----------------------------------------------------------------------------
2068 4096

Session messages binary files usage:

Diskspace usage Num. of files SMBFs Directory
============================================================================
16 191 C:/Program Files/OmniBack/db40/msg

Serverless integrations binary files usage:

Diskspace usage Num. of files SIBFs Directory
============================================================================
0 0 C:/Program Files/OmniBack/db40/meta

I decided to try and purge unused and obsolete filenames from the database:

"omnidbutil -purge -filenames -force" (took a couple of hours) and another "omnidbutil -info" showed this:

Media Management database space usage:

Space used by Diskspace usage Records used Records total
============================================================================
Devices 96 17 20
Libraries 96 2 5
Cartridges 1184 296 852
Compounds 32 0 12
Pools 64 26 30
Media 160 288 297

Catalog database space usage:

Space used by Diskspace usage Records used Records total
============================================================================
Sessions 352 203 284
Objects 256 517 552
Object versions 10080 5354 7810
Positions 12896 77366 97960
Filenames 3175520 13472221 31876336

Detail catalog binary files usage:

Diskspace usage Size limit DC Directory
============================================================================
2099 4096 C:/Program Files/OmniBack/db40/dcbf
----------------------------------------------------------------------------
2099 4096

Session messages binary files usage:

Diskspace usage Num. of files SMBFs Directory
============================================================================
17 197 C:/Program Files/OmniBack/db40/msg

Serverless integrations binary files usage:

Diskspace usage Num. of files SIBFs Directory
============================================================================
0 0 C:/Program Files/OmniBack/db40/meta

Right, apparently there was a lot of unused filename records.

According to HP the best way to clean up and defrag the database would be to export all records and import them right back in again.

I created two empty folders (f:\dump\dp\cdb and f:\dumå\dp\mmdb) and ran

omnidbutil -writedb -mmdb f:\dumå\dp\mmdb -cdb f:\dump\dp\cdb

followed by:

omnidbutil -readdb -mmdb f:\dumå\dp\mmdb -cdb f:\dump\dp\cdb

I then followed the on-screen instructions and took a backup of {omniback}\dcbf and {omniback}\msg
Again I ran "omnidbutil -info"

Media Management database space usage:

Space used by Diskspace usage Records used Records total
============================================================================
Devices 96 17 20
Libraries 96 2 5
Cartridges 448 296 297
Compounds 32 0 12
Pools 64 26 30
Media 160 288 297

Catalog database space usage:

Space used by Diskspace usage Records used Records total
============================================================================
Sessions 288 209 212
Objects 256 517 528
Object versions 6688 5450 5460
Positions 10208 77217 77221
Filenames 1319392 13474587 13474618

Detail catalog binary files usage:

Diskspace usage Size limit DC Directory
============================================================================
2135 4096 C:/Program Files/OmniBack/db40/dcbf
----------------------------------------------------------------------------
2135 4096

Session messages binary files usage:

Diskspace usage Num. of files SMBFs Directory
============================================================================
17 203 C:/Program Files/OmniBack/db40/msg

Serverless integrations binary files usage:

Diskspace usage Num. of files SIBFs Directory
============================================================================
0 0 C:/Program Files/OmniBack/db40/meta

Notice that space usage for the catalog datbase/filenames has decreased by 1.8 GB. Nothing seems to be missing when using the DP Cell Manager GUI and running a omnidbcheck now takes 11 minutes! The slight increase in disk space usage is because I had to run a quite large backup before doing the writedb/readdb operation.

XenServer on HP bl460c servers - nx and sep CPU flags

We just installed two XenServers to check out they hype. People are praising XenServer left and right, so we figured what the heck.

We installed on two HP BL460c G1 blade servers. Installation was a breeze, but when we tried to create a pool with XenCenter and add the two servers we got the following message:

08.12.2009 12:57:22 Error: Adding server 'dnxen2.mydomain.intern' to pool 'DN XenPool' - The hosts in this pool are not homogeneous. cpus differ

A quick look reveals the following difference between the cpus :

[root@dnxen1 ~]# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5440 @ 2.83GHz
stepping : 6
cpu MHz : 2833.454
cache size : 6144 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu de tsc msr pae cx8 apic sep mtrr cmov pat clflush acpi mmx fxsr sse sse2 ss ht nx constant_tsc up pni vmx est
bogomips : 5668.59


[root@dnxen2 ~]# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5440 @ 2.83GHz
stepping : 6
cpu MHz : 2833.456
cache size : 6144 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu de tsc msr pae cx8 apic mtrr cmov pat clflush acpi mmx fxsr sse sse2 ss ht constant_tsc up pni vmx est
bogomips : 5668.07

The flags "sep" and "nx" are enabled on dnxen1, but not on dnxen2. According to cpufeature.h "nx" is is "Execute Disable" and is a AMD-defined CPU feature (Strane since the CPU is Intel) and "sep" is something called "Sysenter/sysexit". Doesnt tell me much - but this page has a better description of NX: http://blog.incase.de/index.php/cpu-feature-flags-and-their-meanings/

"NX No eXecute, a flag that can be set on memory pages to disable execution of code in these pages"

And it seems like "sysenter/sysexit" are cpu instructions.

So let's have a look at the BIOS:

Advanced Options > Processor options > No-Execute memory protection (Changed from disabled to enabled on dnxen2).

The "Sysenter/sysexit" setting is nowhere to be found in the bios. In fact - there were no other differences between the two servers at all.

However after rebooting both servers list the same flags, so apparently enabling "nx" will enable the "sep"-flag as well. Well, XenServer is happy and and dnxen2 is was able to join the pool.

Off to do more testing.

Netapp FAS 2040 CIFS backup performance benchmark

We recently invested in a new filer from Netapp. Unfortunately we can't do agent based backups of the large CIFS-filesystem on the filer, and NDMP dump to tape is also problematic. Actually the NDMP backup itself is not a problem, but restore is. NDMP backups initiated from Data Protector will do a raw block copy, not a file copy of the individual files on the filesystem and thus making restore quite interesting.

We still chose to go for NetApp though, because we trust that the snapshot functionality will allow us to use less backups to tape (hopefully as little as once or twice a month) and thus making high speed a less of a critical issue.

The obvious solution is to do a normal Network share backup from DP (in production this will probably be done on a snapshot, but for now this will do). Basically I'm going to run a series of tests to determine the number of data streams and virtual tape devices which will give the best performance. Note that there are no other users attached to the NetApp filer or the Ethernet switch.

The results are considerably slower than what we can expect from a larger backup due to the relatively large overhead with small jobs.

I noticed that this type of backup is very CPU intensive on the Cell Manager. It would pretty much max out at 100% constantly during. Memory was not a problem though.

Setup

Server: HP Proliant DL380 G3 / Windows 2003 x86 / 2 GB ram / Dual Intel Xeon 2.8 ghz
Backup software: Data Protector 6.0
Network: HP Procurve 2824 switch (single gigabit connection)
Storage system: NetApp FAS 2040
Tape library: HP 6636 VLS (Virtual tape library) connected to the DP Cell Manager by FC.
Test data: A collection of user home folders on 7 shares containg 39.108 files (19.932 MB)


Test Results

Test 1
1 data stream / 1 drive (Load balancing Min:1 / Max:1 and Concurrency: 1
Total 1427 seconds = 13,97 MB/sec

Test 2
2 data streams / 2 drives (Load balancing Min:2 / Max:2 and Concurrency: 1 per drive)
Total 1128 seconds = 17,67 MB/sec

Test 3
4 data streams / 2 drives (Load balancing Min:2 / Max:2 and Concurrency: 2 per drive)
Total 1002 seconds = 19,9 MB/sec

Test 4
2 data streams / 1 drives (Load balancing Min:1 / Max:1 and Concurrency: 2 per drive)
Total 1135 seconds = 17,56 MB/sec

Test 5
3 data streams / 1 drives (Load balancing Min:1 / Max:1 and Concurrency: 3 per drive)
Total 1015 seconds = 19.64 MB/sec

Test 6
4 data streams / 1 drives (Load balancing Min:1 / Max:1 and Concurrency: 4 per drive)
Total 983 seconds = 20,28 MB/sec

Test 7
5 data streams / 1 drives (Load balancing Min:1 / Max:1 and Concurrency: 5 per drive)
Total 994 seconds = 20,05 MB/sec

Test 8
7 data streams / 1 drives (Load balancing Min:1 / Max:1 and Concurrency: 7 per drive)
Total 964 seconds = 20,67 MB/sec

Test 9
8 (7) data streams / 2 drives (Load balancing Min:2 / Max:2 and Concurrency: 4 per drive)
Total 960 seconds = 20,76 MB/sec

Test 10
I decided to increase the amount of data to 37508 mb and rerun the backup with settings from Test 8 to see if the results would be any better.
Total 1597 seconds = 23,49 MB/sec

I was a little disappointed about the results. They are considerably slower than what I experienced with the old HP MSA1000 SAN using a backup agent. On the other hand - I knew that this type of backup is slow.

It also seems that you wont gain much in terms of speed from using > 3 data streams, but you will stress the Netapp filer a little more (I noticed CPU would be higher the more streams you use. One stream would average 9% on the Netapp and 8 streams would average 14%).

Perhaps I can achieve slightly better results by using a faster CPU on the Data Protector Cell Manager server, but I doubt we're talking about anything higher that 30MB/sec at best.

On the other hand - it really does not matter much on a day to day basis if I'm only going to do a full backup to tape only a couple of times a month. A 3 TB volume should take apx 36 hours to finish.

Microsoft Security Essentials Performance test

Today I grabbed the release version (v1.0) of Microsofts new Security Essentials package (aka Morro) and decided to run a quick and dirty performance benchmark to compare it to a couple of other AntiVirus/Malware solutions.

I tested on an old Celeron 2.2ghz laptop with 512mb ram and Windows XP SP3.

I did the following measurements:

- Boot-up time (from switching on until notepad.exe in the startup folder was running).

- Avatilable physical memory (after bootup has finished)

- Total memory usage

- Deep and Standard scan of a folder containing apx 8000 files /10 gb.

- CPU usage during scan.

First out was Avast version 4.8:

- Boot time: 76 seconds

- Available physical memory: 261 mb

- Total memory usage: 249 mb

- Full deep scan: 1140 seconds

- Standard scan: 260 seconds

- Resource usage during scan is shown below:

Next was ESET NOD32 Business 4.0:

- Boot time: 98 seconds

- Available physical memory: 296 mb

- Total memory usage: 182 mb

- Full deep scan: 290 seconds

- Standard scan: 175 seconds

- Resource usage during scan is shown below:

And finally Microsoft Security Essentials :

- Boot time: 80 seconds

- Available physical memory: 251 mb

- Total memory usage: 261 mb

- Full deep scan: 757 seconds (when selecting a foldertree you're not able to select scanning method)

- Standard scan: n/a

- Resource usage during scan is shown below:

Conclusions:

NOD32 has the lowest memory footprint and the fastest full scanning (although I suspect that it didn't scan inside all archives), but boot-up time is slightly slower than the Avast and MSE. CPU usage during scan is a little lower with Avast than MSE and NOD32, but for all practical purposes you probably won't notice any difference. In fact all these solutions seems to do fairly well on my old, slow laptop.

All three solutions found the malware and viruses I had put in there for the test.

So which one would I choose?

Probably NOD32 if I had the money to spend (especially for business purposes since the business version integrates with a central policy server).

If I had to go for a free solution I would choose MSE. It seems to perform as well as Avast, but has a better and more intuitive GUI (although I miss some advanced options that Avast have). In addition I won't have to go to Avasts webpage to register every year (I have previously installed Avast for friends and family, and this renewal process that Avast requires is really confusing for a lot of people). MSE is basically set-it-and-forget-it.

My only worry is that I have a feeling that Malware/virus-makers are going to target MSE directly if too many people start using it :-)