Alasdair's Blog

Hard-core technical stuff - not for human consumption!

You can read the blog below or see the original at http://alasdaircs.spaces.live.com/blog/

24 February

Adventures with .NET and COM

I was working with PDFCreator and downloaded the 0.9.7 release. It refused to install the pdfforge .NET dll because it said I hadn’t got .NET 1.1. Well, as most .NET 1.1 assemblies will run happily in .NET 2.0, I thought I’d have a go with it.

I got the pdfforge dll and test scripts from SourceForge and tried the scripts. No dice. So next I looked at the assembly in Reflector and found an unresolved reference to iTextSharp, which also thinks its a .NET 1.1 assembly. The plot thickens though, as when I got the iTextSharp DLL, it still wouldn’t load it. It turns out that in the comments in SVN, the pdfCreator guys did say they built iTextSharp from source, and they’re reference is to v 4.1.2.1 without a strong name, whereas the latest release binary of iTextSharp is 4.1.2.0 with a strong name. In order to get them to “talk” I needed to either change the reference baked in to pdfforge.dll or change the assembly version baked in to iTextSharp (and remove it’s strong name), or build iTextSharp from source…

Well, enter the Reflexil add-in for Reflector. Wow! With this I had a choice – it’ll let me fiddle with assembly versions, strip strong names, fiddle with assembly references, even hack the code & save it back. Get it from SourceForge.

So, with my demo code running nicely in c#:

using System;
using pdfforge.PDF;

namespace AcsComms
{
    class Program
    {
        static void Main( string[] args )
        {
            PDF pdf = new PDF();
            pdf.CreatePDFTestDocument( @"C:\Temp\PDF\Testy.PDF", 1, "Have at it, matey!" );
        }
    }
}

I tried to RegAsm the pdfforge.dll. Two gotchas here.

The first time I did it, I forgot to use the /codebase option, so COM & mscoree couldn’t find the DLL.
On my x64 Vista, I used the RegAsm in C:\Windows\Microsoft.NET\Framework\v2.0.50727, which is 32-bit. Consequently the registry entries got written to HKEY_CLASSES_ROOT\Wow6432Node\CLSID, and I could only use the DLL in 32-bit land, e.g.

C:\Windows\SysWOW64\wscript.exe "C:\Program Files (x86)\PDFCreator\PlugIns\pdfforge\nUp.vbs"

This was fixed by re-running RegAsm in the 64-bit framework directory:

C:\Windows\Microsoft.NET\Framework64\v2.0.50727\regasm "c:\Program Files (x86)\DFCreator\PlugIns\pdfforge\pdfforge.dll" /codebase

So now I can just double-click on the demo .vbs scripts and they work just fine!

00:57 GMT | Read comments(0)

21 May

SCDPM 2010 Protecting Exchange Server 2003 SP2 – Selecting the correct ESEUTIL

We’ve recently commissioned a server running the new Microsoft System Center Data Protection Manager 2010 to provide continuous data protection for business critical servers, including Hyper-V hosts and their guests.

At the moment, there’s still an old 32-bit Windows Server 2003 SP2 server running Exchange Server 2003 SP2.

After reading the documentation on Technet I determined that we needed a copy of the 32-bit ESEUTIL from Exchange 2003 on the SCDPM server.

Specifically, it says:

The eseutil.exe and ese.dll versions that are installed on the most recent release of Exchange Server must be the same versions that are installed on the DPM server. In addition, you must update eseutil.exe and ese.dll on the DPM server if they are updated on a computer running Exchange Server after applying an upgrade or an update.

...

The versions of the Exchange Server Database Utilities (eseutil.exe) and ese.dll that are installed on the computer running the most recent edition of Exchange Server must be the same versions that are installed on the DPM server.

You use the Microsoft Exchange Server 2007 management tools to maintain up-to-date copies of eseutil.exe and ese.dll. Exchange Server 2007 is a native 64-bit application that includes 64-bit management tools. You can use the management tools to remotely administer your Exchange Server environment. If the remote computer is running a 32-bit operating system, you must download the Microsoft Exchange Server 2007 32-bit management tools (http://go.microsoft.com/fwlink/?LinkId=112325). The management tools are supported by all update methods such as Microsoft Update and Microsoft Systems Management Server (SMS).

From this I inferred that

The computer running the most recent version of Exchange is running 2003 SP2, so that’s the correct version of ESEUTIL.EXE and ESE.DLL to use.
The 32-bit versions are the correct ones.

I can categorically state now that the docs are wrong – if the DPM server is x64 (and DPM2010 is only x64), then the ESEUTIL.EXE and ESE.DLL must also be x64.

Anyway, this is what the symptoms look like…

The problem is with the First Storage Group in Exchange. This is the error in DPM2010 administrator console:

Affected area: First Storage Group
Occurred since: 20/05/2010 10:02:11
Description:    The replica of Storage group First Storage Group on tsp-svr-01.stokes.msi is inconsistent with the protected data source. All protection activities for data source will fail until the replica is synchronized with consistency check. You can recover data from existing recovery points, but new recovery points cannot be created until the replica is consistent.
For SharePoint farm, recovery points will continue getting created with the databases that are consistent. To backup inconsistent databases, run a consistency check on the farm. (ID 3106)
     Data consistency verification check failed for LOGS of Storage group First Storage Group on tsp-svr-01.stokes.msi. (ID 30146 Details: Unknown error (0xfffffc01) (0xFFFFFC01))
     More information
Recommended action: Either the database files are corrupt or the proper versions of the Eseutil.exe and Ese.dll files are missing.
If you have recently upgraded your Exchange server, copy them from that server to the DPM server.Contact your Exchange server administrator, and then verify the issue. You can recover the last known good backup to address the corrupted state.
     Synchronize with consistency check.
     Run a synchronization job with consistency check...
Resolution:     To dismiss the alert, click below
     Inactivate alert

I had copied the ESE.DLL and ESEUTIL.EXE 32-bit files directly from the protected Exchange server’s Exchsrvr\bin folder to C:\Program Files\Microsoft DPM\DPM\bin on the DPM2010 computer.

DPM was finding and starting the ESEUTIL, as the application event log shows the following:

Log Name:      Application
Source:        ESE
Date:          20/05/2010 10:01:51
Event ID:      100
Task Category: General
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      TSP-DPM-01.whatever
Description:

eseutil (2576) The database engine 6.05.7638.0002 started.

Followed by numerous errors such as this:

Log Name:      Application
Source:        ESE
Date:          20/05/2010 10:01:51
Event ID:      489
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      TSP-DPM-01.whatever
Description:
eseutil (2576) An attempt to open the file "C:\Windows\system32\Volume{167a68cd-62b8-11df-b24a-0026b98c66d4}\cf0d079c-a3b3-4eb7-885c-6e98e257b802\Logs\Logs\E001FEFF.log" for read only access failed with system error 3 (0x00000003): "The system cannot find the path specified. ". The open file operation will fail with error -1023 (0xfffffc01).

Each error quotes a different exchange log file name, but in the same (non-existent) folder.

Looking at it more closely, I think reason that the 32-bit ESEUTIL.EXE failed is because DPM mounts the volume containing the log files in a folder inside the System32 directory, and on an x64 machine, 32-bit processes will have the SysWOW64 folder mapped as System32, so they won’t see the mounted volumes as they’re mounted in the real System32, not SysWOW64...

One more thing - Initially I had difficulty deleting the 32-bit copy of ESE.DLL from the DPM\Bin folder - it said the file was in use by DHCHP Client, and when I stopped that service it said the file was in use by Event Log. Both of these must have been untrue, as it’s a 32-bit DLL and won’t load into a 64-bit process, but I did notice that the DPM\Bin folder is appended to the system Path. I got around this by removing DPM\Bin from the Path, rebooting the DPM server, deleting the ESE.DLL and then re-adding DPM\Bin folder to the Path. That wasted an hour.

After the reboot & successful delete, I put the Exchange Server 2007 SP1 x64 ESEUTIL.EXE and ESE.DLL in a separate folder and created hard links to them in the DPM\Bin folder.

Next, we ran the “Perform consistency check...” on First Storage Group and we now have a Green Tick – Yay!

00:56 GMT | Read comments(0)

11 April

Clustered Hyper-V

I’m working on a project to virtualise all of the business-critical servers for a client of mine (and then again, who isn’t at the moment). There is already an incumbent SAN provided by Datacore SANMelody running on HP Storage Server boxes. The three main servers have Fibre Channel connections to this SAN. They all boot from local C: drives, and also have one or two drive letters each provided by LUNs on the SAN, This configuration protects the data but not the C: drives, and does not provide any failover support in the event of a hardware fault.

The new configuration is going to be two new Dell R710 servers with 48GB RAM and Dual Xeon 5550s running Windows 2008 R2 with the Hyper-V role and Failover Clustering feature. We’ve got this running in a lab at the moment with a 30-day trail version of SAN Melody running on a 64-bit capable workstation and the two R710s connected to the SAN by iSCSI.. We have learned some important lessons from this which I share below.

Live Migration

I have to say, the live migration is very cool. However, it’s a real test of whether you’ve got everything configured correctly. I strongly recommend ensuring that any VM you create is tested with live migration several times over a couple of days before allowing it to be used for live workloads.

Network adapter binding order

I’m going to cover the build of the cluster hosts elsewhere. This post is mainly concerned with provisioning a new guest VM using CSV for storage of configuration and pass-through disks for the OS and data.

Cluster Shared Volumes(CSV) for VM Configuration Data

Well, apart from a clash of TLAs (and with 26^3=17576 available TLSs that does seem a bit unfortunate), CSV is a clever idea. There are some preconditions, for example all nodes in the cluster should share the same system driver (usually C:), and you’re only allowed to use CSV for Hyper-V storage, but apart from that, it seems a good idea, especially in VDI rollouts. I’ve found a slightly different use for them though… I’ve been warned off using CSV by my SAN vendor, who says there are SCSI command queuing issues or some such techie nonsense with putting lots of IO on a single LUN. More to the point though, we’re planning on using pass-through disks for performance and manageability, and that leads us to a little problem. In a Virtual Desktop Infrastructure (VDI) rollout you generally need storage for the VM configuration, the VHD for the C: drive and somewhere for the snapshot data, and a CSV is the perfect place for it all. But I want my C: drives to be on pass-through disks, and that means I can’t put the VM configuration on the same LUN as the C drive as the host can’t see the filesystem on the pass-through disk. So I’m then stuck with needing somewhere to store my VM configuration data. I can’t use a single ordinary LUN as it will need to be owned by a single node and all the VMs will depend on it. And I really don’t want to have to provision a whole separate LUN for each individual VM’s configuration & snapshot data. The answer is to use a single CSV to store the configuration and snapshot data for all my VMs. This helps speed up Live Migration, and significantly reduces the number of LUNs I need to provision, without putting much IO onto the CSV.

Provisioning Storage to my Guests

In server virtualisation there are often (usually?) several volumes that need to be served up to each guest. In an iSCSI world, you can mount additional disks in the guest using the iSCSI initiator inside the guest. When the guest moves from one host to another, it can keep it’s own connections to the SAN, and they can have multi-path IO and so on.. The same is theoretically possible in the Fibre Channel world using N-port ID virtualisation (NPIV – there are 456976 FLAs for no clash there as far as I’m aware). However, I’m again specifically and categorically warned off using NPIV with my SAN – it’s not supported (yet). This means I have to present all storage to the VM host, and then pass it on to the guests. The “Thin Provisioning” feature of my SAN means I can serve up dozens of huge LUNs from a relatively small pool of storage. Only the blocks actually used get allocated. I want all my guests to get direct access to this feature, I don’t like variable-size VHDs and so it’s pass-through disks for me. The important thing about this is that the pass-through disk has to be equally visible to all cluster nodes, and that means that the usual naming convention won’t be portable.

The fun starts with the SAN – you’ve got to provision the right sort of disk. Firstly, all disks which will be involved in clustering need to have SCSI3 Persistent Reservation support. Secondly, a VM can only boot from Virtual IDE, not from virtual SCSI disk, and there’s a maximum size for the disk presented to the Virtual IDE controller which is slightly less hat the default 2047.5GB (2096640 MB) presented by SANMelody by default. Here I am fixing the size and persistent reserve:

I’ve also selected Asymmetric Logical Unit Access as this give flexibility in assigning a preference in a multi-path IO (MPIO) mirrored active-active SAN configuration. I then mapped the volume to both servers in the cluster.

You need to select Rescan Disks to see the newly provisioned LUN showing up as disk 4 in Disk Administrator. Furthermore, you need to do this on all cluster hosts

So, it’s a Basic disk, 2TB (thin provisioned from the SAN), but it won’t work as a pass-through disk in Hyper-V unless we take it offline. Once it’s offline, I' add the disk to the cluster:

It’s a good idea to rename the clustered disk resource to something more memorable than “Cluster Disk 1”:

So, now I’m going to create a new clustered guest VM. The first thing I do is create the VM from the Services and Applications node of the cluster manager:

I put the configuration on my Clustered Shared Volume:

The other thing I need to do is not connect a VHD, because I’m not going to be using VHDs:

When I click Finish, the cluster software does its thing to make my VM highly available:

But remember – at this point it knows nothing about the pass-through disk. This is what it looks like in Failover Cluster Manager:

So next we edit the VM configuration:

We need to add a pass-through disk. Remember that only off-line disks on the VMs current host will show in the list,:

The cluster stuff will read the Hyper-V settings and work out the dependencies now. If you do things in a different order, you’ll most likely get errors in the virtual machine configuration refresh,. It will be very important to have a clean configuration refresh before you try to do anything with the VM.

At this point the Clustered VM should look like this:

Its time to start the VM and load the OS! I Connect to the VM, Capture the D: drive (which contains my Windows 2008 R2 DVD) to the virtual DVD and click start. Here you can see the pass-through disk in the installer:

Once the OS is installed there will be one or two partitions on your new LUN.

Shut down the guest and run a refresh on the storage node in the Cluster Manager. You will see the newly created partition(s) and volumes on the LUN. Note that Windows 7 editions which support BitLoccker and Windows Server 2008R2 create the 100MB partitionas well as the main partition for the C: drive:

Note that it is vitally important that the volume paths start with \\?\GLOBALROOT\. If they don’t, they won’t migrate between cluster nodes and you will become very unhappy.

Refresh the VM in Cluster Manager:

This is how it’s supposed to look:

I’d recommend a final refresh in Disk Administrator on the other cluster nodes.

Check your VM has the correct Live Migration network selected:

And that the dependencies look OK. Note that these are created by the Failover Cluster manager, not by me:

The final thing to do before try a live migration is a refresh of the VM Configuration. If you’re forgetful like me you’ll have left a captured DVD drive or ISO image mounted which isn’t cluster compatible:

The error will look like this:

Disk path 'IDE\CDROMPLDS_DVD+-RW_DS-8A4S____________________JD51____\5&36A28B5E&0&0.0.0' is not a path to storage in the cluster or to storage that can be added to the cluster. You must ensure this storage is available to every node in the cluster to make this virtual machine highly available.

Don’t panic, just remove the captured DVD from the VM. It’ll automatically do a refresh, but you can do another one to convince yourself it’s a clean configuration then we’re good to go with our 1st live migration attempt.

And it works.

Summary:

Provision volumes from the SAN with SCSI3 Persistent Reservation and (if available) ALUA support.

Bootable LUNs must be smaller that 2TB. Second and subsequent volumes can be as big as you like as they’ll be connected to the virtual SCSI controller on the VM, not the virtual IDE controller.
After provisioning, rescan the disks in Disk Administrator on all cluster nodes.
Bring the new disk into the cluster as a storage resource and give it a sensible name.
Take the disk offline (from the storage node of the Cluster Manager)
Create the new guest VM with its configuration on the CSV, but with no VHD attached
Edit the settings of the new VM (or an existing one) and add the LUN as a physical disk
Fire up the VM and install the OS (or partition & format the disk for non-boot disks) – remember always to use Quick Format or you’ll break the thin provisioning.
Shut down the VM.
Refresh disk administrator on all cluster nodes
Refresh the Storage node and expand the disk resource to see the volumes. They must start with \\?\GLOBALROOT\.
Check Clustered VM properties for correct live migration network and dependencies. Note that dependencies should be built for you.
Remove any non-clustered resources, such as the DVD or ISO you used to install the OS
Do a final refresh of the VM configuration in cluster manager and check it’s clean
Start your VM and try a live migration!

I hope this helps somebody.

05:24 GMT | Read comments(1)

24 October

Windows Internal Database

On Wikipedia there's a short article about Windows Internal database. There are several important facts they mention, and some they don't:

You can connect to it with any version of SQL Server Management Studio, including SQL Server Management Studio Express
It only allows connections for the local machine - it does not listen on TCP/IP or externally accessible named pipes.
The pipe name you type into the server name is \\.\pipe\mssql$microsoft##ssee\sql\query
Although you can create databases and drop objects, the super-secure mechanism to stop you using it for your own purposes is that you can't create objects (tables, stored procs, etc). What you get is:

Msg 33003, Level 16, State 1, Line 1
DDL statement is not allowed.
The workaround for point 4 above is to put the database in single user mode while you're doing the DDL: