Andre's Blog • Printing from a 32-bit IIS process on 64-bit Windows

Printing from a 32-bit IIS process on 64-bit Windows Posted Tue Jul 21 21:57:58 EDT 2009 in Computing by Andre Last week I came across a very strange problem with Windows printing - an attempt to create a printer device context through `CreateDC` within a 32-bit IIS worker process would fail and return `NULL`, but `GetLastError` would indicate that there was no error and there was nothing in the system or application event log to help me identify the problem. Naturally, my first thought was that the IIS anonymous user is missing some access rights and I spent some time double-checking various permissions and privileges, but found nothing that would be relevant in this case. Suspecting that the problem lies elsewhere, I added the anonymous user to the Administrators group, which actually made things worse. Now not only `CreateDC` would still return `NULL`, but it would actually take about a minute for this call to fail! The only way I could make `CreateDC` work was to call it from the security context of the Network Service user, which was undesirable in the context of the application I was working with. This started to get interesting. What's my session ID? I found the first bug quite quickly. It turned out that IIS sets up the worker process (`w3wp.exe`) such that the anonymous user cannot query the process. `CreateDC`, on the other hand, calls `ProcessidToSessionId` in order to figure out its Terminal Services session ID, which requires the user to have process query rights. So, `ProcessIdToSessionId` failed with the error Access Denied, which was thrown away by the caller, so upon return `CreateDC` returned no error. In order to test this theory, I granted the Users group, which includes all authenticated users, the right to query the worker process using code similar to this simplified fragment: CDacl dacl; CSecurityDesc secdesc; SECURITY_DESCRIPTOR _secdesc = NULL; ACL _dacl = NULL; if(GetSecurityInfo(GetCurrentProcess(), SE_KERNEL_OBJECT, DACL_SECURITY_INFORMATION, NULL, NULL, &_dacl, NULL, (PSECURITY_DESCRIPTOR) &_secdesc) == ERROR_SUCCESS) { secdesc = _secdesc; dacl = *_dacl; dacl.AddAllowedAce(Sids::Users(), PROCESS_QUERY_INFORMATION); SetSecurityInfo(GetCurrentProcess(), SE_KERNEL_OBJECT, DACL_SECURITY_INFORMATION, NULL, NULL, (PACL) dacl.GetPACL(), NULL); } , and then restored the original access rights after calling `CreateDC`. This allowed me to go past the first problem, but now `CreateDC` started to pause for about one minute before failing the same was it did when the anonymous user was a member of the Administrators group. What RPC server?! I attached a debugger to the worker process and after jumping through a few call stacks figured out that the process was busy calling `__rpc_mgmt_is_server_listening@8`, which was consistently failing to locate some RPC server, then calling `__imp__Sleep@4` and repeating the entire sequence for about a minute. The call stack indicated that IIS was trying to create a proxy object to allow the 32-bit IIS worker process call 64-bit printer driver: rpcrt4.dll!_RpcMgmtIsServerListening@4() + 0x28 bytes winspool.drv!_ConnectToLd64In32ServerWorker@8() + 0x19b bytes winspool.drv!_ExternalConnectToLd64In32Server@4() + 0x1a bytes gdi32.dll!PROXYPORT::PROXYPORT() + 0x189 bytes gdi32.dll!_LoadUserModePrinterDriverEx@20() + 0xfda bytes gdi32.dll!_LoadUserModePrinterDriver@16() + 0x40 bytes gdi32.dll!_hdcCreateDCW@20() + 0x51e3 bytes gdi32.dll!_bCreateDCW@20() + 0x91 bytes gdi32.dll!_CreateDCW@16() + 0x18 bytes After spending some time going through all of the RPC endpoints on my machine, I realized that the one IIS was trying to connect to was not available, so I spent some time looking for a reason this RPC server wasn't running. Where did it go?! After more digging through numerous screens of assembly instructions, I found out that the RPC server is supposed to be started by the worker process itself and, interestingly enough, the `CreateProcess` call: winspool.drv!_ConnectToLd64In32ServerWorker@8() + 0x20f bytes winspool.drv!_ExternalConnectToLd64In32Server@4() + 0x1a bytes gdi32.dll!PROXYPORT::PROXYPORT() + 0x189 bytes gdi32.dll!_LoadUserModePrinterDriverEx@20() + 0xfda bytes gdi32.dll!_LoadUserModePrinterDriver@16() + 0x40 bytes gdi32.dll!_hdcCreateDCW@20() + 0x51e3 bytes gdi32.dll!_bCreateDCW@20() + 0x91 bytes gdi32.dll!_CreateDCW@16() + 0x18 bytes was actually successful. The name of the executable that was being launched was `splwow64.exe`, which was the missing RPC server, whose purpose is to host 64-bit printer drivers and communicate with 32-bit processes that need to print. I started Process Monitor and set the filters to record all activity generated by `splwow64.exe`. Much to my surprise, none of what this process did triggered any errors (aside from some of the debug entries it did not find in the registry). `splwow64.exe` simply started, read a bunch of registry entries, loaded a few DLLs and then exited, leaving IIS puzzled about what just happened. Hacky x86/x64 printing Seeing some of the code and reading about `splwow64.exe`, I couldn't shake the feeling that this whole x86/x64 printing bridge was a hack somebody put together at the last minute before the release and then just left it there as is. For example, one knowledge base article indicates that only one user can use `splwow64.exe` at a time: http://support.microsoft.com/kb/923357 It seems that printing from 32-bit processes definitely wasn't too high on the list of Windows deliverables. Dear Microsoft At this point I was running out of time and decided to contact Microsoft, to let people with source code take a stab at the problem. I created a sample project demonstrating the problem and accompanied it with a detailed description of what was going on. Unlike with many other companies, I got through to Microsoft quite quickly. May be having to pay $300 for a premium support incident was helping a bit. A support engineer connected to my machine and I walked him through the code and demonstrated all the problems I identified. A few hours later I had to repeat everything to another guy who specialized in IIS/.Net and then again the next day to somebody from the printing team. September 16th, 2009 Microsoft finally called me to inform that they have completed the investigation and created a bug. They also told me that this bug is low priority and will not be fixed in the foreseeable future, if ever due to small number of complaints. All customers who encountered this bug are advised to move to Windows 7 and Windows Server 2008 RC2, where all listed issues have been fixed. I hope they are right. Name: Comment:	Index Login All Computing Employment Gaming Photography Programming Security Stone Steps Webalizer 2010 August (2) 2010 April (1) 2010 January (1) 2009 December (1) 2009 October (3) 2009 August (1) 2009 July (2) 2009 June (1) 2009 May (2) 2009 April (1) 2009 January (2) 2008 December (3) 2008 November (3) 2008 October (3) 2008 August (2) 2008 July (4) 2008 June (5) 2008 May (4) 2008 April (8) 2008 March (3)

Printing from a 32-bit IIS process on 64-bit Windows

Posted Tue Jul 21 21:57:58 EDT 2009 in Computing by Andre

Last week I came across a very strange problem with Windows printing - an attempt to create a printer device context through CreateDC within a 32-bit IIS worker process would fail and return NULL, but GetLastError would indicate that there was no error and there was nothing in the system or application event log to help me identify the problem.

Naturally, my first thought was that the IIS anonymous user is missing some access rights and I spent some time double-checking various permissions and privileges, but found nothing that would be relevant in this case. Suspecting that the problem lies elsewhere, I added the anonymous user to the Administrators group, which actually made things worse. Now not only CreateDC would still return NULL, but it would actually take about a minute for this call to fail!

The only way I could make CreateDC work was to call it from the security context of the Network Service user, which was undesirable in the context of the application I was working with. This started to get interesting.

What's my session ID?

I found the first bug quite quickly. It turned out that IIS sets up the worker process (w3wp.exe) such that the anonymous user cannot query the process. CreateDC, on the other hand, calls ProcessidToSessionId in order to figure out its Terminal Services session ID, which requires the user to have process query rights. So, ProcessIdToSessionId failed with the error Access Denied, which was thrown away by the caller, so upon return CreateDC returned no error.

In order to test this theory, I granted the Users group, which includes all authenticated users, the right to query the worker process using code similar to this simplified fragment:

CDacl dacl;
CSecurityDesc secdesc;
SECURITY_DESCRIPTOR *_secdesc = NULL;
ACL *_dacl = NULL;

if(GetSecurityInfo(GetCurrentProcess(), 
          SE_KERNEL_OBJECT, DACL_SECURITY_INFORMATION, 
          NULL, NULL, &_dacl, NULL, 
          (PSECURITY_DESCRIPTOR*) &_secdesc) == ERROR_SUCCESS) {
	secdesc = *_secdesc;
	dacl = *_dacl;
		
	dacl.AddAllowedAce(Sids::Users(), PROCESS_QUERY_INFORMATION);

	SetSecurityInfo(GetCurrentProcess(), SE_KERNEL_OBJECT,
          DACL_SECURITY_INFORMATION, NULL, NULL, 
          (PACL) dacl.GetPACL(), NULL);
}

, and then restored the original access rights after calling CreateDC. This allowed me to go past the first problem, but now CreateDC started to pause for about one minute before failing the same was it did when the anonymous user was a member of the Administrators group.

What RPC server?!

I attached a debugger to the worker process and after jumping through a few call stacks figured out that the process was busy calling __rpc_mgmt_is_server_listening@8, which was consistently failing to locate some RPC server, then calling __imp__Sleep@4 and repeating the entire sequence for about a minute.

The call stack indicated that IIS was trying to create a proxy object to allow the 32-bit IIS worker process call 64-bit printer driver:

rpcrt4.dll!_RpcMgmtIsServerListening@4()  + 0x28 bytes	
winspool.drv!_ConnectToLd64In32ServerWorker@8()  + 0x19b bytes	
winspool.drv!_ExternalConnectToLd64In32Server@4()  + 0x1a bytes	
gdi32.dll!PROXYPORT::PROXYPORT()  + 0x189 bytes	
gdi32.dll!_LoadUserModePrinterDriverEx@20()  + 0xfda bytes	
gdi32.dll!_LoadUserModePrinterDriver@16()  + 0x40 bytes	
gdi32.dll!_hdcCreateDCW@20()  + 0x51e3 bytes	
gdi32.dll!_bCreateDCW@20()  + 0x91 bytes	
gdi32.dll!_CreateDCW@16()  + 0x18 bytes

After spending some time going through all of the RPC endpoints on my machine, I realized that the one IIS was trying to connect to was not available, so I spent some time looking for a reason this RPC server wasn't running.

Where did it go?!

After more digging through numerous screens of assembly instructions, I found out that the RPC server is supposed to be started by the worker process itself and, interestingly enough, the CreateProcess call:

winspool.drv!_ConnectToLd64In32ServerWorker@8()  + 0x20f bytes	
winspool.drv!_ExternalConnectToLd64In32Server@4()  + 0x1a bytes	
gdi32.dll!PROXYPORT::PROXYPORT()  + 0x189 bytes	
gdi32.dll!_LoadUserModePrinterDriverEx@20()  + 0xfda bytes	
gdi32.dll!_LoadUserModePrinterDriver@16()  + 0x40 bytes	
gdi32.dll!_hdcCreateDCW@20()  + 0x51e3 bytes	
gdi32.dll!_bCreateDCW@20()  + 0x91 bytes	
gdi32.dll!_CreateDCW@16()  + 0x18 bytes

was actually successful. The name of the executable that was being launched was splwow64.exe, which was the missing RPC server, whose purpose is to host 64-bit printer drivers and communicate with 32-bit processes that need to print.

I started Process Monitor and set the filters to record all activity generated by splwow64.exe. Much to my surprise, none of what this process did triggered any errors (aside from some of the debug entries it did not find in the registry). splwow64.exe simply started, read a bunch of registry entries, loaded a few DLLs and then exited, leaving IIS puzzled about what just happened.

Hacky x86/x64 printing

Seeing some of the code and reading about splwow64.exe, I couldn't shake the feeling that this whole x86/x64 printing bridge was a hack somebody put together at the last minute before the release and then just left it there as is. For example, one knowledge base article indicates that only one user can use splwow64.exe at a time:

http://support.microsoft.com/kb/923357

It seems that printing from 32-bit processes definitely wasn't too high on the list of Windows deliverables.

Dear Microsoft

At this point I was running out of time and decided to contact Microsoft, to let people with source code take a stab at the problem. I created a sample project demonstrating the problem and accompanied it with a detailed description of what was going on.

Unlike with many other companies, I got through to Microsoft quite quickly. May be having to pay $300 for a premium support incident was helping a bit. A support engineer connected to my machine and I walked him through the code and demonstrated all the problems I identified. A few hours later I had to repeat everything to another guy who specialized in IIS/.Net and then again the next day to somebody from the printing team.

September 16th, 2009

Microsoft finally called me to inform that they have completed the investigation and created a bug. They also told me that this bug is low priority and will not be fixed in the foreseeable future, if ever due to small number of complaints. All customers who encountered this bug are advised to move to Windows 7 and Windows Server 2008 RC2, where all listed issues have been fixed. I hope they are right.