COM Interop crashes (0x8000FFFF) when IDispatch marshals

Question

I am developing an IM provider for Outlook 2010. To do this, I need to implement some of the IMessenger * COM interfaces that load at startup. I want to do this using C # and the out-of-line server for which I followed the MSDN example here .

This is what I have already achieved:

• Register everything (CLSID, enter libraries, etc.) correctly and configure Outlook to try to load my server at startup
• Deploy the COM server as a native ATL / C ++ server and make sure that it works
• Deploy the base interface in C # /. NET and make sure it is loaded

At the moment, I can start my .EXE server, and then start the look and see the console windows of my server, while it displays all the method calls coming from the COM subsystem (I added registration for each method). The problem is that at some point (only in the .NET implementation!) I got an error that looks like this in Outlook logs:

CMsoIMProviderOC20::HrEnsureServiceData !failed! Line: 402 hr = 0x8000FFFF 

I know exactly where this happens in my program, but I can not do anything about it (I tried to find a solution for this for several days). Remember that if I implement it using C ++ / ATL, it really works, so I assume that it should have something to do with how .NET Interop marshaling works.

Here are the details:

There are 3 interfaces to this problem: IMessenger, IMessengerServices and IMessengerService:

 [ uuid(D50C3186-0F89-48f8-B204-3604629DEE10), // IID_IMessenger helpstring("Messenger Interface"), helpcontext(0x0000), dual, oleautomation ] interface IMessenger : IDispatch { ... [id(DISPID_MUAM_SERVICES), propget, helpstring("Returns services list."), helpcontext(0x0000)] HRESULT Services([out, retval] IDispatch ** ppdispServices); ... } 

While I know that this Services property should actually return this information:

 [ uuid(2E50547B-A8AA-4f60-B57E-1F414711007B), // IID_IMessengerServices helpstring("Messenger Services Interface"), helpcontext(0x0000), dual, oleautomation ] interface IMessengerServices : IDispatch { ... [id(DISPID_NEWENUM), propget, restricted, helpstring("Enumerates the services."), helpcontext(0x0000)] HRESULT _NewEnum([out, retval] IUnknown **ppUnknown); ... } 

This interface, in turn, should return IMesengerService interfaces. However, this is not important for the purpose of this question.

In C ++ / ATL, I implement the Services property as follows:

 STDMETHOD(get_Services)(IDispatch ** ppdispServices) { (*ppdispServices) = (IDispatch*)new CComObject<CMessengerServices>(); (*ppdispServices)->AddRef(); return S_OK; } 

And here is the C # implementation:

 public object Services { get { return new MessengerServices(); } } 

Until everything is in order in both implementations! Now the problem begins ...

The first thing that is strange is that in C ++ / ATL, the get__NewEnum(IUnknown **pUnkown) my CMessengerServices class will never be executed. Instead, other functions are called. It's good.

In C #, however, the first member of my MessengerServices class that gets called is the GetEnumerator () method. I can set a breakpoint there and clearly see that this is the last line in C # code (under my control), which is executed before Outlook stops starting and reports error 0x8000FFFF in its log. After that, no other line of code on my .EXE server is called.

The most important parts of implementing the MessengerServices class are:

 [ClassInterface(ClassInterfaceType.None)] [ComVisible(true), Guid("DF394E2C-38E2-4B70-B707-9749A4F857B0")] public class MessengerServices : IMessengerServices { IEnumerator IMessengerServices.GetEnumerator() { return messengerServices.GetEnumerator(); } private readonly ArrayList messengerServices; public MessengerServices() { messengerServices = new ArrayList { new MessengerService() }; } ... } 

And here is the CCW proxy created by tlbimp.exe:

 [ComVisible(true), Guid("2E50547B-A8AA-4F60-B57E-1F414711007B")] public interface IMessengerServices : IEnumerable { [TypeLibFunc(TypeLibFuncFlags.FRestricted)] [DispId(-4)] [MethodImpl(MethodImplOptions.InternalCall, MethodCodeType = MethodCodeType.Runtime)] [return: MarshalAs(UnmanagedType.CustomMarshaler, MarshalType = "System.Runtime.InteropServices.CustomMarshalers.EnumeratorToEnumVariantMarshaler, CustomMarshalers, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a")] new IEnumerator GetEnumerator(); ... } 

One final hint if this can be useful: when I set a breakpoint in the implementation of GetEnumerator () and hit F10 to jump, this is the last line in the Visual Studio output window:

Navigating the non-user code 'System.Runtime.InteropServices.CustomMarshalers.EnumeratorToEnumVariantMarshaler.MarshalManagedToNative

Another interesting thing (at least for me) is that if I completely remove the Enumerator object from the Interface declaration and the definition of my class, Outlook still calls my interface, but instead uses a different method (PrimaryService for completeness). but that basically has the same error.

Honestly, I absolutely do not know what to do with this error? Thanks in advance for any help!

Answer 1

Now I have found a solution to the problem. Hans Passants answer led me in the right direction (thanks!), However the problem was this:

Tlbimp.exe does not preserve the order of the methods in the vtable, as they were originally defined in IDL / TLB. Instead, methods and properties are re-sorted alphabetically!

So, when you implement the interface and reference the type library using Visual Studio, there is a chance that the virtual table created by CCW will be confused. The solution is to copy Tlbimp (the tool used by Visual Studio to generate the interaction assembly), the generated shell into your own code and reorder methods and properties that are identical to the order in the type library.

Answer 2

The standard diagnosis for such problems is that your [ComVisible] interface has the wrong layout. This causes the wrong method to execute when the client code calls, say, IDispatch :: GetTypeInfoCount (). Which is the pointer to the 4th method in the IMessengerServices v-table, your GetEnumerator () method is the 4th method in your v-table interface. Outlook crashes when it receives a strange return value that it does not know how to handle.

I don’t know where you got the "CCW proxy server" from, it looks like Tlbimp.exe generated it. An important attribute that is missing is [InterfaceType(ComInterfaceType.InterfaceIsDual)] . This is what the CLR says it needs to implement IDispatch. Thus, your interface does not have four IDispatch methods.

Besides fixing the interface declaration, the best way to solve this problem is by far to import the interface definition from the type library so that there is never a mismatch. I don’t have it on my machine, you can find it on your computer by looking in the registry HKCR\CLSID\{2E50547B-A8AA-4F60-B57E-1F414711007B} .