[API Proposal]: Marshaller for OLE `VARIANT`

[AaronRobinsonMSFT]

Background and motivation

The VARIANT type is a discriminated union data structure that is common in COM interop. The built-in system has various APIs off of Marshal that help with marshalling of VARIANT (for example, Marshal.GetObjectForNativeVariant()). There is also support in the built-in COM Interop for marshalling as a VARIANT (for example, [MarshalAs(UnmanagedType.Struct)] object a). This proposal is for add a marshaller for consumers that helps bridge the gap. Precise support what this type is permitted to hold will initially be limited to primitives and IUnknown types. However this can change but will largely be driven by user need/feedback.

The primary goal would be to support the WinForms effort to become AOT compatible.

Support for this would require usage of MarshalUsingAttribute as opposed to existing usages with MarshalAsAttribute in the example above. This decision is being made since current usages of the MarshalAs pattern are too permissive and we are attempting to narrow when usage is supported.

API Proposal (Updated 21-09-2023)

using System.Diagnostics.CodeAnalysis;
using System.Runtime.CompilerServices;
namespace System.Runtime.InteropServices.Marshalling;

[StructLayout(LayoutKind.Explicit)]
public struct OleVariant : IDisposable
{
    // private fields to match the layout of VARIANT

    public OleVariant();

    public void Dispose();

    // Determine the VarEnum for the variant based on T.
    // Respects the System.Runtime.InteropServices.*Wrapper types for specifying the variant type.
    // We will not allow System.Object as T (we will throw an InvalidOperationException).
    // If a System.Runtime.InteropServices.*Wrapper type that corresponds to a COM interface is specified,
    // the following behavior occurs:
    // - If Marshal.IsComObject() returns true, call Marshal.GetIUnknown/IDispatchForObject
    // - Otherwise, use ComInterfaceMarshaller<T> to get the interface pointer.
    // Types without a corresponding VarEnum type are not allowed.
    public static OleVariant Create<T>([DisallowNull] T value);

    // Explicitly specify the VarEnum for the variant and the raw value.
    // Throws if T is larger than the VARIANT's data union.
    // Throws for VT_DECIMAL.
    // Throws if T does not match the size of the data union member that vt corresponds to.
    public static OleVariant CreateRaw<T>(VarEnum vt, T rawValue) where T : unmanaged;

    // Create VT_NULL variant
    public static OleVariant Null { get; };

    // Get a .NET object of the specified type that represents the underlying VARIANT value.
    // For COM object-based VARIANTs, uses ComInterfaceMarshaller to create the managed object.
    // If passed one of the System.Runtime.InteropServices.*Wrapper types, the wrapper type will be used
    // if it matches the variant type. Using a wrapper type is not required.
    // If the requested type does not match the underlying type, throws an exception.
    public T As<T>();

    // For cases where the built-in Create/As methods are not sufficient:
    // Expose accessors for the underlying fields of the VARIANT struct
    // but in a limited fashion.
    // This will allow consumers such as WinForms to support VARIANT types
    // that we do not plan to support immediately (like SAFEARRAY).
    public VarEnum VarType { get; }

    // Returns a reference to the data union within the variant.
    // Throws if T is larger than the VARIANT's data union.
    [UnscopedRef]
    public ref T GetRawDataRef<T>() where T : unmanaged;
}

// Let's start with a marshaller for object<->OleVariant as this will cover the majority of cases.
// We can add a generic marshaller for T<->OleVariant later if needed (e.g. for an IDispatch-based generator)
[CustomMarshaller(typeof(object?), MarshalMode.Default, typeof(OleVariantMarshaller))]
// To ensure correct behavior, we will need to update the generator to use the "ref" shape for both of the following cases.
// Is this behavior we want to generalize in an opt-in way? Or should we just special-case this case for now?
[CustomMarshaller(typeof(object?), MarshalMode.UnmanagedToManagedIn, typeof(OleVariantMarshaller.RefPropogate))]
[CustomMarshaller(typeof(object?), MarshalMode.UnmanagedToManagedRef, typeof(OleVariantMarshaller.RefPropogate))]
public static class OleVariantMarshaller
{
    public static OleVariant ConvertToUnmanaged(object? managed);
    public static object? ConvertToManaged(OleVariant unmanaged);
    public static void Free(OleVariant unmanaged);

    public struct RefPropogate
    {
        public void FromUnmanaged(OleVariant unmanaged);
        public void FromManaged(object? managed);
        public OleVariant ToUnmanaged();
        public object? ToManaged();
        public void Free();
    }
}

API Usage

OleVariant variant = OleVariant.Create(1.0m);
OleVariant variant2 = OleVariant.Create(new ErrorWrapper(0));


if (variant2.VarType == VarEnum.VT_ERROR)
{
    variant2.GetRawDataRef<int>() = 1;
}

[LibraryImport("Foo")]
public static partial Bar([MarshalUsing(typeof(OleVariantMarshaller))] object x);

Alternative Designs

No response

Risks

Minimal. The primary risk is narrowly supporting VARIANT in ways that limit future efforts. The WinForms team has rewritten a majority of the VARIANT support for their purposes so we can learn from them and mitigate much of the risk. Ideally the Interop team will simply take ownership of the WinForms code so the entire .NET ecosystem can benefit.

Tagging subscribers to this area: @dotnet/interop-contrib
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and motivation

The VARIANT type is a discriminated union data structure that is common in COM interop. The built-in system has various APIs off of Marshal that help with marshalling of VARIANT (for example, Marshal.GetObjectForNativeVariant()). There is also support in the built-in COM Interop for marshalling as a VARIANT (for example, [MarshalAs(UnmanagedType.Struct)] object a). This proposal is for add a marshaller for consumers that helps bridge the gap. Precise support what this type is permitted to hold will initially be limited to primitives and IUnknown types. However this can change but will largely be driven by user need/feedback.

The primary goal would be to support the WinForms effort to become AOT compatible.

Support for this would require usage of MarshalUsingAttribute as opposed to existing usages with MarshalAsAttribute in the example above. This decision is being made since current usages of the MarshalAs pattern are too permissive and we are attempting to narrow when usage is supported.

API Proposal

namespace System.Runtime.InteropServices.Marshalling;

[CustomMarshaller(...)]
public static class VariantMarshaller<T>
{
  ...
}

API Usage

[GeneratedComInterface]
interface IFoo
{
    void Method1([MarshalUsing(typeof(VariantMarshaller<int>))] int i); // The supplied VARIANT always contains an int
    void Method2([MarshalUsing(typeof(VariantMarshaller<object>))] object o); // Marshal the containing object and box
}

Alternative Designs

No response

Risks

Minimal. The primary risk is narrowly supporting VARIANT in ways that limit future efforts. The WinForms team has rewritten a majority of the VARIANT support for their purposes so we can learn from them and mitigate much of the risk. Ideally the Interop team will simply take ownership of the WinForms code so the entire .NET ecosystem can benefit.

Author:	AaronRobinsonMSFT
Assignees:	-
Labels:	api-suggestion, area-System.Runtime.InteropServices
Milestone:	9.0.0

[AaronRobinsonMSFT]

/cc @JeremyKuhne

[jtschuster]

Would we support marshalling something as a *int or **IUnknown variant? Or would we only do the least indirection variant and use ref in the signature?

edit: I think I see, when T is object, we add a layer of indirection to the variant? So [Marshalusing(VariantMarshaller<object>] int p marshals as *int variant?

[jkoritzinsky]

This is the built-in marshaller's behavior:

Variant and VarType marshalling behavior	Native Signature	VarType
Managed->Unmanaged by value	VARIANT	no VT_REF
Managed->Unmanaged by ref	VARIANT*	no VT_REF
Unmanaged->Managed by value	VARIANT	whatever unmanaged set it to
Unmanaged->Managed by ref with in or ref	VARIANT*	Preserve VT_REF from unmanaged code and propagate back if VT_REF was set.
Unmanaged->Managed by ref with out	VARIANT*	no VT_REF

I guess this behavior does make sense, but we might have to do some work on the design to provide users with equivalent behavior.

[JeremyKuhne]

The WinForms interop code is here. The struct itself is generated by CSWin32. A notable gap in our implementation is converting from VARIANT to object: dotnet/winforms#8596

[jkoritzinsky]

Here's an API proposal for a type for the VARIANT that would implement "enough" of the base behavior that WinForms can build on in their code base as well as a marshaller type that will allow us to integrate support into our generators:

using System.Runtime.CompilerServices;
namespace System.Runtime.InteropServices.Marshalling;

[StructLayout(LayoutKind.Explicit)]
public struct OleVariant
{
    // private fields to match the layout of VARIANT

    public OleVariant();

    // Determine the VarEnum for the variant based on T.
    // Respects the System.Runtime.InteropServices.*Wrapper types for specifying the variant type.
    // Should we allow System.Object?
    public static OleVariant Create<T>(T? value);

    // Determine the VarEnum for the variant based on the runtime type of value.
    // Respects the System.Runtime.InteropServices.*Wrapper types for specifying the variant type.
    public static OleVariant Create(object value);

    // Explicitly specify the VarEnum for the variant.
    // Will throw ArgumentException if the VarEnum is not valid for the given T.
    public static OleVariant Create<T>(VarEnum vt, T value);

    public T ToObject<T>();

    // For cases where the built-in Create/ToObject methods are not sufficient:
    // Expose accessors for the underlying fields of the VARIANT struct
    // but in a limited fashion.
    // This will allow consumers such as WinForms to support VARIANT types
    // that we do not plan to support immediately (like SAFEARRAY).
    public VarEnum VarType { get; }

    [UnscopedRef]
    public ref byte Data { get; }
}

// Let's start with a marshaller for object<->OleVariant as this will cover the majority of cases.
// We can add a generic marshaller for T<->OleVariant later if needed (e.g. for an IDispatch-based generator)
[CustomMarshaller(typeof(object?), MarshalMode.Default, typeof(OleVariantMarshaller))]
// To ensure correct behavior, we will need to update the generator to use the "ref" shape for both of the following cases.
// Is this behavior we want to generalize in an opt-in way? Or should we just special-case this case for now?
[CustomMarshaller(typeof(object?), MarshalMode.UnmanagedToManagedIn, typeof(OleVariantMarshaller.RefPropogate))]
[CustomMarshaller(typeof(object?), MarshalMode.UnmanagedToManagedRef, typeof(OleVariantMarshaller.RefPropogate))]
public static class OleVariantMarshaller
{
    public static OleVariant ConvertToUnmanaged(object? managed);
    public static object? ConvertToManaged(OleVariant unmanaged);
    public static void Free(OleVariant unmanaged);

    public struct RefPropogate
    {
        public void FromUnmanaged(OleVariant unmanaged);
        public void FromManaged(object? managed);
        public OleVariant ToUnmanaged();
        public object? ToManaged();
        public void Free();
    }
}

[AaronRobinsonMSFT]

implement "enough" of the base behavior that WinForms can build on in their code base

Is this close enough so that existing definitions of VARIANT in .NET so adoption is low cost?

public T ToObject<T>();

I assume this will throw if T doesn't match the enclosed VarEnum value?

public struct OleVariant

How do you envision cleanup being performed? Consider the case that a VARIANT can contain IUnknown based types.

[jkoritzinsky]

I took a look at the Variant types in dotnet/runtime as well as the VARIANT types I could find across GitHub with grep.app.

I think for clearing, we can make the type implement IDisposable.

I have a few modifications to the shape below that I think make the shape work a little better:

using System.Runtime.CompilerServices;
namespace System.Runtime.InteropServices.Marshalling;

[StructLayout(LayoutKind.Explicit)]
public struct OleVariant : IDisposable
{
    // private fields to match the layout of VARIANT

    public OleVariant();

    public void Dispose();

    // Determine the VarEnum for the variant based on T.
    // Respects the System.Runtime.InteropServices.*Wrapper types for specifying the variant type.
    // Should we allow System.Object?
    public static OleVariant Create<T>(T? value);

    // Determine the VarEnum for the variant based on the runtime type of value.
    // Respects the System.Runtime.InteropServices.*Wrapper types for specifying the variant type.
    public static OleVariant Create(object value);

    // Explicitly specify the VarEnum for the variant.
    // Will throw ArgumentException if the VarEnum is not valid for the given T.
    public static OleVariant Create<T>(VarEnum vt, T value);

    // Get a .NET object of the specified type that represents the underlying VARIANT value.
    // For COM object-based VARIANTs, uses the same rules as [MarshalAs(UnmanagedType.Interface)]
    // on a LibraryImport to create the managed object.
    // If passed one of the System.Runtime.InteropServices.*Wrapper types, the wrapper type will be used
    // if it matches the variant type. Using a wrapper type is not required.
    // If the requested type does not match the underlying type, throws an exception.
    public T ToObject<T>();

    // Create a boxed .NET object of the underlying value.
    // For COM object-based VARIANTs, uses the same rules as [MarshalAs(UnmanagedType.Interface)]
    // on a LibraryImport to create the managed object.
    // If in a mode that has a corresponding System.Runtime.InteropServices.*Wrapper type,
    // the returned object will be wrapped in an instance of that type.
    public object ToObject();

    // For cases where the built-in Create/ToObject methods are not sufficient:
    // Expose accessors for the underlying fields of the VARIANT struct
    // but in a limited fashion.
    // This will allow consumers such as WinForms to support VARIANT types
    // that we do not plan to support immediately (like SAFEARRAY).
    public VarEnum VarType { get; }

    // Returns a reference to the data union within the variant.
    // Throws if the type does not match the VarEnum value associated with the VarType field.
    // Does not support OleVariants that are VT_DECIMAL.
    [UnscopedRef]
    public ref T GetRawDataRef<T>() where T : unmanaged;
}

// Let's start with a marshaller for object<->OleVariant as this will cover the majority of cases.
// We can add a generic marshaller for T<->OleVariant later if needed (e.g. for an IDispatch-based generator)
[CustomMarshaller(typeof(object?), MarshalMode.Default, typeof(OleVariantMarshaller))]
// To ensure correct behavior, we will need to update the generator to use the "ref" shape for both of the following cases.
// Is this behavior we want to generalize in an opt-in way? Or should we just special-case this case for now?
[CustomMarshaller(typeof(object?), MarshalMode.UnmanagedToManagedIn, typeof(OleVariantMarshaller.RefPropogate))]
[CustomMarshaller(typeof(object?), MarshalMode.UnmanagedToManagedRef, typeof(OleVariantMarshaller.RefPropogate))]
public static class OleVariantMarshaller
{
    public static OleVariant ConvertToUnmanaged(object? managed);
    public static object? ConvertToManaged(OleVariant unmanaged);
    public static void Free(OleVariant unmanaged);

    public struct RefPropogate
    {
        public void FromUnmanaged(OleVariant unmanaged);
        public void FromManaged(object? managed);
        public OleVariant ToUnmanaged();
        public object? ToManaged();
        public void Free();
    }
}

[AaronRobinsonMSFT]

public static OleVariant Create<T>(T? value);

I'm not overly keen on this signature. If T is int then the VARIANT becomes odd. I would prefer to make this T and have a public static OleVariant CreateNull(); or something similar.

public struct OleVariant : IDisposable

The following OleVariant v = default; would yield a VT_EMPTY. This is fine with me, what do we think about its utility and would all members behave as expected?

public VarEnum VarType { get; }

How would this help with creation on the managed side? Consuming makes sense, but I don't see the going down mechanism if the Create() APIs don't support all values. Do we need a Create(VarEnum vt) where the user is expected to fill in the data via GetRawDataRef<T>()?

[UnscopedRef]
public ref T GetRawDataRef<T>() where T : unmanaged;

This is concerning. There is nothing from T being some large value type that could then trigger memory corruption. Perhaps we could be more explicit this is an unsafe operation by returning void*?