[cpp] Marshalling Extern Types #11981
Conversation
|
Thanks for looking into it! When it is merged, would love to see a rebase so I can test out the binary built from this PR |
|
I've merged dev into this branch so new builds should have the fix. |
|
Okay I can compile the existing code with the haxe binary now. So I can actually start testing this out. I just found that on the cpp side the marshalling wrapper does not seem to be compatible with cppia: Host.hx @:headerCode('enum MyEnum { X = 3, Y = 27 };')
class Host {
public static function main() {}
public function new(type:MyEnum) {}
static function getValue():MyEnum {
return MyEnum.X;
}
}
@:semantics(reference)
@:cpp.ValueType({type: 'MyEnum', namespace: []})
extern enum abstract MyEnum(Int) to Int {
final X;
final Y;
}build.hxml generated cpp ::cpp::marshal::ValueType< ::MyEnum > Host_obj::getValue();
static void CPPIA_CALL __s_getValue(::hx::CppiaCtx *ctx) {
ctx->returnInt(Host_obj::getValue());
}Outcome: also it can't convert an int to the marshalled enum for a cppia function arg: |
|
This should now be working, or, atleast, the marshalling tests now compile with the scriptable define which is a good start. |
|
hmm, I am still unable to compile the code snippet in my previous post. Got this: Does it happen on your side as well? |
|
Yep, it was happening on my side as well. I got the marshalling tests compiling with the scriptable flag and forgot to check your original sample. There's a fair bit of duplication in the cppia support code generation which could probably do with being cleaned up... |
|
I guess there is another missing bit for member functions returning a value type: @:headerCode('enum MyEnum { X = 3, Y = 27 };')
class Host {
public static function main() {}
public function new(type:MyEnum) {}
static function staticGetValue(input:MyEnum):MyEnum {
return MyEnum.X;
}
function getValue(input:MyEnum):MyEnum {
return MyEnum.X;
}
}
@:semantics(reference)
@:cpp.ValueType({type: 'MyEnum', namespace: []})
extern enum abstract MyEnum(Int) to Int {
final X;
final Y;
}I guess the generated |
|
yes, that one should be fixed now as well. I've done some cleanup of the cppia scaffolding generation so that should hopefully be the last of the issues (last famous words...). |
|
Edited: In short I would like to know how to consistently obtain a pointer to a ValueType, and the actual value of a ValueType. For the purpose of interop-ing with native code that expects pointer of the value type and actual value of the value type. Original:
How does it work if I want to get the pointer to a ValueType value regardless of its promotion state? Besides pointer, I am also curious in how to make haxe emit code that correctly dereferences a ValueReference if it decided to produce a temp var. For example, I can see that currently the following is generated for the haxe code ::cpp::marshal::ValueReference< ::godot::Variant > p_rhs1 = ( (::cpp::marshal::ValueReference< ::godot::Variant >)(p_rhs) );
return x == p_rhs1;But it is not correct because I expect the actual value be used in the comparison, not the reference created by haxe. |
|
There are two ways you can deal with pointers to value types. If you have a function which accepts a pointer to a type you've externed as a value type the easiest way is to just extern that function using the existing haxe extern and C++ operator conversions will take care of it. I used this method in the directx gist in the original post. E.g. I have a value type extern for @:semantics(reference)
@:cpp.ValueType({ type : 'DXGI_ADAPTER_DESC' })
extern class DxgiAdapterDescription {
@:native('VendorId')
var vendorId : Int;
@:native('DeviceId')
var deviceId : Int;
}and I want to use a function @:native('GetDesc')
function getDesc(desc : DxgiAdapterDescription) : Int;Then it will "Just Work" as C++ operators take care of getting a pointer to the value type passed in, no matter what the state of it is. The alternative if you want to be a bit more explicit is to use something like @:native('GetDesc')
function getDesc(desc : cpp.Star<DxgiAdapterDescription>) : Int;When you use those existing pointer interop types with value types it will use the underlying type, not the hxcpp wrapper types. i.e. There are more value type interop examples in the hxcpp tests. https://github.com/HaxeFoundation/hxcpp/blob/7fcb9203ae2da9e01e82877232d7cac7122ec7bb/test/native/tests/marshalling/classes/TestValueTypeInterop.hx For variables, only those typed by the user will be given the stack or promoted state, any internal compiler generated variables will be given the reference state. This is to avoid the sort of issues you see with |
|
Thanks for the explanation, I seem to get the pointer/value part working (although I reckon it can be optimized a little bit) For my use case I need to pass a pointer to native API, the pointer cast in Haxe works well. On the other hand I also need to be able to get the actual value for native operator overloads, which can be done via casting it to cpp.Pointer again then use its The other topic I would like to explore is about using the Haxe generated code from native side. class Foo {
var x:MyValueType;
public function new(x:MyValueType) this.x = x;
}The generated c++ will be like: // this->x is typed as Boxed<MyValueType>
void Foo_obj::__construct(::cpp::marshal::ValueType< MyValueType > v) {
this->x = (::cpp::marshal::ValueReference< MyValueType >)(v);
}If I understand correctly ValueType is stack allocated and I don't get how it ends up being allocated in the Heap (for ValueReference and Boxed) during the the various casts. One more side note: extern enum abstract annotated with ValueType is not compatible with string concatenation any more. I need to do a manual int cast so this works: |
I'm not sure I follow this paragraph, do you have some concrete examples?
Sometimes value type arguments and return types need to be boxed to work with certain haxe function "features" related to dynamic. The good news is I've worked on improving function calling in #11151 so that you only pay the dynamic cost if you're explicitly using it, so in the future value type arguments won't need to be boxed as often. The types One of the conversions is an implicit conversion to a this->x = (::cpp::marshal::ValueReference< MyValueType >)(v);The Most of this should be implementation details, even if you're writing C++ glue code you shouldn't need to manually deal with the value, boxed, or reference state types (although I'm sure there are some extreme edge cases), you can just deal with standard C/C++ pointers, references, and values. Appropriate conversions should automatically happen when those type pass between haxe and C/C++.
I'll take a look, wonder if this is a general limit of extern enum abstracts as opposed to something wrong with the C++ stuff. |
So basically I am dealing with cases where the expected type is not explicit , such as c++ operator overloads. Let's say a class X has a op overload for value type Y: // c++
class X {
bool operator==(const Y& y);
}When trying to generate the comparison code in Haxe: // haxe
// x is instance of `extern class X {}`
// y is instance of `@:cpp.ValueType extern class Y {}`
untyped __cpp__('{0} == {1}', x, y);It is not going to work because the c++ compiler would not know that it should convert I could manually insert a double cast via // haxe
untyped __cpp__('{0} == static_cast<Y&>(static_cast<ValueReference<Y>>({1}))', x, y);Hence what I was trying to ask is that we could possibly make it nicer with some sort of conversion API on the Haxe side. e.g.: // haxe
untyped __cpp__('{0} == {1}', x, cpp.ValueType.getReference(y));and the case is similar when the actual value type value is needed. Maybe Not very sure if all these make sense at all, please let me know! |
|
I also have question regarding null. Consider the following example regarding optional function argument: // Foo is a @:cpp.PointerType extern class
function f(?native:Foo) {
trace("native is null", native == null);
}
f(); // false, but I think it should be true from a Haxe point of view?The generated c++ is roughly: void f(Boxed<Foo*> native){
::hx::IsEq( native,Boxed<Foo*>( new Boxed_obj<Foo*> (null())) );
}
f(Boxed<Foo*>( new Boxed_obj<Foo*> (null()))); |
|
I see what you mean now, in that situation I'd write a small wrapper C++ functions, something like bool _hx_x_eq_y(X* lhs, Y* rhs) {
return (*lhs) == (*rhs);
}and extern it to haxe with something like function x_eq_y(lhs : cpp.Star<X>, rhs : cpp.Star<Y>) : Bool;I'm a bit hessitant to add escape hatch functions because it opens up untyped cpp and I want to reduce the amount of that I see as much as possible. But with the above solution it does require something like an abstract to make it nice to use from haxe and requires writing a fair bit of boiler plate code. Speaking of abstracts, before starting all this one alternative I did consider was using core type abstracts instead of extern classes as that would make all those operator overload scenarios possible. function main() {
final x = new X();
final y = new Y();
trace(x == y);
}
@:coreType
@:notNull
@:cpp.ValueType({ type : 'X' })
abstract X {
public function new() : Void;
@:op(A == B)
public function eqY(rhs:Y) : Bool;
}
@:coreType
@:notNull
@:cpp.ValueType({ type : 'Y' })
abstract Y {
public function new() : Void;
}The one main kicker with this is that there's no nice way to represent inheritance, so I opted against it in the end and went with traditional extern classes. I like this idea but I'm not sure what to do with it since a lack of inheritance support is a pretty big issue.
I think that was a bit of an oversight. I guess null checks involving boxed pointers should check the null-ness of the GC container and the null-ness of the held pointer. I don't think there are any scenarios where you'd want to differentiate between the two. |
|
It is unfortunate that the example was a little bit misleading, the main culprit is not the operator overload per se, but how the c++ compiler sees a type and select the correct construct (function call, operator, etc). So it will still happen for templated function such as: template<typename T>
void foo(T& ref);
template<typename T>
void bar(T* ptr);In that case there is basically no way to call For me the way to call bar with a PointerType value // works
extern static inline function bar<T>(v:T):Void
untyped __cpp__('bar({0})', ( v: cpp.Star<T> ));
// does not work
@:native("bar") extern static function bar<T>(v:cpp.Star<T>):Void;And for c++ references unless we have something like cpp.Ampersand that does the same job as cpp.Star, we are left with doing the cast in untyped cpp. |
|
This is possible as long as you introduce a partial template specialisation or something similar. This specialisation would just be an intermediate which is specialised to a @:cpp.ValueType
@:semantics(reference)
extern class Foo {
var i : Int;
function new() : Void;
static function bar<T>(v : T) : Void;
}
@:headerNamespaceCode('
struct Foo {
int i;
template<class T>
static void bar(T& v) {
printf("base template %i\\n", v.i);
v.i = 10;
}
template<class T>
static void bar(::cpp::marshal::ValueReference<T> v) {
printf("partial specialisation\\n");
bar(*v.ptr);
}
};
')
class Main {
static function main() {
final foo = new Foo();
foo.i = 7;
Foo.bar(foo);
trace(foo.i);
}
}We trace the value of
|
|
I think the compiler should be aware of any ValueType that is created inline and immediately converted to a reference. Because in that case the reference will become invalid as the value itself is out of scope straight away. Consider the following pseudo generated code: foo( (ValueReference<T>) (ValueType<T>(args)) )When foo gets the reference and use it in the body, the reference is already invalid as the actual value itself is already out of scope. In this case I believe the compiler needs to make sure the actual value can survive the ValueType<T> tmp(args);
foo((ValueReference<T>)(tmp) |
Another long one, so make sure you're sitting comfortably.
Corresponding hxcpp PR: HaxeFoundation/hxcpp#1189
The Problem
Working with the current interop types comes with many pitfalls which are not immediately obvious, these include.
cpp.Struct/cpp::Struct) do not work as expected when captured in a closure. If you mutate one of these captured value types you are mutating a copy, not the object referenced by the variable name.hx::Objectsub class some of the necessary GC code isn’t generated for the generational GC.In short, the current interop handlers mostly work with basic c structs in local variables, but if you want to interop directly with C++ classes, you’re going to have a painful time in anything but the most basic cases.
If you just want to see a quick example of it all in action here's a gist which will compile on Windows and use DXGI to print out all displays connected to the first graphics adapter. DXGI uses pointers to pointers, pointers to structs, pointers to void pointers, and other C++ concepts which have been very difficult to map onto haxe in the past. But hopefully you'll agree that it looks like pretty "normal" haxe.
https://gist.github.com/Aidan63/07364c227335f02fbe50b9c9576f7544
New Metadata
In my mind there are three categories of things you might want to extern, native value types, native pointer types, and "managed" (custom hx::Object sub classes) types. This merge introduces three new bits of metadata to represent these categories and solve the above issues.
cpp.ValueType
Using the
@:cpp.ValueTypemetadata on an extern class will cause it to be treated as a value type, so copies will be created when passing into functions, assigning to variables, etc, etc.I've chosen the metadata to take a struct which currently supports a
typefield for the underlying native type name andnamespacewhich must be an array of string literals for the namespace the type is in. Iftypeis omitted then the name of the extern class is used, ifnamespaceis omitted then the type is assumed to be in the global namespace.Using this metadata provides several guarantees old struct types. First it behaves how you'd expect when captured in a closure.
Destructors are guaranteed to be called. When a value type is captured in a closure, stored in a class field, enums, anons, or any other GC objects it is "promoted" to the GC heap and the holder class its contained within registers a finaliser which will call the destructor.
Operators on the defined native type are always used, no memcmp or memcpy. Copy constructors, copy assignment, and standard equality operators are always used no matter the case.
The same sort of null checks are performed with references to these value types as standard haxe classes so you will get standard null exceptions instead of the program crashing with a memory error.
The nullability of these values types is unfortunately a bit odd... If you have a explicitly nullable TVar value type then its always promoted and can be null. But a null value type doesn't make much sense so I've disallowed value type variable declarations with no expression or with a null constant. Trying to assign a null value to a stack value type will result in a runtime null exception. Since value types in class fields and the likes are always promoted they are null if uninitialised. Ideally value type externs could have the same "not null" handling as core type abstracts, but that doesn't seem possible.
Interop with the existing pointer types is also provided as well implicit conversions to pointers on the underlying types for easier interop.
This value type metadata is also supported on extern enum abstracts. Historically externing enums have been a bit of a pain but it works pretty nicely now.
cpp.PointerType
Using the
@:cpp.PointerTypemetadata on an extern class will cause it to be treated as a pointer, this metadata supports the same fields as the above value type one.Extern classes annotated with the pointer type metadata cannot have constructors as they are designed to be used with the common C/C++ pattern of free functions which allocate and free some sort of opaque pointer, or the pointer to pointer pattern.
E.g. the following native API could be externed and used as the following.
The pointer to pointer pattern which is pretty common is quite difficult to extern without custom C++ glue code, but the new pointer type externs understand this pattern and can be converted to a double pointer of the underlying type as well as a pointer to a void pointer which is also seen in many places.
Internally pointer types and value types are treated almost identically so most of the previous points apply here as well, the main exceptions being that promoted pointers don't have finalisers assigned and that null is always an allowed value.
cpp.ManagedType
When you want to extern a custom
hx::Objectsubclass then this is the metadata to use as it ensures the write barriers are generated for the generational gc. Like the above two metadata it supports thetypeandnamespacefields.In the above sample
Barwill be generated as::hx::ObjectPtr<bar>in most cases.There is one extra field to the managed type,
flags, which is expected to be an array of identifiers and currently there is one flag,StandardNaming. If in C++ you use the hxcpp class declaration macro to create a custom subclass with the same naming scheme as haxe generated classes then this flag is designed for that.In the above case
Barwill be used instead of the manual::hx::ObjectPtrwrapping butBar_objwill be used when constructing the class.Implementation Details and Rational
Marshalling State
Value and pointer types are represented by the new
TCppNativeMarshalTypeenum which can be in one of three states,Stack,Promoted, orReference. This is the key to working around optimiser issues, capturing, and some interop features. AllTCppNativeMarshalTypefields are given the promoted state andTVars can be given any three of the states. AnyTLocalto a native marshal type is given the reference state. How TVars are given their state is important, variables allocated by the compiler are given the reference state, only variables typed by the user are given one of the stack (uncaptured) or promoted (captured or nullable) state. This means we dodge the issue with cpp.Struct where you could be operating on a copy due to compiler created variables.TLocals of the reference state are generated with the new
cpp::marshal::Reference<T>type which holds a pointer to a specific type and is responsible for any copying, promotion, checking, and just about everything. For the value type case it's T will be a pointer to the value type, and for pointer types will be a pointer to the pointer.Semantics
You are required to put the
@:semantics(reference)metadata on an extern class when using the value or pointer type metadata, this does feel like a bit of a bodge... I was initially hoping that thevaluesemantic would be what was needed, but tests start to fail when the analyser is enabled with value semantics. Maybe I'm just misinterpreting what these semantics are actually used for. With the reference semantics the tests do pass with the analyser, but from a quick glace that appears to be because many optimisations are disabled on types with that semantic meta...Compiler Error Numbers
There are several errors you may now get if you try and do things wrong (invalid meta expression) or which are not supported (function closures) instead of vague C++ errors. In these cases I've given them distinct error numbers in the CPPXXXX style, similar to MSVC and C# errors. I plan on documenting these since they're things users might naturally cause as opposed to internal bugs, so I thought it might be nice to give then concrete numbers for easier searching.
Scoped Metadata
I can never remember the names of the C++ specific metadata and end up sifting through the dropdown list every time, so I decided to prefix these ones with
cpp.to make it easier.Metadata Style
I wanted to avoid re-using the
@:nativemetadata for the extern classes as its already pretty common to do stuff like@:native("::cpp::Struct<FooStruct>")so by having atypeandnamespacefield I wanted to make it clear it should be just the type, nothing else. Also with this we can prefix::to the type / namespace combo to avoid any relative namespace resolution issues.Eager Promotion
Due to the very dynamic nature of hxcpp's function arguments and return types there are many places where value types which could be on the stack have to be promoted to satisfy the dynamic handling. With my callable type PR this should be solvable.
Future Stuff
Closures
Currently trying to use a function closure of a value or pointer type will result in a compilation error, but now that the variable promotion stuff is in place it should be possible to generate closures which capture the object to support this. Again I wouldn't want to do this until the callable change is in since that will greatly simplify things.
Arrays and Vectors
Value types stored in contains such as arrays are in their promoted state, not a contiguous chunk of memory which I originally wanted. Preserving C++ copying / construction semantics with haxe's resizable array looked to be a massive pain so I decided not to.
I do think having
haxe.ds.Vectors of value types be contiguous should be possible and open up more easier interop possibilities.Un-dynamicification
Lots of the cpp std haxe types have a
Dynamiccontext object which is then passed into C++ where its cast to a specific type. With the managed type meta we should be able to "un-dynamic" a lot of the standard library implementation.Closing
I'm sure there's stuff I've missed but this seems to be much more consistent in behaviour and nicer to use than the existing interop types, I've also added a bunch of tests on the hxcpp side to capture all sorts of edge cases I came across. I will also try and write some formal documentation for all this to encourage this over the existing types.