How to change a C-style array as a D-style array?

Question

Question

What is the best way to take a C-style array as a parameter, change it as a D-style array (including changing the length), and return it as a C-style array?

---

In the context

I am writing a library in D that compiles into a DLL with a C interface (I will call my D DLL from C ++, so I need a C interface). Requires byte arrays and modifies their contents, sometimes changing the length of the array .

Since I use the C interface, my functions should accept arrays as C-style. Ideally, I would like to be able to allocate more memory (i.e. Extend bufferMaxSize ) if this buffer is too small.

This is how my D DLL takes parameters right now:

 // D library code; compiles to DLL with C interface. // bufferSize is the data length, and is a pointer because I may modify the data length. // bufferMaxSize is the total allocated buffer size. export extern(C) void patchData(const size_t bufferMaxSize, size_t * bufferSize, byte * buffer) { ... } 

In my D library, I have existing code that accepts D-style arrays. Somewhere along the line there should be a conversion of the C-style array to the D-style array .

I am currently doing such a conversion (simplified example):

 // D library code; compiles to DLL with C interface. export extern(C) void patchData(const size_t bufferMaxSize, size_t * bufferSize, byte * buffer) { // Convert from C-style array to D-style. byte[] dStyleArray = buffer[0 .. *bufferSize]; // Modify data. dStyleArray[0] = cast(byte) 0xab; dStyleArray[1] = cast(byte) 0xbc; dStyleArray.length = dStyleArray.length + 16; // Return modified data as C-style array. buffer[0 .. dStyleArray.length] = dStyleArray[0 .. dStyleArray.length]; *bufferSize = dStyleArray.length; } 

This works, but I'm not sure what is really going on here. My main problem is speed. If I focus on this function, I don’t want to constantly allocate new memory and copy its contents back and forth .

When I do byte[] dStyleArray = buffer[0 .. *bufferSize] , D allocates a new piece of memory and copies everything to a D-style array or points to an already allocated C-style array?

What happens when I do dStyleArray.length = dStyleArray.length + 16 ? Since dStyleArray was sliced ​​from buffer , am I now allocating new memory / copy memory? Or am I spreading to buffer ?

When I do buffer[0 .. dStyleArray.length] = dStyleArray[0 .. dStyleArray.length]; I'm copying memory, right?

Is it possible to simply bind a D-style array to a C-style array and access the previously allocated memory through the D-style array interface?

Answer 1

When I do a byte [] dStyleArray = buffer [0 .. * bufferSize], D allocates a new piece of memory and copies everything into a D-style array or points to an already allocated C-style array?

It just points to it. The slice operator on the right side is the same (conceptually) as array.pointer = &first_element; array._length = length; array.pointer = &first_element; array._length = length; - very fast and easy operation. (I called it _length instead of length btw, because setting the length property can actually call the function following.)

What happens when I do dStyleArray.length = dStyleArray.length + 16?

This will allocate new memory. When the length extends, if the runtime cannot prove that it is safe (or you tell her that she is, and she knows that she came from the GC), the array is copied to a new location. It basically calls realloc() pointer - although not literally, it is incompatible with C realloc.

Since it came from C, the runtime simply knows that it does not own the memory, that it is somehow managed elsewhere and will always allocate a new one when trying to expand. If you want to expand some other ways, you need to do it yourself.

When I make a buffer [0 .. dStyleArray.length] = dStyleArray [0 .. dStyleArray.length] ;, I copy the memory, right?

right, this copies because you cut the left side.

Is it possible to simply bind a D-style array to a C-style array and access the previously allocated memory through the D-style array interface?

Simple rectangular cut:

auto d_array = c_array[0 .. c_array_length];

handles it for all but length extensions. It holds the pointer, so writing to elements instantly affects the original thing. (By the way, since this is C's shared memory, make sure you don’t free it, and D is still using it! You should be fine while you use it only inside this function, but not saving the slice anywhere.)

If you need to lengthen the length, you need to do it yourself. The way I like to do this is to cut the entire potential array, full capacity, and then cut it again to get a window of limited capacity.

So maybe:

 auto whole_array = buffer[0 .. bufferMaxSize]; // assuming buffer is already fully allocated on the C side auto part_youre_using = whole_array[0 .. *bufferSize]; // to extend: *bufferSize += 16; // extend the size part_your_using = whole_array[0 .. *bufferSize]; // and reslice from the original 

The reason I did the whole_array thing instead of reslicing buffer is that D can catch the constraints for me. He does not do this on a bare pointer, but does on a threaded pointer, since he knows the maximum size as length.

If you need to expand the buffer, do it with the correct C function, like realloc or something else, and then cut the whole_addresses and part_youre_using again.

Answer 2

When I do a byte [] dStyleArray = buffer [0 .. * bufferSize], D allocates a new piece of memory and copies everything into a D-style array or points to an already allocated C-style array?

It indicates. Phobos uses the same trick to convert C "string" to D: https://github.com/D-Programming-Language/phobos/blob/67c95e6de21d5d627e3c57128b4d6e332c82f785/std/string.d#L208-L211

What happens when I do dStyleArray.length = dStyleArray.length + 16? Since dStyleArray was cut out of the buffer, am I now allocating new memory / copy of memory? Or did I go to the buffer?

You are probably not doing what you want / expect. He will allocate a new block to the garbage collection and copy it to it. It cannot be expanded since the runtime does not have information about the memory block (it does not control it). Are you sure you want to expand the buffer or move the pointer (which will slice to D)?

When I make a buffer [0 .. dStyleArray.length] = dStyleArray [0 .. dStyleArray.length] ;, I copy the memory, right?

Yes. It went down to memcpy.

Is it possible to simply bind a D-style array to a C-style array and access the previously allocated memory through the D-style array interface?

Yes, this is what you did in the beginning;)

If you just want to change the first 2 elements of the array, just bind and change them, it will “just work”.

If you want to test the behavior, I would recommend that you put the unittest block under the function, so you can check what happens by specifying a pointer. Also, if you want to make sure that you are not performing GC allocation, you may need to put @nogc on your function to statically check this (and nothrow usually a good idea for a C function).

Answer 3

Code Transition:

 byte[] dStyleArray = buffer[0 .. *bufferSize]; 

It looks good. Please note that this will not allocate memory. D is essentially a pointer and a length, so this line is equivalent to this pseudocode :

 struct DByteArray { byte* ptr; size_t length; } DByteArray dStyleArray; dStyleArray.ptr = buffer; dStyleArray.length = *bufferSize; 

In this case, access to the dStyleArray elements will have access to the same data pointed to by buffer , which means:

 dStyleArray[0] = cast(byte) 0xab; dStyleArray[1] = cast(byte) 0xbc; 

will also "modify buffer ".

Further:

 dStyleArray.length = dStyleArray.length + 16; 

Increasing the length dynamic array D will result in redistribution. In this runtime D mode, the memory will be cut out using dStyleArray and copied to the newly allocated memory block. If you want dStyleArray point to the first byte , but with a longer length , you need to slice the pointer again:

 dStyleArray = dStyleArray.ptr[0 .. dStyleArray.length + 16]; 

or

 dStyleArray = buffer[0 .. *bufferSize + 16]; 

Then the line:

 buffer[0 .. dStyleArray.length] = dStyleArray[0 .. dStyleArray.length]; 

(which copies the memory, you guessed it) becomes redundant because two points to the same memory block.

Is it possible to simply bind a D-style array to a C-style array and access the previously allocated memory through the D-style array interface?

Yes, this is exactly what the pointer slices.