Best way to wrap items in an array?

Question

I have a lot of data arrays on my website. for example: an array of vertices, an array of colors, an array of sizes ...

I work with a lot of items. Up to tens of millions.

Before adding data to arrays, I need to process them. So far, I have been doing this in the main topic, and this has caused my site to freeze for X seconds. It froze due to processing and due to the addition of processed data to arrays.

Today I moved (done a lot of work) processing to web workers, but the processed data is added to the main stream. I managed to save processing freeze time, but not add.

Adding is simply done using array.push() or array.splice() .

I read several articles about how an array works, and found out when we add an element to the array, the array is completely copied to a new location in memory with the size of array.length + 1 and adds the value. This makes my data slow.

I also read that typed arrays are much faster. But for this, I would need to know the size of the array, which I do not know, and to create a large typed array with an additional counter and control the addition of elements in the middle (and not at the end of the array) there would be many code changes that I do not want to do in present.

So, in my question, I have a TypedArray that returns from a web worker, and I need to add it to a regular array. What is the best way to do this? (today I run in a loop and click one after another)

EDIT

An example of using a website: a client adds the number of elements, say 100,000. The source data of the elements is collected and sent to the employee. The worker processes all the information and sends the processed data in the form of a typed array (for use as portable objects). In the main stream, we add the processed data to arrays - to the end or to a specific index. 2nd round. the client will add another 100,000 items. sending to the employee and adding the result to the main stream arrays. The 3rd round can be 10 items, the 4th round 10000, the 5th round can remove the indices 10-2000, ...

Answer 1

We did some more research using comments and thought about a different direction.

I tried using the typedArray.set method and found that it is very fast.

10 million items using sets took 0.004 seconds, compared to array.push seconds. I split 10 million into 10 arrays to make sure the set method doesn't work faster starting at index 0.

Thus, I think I would even use my insertAtIndex my own, using TypedArray, which would drag all the elements forward and set a new one in the right index.

In addition, I can use TypedArray.subArray to retrieve my auxiliary data according to the actual amount of data in the array (which does not copy data) - useful for loading data into a buffer (WebGL)

I said that I want to work with regular arrays, but this is a performance improvement. I do not think that I will get another reasonable. And that is not much work when I wrap MyNewTypedArray as a TypedArray with all push , splice , my own implementation.

Hope this information has helped someone

 var maxCount = 10000000; var a = new Float32Array(maxCount); var aSimple = []; var arrays = []; var div = 10; var arrayLen = maxCount / div; for (var arraysIdx = 0; arraysIdx < div; arraysIdx++) { var b = new Float32Array(arrayLen); for (var i = 0; i < b.length; i++) { b[i] = i * (arraysIdx + 1); } arrays.push(b); } var timeBefore = new Date().getTime(); for (var currArrayIdx = 0; currArrayIdx < arrays.length; currArrayIdx++) { a.set(arrays[currArrayIdx], currArrayIdx * arrayLen); } var timeAfter = new Date().getTime(); good.innerHTML = (timeAfter - timeBefore) / 1000 + " sec.\n"; timeBefore = new Date().getTime(); for (var currArrayIdx = 0; currArrayIdx < arrays.length; currArrayIdx++) { for (var i = 0; i < arrayLen; i++) { aSimple.push(arrays[currArrayIdx][i]); } } timeAfter = new Date().getTime(); bad.innerHTML = (timeAfter - timeBefore) / 1000 + " sec.\n"; 

 Using set of TypedArray: <div id='good' style='background-color:lightGreen'>working...</div> Using push of Array: <div id='bad' style='background-color:red'>working...</div>