I have an application that transfers 250 MB of data using a simple and fast neural network threshold function for data blocks (2 32-bit words in total). Based on the result of a (very simple) calculation, a piece is unpredictably inserted into one of 64 bins. So one big stream in and 64 shorter (variable length) flows.
This is repeated many times with various detection functions.
Calculation is limited by bandwidth. I can say this because there is no change in speed, even if I use the discriminant function, which is much more computationally intensive.
What is the best way to structure recording new threads to optimize memory bandwidth? I especially think that understanding cache usage and cache line size can play a big role in this. Imagine the worst case where I have 64 output streams and, unfortunately, many are mapped to the same cache line. Then, when I write the next 64 bits of data to the stream, the CPU must pop the obsolete cache line into main memory and load it into the corresponding cache line. Each of them uses 64 bytes of bandwidth ... so a limited bandwidth application can spend 95% of the memory bandwidth (although in this hypothetical worst case).
It is difficult to even try to measure the effect, so the development of the paths around it is even more vague. Or am I even chasing a bottleneck in a ghost that is somehow optimizing the hardware better than I could?
I use Core II x86 processors if that matters.
Edit: Here is a sample code. It passes through an array and copies its elements to various output arrays selected pseudo-randomly. Running the same program with different numbers of destination cells gives different time intervals, although the same amount of calculations and reading and writing in memory were done:
2 output streams: 13 seconds
8 output streams: 13 seconds
32 output streams: 19 seconds
128 output streams: 29 seconds
512 output streams: 47 seconds
512 2 4X (,), .
#include <stdio.h>
#include <stdlib.h>
#include <ctime>
int main()
{
const int size=1<<19;
int streambits=3;
int streamcount=1UL<<streambits;
int *instore=(int *)malloc(size*sizeof(int));
int **outstore=(int **)malloc(streamcount*sizeof(int *));
int **out=(int **)malloc(streamcount*sizeof(int));
unsigned int seed=0;
for (int j=0; j<size; j++) instore[j]=j;
for (int i=0; i< streamcount; ++i)
outstore[i]=(int *)malloc(size*sizeof(int));
int startTime=time(NULL);
for (int k=0; k<10000; k++) {
for (int i=0; i<streamcount; i++) out[i]=outstore[i];
int *in=instore;
for (int j=0; j<size/2; j++) {
seed=seed*0x1234567+0x7162521;
int bin=seed>>(32-streambits);
*(out[bin]++)=*(in++);
*(out[bin]++)=*(in++);
}
}
int endTime=time(NULL);
printf("Eval time=%ld\n", endTime-startTime);
}