Effective Copy Memory

Question

I inherited the following code:

void *buf1 = calloc(1,total_buf_size); //build buf1 up with memcpy int index; void *buf2 = buf1; for (index = 0; index < log->numentries; index++) { logentry_t *entry = (logentry_t*)&log->entries[index]; memcpy(buf2,entry->entryname,sizeof(entry->entryname)); buf2 = buf2 + (int)sizeof(entry->entryname); memcpy(buf2,&entry->entrysize,sizeof(entry->entrysize)); buf2 = buf2 + (int)sizeof(entry->entrysize); memcpy(buf2,&entry->updatesize,sizeof(entry->updatesize)); buf2 = buf2 + (int)sizeof(entry->updatesize); memcpy(buf2,&entry->numupdates,sizeof(entry->numupdates)); buf2 = buf2 + (int)sizeof(entry->numupdates); int j; for (j = 0; j < entry->numupdates; j++) { memcpy(buf2,&entry->offsets[j],sizeof(entry->offsets[j])); buf2 = buf2 + (int)sizeof(entry->offsets[j]); } int k; for (k = 0; k < entry->numupdates; k++) { memcpy(buf2,&entry->sizes[k],sizeof(entry->sizes[k])); buf2 = buf2 + (int)sizeof(entry->sizes[k]); } memcpy(buf2,entry->data,entry->updatesize); } 

I have a transaction log in which I iterate, and I need to write data for each log entry to a line in a file. Currently, it uses memcpy to create a buffer for all entries and immediately writes them to a file. Is there a better way to copy and expand memory in buf2?

Answer 1

I'm not sure exactly what kind of performance you're looking for, but here is an example of writing each log entry to a file and then flushing the buf2 buffer for each iteration. Thus, you can reduce the size of buf1 (defined by total_buf_size) to match only one log entry:

 void *buf1 = calloc(1,total_buf_size); //With this method total_buf_size can be reduced to fit just one log entry record //build buf1 up with memcpy int index; void *buf2 = buf1; FILE * pFile; pFile = fopen ("myfile.txt","w"); for (index = 0; index < log->numentries; index++) { logentry_t *entry = (logentry_t*)&log->entries[index]; memcpy(buf2,entry->entryname,sizeof(entry->entryname)); buf2 = buf2 + (int)sizeof(entry->entryname); memcpy(buf2,&entry->entrysize,sizeof(entry->entrysize)); buf2 = buf2 + (int)sizeof(entry->entrysize); memcpy(buf2,&entry->updatesize,sizeof(entry->updatesize)); buf2 = buf2 + (int)sizeof(entry->updatesize); memcpy(buf2,&entry->numupdates,sizeof(entry->numupdates)); buf2 = buf2 + (int)sizeof(entry->numupdates); int j; for (j = 0; j < entry->numupdates; j++) { memcpy(buf2,&entry->offsets[j],sizeof(entry->offsets[j])); buf2 = buf2 + (int)sizeof(entry->offsets[j]); } int k; for (k = 0; k < entry->numupdates; k++) { memcpy(buf2,&entry->sizes[k],sizeof(entry->sizes[k])); buf2 = buf2 + (int)sizeof(entry->sizes[k]); } memcpy(buf2,entry->data,entry->updatesize); fwrite(buf2, sizeof(logentry_t), sizeof(buf2), pFile); memset(&buf2, 0, sizeof(buf2)); //clear it out buf2 = buf1; //reset the pointer } fclose(pFile); free(buf2); 

Answer 2

In my comment below the question, I had in mind something like:

 FILE *file = fopen("log.file", "a"); // assert(file != NULL); for (int i = 0; i < log->numentries, i++) { logentry_t *e = (logentry_t*)&log->entries[i]; #define OUT(x) do { fwrite(x, sizeof(x), 1, file) } while (0) OUT(e->entryname); OUT(e->entrysize); OUT(e->updatesize); OUT(e->numupdates); for (int j = 0; j < e->numupdates; j++) OUT(&e->offsets[j]); for (int k = 0; k < e->numupdates; k++) OUT(&entry->sizes[k]); fwrite(e->data, e->updatesize, 1, file); #undef OUT } fflush(file); ... 

those. you don't need buffers at all, because fwrite already buffering at the stdio.h level. This code has not been tested and may contain errors of any type.

Also note that writing raw data to a log file can lead to small / large problems with finite and integer size on different platforms or even with compilers (although the latter is less likely, depending on the actual types of the logentry_t fields). You can consider preformatting integer data fields for a portable fixed size with something like htons / htonl or even printf("%ld", (long)x) as text.