Geek Answers Handbook

Available process memory

I am trying to get available memory in a process to make sure that I am not getting an OutOfMemoryException. I searched the internet and found some examples of how to use memory, but not available.

Let me give an example of use ...

I have a process that does bulk insert (using SqlBulkCopy). I pass the DataTable to the WriteToServer method. I can not use DataReader because I have the opportunity to repeat the process after a failure. My first thought was to select an arbitrary number of rows to insert at a time, say, 50,000. But this is a general process that does not know the data; he does not know the number of columns and the amount of data in each row. Therefore, I thought I could control the memory by adding rows to the DataTable and then sending them to SqlBulkCopy when it is close to running out of memory.

Is this the right approach or is there a better way?
If this is a valid approach, what function could I use to determine the amount of available memory?

Here is my code so far ... AvailableMemoryIsLow is something that I cannot figure out how to determine.

 // m_buffer is a read-once cache (implements IDataReader) that pulls // data from an external source as needed so it uses very little memory. // My original implementation just used m_buffer as the parameter of // WriteToServer but now I have to add retry logic into the process. DataTable dataTable = new DataTable(m_tableName); foreach (DataField d in m_buffer.GetColumns()) dataTable.Columns.Add(new DataColumn(d.FieldName, d.FieldType)); while (m_buffer.Read()) { DataRow row = dataTable.NewRow(); for (int i = 0; i < m_buffer.FieldCount; i++) row[i] = m_buffer.GetValue(i); dataTable.Rows.Add(row); // How do I determine AvailableMemoryIsLow if (rowCount++ >= 50000 || AvailableMemoryIsLow) { PutDataIntoDatabase(dataTable); dataTable.Clear(); rowCount = 0; } } if (dataTable.Rows.Count > 0) PutDataIntoDatabase(dataTable);
+4
source share
3 answers

Obviously, you are running this code on a 32-bit machine or you will not have this problem. In general, pushing a program to consume almost all of the available virtual memory space (2 gigabytes) is not a reasonable task. In short from the ever-present danger of OOM, the data you manage is “live data,” it will most likely be displayed in RAM. A program that requires almost all of the available RAM is very harmful for this program, the operating system, and other processes that run on this machine.

You force the operating system to choose how to allocate RAM between what the processes need and what it reserves for the file system cache. Such a choice always leads to the crowding out of data from RAM to the swap file. This can significantly slow down the work, both when it will be recorded, and again, when the process needs it back to RAM. The primary problem of the operating system, called "shredding."

Just don't do this, dunking so much data in RAM just doesn't make your program faster. This makes it slower. The reasonable upper limit on the amount of RAM that you consume in a 32-bit operating system is somewhere around 500 megabytes. There is no need to hit this particular limit; row counting is good enough.

+3
source

You mentioned you found methods that tell you how much memory is allocated

  GC.GetTotalMemory(false);

- one of these methods (which, I believe, you have already found).

One thing I would like to point out from the MSDN documentation.

Retrieves the number of bytes that are currently considered allocated

This is at the very top of the documentation for the GC.GetTotalMemory method. I would like to indicate the word thought in the above phrase. Now I know that you know how to find the allocated amount, as it is mentioned in your question, however I will give this to illustrate that C # is a managed language. Using and consuming memory is distracting from you, and even GC methods are simply necessary to give you a vague idea of ​​what is going on in your process. Working with memory levels manually sounds risky and unreliable to me.

I would recommend going with your original approach, but bringing the batch size back to level, so it is unlikely that you will get an exception from memory, no matter how many columns you work on. Think of hundreds, perhaps several thousand, not tens of thousands. Any performance gains that you get in large batches will likely outweigh the risk of memory problems at these levels, even if you try to find it. The performance tools mentioned in another answer would be a great way to determine how much batch size should be, and if that's even a problem.

+1
source

The problem is that there are many different “resources”; any of which can prove itself with "OutOfMemoryException".

Probably your best bet, however, is GC.GetTotalMemory(false) .

And even a better approach would be to get a tool like JetBrains dotTrace or RedGate ANTS .

IMHO ...

PS:

If you are doing SQL Bulk Copy, be sure to set EnableStreaming:

0
source

More articles:


All Articles