1

so I have the same problem as many others. I have read through articles and SO questions that are similar but never clearly understood how to resolve this problem.

I am using a SQLDataReader to run a select query against a DB, and the query returns 180k records, I am looping through reader.Read() and creating new instance of an object for each loop, assigning values to properties from reader, and at the end of the loop adding this object to a list, I am doing this for all 180k rows of data. Quite obviously I am running into Out of memory exception. Can someone suggest a workaround for this?

Relevant code:

List<MyObject> collection = new List<MyObject>();
var reader = cmd.ExecuteReader();

while(reader.Read())
{
   var obj = new MyObject();
   obj.Property1 = Convert.ToInt32(reader["Column1"]);
   obj.Property2 = Convert.ToString(reader["Column2"]);
   ...
   ...
   obj.Property178 = Convert.ToString(reader["Column178"]);

   collection.Add(obj);
}

This collection at the end of the loop is being returned to a calling method which is doing further processing on the records. Alternatives I tried:

  1. Use DataTable and dump records from reader into it instead of looping.
    • This did not show any change in memory usage.
  2. Use a single instance of MyObject and reassign values to the same instance on every loop
    • This significantly reduced the memory being used, but at the end of the loop, all 180k records in the collection had the exact same value since obj is reference type and modifying its value will modify the list as well.

Any other suggestions?

3
  • 2
    the 1st question you need to answer is: why you need to load 180K records, each of them has 178 properties? is it really need it this way? what I mean is: has to have a way to filter this before you load it. Currently, you created 180K * 178, about 32M objects, and most are string. not good. Commented Jun 7, 2023 at 15:43
  • You might be able to pre-allocate the list size new List<MyObject>(190000); but that probably won't be enough. you need to rethink why you need so much data in the first place, and whether they all need to be strings. Commented Jun 7, 2023 at 22:07
  • 1
    A multidimensional collection of 180000x178 is way beyond the maximum size which is advisable to be managed via memory in any scenario. There are other alternatives to deal with so much information efficiently. DBs are a good example. You aren't maximising your current setup, your available resources, right the contrary. You should let the DB do its work and memory/code deal with other problems. You could either get 1 column (primary key/ID) for all the records, iterate through/query each of them, or divide it into batches (25-50 elements seems more than enough) via said main column/ID. Commented Jun 8, 2023 at 6:27

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.