2

I wrote a WPF application, targeting .NET 4.5.2 64-bit. It runs on Windows 8.1 Professional 64-bit.

The application is used to record multiple videos simultaneously. Because the video encoder's speed can vary greatly during operation, I make some very large arrays of byte arrays in memory (totalling 6 GiB), which is enough for me to buffer 40 seconds of video, and I let the encoding happen on a separate thread, so I don't risk getting blocked by the encoding and miss incoming frames from the cameras.

I have read somewhere that (byte) arrays are lazily allocated in .NET - i.e. just writing byte[] foo = new byte[1024*1024]; does not mean that 1 MiB is immediately allocated; the memory is not actually allocated until the first access. Though I can't find where I read that anymore. Anyhow, knowing that, I make sure to call Array.Clear() on my arrays, which should force them to be immediately allocated. I also need to make sure the buffer stays in RAM at all times, so I have disabled the page file altogether.

But for some reason, it seems that either the .NET runtime or Windows is still doing something crazy: right after initializing my byte arrays and calling Array.Clear() on all of them, I could see in Task Manager that 7.4 GiB is in use. And it sat there until I pressed the "Record" button, at which point I started to have real data to fill the buffers with. And the strange thing is, the In Use memory started to drop to about 3.6 GiB, and then slowly climbed back to 7.4 GiB. At no point did my application discard and reallocate arrays. Also keep in mind that because I have disabled the page file, there is no place for Windows to page out to.

What is going on? How should I interpret this weirdness?

What I saw in Task Manager

Improve this question
2
  • 4
    arrays are lazily allocated in .NET No, that would be a result of the operating system handling virtual memory. When you allocate an array in .NET you reserve all the virtual memory required. However, some of this virtual memory may not be assigned to physical RAM. To better understand what is going on I suggest that you look at performance counters for your process; not the performance of your entire computer. And do you have a problem, by the way? Commented Dec 9, 2015 at 9:06
  • 4
    Might one suggest that you've picked the wrong tool for the job. Using a programming language that abstracts your use of memory, and then having to fight with the OS to prevent it doing it's job. If you want explicit control over memory, you should be working at a lower level of abstraction where you can make use of the Windows API, such as VirtualLock Commented Dec 9, 2015 at 9:11

2 Answers 2

Reset to default
2

What is going on? How should I interpret this weirdness?

This isn't weird, this is simply how garbage collection works.

When you use Array.Clear, you're only resetting those bytes to 0 (Thanks @Martin). This does not mean that a garbage collection will occur at that same moment. If the array is still referenced, this won't even make it eligable for collection. The running time of the GC is non-deterministic.

What you're seeing actually makes sense, since you see that once you press "Record", and the runtime needs to allocate more resources to use, it first invokes a collection, and then starts slowly increasing as those bytes are consumed by your application.

As others have said in the comments, if you need fine grained granularity over memory managed in your app, you should preferably be using a lower level language which enables you to control (as best as it can) the allocation and release of memory.

Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

Calling Array.Clear on an array of bytes will not make any memory available for memory collection. It will simply set the bytes to 0.
I do know that Array.Clear sets the bytes to 0. This is why I thought it would essentially force access on my array and cause physical RAM to be allocated. If I combine this information with @Luaan's answer, am I to understand that Array.Clear either is a no-op on a pre-zeroed page or turns my array into a pre-zeroed page as far as the OS sees it (and so the OS decides to point it to one pre-zeroed page?
And it doesn't really explain the drop. When I press the "Record" button, 1) there is no additional resource that I need; 2) even if GC starts a collection, it shouldn't be collecting my arrays, I still have references to them.
@Kal Since you haven't provided us with a minimal reproducible example, I can't help further as to understand that drop.
@YuvalItzchakov ok, I'll try to make a minimal example that triggers this behaviour.
|
1

No, .NET arrays are not allocated lazily. However, the OS does a little trick - when you allocate a pre-zeroed page in memory (and .NET always allocates pre-zeroed pages), it will not actually give you physical RAM - instead, all of your pages point to the (single) zero-page. Until you actually write data to your huge empty array, all of the pages that store the array's data are that one page.

This means that while the virtual memory of the process is always what you'd expect (e.g. the megabyte of the 1024*1024 byte array), the physical RAM usage is not. This is similar to when the OS decides to page your memory out to the page file - you have no control over that, and you really shouldn't have.

In practice, this shouldn't make any difference whatsoever for your application. You still have your contiguous virtual memory space, and the OS will give you real memory as soon as you actually need it. Unless your system is under memory pressure, you get the real memory pretty much immediately (it may need to be zeroed).

Improve this answer

2 Comments

This may be exactly what I meant about lazy allocation: the OS being clever with pre-zeroed pages. Do you have links to articles that describe how this pre-zeroed page handling works?
@Kal It's really the very basics of Windows memory management, it shouldn't be hard to find all the details. For example, see this MSDN blog post - blogs.msdn.com/b/tims/archive/2010/10/29/….

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.