0

How can such an output of free -m be explained?

               total        used        free      shared  buff/cache   available
Mem:           32036        1012         225           3        8400       31024
Swap:          32767       24138        8629

I understand having free memory to be low is no sign of alarm as Linux uses unused memory for buffers and file system caches (buff/cache). What's important to have enough available memory.

But why is the kernel not swapping in again? Nearly all memory is available.

I took this output from a continuously log-to-disk I setup as "every minute" cronjob. At that point in time the system was so unresponsive I could not even locally login anymore. After slowly typing username and password, there was a timeout (Login timed out after 60 seconds.), so I could not reach a shell and had to power-cycle the server to recover.

The journal is full of take too long, timeout and broken pipe messages as everything on the system is crawling and therefore malfunctioning.

I played around with vm.swappiness, having the default value of 60 reduced to 10 (to put the kernel more onto "only swap if it's really necessary"), but I have similar results.

I was hesitant to try a swapoff && swapon to bring the available memory back into play. Does the oom-killer take over if not everything fits into RAM? Or does the system crash then?

A little more background information about the concrete case:
I have a Proxmox setup, evaluating how stable everything runs. I really stress the machine having allocated more RAM to the VMs in total than I have. To my unterstanding, this should still work with paying a little price of using swap space, slowing things down.
I noticed that everything works stabile as I expected. I play around suspending VMs to disk, then starting other VMs. Swap gets used if needed and when VMs are suspended, Swap is being freed again.

But lately I added backup into my evaluation and this really crashes the machine. Over night, when PVE Backup is started RAM gets more and more available by consuming Swap. Backup speed falls from "1% per few seconds" to "1% per several hours" and eventually no progress at all. The machine gets unresponsive with that memory picture. The VMs are still running, but also their applications are malfunctioning as their system gets errors like interrupt took 2.2s, Watchdog timeout (limit 3min)!, CPU stuck for 23s!. In the morning I find myself an unresponsive host.

Improve this question
3
  • cat /proc/meminfo would give a lot more detail than free and might give a clue how the "available" memory is actually being used. Commented May 17, 2024 at 0:48
  • 1
    The kernel doesn't idly "swap something (a page or group of pages) in". It waits until a process references (reads, writes, executes, or does I/O) it. If the process memory represented by swapped out pages is freed, the swap space is likewise released. Commented May 17, 2024 at 1:23
  • Linux box with 100% used swap and 60% available RAM - is it normal? and more generally this search Commented May 17, 2024 at 8:34

2 Answers 2

Reset to default
2

Swapping in results in massive amounts of IO activity which could negatively affect system performance. If the swapped out applications are dormant, it makes little to no sense to swap them back in.

The free snapshot that you're showing doesn't contain historical data, thus we may never know what you were running before. It's possible that you had very heavy applications running, thus the kernel have swapped out so much.

Does the oom-killer take over if not everything fits into RAM? Or does the system crash then?

It takes into account the amount of virtual RAM available (RAM + swap), thus, no, it won't kill applications until your RAM and swap are both completely consumed.

Generally if you have enough RAM to accommodate all the running applications and still have enough free RAM, you could simply disable swap altogether. In my experience swap has more and more serious negatives than positives and it's only suitable for systems where there's not enough RAM to run applications at all.

Improve this answer
10
  • Fully agree with your answer, but the last paragraph is wrong, isn't it? Without swap, whatever caused swapping in the first place would have run out of memory, and something would have had to die. Commented May 17, 2024 at 11:06
  • (swap is generally very useful, not sure where you see the negatives. It's not like it's going to be used if it's not necessary. I'd compare it to antibiotics: sure, terrible side effects, but a patient wouldn't be taking antibiotics if they didn't have a very problematic bacterial infection) Commented May 17, 2024 at 11:06
  • SWAP negatives: 1) unpredictable major latency spikes 2) unpredictable applications and system slow downs 3) unpredictable system behavior 4) storage wear and tear 5) possible information leak/security concerns. SWAP positive 1) Allowing to run more applications than there's physical RAM. That's all. In my company we have over 200 servers where zero have SWAP enabled. We've observed a critical performance loss and major throughput issues on servers with SWAP enabled. Personally I've not had SWAP enabled on any of my systems since 2004 with zero issues so far. Commented May 17, 2024 at 12:05
  • 1) much better than unpredictable timing of inability to allocate memory 2) much better than unpredictable application crashes and OOM killings 3) you're confusing things, running out of memory leads to unpredictable behaviour, so here swap helps 4) agreeing with you on that, on old or low-quality storage media like SD cards, that's a problem. not a problem on HDDs or SSDs of the last 1.5 decades Commented May 17, 2024 at 12:25
  • But I do think you have a point. Something in your company's distributed system prefers running out of memory over having unpredictable latency (that might be standard things like file servers, or it might be custom things like say your company's own applications): that can be a valid design choice, but the way you portrait it, it would sound like there's something problematic in a large distributed system about it. I'd assume, knowing how well you usually know your stuff, that some problems are hard, and maybe sometimes crashing fast being better than becoming slow is the right choice. Commented May 17, 2024 at 12:27
0

To my question

But why is the kernel not swapping in again?

waltinator gave the answer clear and concise in the comments:

The kernel doesn't idly "swap something (a page or group of pages) in".

It turned out that my described problems (timeouts and the system not reacting anymore) were not related to memory shortage, but an unrelated issue with CIFS.

I was set on the wrong track by the CIFS issue, I suspected memory and therefore swapping to be the cause. Once CIFS was replaced with NFS, the system stayed responsive again.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.