[OpenIndiana-discuss] Swap/stack space continuously shrinking

Discussion:

Markus Wernig

2018-02-21 11:27:43 UTC

Hi all

I have 2 OI installations in kvm/qemu virtual machines. Both are showing
the same symptom: The available swap/stack space is continuously
decreasing, up to the point (roughly every 6-7 weeks) where none is left
and no new processes can be forked. Meaning a forced reboot, as also no
login shells can be spawned any more.
I have already moved /tmp to a separate dataset in the global zone (and
to plain directories within the two other zones).

What I see is a slow, continuous decrease in the sisze of all file
systems marked as swap (~6.5 GB after reboot):

df -h | grep swap over the last 24 hours:

swap 6.45G 1020K 6.45G 1% /etc/svc/volatile
swap 6.45G 60K 6.45G 1% /var/run
..
swap 5.94G 1020K 5.94G 1% /etc/svc/volatile
swap 5.94G 60K 5.94G 1% /var/run
..
swap 5.89G 204K 5.89G 1% /etc/svc/volatile
swap 5.89G 12K 5.89G 1% /var/run
..
swap 5.62G 204K 5.62G 1% /etc/svc/volatile
swap 5.62G 12K 5.62G 1% /var/run
..
swap 5.04G 1020K 5.04G 1% /etc/svc/volatile
swap 5.04G 48K 5.04G 1% /var/run

The swap device on the other hand remains stable:

# swap -lh
swapfile dev swaplo blocks free
/dev/zvol/dsk/rpool/swap 270,2 4K 512M 512M

As nothing seems to get stored in that space, I assume it must be used
by something else (processes not freeing it).

Does anybody have an idea on how to start debugging this? All I get in
the log files are these messages, starting some hours before the boxes
stand still:

Feb 14 03:51:09 xfer-srv05 tmpfs: [ID 518458 kern.warning] WARNING:
/etc/svc/volatile: File system full, swap space limit exceeded
Feb 14 03:51:10 xfer-srv05 last message repeated 1 time
Feb 14 04:03:20 xfer-srv05 genunix: [ID 470503 kern.warning] WARNING:
Sorry, no swap space to grow stack for pid 8106 (httpd)
Feb 14 04:28:36 xfer-srv05 genunix: [ID 470503 kern.warning] WARNING:
Sorry, no swap space to grow stack for pid 8920 (httpd)
Feb 14 05:58:20 xfer-srv05 genunix: [ID 470503 kern.warning] WARNING:
Sorry, no swap space to grow stack for pid 11050 (httpd)
Feb 14 05:58:24 xfer-srv05 last message repeated 1 time

Thanks for any ideas.

/markus

Till Wegmüller

2018-02-21 13:34:37 UTC

Permalink

Hi Markus

This happens when the Memory Of the Machine runs out.
It is named swap but it is an in memory Filesystem that is mounted at
boot on these specific directories. (Much like TMPFS on Linux)

Have you overcomited the RAM?

Hope this helps
Greetings
Till

Post by Markus Wernig
Hi all
I have 2 OI installations in kvm/qemu virtual machines. Both are showing
the same symptom: The available swap/stack space is continuously
decreasing, up to the point (roughly every 6-7 weeks) where none is left
and no new processes can be forked. Meaning a forced reboot, as also no
login shells can be spawned any more.
I have already moved /tmp to a separate dataset in the global zone (and
to plain directories within the two other zones).
What I see is a slow, continuous decrease in the sisze of all file
swap 6.45G 1020K 6.45G 1% /etc/svc/volatile
swap 6.45G 60K 6.45G 1% /var/run
..
swap 5.94G 1020K 5.94G 1% /etc/svc/volatile
swap 5.94G 60K 5.94G 1% /var/run
..
swap 5.89G 204K 5.89G 1% /etc/svc/volatile
swap 5.89G 12K 5.89G 1% /var/run
..
swap 5.62G 204K 5.62G 1% /etc/svc/volatile
swap 5.62G 12K 5.62G 1% /var/run
..
swap 5.04G 1020K 5.04G 1% /etc/svc/volatile
swap 5.04G 48K 5.04G 1% /var/run
# swap -lh
swapfile dev swaplo blocks free
/dev/zvol/dsk/rpool/swap 270,2 4K 512M 512M
As nothing seems to get stored in that space, I assume it must be used
by something else (processes not freeing it).
Does anybody have an idea on how to start debugging this? All I get in
the log files are these messages, starting some hours before the boxes
/etc/svc/volatile: File system full, swap space limit exceeded
Feb 14 03:51:10 xfer-srv05 last message repeated 1 time
Sorry, no swap space to grow stack for pid 8106 (httpd)
Sorry, no swap space to grow stack for pid 8920 (httpd)
Sorry, no swap space to grow stack for pid 11050 (httpd)
Feb 14 05:58:24 xfer-srv05 last message repeated 1 time
Thanks for any ideas.
/markus
_______________________________________________
openindiana-discuss mailing list
https://openindiana.org/mailman/listinfo/openindiana-discuss

Jim Klimov

2018-02-21 13:50:22 UTC

Permalink

Do you have anything writing to tmpfs (e.g. /tmp in global or local zones)? Note that unlike Linux, there is no default service to wipe older files here - so if you have e.g. regular jobs dumping something into /tmp and not cleaning up after themselves, that would eat your virtual memory and stay in it until deleted (or until reboot kills the tmpfs).

Jim

--
Typos courtesy of K-9 Mail on my Android

Richard L. Hamilton

2018-02-21 14:38:03 UTC

Permalink

Once a week, save the output of

df -h -F tmpfs

and

top -o size -b -c 10

Look for filesystem growth in the former, and a process that's getting larger in the latter.

If it's the kernel, you'd have to use mdb -K from the console or boot with kmdb, and then use ::findleaks

The following covers some of that, if differently; but their use of mdb is for a process, not for the kernel.
https://www.joyent.com/blog/bruning-questions-where-s-my-memory-going

For the kernel, google for

mdb findleaks

and look around a little.

Reginald Beardsley via openindiana-discuss

2018-02-21 15:01:07 UTC

Permalink

I've been forced to power cycle my Hipster system a couple of times lately because it starts thrashing so badly I can't even move the mouse pointer.

This is in a situation where I have a paused VM. If LRU paging was in effect, the paused VM would eventually get swapped to disk and the other processes would recover. That is not happening. It's behaving as if the resident working set was being made too large in an attempt to reduce latency when users switch windows.

I've been meaning to ask about this as I had to power cycle the system yesterday because of the memory leaks in Firefox. I paused the Debian VM and started the Win 7 VM at which point everything ground to a halt. On a previous occasion I let the machine run and the mouse pointer would eventually move, then freeze, move a little more, freeze, etc. It was often 5-10 minutes before the pointer moved again.

Historically going back to SunOS 4.1 I have always configured swap as 8x DRAM so that idle processes could be paged to disk. I'll do some experiments on my Solaris 10 u8 system and see how it behaves if I start several programs that consume all of core and then try to edit a file in an xterm.

Prior to encountering Unix I had tuned a MicroVAX with a couple of batch queues to run at 100% CPU utilization for weeks at a time. If an interactive user logged in, the system responded as if it were completely idle. That's relatively easy to configure in a transient process OS, but the bookkeeping makes it impractical in a fork-exec OS.

Reg

Markus Wernig

2018-02-21 16:33:09 UTC

Permalink

Hi all

Thanks for the pointers, will try to follow up on them.

The fact that Solaris is interleaving disk and memory-backed storage
into the tmpfs hadn't struck me until now, as there never seems to be an
increase of the used space in these file systems, but just a decrease of
the available overall (presumably memory) space.

Just one other question: Shouldn't an increase in memory consumption by
any process be visible in the output of tools like eg. top? (Which it
isn't: no matter how low the available "swap" space reported by df
becomes, top keeps reporting ~6GB free memory:

# top
Memory: 8192M phys mem, 5900M free mem, 512M total swap, 512M free swap
ARC: 816M Total, 607M MRU, 159M MFU, 1579K Anon, 4144K Header, 45M Other

# df -h -F tmpfs
Filesystem Size Used Available Capacity Mounted on
swap 4.98G 1020K 4.98G 1% /etc/svc/volatile
swap 4.98G 48K 4.98G 1% /var/run)

best
/markus

Bob Friesenhahn

2018-02-21 20:05:03 UTC

Permalink

While it may not make a difference, the Solaris equivalent to 'top' is
'prstat'. The 'prstat' utility supports a number of options to show
different things. Since 'prstat' is written specifically for Solaris,
it might provide some value-add over what 'top' (a generic program) is
able to show.

Bob

--
Bob Friesenhahn
***@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/