Swap Usage in Linux Host

Today I found two interesting questions, which may sound pretty easy but actually requires lots of expertise to answer.

Q1 How to identify which process is consuming swap space in a linux host?
Q2 Why SWAP shows significant usage in top while free returns 0 usage for swap?

Let’s start our voyage of discovery now.

My test host:
OS: Redhat Linux 5.4
DB: Oracle 11.2.0.3

# top command output

top - 19:10:57 up 14:16, 5 users, load average: 0.00, 0.05, 0.07
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.6%us, 0.6%sy, 0.0%ni, 98.4%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
Mem: 1555320k total, 1503844k used, 51476k free, 89296k buffers
Swap: 1048568k total, 0k used, 1048568k free, 1155884k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP COMMAND

2559 oracle 15 0 683m 68m 66m S 0.0 4.5 0:01.11 614m ora_smon_PROD1

Note the VIRT, RES and SWAP column. We have an Oracle instance running on this host and pid of smon process is 2559.

From the following command we can be sure that swap space is not being used.

[oracle@odd ~]$ grep -i swap /proc/meminfo
SwapCached: 0 kB
SwapTotal: 1048568 kB
SwapFree: 1048568 kB

[root@odd ~]# free -k
total used free shared buffers cached
Mem: 1555320 1519620 35700 0 87608 1166348
-/+ buffers/cache: 265664 1289656
Swap: 1048568 0 1048568

[root@odd ~]# swapon -s
Filename Type Size Used Priority
/dev/mapper/VolGroup00-LogVol01 partition 1048568 0 -1

Now we need to build a program to populate huge chunk of memory and check whether we can find who is using SWAP.

vi kk.c

#include <stdio.h>
#include <stdlib.h>

#define ALLOC_SIZE(e)( e * 1024 )

int main(){
  char *p = malloc(ALLOC_SIZE(1024 * 1024));
  int x,y,z;
  for ( x=0 ; x<1024; x++ ) {
    for ( y=0; y<1024; y++) {
      for( z=0; z<1024; z++){
        p[x*1024*1024+y*1024+z]='a';
      }
    }
  }
  sleep(190000000);
}

This program applies for 1024M memory and populate it with the character ‘a’. We may need to be a little patient before swap actually happened in our host.

[root@odd ~]# gcc -o kk kk.c
[root@odd ~]# ./kk &
[1] 2841

[root@odd ~]# free -m
total used free shared buffers cached
Mem: 1518 1481 37 0 1 248
-/+ buffers/cache: 1231 287
Swap: 1023 8 1015

[root@odd ~]#
[root@odd ~]#
[root@odd ~]# for file in /proc/*/smaps
do
awk -v Path="$file" '{if (($0 ~ /Swap/) && ($2>0)) {print Path,$2}}' $file;
done

/proc/2841/smaps 920

Now we found that free is reporting swap usage and we did find out process 2841 is using swap. In addition, if you wait for another several minutes, you may find that other processes are subject to swap as well.

for file in /proc/*/smaps
do
awk -v Path="$file" '{if (($0 ~ /Swap/) && ($2>0)) {print Path,$2}}' $file;
done | sort -k2n

/proc/1971/smaps 1152
/proc/2841/smaps 1872
/proc/2380/smaps 2544

Having got the answer for the first question, now let’s move to the second question.

[root@odd ~]# pmap 2559 | grep /dev/shm | grep 4096K | wc -l
128

/dev/shm is used to store shared segments in Linux. 128*4096K = 512M, which means our db SGA is 512M and granule size is 4096K. We can prove it in our database.

SQL> show parameter sga

NAME TYPE VALUE
------------------------------------ -------------------------------- ------------------------------
lock_sga boolean FALSE
pre_page_sga boolean FALSE
sga_max_size big integer 512M
sga_target big integer 512M

SQL> select * from v$sgainfo;

NAME BYTES RES
-------------------------------- ---------- ---
Fixed SGA Size 1346140 No
Redo Buffers 5832704 No
Buffer Cache Size 243269632 Yes
Shared Pool Size 251658240 Yes
Large Pool Size 4194304 Yes
Java Pool Size 20971520 Yes
Streams Pool Size 8388608 Yes
Shared IO Pool Size 0 Yes
Granule Size 4194304 No
Maximum SGA Size 535662592 No
Startup overhead in Shared Pool 64967656 No
Free SGA Memory Available 0

Let’s see how top calculates VIRT, RES and SWAP.

[oracle@odd 2559]$ cd /proc/2559
[oracle@odd 2559]$ grep Size smaps | awk '{print $2}' | awk 'BEGIN {sum=0}{sum+=$0}END{print sum/1024}'
683.273

[oracle@odd 2559]$ grep Rss smaps | awk '{print $2}' | awk 'BEGIN {sum=0}{sum+=$0}END{print sum/1024}'
69.0703

SWAP = VIRT - RES

man top for more details:

o: VIRT -- Virtual Image (kb)
The total amount of virtual memory used by the task. It includes all code, data and shared libraries
plus pages that have been swapped out and pages that have been mapped but not used.

p: SWAP -- Swapped size (kb)
Memory that is not resident but is present in a task. This is memory that has been swapped out but
could include additional non-resident memory. This column is calculated by subtracting physical mem‐
ory from virtual memory.

q: RES -- Resident size (kb)
The non-swapped physical memory a task has used.

Until now, SWAP in top seems quite misleading and it would be appreciated to leave your comment if you happen to have the answer.

Reference:

http://www.cyberciti.biz/faq/linux-which-process-is-using-swap/
http://serverfault.com/questions/424213/how-to-read-top-process-list-says-processes-have-swap-but-overview-says-no-sw
http://www.linuxquestions.org/questions/linux-server-73/identify-which-processes-are-consuming-swap-892549/

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s