Interpreting data from / proc / [id] / mountstats or / proc / self / mountstats

Question

I would like to control the IO of my NFS shares. There are many programs that interpret data in /proc/[id]/mountstats ([id] = id / client / server nfs) or /proc/self/mountstats . The problem is that these programs calculate several IOs per second. This is very good on the command line, but it is completely useless for my monitoring solution. Therefore, I want to interpret mountstats counters. My problem is that I need mountstats documentation for nfs v4.

So far, my search has not yielded anything useful.

thanks for the help

sample output / proc / self / mountstats on Ubuntu (nfs client):

 ... device IPADRESS:/mountpoint mounted on /mnt/mountpoint with fstype nfs4 statvers=1.0 opts: rw,vers=4,rsize=1048576,wsize=1048576,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,nointr,proto=tcp,timeo=600,retrans=2,sec=sys age: 77663 caps: caps=0x1e,wtmult=512,dtsize=4096,bsize=0,namelen=255 nfsv4: bm0=0xfdffbfff,bm1=0xf9be3e,acl=0x3 sec: flavor=1,pseudoflavor=1 events: 1462723 25756220 1025148 1164028 1083099 2027020 31347107 1984730 279773 1915700 1346583 21077199 520498 31825 525298 39026 2180470 1083041 0 16822 990144 11011 0 0 0 bytes: 46686144592 3990939437 0 0 30238687174 3759616491 7739053 1346583 RPC iostats version: 1.0 p/v: 100003/4 (nfs) xprt: tcp 852 1 2 0 0 11822378 11822377 0 17669710 13357205 per-op statistics NULL: 0 0 0 0 0 0 0 0 READ: 1950114 1950114 0 330348288 30355844804 105050 6678050 6907640 WRITE: 911822 911822 0 3932410080 215189360 10653830 1472320 12211620 COMMIT: 389509 389509 0 65718096 88808052 8060 253550 284360 OPEN: 1168845 1168845 0 292577612 554554444 430040 628900 1229990 OPEN_CONFIRM: 43 43 0 7048 2924 0 10 10 OPEN_NOATTR: 0 0 0 0 0 0 0 0 OPEN_DOWNGRADE: 54 54 0 9952 12744 90 0 120 CLOSE: 1081161 1081161 0 191495392 255153996 201030 404620 767870 SETATTR: 43303 43303 0 8753024 10046296 1060 25650 30380 FSINFO: 2 2 0 268 184 0 0 0 RENEW: 0 0 0 0 0 0 0 0 SETCLIENTID: 0 0 0 0 0 0 0 0 SETCLIENTID_CONFIRM: 0 0 0 0 0 0 0 0 LOCK: 1140050 1140050 0 246277244 77523400 76270 326950 469890 LOCKT: 1 1 0 188 52 0 0 0 LOCKU: 1037895 1037895 0 187916912 70576860 18130 357530 441480 ACCESS: 450451 450451 0 72511716 102702828 21440 222460 280950 GETATTR: 1462724 1462724 0 224029496 310097488 21290 390910 485350 LOOKUP: 2043380 2043380 0 361899452 532433688 18200 499200 585170 LOOKUP_ROOT: 1 1 0 116 232 0 0 0 REMOVE: 32356 32356 0 5953448 7765440 600 15450 18560 RENAME: 20388 20388 0 5593088 9215376 510 7740 9750 LINK: 537 537 0 132024 231984 0 220 270 SYMLINK: 0 0 0 0 0 0 0 0 CREATE: 999 999 0 227376 293140 30 650 830 PATHCONF: 1 1 0 140 72 0 0 0 STATFS: 2872 2872 0 419360 333152 580 3980 4880 READLINK: 0 0 0 0 0 0 0 0 READDIR: 82718 82718 0 14625720 118955760 770 48470 53310 SERVER_CAPS: 3 3 0 400 264 0 0 0 DELEGRETURN: 0 0 0 0 0 0 0 0 GETACL: 0 0 0 0 0 0 0 0 SETACL: 0 0 0 0 0 0 0 0 FS_LOCATIONS: 0 0 0 0 0 0 0 0 ... 

Answer 1

Full source: http://www.fsl.cs.stonybrook.edu/~mchen/mountstat-format.txt

  Format of NFS mountstat 

Linux kernel version: 2.6.32-358.el6.x86_64

Important files for statistics:

 - linux/fs/nfs/super.c: nfs_show_stats - linux/include/linux/nfs_iostat.h: nfs_stat_eventcounters - linux/net/sunrpc/stats.c: rpc_print_iostats - linux/net/sunrpc/xprtsock.c: xs_tcp_print_stats 

Server features (caps)

• caps: server features
• wtmult: server block size
• dtsize: readdir size
• bsize: server block size
• namlen:

nfsv4 (nfs_show_stats)

• bm0: attribute_bitmask [0], which attributes are supported
• bm1: attribute_bitmask [1], which attributes are supported
• acl: acl_bitmask, supported acl attributes

sec, the security effect for this mount (nfs_show_stats)

• Fragrance: auth-> au_ops-> au_flavor
• pseudoflavor: auth-> au_flavor

Events (linux / nfs_iostat.h: nfs_stat_eventcounters)

• inoderevalidate
• dentryrevalidate
• datainvalidate
• attrinvalidate
• vfsopen
• vfslookup
• vfsaccess
• vfsupdatepage
• vfsreadpage
• vfsreadpages
• vfswritepage
• vfswritepages
• vfsgetdents
• vfssetattr
• vfsflush
• vfsfsync
• vfslock
• vfsrelease
• congestionwait
• setattrtrunc
• extendwrite
• sillyrename
• shortread
• shortwrite
• delay
• pnfs_read
• pnfs_write

Byte (linux / nfs_iostat.h: nfs_stat_bytecounters)

• normalreadbytes
• normalwritebytes
• directreadbytes
• directwritebytes
• serverreadbytes
• serverwritebytes (serverwrittenbytes)
• readpages
• writepages

xprt (tcp)

• linux / net / sunrpc / xprtsock.c: xs_tcp_print_stats
• linux / include / linux / sunrpc / xprt.h: struct rpc_xprt
  • srcport: Ephemeral port
  • bind_count: how many rpcbind operations
  • connect_count: how many TCP connections
  • connect_time: how much time has passed the connection
  • idle_time: how long the migration has been idle
  • rpcsends: how many sockets sends
  • rpcrecvs: how many sockets it receives
  • badxids: how many unmatchable xid received
  • req_u: Average wire requests (using the slot table)
  • bklog_u: using the delay queue (average queue length in the backlog)
  • max_slots: used max rpc_slots
  • send_u: send q use
  • pending_u: pend q usage

Each type of RPC operation shows (rpc_print_iostats)

Statistics record: rpc_exit_task ---> xprt_release ---> rpc_count_iostats

All NFS protocols are described here: http://www.freesoft.org/CIE/Topics/115.htm

• Type name op
• ops How many requests of this type were requested
• trans: how many transmissions of this type op were sent
• timeouts: how many timeouts of this type of operation occurred
• bytes_sent: how many bytes were sent for this type op
• bytes_recv: how many bytes received for this type op
• queue: how long did ops of this type wait in line before (microsecond)
• rtt: How long has the client been waiting for responses about this operator type from the server (microsecond)
• execute: how long do operations of this type take to execute (from rpc_init_task - rpc_exit_task) (microsecond)

rpc_run_task ---> rpc_new_task ---> rpc_init_task

The last three time statistics are collected using ns, but they are converted to ms when they are printed.

Runtime should approach (queue + rtt).

Please note: in Linux source code, field names are prefixed, for example, "om_", the first field is "om_ops" instead of "ops".

NFSv4.0 EXAMPLE

 device 130.245.177.235:/nfsdata/ mounted on /mnt with fstype nfs4 statvers=1.1 opts: rw,vers=4,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60, acdirmin=30,acdirmax=60,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys, clientaddr=130.245.177.234,minorversion=0,local_lock=none age: 19 (time, in seconds, since the NFS is mounted) caps: caps=0xfff7,wtmult=512,dtsize=32768,bsize=0,namlen=255 nfsv4: bm0=0xfdffafff,bm1=0xf9be3e,acl=0x0 sec: flavor=1,pseudoflavor=1 events: 0 168232 0 0 0 10095 217808 0 2 9797 0 9739 0 0 19739 19739 0 19739 0 0 0 0 0 0 0 0 0 bytes: 1612840960 0 0 0 627536112 0 158076 0 RPC iostats version: 1.0 p/v: 100003/4 (nfs) xprt: tcp 737 0 1 0 0 69698 69697 0 81817 0 2 1082 12119 per-op statistics NULL: 0 0 0 0 0 0 0 0 READ: 9797 9797 0 1567520 628138612 71 7953 8200 WRITE: 0 0 0 0 0 0 0 0 COMMIT: 0 0 0 0 0 0 0 0 OPEN: 19740 19740 0 4737600 7343280 505 3449 4172 OPEN_CONFIRM: 10211 10211 0 1552072 694348 74 836 1008 OPEN_NOATTR: 0 0 0 0 0 0 0 0 OPEN_DOWNGRADE: 0 0 0 0 0 0 0 0 CLOSE: 19739 19739 0 3316152 2605548 334 3045 3620 SETATTR: 0 0 0 0 0 0 0 0 FSINFO: 1 1 0 132 108 0 0 0 RENEW: 0 0 0 0 0 0 0 0 SETCLIENTID: 0 0 0 0 0 0 0 0 SETCLIENTID_CONFIRM: 0 0 0 0 0 0 0 0 LOCK: 0 0 0 0 0 0 0 0 LOCKT: 0 0 0 0 0 0 0 0 LOCKU: 0 0 0 0 0 0 0 0 ACCESS: 96 96 0 14584 19584 0 8 10 GETATTR: 1 1 0 132 188 0 0 0 LOOKUP: 10095 10095 0 1655576 2382420 36 898 1072 LOOKUP_ROOT: 0 0 0 0 0 0 0 0 REMOVE: 0 0 0 0 0 0 0 0 RENAME: 0 0 0 0 0 0 0 0 LINK: 0 0 0 0 0 0 0 0 SYMLINK: 0 0 0 0 0 0 0 0 CREATE: 0 0 0 0 0 0 0 0 PATHCONF: 1 1 0 128 72 0 0 0 STATFS: 0 0 0 0 0 0 0 0 READLINK: 0 0 0 0 0 0 0 0 READDIR: 0 0 0 0 0 0 0 0 SERVER_CAPS: 2 2 0 256 176 0 0 0 DELEGRETURN: 0 0 0 0 0 0 0 0 GETACL: 0 0 0 0 0 0 0 0 SETACL: 0 0 0 0 0 0 0 0 FS_LOCATIONS: 0 0 0 0 0 0 0 0 RELEASE_LOCKOWNER: 0 0 0 0 0 0 0 0 SECINFO: 0 0 0 0 0 0 0 0 EXCHANGE_ID: 0 0 0 0 0 0 0 0 CREATE_SESSION: 0 0 0 0 0 0 0 0 DESTROY_SESSION: 0 0 0 0 0 0 0 0 SEQUENCE: 0 0 0 0 0 0 0 0 GET_LEASE_TIME: 0 0 0 0 0 0 0 0 RECLAIM_COMPLETE: 0 0 0 0 0 0 0 0 LAYOUTGET: 0 0 0 0 0 0 0 0 GETDEVICEINFO: 0 0 0 0 0 0 0 0 LAYOUTCOMMIT: 0 0 0 0 0 0 0 0 LAYOUTRETURN: 0 0 0 0 0 0 0 0 NFSv4.1 EXAMPLE =============== device 130.245.177.235:/nfsdata mounted on /mnt with fstype nfs4 statvers=1.1 opts: rw,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=130.245.177.234,local_lock=none age: 44 impl_id: name='',domain='',date='0,0' caps: caps=0x3ffff,wtmult=512,dtsize=32768,bsize=0,namlen=255 nfsv4: bm0=0xfdffbfff,bm1=0xf9be3e,bm2=0x800,acl=0x3,sessions,pnfs=not configured sec: flavor=1,pseudoflavor=1 events: 67379 1692812 0 30749 0 980 1984612 0 0 0 0 0 0 0 96611 96611 0 96611 0 0 0 0 0 0 0 0 0 bytes: 0 0 0 0 0 0 0 0 RPC iostats version: 1.0 p/v: 100003/4 (nfs) xprt: tcp 674 0 1 0 0 279772 279771 0 649944 0 5 1711 90409 per-op statistics NULL: 0 0 0 0 0 0 0 0 READ: 0 0 0 0 0 0 0 0 WRITE: 0 0 0 0 0 0 0 0 COMMIT: 0 0 0 0 0 0 0 0 OPEN: 96612 96612 0 27824256 34007424 13349 16253 32061 OPEN_CONFIRM: 0 0 0 0 0 0 0 0 OPEN_NOATTR: 0 0 0 0 0 0 0 0 OPEN_DOWNGRADE: 0 0 0 0 0 0 0 0 CLOSE: 96610 96610 0 19708440 17003360 2936 11082 15386 SETATTR: 0 0 0 0 0 0 0 0 FSINFO: 1 1 0 168 152 0 0 0 RENEW: 0 0 0 0 0 0 0 0 SETCLIENTID: 0 0 0 0 0 0 0 0 SETCLIENTID_CONFIRM: 0 0 0 0 0 0 0 0 LOCK: 0 0 0 0 0 0 0 0 LOCKT: 0 0 0 0 0 0 0 0 LOCKU: 0 0 0 0 0 0 0 0 ACCESS: 16932 16932 0 3183200 2844576 83 1919 2308 GETATTR: 67380 67380 0 12128380 15632160 336 8367 10644 LOOKUP: 1236 1236 0 247176 167784 7 153 197 LOOKUP_ROOT: 0 0 0 0 0 0 0 0 REMOVE: 0 0 0 0 0 0 0 0 RENAME: 0 0 0 0 0 0 0 0 LINK: 0 0 0 0 0 0 0 0 SYMLINK: 0 0 0 0 0 0 0 0 CREATE: 980 980 0 219516 305760 5 168 197 PATHCONF: 1 1 0 164 116 0 0 0 STATFS: 0 0 0 0 0 0 0 0 READLINK: 0 0 0 0 0 0 0 0 READDIR: 0 0 0 0 0 0 0 0 SERVER_CAPS: 2 2 0 328 280 0 0 0 DELEGRETURN: 0 0 0 0 0 0 0 0 GETACL: 0 0 0 0 0 0 0 0 SETACL: 0 0 0 0 0 0 0 0 FS_LOCATIONS: 0 0 0 0 0 0 0 0 RELEASE_LOCKOWNER: 0 0 0 0 0 0 0 0 SECINFO: 0 0 0 0 0 0 0 0 EXCHANGE_ID: 0 0 0 0 0 0 0 0 CREATE_SESSION: 0 0 0 0 0 0 0 0 DESTROY_SESSION: 0 0 0 0 0 0 0 0 SEQUENCE: 0 0 0 0 0 0 0 0 GET_LEASE_TIME: 0 0 0 0 0 0 0 0 RECLAIM_COMPLETE: 0 0 0 0 0 0 0 0 LAYOUTGET: 0 0 0 0 0 0 0 0 GETDEVICEINFO: 0 0 0 0 0 0 0 0 LAYOUTCOMMIT: 0 0 0 0 0 0 0 0 LAYOUTRETURN: 0 0 0 0 0 0 0 0 SECINFO_NO_NAME: 0 0 0 0 0 0 0 0 TEST_STATEID: 0 0 0 0 0 0 0 0 FREE_STATEID: 0 0 0 0 0 0 0 0 GETDEVICELIST: 0 0 0 0 0 0 0 0 BIND_CONN_TO_SESSION: 0 0 0 0 0 0 0 0 DESTROY_CLIENTID: 0 0 0 0 0 0 0 0 # vim: tabstop=2:shiftwidth=2:expandtab:foldmethod=indent 

Also https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/include/linux/nfs_iostat.h and https://utcc.utoronto.ca/~cks/space/ blog / linux / NFSMountstatsIndex for additional reading.

Answer 2

There seems to be no better “documentation" than the mountstats utility code .

Answer 3

What Linux are you doing this on, because for me, on Redhat, mountstats (for v4) have something like this:

 [ root@riau petri]# cat /proc/1511/mountstats device rootfs mounted on / with fstype rootfs device /proc mounted on /proc with fstype proc device /sys mounted on /sys with fstype sysfs device udev mounted on /dev with fstype devtmpfs device devpts mounted on /dev/pts with fstype devpts device tmpfs mounted on /dev/shm with fstype tmpfs device /dev/mapper/vg_riau-lv_root mounted on / with fstype ext4 device /proc/bus/usb mounted on /proc/bus/usb with fstype usbfs device /dev/sda1 mounted on /boot with fstype ext4 device /dev/mapper/vg_riau-lv_home mounted on /home with fstype ext4 device none mounted on /proc/sys/fs/binfmt_misc with fstype binfmt_misc device cgroup mounted on /cgroup/cpuset with fstype cgroup device cgroup mounted on /cgroup/cpu with fstype cgroup device cgroup mounted on /cgroup/cpuacct with fstype cgroup device cgroup mounted on /cgroup/memory with fstype cgroup device cgroup mounted on /cgroup/devices with fstype cgroup device cgroup mounted on /cgroup/freezer with fstype cgroup device cgroup mounted on /cgroup/net_cls with fstype cgroup device cgroup mounted on /cgroup/blkio with fstype cgroup device sunrpc mounted on /var/lib/nfs/rpc_pipefs with fstype rpc_pipefs device /etc/auto.misc mounted on /misc with fstype autofs device -hosts mounted on /net with fstype autofs device nfsd mounted on /proc/fs/nfsd with fstype nfsd