31119: больше 30000 соединений в системе ------------------------------------------------------ /usr/local/alertmon/store_log.sh: last pid: 85176; load averages: 0.85, 0.66, 0.62 up 24+21:27:18 16:40:02 122 processes: 1 running, 120 sleeping, 1 zombie Mem: 224M Active, 1385M Inact, 278M Wired, 96M Cache, 169M Buf, 18M Free Swap: 2048M Total, 256K Used, 2048M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 85160 root 1 -8 0 12936K 10596K piperd 1 0:01 51.00% perl5.8.8 85158 opennet 1 -8 0 2912K 2280K biowr 0 0:01 34.00% perl5.8.8 85169 postgres 1 98 0 5004K 4072K select 1 0:00 6.00% perl5.8.8 8882 web 1 4 0 4444K 3760K sbwait 0 0:31 0.15% httpd 1398 postgres 1 96 0 30620K 27096K select 0 3:44 0.00% postgres 1041 root 1 96 0 2972K 2268K select 0 3:40 0.00% httpd 1096 bind 1 96 0 7688K 6876K select 1 3:09 0.00% named 1033 postgres 1 96 0 30460K 2424K select 1 1:43 0.00% postgres 79121 opennet 1 4 0 10296K 4568K accept 1 1:32 0.00% perl5.8.8 782 root 1 96 0 6344K 4180K select 1 1:08 0.00% snmpd 974 pop 1 4 0 3936K 3268K kqread 0 1:02 0.00% memcached 40382 web 1 20 0 4788K 4076K lockf 0 0:58 0.00% httpd 79115 opennet 1 4 0 6092K 5260K accept 1 0:50 0.00% perl5.8.8 66909 web 1 4 0 4468K 3800K sbwait 1 0:50 0.00% httpd 64679 web 1 4 0 4588K 3896K sbwait 1 0:49 0.00% httpd 66308 web 1 20 0 4904K 4216K lockf 0 0:49 0.00% httpd 74883 web 1 20 0 4792K 4092K lockf 0 0:48 0.00% httpd 956 root 1 96 0 1520K 984K select 0 0:45 0.00% master ====================================== net.inet.tcp.syncache ====================================== vmstat ITEM SIZE LIMIT USED FREE REQUESTS FAILURES UMA Kegs: 140, 0, 71, 1, 71, 0 UMA Zones: 480, 0, 71, 1, 71, 0 UMA Slabs: 64, 0, 3506, 1155, 1089385, 0 UMA RCntSlabs: 104, 0, 397, 269, 721534, 0 UMA Hash: 128, 0, 3, 27, 7, 0 16 Bucket: 76, 0, 24, 26, 79, 0 32 Bucket: 140, 0, 32, 52, 124, 0 64 Bucket: 268, 0, 69, 43, 307, 92 128 Bucket: 524, 0, 2548, 98, 262809, 278077 VM OBJECT: 132, 0, 35714, 58072, 744082375, 0 MAP: 192, 0, 7, 13, 7, 0 KMAP ENTRY: 68, 83944, 887, 3705, 7619883, 0 MAP ENTRY: 68, 0, 4374, 3298, 1673767739, 0 PV ENTRY: 24, 2154990, 94701, 176304, 26110174046, 0 DP fakepg: 72, 0, 0, 159, 187, 0 mt_zone: 1024, 0, 202, 130, 202, 0 16: 16, 0, 3209, 1054, 88866411, 0 32: 32, 0, 1824, 1453, 22801849, 0 64: 64, 0, 4720, 885, 291397064, 0 128: 128, 0, 2895, 1665, 657336777, 0 256: 256, 0, 1228, 1292, 68914041, 0 512: 512, 0, 322, 350, 3033778, 0 1024: 1024, 0, 83, 373, 15427106, 0 2048: 2048, 0, 392, 404, 5521784, 0 4096: 4096, 0, 254, 327, 27822879, 0 Files: 72, 0, 387, 673, 1032832831, 0 TURNSTILE: 52, 0, 601, 107, 601, 0 PROC: 536, 0, 171, 403, 17250656, 0 THREAD: 384, 0, 574, 26, 574, 0 KSEGRP: 88, 0, 574, 66, 574, 0 UPCALL: 44, 0, 0, 0, 0, 0 SLEEPQUEUE: 32, 0, 601, 190, 601, 0 VMSPACE: 296, 0, 122, 398, 17250627, 0 mbuf_packet: 256, 0, 416, 378, 786102352, 0 mbuf: 256, 0, 13, 573, 1309423807, 0 mbuf_cluster: 2048, 101376, 794, 0, 34148467, 0 mbuf_jumbo_pagesize: 4096, 0, 0, 0, 0, 0 mbuf_jumbo_9k: 9216, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 0, 0, 0, 0, 0 ACL UMA zone: 388, 0, 0, 0, 0, 0 g_bio: 132, 0, 15, 1116, 313315284, 0 ata_request: 204, 0, 4, 509, 87137629, 0 ata_composite: 196, 0, 0, 0, 0, 0 VNODE: 272, 0, 57895, 39251, 622952749, 0 VNODEPOLL: 76, 0, 0, 100, 2, 0 S VFS Cache: 68, 0, 60142, 7954, 687673790, 0 L VFS Cache: 291, 0, 176, 435, 3447964, 0 NAMEI: 1024, 0, 0, 296, 5778920668, 0 DIRHASH: 1024, 0, 7832, 292, 6940321, 0 NFSMOUNT: 480, 0, 0, 0, 0, 0 NFSNODE: 464, 0, 0, 0, 0, 0 PIPE: 408, 0, 46, 386, 27243638, 0 KNOTE: 68, 0, 5, 387, 4770118, 0 socket: 356, 36003, 352, 495, 67634777, 0 unpcb: 144, 36018, 69, 390, 41065089, 0 ipq: 32, 3277, 0, 339, 1193599, 0 udpcb: 180, 36014, 6, 148, 1300549, 0 inpcb: 180, 36014, 31128, 112, 25269006, 0 tcpcb: 464, 36000, 277, 539, 25269006, 0 tcptw: 48, 41028, 30851, 193, 16513268, 0 syncache: 100, 15366, 4, 776, 24698106, 0 hostcache: 76, 15400, 4235, 815, 1070878, 0 tcpreass: 20, 6422, 0, 507, 148191, 0 sackhole: 20, 0, 0, 507, 7100340, 0 ripcb: 180, 36014, 0, 66, 132, 0 rtentry: 132, 0, 15, 72, 15, 0 IPFW dynamic rule: 108, 0, 353, 691, 31765808, 0 SWAPMETA: 276, 121576, 23, 201, 1209, 0 Mountpoints: 664, 0, 16, 8, 18, 0 FFS inode: 132, 0, 57835, 6835, 622883321, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 57835, 5315, 622883321, 0 md1: 512, 0, 622, 130, 2516, 0 interrupt total rate irq14: ata0 58 0 irq16: em0 uhci3 647842845 301 irq19: uhci1+ 83069504 38 cpu0: timer 6646431 3 cpu1: timer 6646192 3 Total 744205030 346 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad4 ad5 in sy cs us sy id 0 5 0 307812 116536 282 1 0 0 1738 122 0 0 1552 638 1610 18 6 76 ====================================== iostat ====================================== netstat 426/954/1380 mbufs in use (current/cache/total) 413/381/794/101376 mbuf clusters in use (current/cache/total/max) 413/381 mbuf+clusters out of packet secondary zone in use (current/cache) 0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/0 9k jumbo clusters in use (current/cache/total/max) 0/0/0/0 16k jumbo clusters in use (current/cache/total/max) 932K/1000K/1933K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/225/8704 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 416785 requests for I/O initiated by sendfile 20781 calls to protocol drain routines Current listen queue sizes (qlen/incqlen/maxqlen) Proto Listen Local Address tcp4 0/0/32 *.21 tcp4 0/0/80 *.5432 tcp6 0/0/80 *.5432 tcp4 0/0/128 127.0.0.1.953 tcp4 0/0/3 77.234.201.246.53 tcp4 0/0/1024 77.234.201.244.80 tcp4 0/0/511 127.0.0.1.80 tcp4 0/14/511 77.234.201.242.80 tcp4 0/0/511 77.234.201.242.8100 tcp4 0/0/511 77.234.201.242.8101 tcp4 0/0/511 77.234.201.242.8102 tcp4 0/0/511 77.234.201.242.8103 tcp4 0/0/511 77.234.201.242.8104 tcp4 0/0/128 *.22 tcp6 0/0/128 *.22 tcp4 0/0/1024 127.0.0.1.11211 tcp4 0/0/128 *.199 tcp: 559731328 packets sent 418477876 data packets (447278633 bytes) 14413229 data packets (2190080930 bytes) retransmitted 230752 data packets unnecessarily retransmitted 4393 resends initiated by MTU discovery 109339634 ack-only packets (0 delayed) 0 URG only packets 58012 window probe packets 186764 window update packets 17256447 control packets 450050990 packets received 322941300 acks (for 1707927532 bytes) 37323777 duplicate acks 1329155 acks for unsent data 77733792 packets (2941973567 bytes) received in-sequence 1455527 completely duplicate packets (167364973 bytes) 0 old duplicate packets 13389 packets with some dup. data (4985598 bytes duped) 147825 out-of-order packets (150885747 bytes) 1722 packets (131558 bytes) of data after window 61 window probes 5158555 window update packets 302991 packets received after close 13139 discarded for bad checksums 59 discarded for bad header offset fields 0 discarded because packet too short 671119 connection requests 24561927 connection accepts 828433 bad connection attempts 0 listen queue overflows 609 ignored RSTs in the windows 25177806 connections established (including accepts) 25237878 connections closed (including 2353819 drops) 10841867 connections updated cached RTT on close 11035278 connections updated cached RTT variance on close 6977589 connections updated cached ssthresh on close 1024 embryonic connections dropped 114982435 segments updated rtt (of 123165873 attempts) 8923984 retransmit timeouts 186886 connections dropped by rexmit timeout 68645 persist timeouts 50 connections dropped by persist timeout 30233 keepalive timeouts 27288 keepalive probes sent 2944 connections dropped by keepalive 25476085 correct ACK header predictions 47696324 correct data packet header predictions 24698088 syncache entries added 533954 retransmitted 697516 dupsyn 20 dropped 24561927 completed 0 bucket overflow 0 cache overflow 70399 reset 64950 stale 0 aborted 0 badack 808 unreach 0 zone failures 24698108 cookies sent 0 cookies received 3050778 SACK recovery episodes 4272923 segment rexmits in SACK recovery episodes 1699084548 byte rexmits in SACK recovery episodes 21226481 SACK options (SACK blocks) received 122811 SACK options (SACK blocks) sent 0 SACK scoreboard overflow ====================================== sysctl net.inet.ip.fw.*dyn* ====================================== counters (connection/stablished/open sockets/open files) 31210 187436 2403688 42 252 3318 ------------------------------------------------------ /usr/local/alertmon/store_log.sh: last pid: 85240; load averages: 0.85, 0.66, 0.62 up 24+21:27:19 16:40:03 124 processes: 3 running, 119 sleeping, 2 zombie Mem: 224M Active, 1386M Inact, 278M Wired, 84M Cache, 169M Buf, 29M Free Swap: 2048M Total, 256K Used, 2048M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 85160 root 1 -8 0 12936K 10596K piperd 0 0:01 38.44% perl5.8.8 85194 opennet 1 8 0 1728K 1084K wait 1 0:00 1.00% sh 8882 web 1 4 0 4444K 3760K sbwait 0 0:31 0.10% httpd 85104 web 1 4 0 3148K 2448K sbwait 1 0:00 0.09% httpd 6149 web 1 20 0 4680K 4000K lockf 1 0:34 0.05% httpd 8876 web 1 4 0 4664K 3980K sbwait 1 0:32 0.05% httpd 14054 web 1 96 0 4600K 3896K select 1 0:30 0.05% httpd 1398 postgres 1 96 0 30620K 27096K select 0 3:44 0.00% postgres 1041 root 1 96 0 2972K 2268K select 0 3:40 0.00% httpd 1096 bind 1 96 0 7688K 6876K select 1 3:09 0.00% named 1033 postgres 1 96 0 30460K 2424K select 0 1:43 0.00% postgres 79121 opennet 1 4 0 10296K 4568K accept 0 1:32 0.00% perl5.8.8 782 root 1 96 0 6344K 4180K select 1 1:08 0.00% snmpd 974 pop 1 4 0 3936K 3268K kqread 0 1:02 0.00% memcached 40382 web 1 96 0 4788K 4076K select 0 0:58 0.00% httpd 79115 opennet 1 4 0 6092K 5260K accept 0 0:50 0.00% perl5.8.8 66909 web 1 20 0 4468K 3800K lockf 1 0:50 0.00% httpd 64679 web 1 4 0 4588K 3896K sbwait 1 0:49 0.00% httpd ====================================== net.inet.tcp.syncache ====================================== vmstat ITEM SIZE LIMIT USED FREE REQUESTS FAILURES UMA Kegs: 140, 0, 71, 1, 71, 0 UMA Zones: 480, 0, 71, 1, 71, 0 UMA Slabs: 64, 0, 3506, 1155, 1089391, 0 UMA RCntSlabs: 104, 0, 397, 269, 721534, 0 UMA Hash: 128, 0, 3, 27, 7, 0 16 Bucket: 76, 0, 24, 26, 79, 0 32 Bucket: 140, 0, 32, 52, 124, 0 64 Bucket: 268, 0, 69, 43, 307, 92 128 Bucket: 524, 0, 2548, 98, 262809, 278077 VM OBJECT: 132, 0, 35747, 58039, 744084316, 0 MAP: 192, 0, 7, 13, 7, 0 KMAP ENTRY: 68, 83944, 887, 3705, 7619897, 0 MAP ENTRY: 68, 0, 4413, 3259, 1673772314, 0 PV ENTRY: 24, 2154990, 95447, 175558, 26110257951, 0 DP fakepg: 72, 0, 0, 159, 187, 0 mt_zone: 1024, 0, 202, 130, 202, 0 16: 16, 0, 3212, 1051, 88866606, 0 32: 32, 0, 1818, 1459, 22802156, 0 64: 64, 0, 4755, 850, 291397779, 0 128: 128, 0, 3034, 1526, 657338347, 0 256: 256, 0, 1230, 1290, 68914186, 0 512: 512, 0, 322, 350, 3033793, 0 1024: 1024, 0, 83, 373, 15427346, 0 2048: 2048, 0, 399, 397, 5521804, 0 4096: 4096, 0, 256, 325, 27822966, 0 Files: 72, 0, 399, 661, 1032834734, 0 TURNSTILE: 52, 0, 601, 107, 601, 0 PROC: 536, 0, 173, 401, 17250721, 0 THREAD: 384, 0, 574, 26, 574, 0 KSEGRP: 88, 0, 574, 66, 574, 0 UPCALL: 44, 0, 0, 0, 0, 0 SLEEPQUEUE: 32, 0, 601, 190, 601, 0 VMSPACE: 296, 0, 123, 397, 17250692, 0 mbuf_packet: 256, 0, 441, 353, 786102847, 0 mbuf: 256, 0, 17, 569, 1309424867, 0 mbuf_cluster: 2048, 101376, 794, 0, 34148467, 0 mbuf_jumbo_pagesize: 4096, 0, 0, 0, 0, 0 mbuf_jumbo_9k: 9216, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 0, 0, 0, 0, 0 ACL UMA zone: 388, 0, 0, 0, 0, 0 g_bio: 132, 0, 0, 1131, 313315486, 0 ata_request: 204, 0, 0, 513, 87137678, 0 ata_composite: 196, 0, 0, 0, 0, 0 VNODE: 272, 0, 57831, 39315, 622952811, 0 VNODEPOLL: 76, 0, 0, 100, 2, 0 S VFS Cache: 68, 0, 60056, 8040, 687674004, 0 L VFS Cache: 291, 0, 176, 435, 3447964, 0 NAMEI: 1024, 0, 0, 296, 5778930862, 0 DIRHASH: 1024, 0, 7832, 292, 6940321, 0 NFSMOUNT: 480, 0, 0, 0, 0, 0 NFSNODE: 464, 0, 0, 0, 0, 0 PIPE: 408, 0, 44, 388, 27243674, 0 KNOTE: 68, 0, 5, 387, 4770119, 0 socket: 356, 36003, 372, 475, 67634862, 0 unpcb: 144, 36018, 69, 390, 41065111, 0 ipq: 32, 3277, 0, 339, 1193599, 0 udpcb: 180, 36014, 6, 148, 1300567, 0 inpcb: 180, 36014, 31159, 81, 25269051, 0 tcpcb: 464, 36000, 297, 519, 25269051, 0 tcptw: 48, 41028, 30862, 182, 16513279, 0 syncache: 100, 15366, 2, 778, 24698140, 0 hostcache: 76, 15400, 4236, 814, 1070879, 0 tcpreass: 20, 6422, 0, 507, 148191, 0 sackhole: 20, 0, 0, 507, 7100340, 0 ripcb: 180, 36014, 0, 66, 132, 0 rtentry: 132, 0, 15, 72, 15, 0 IPFW dynamic rule: 108, 0, 387, 657, 31765848, 0 SWAPMETA: 276, 121576, 23, 201, 1209, 0 Mountpoints: 664, 0, 16, 8, 18, 0 FFS inode: 132, 0, 57770, 6900, 622883382, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 57770, 5380, 622883382, 0 md1: 512, 0, 622, 130, 2516, 0 interrupt total rate irq14: ata0 58 0 irq16: em0 uhci3 647843217 301 irq19: uhci1+ 83069554 38 cpu0: timer 6649029 3 cpu1: timer 6648790 3 Total 744210648 346 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad4 ad5 in sy cs us sy id 0 3 1 308848 114984 282 1 0 0 1738 122 0 0 1552 638 1610 18 6 76 ====================================== iostat ====================================== netstat 479/901/1380 mbufs in use (current/cache/total) 463/331/794/101376 mbuf clusters in use (current/cache/total/max) 463/331 mbuf+clusters out of packet secondary zone in use (current/cache) 0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/0 9k jumbo clusters in use (current/cache/total/max) 0/0/0/0 16k jumbo clusters in use (current/cache/total/max) 1045K/887K/1933K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/225/8704 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 416785 requests for I/O initiated by sendfile 20781 calls to protocol drain routines Current listen queue sizes (qlen/incqlen/maxqlen) Proto Listen Local Address tcp4 0/0/32 *.21 tcp4 0/0/80 *.5432 tcp6 0/0/80 *.5432 tcp4 0/0/128 127.0.0.1.953 tcp4 0/0/3 77.234.201.246.53 tcp4 0/0/1024 77.234.201.244.80 tcp4 0/0/511 127.0.0.1.80 tcp4 0/22/511 77.234.201.242.80 tcp4 0/0/511 77.234.201.242.8100 tcp4 0/0/511 77.234.201.242.8101 tcp4 0/0/511 77.234.201.242.8102 tcp4 0/0/511 77.234.201.242.8103 tcp4 0/0/511 77.234.201.242.8104 tcp4 0/0/128 *.22 tcp6 0/0/128 *.22 tcp4 0/0/1024 127.0.0.1.11211 tcp4 0/0/128 *.199 tcp: 559731898 packets sent 418478196 data packets (447651617 bytes) 14413230 data packets (2190082382 bytes) retransmitted 230752 data packets unnecessarily retransmitted 4393 resends initiated by MTU discovery 109339845 ack-only packets (0 delayed) 0 URG only packets 58012 window probe packets 186764 window update packets 17256485 control packets 450051530 packets received 322941628 acks (for 1708262598 bytes) 37323811 duplicate acks 1329155 acks for unsent data 77733953 packets (2942077788 bytes) received in-sequence 1455527 completely duplicate packets (167364973 bytes) 0 old duplicate packets 13389 packets with some dup. data (4985598 bytes duped) 147825 out-of-order packets (150885747 bytes) 1722 packets (131558 bytes) of data after window 61 window probes 5158555 window update packets 302991 packets received after close 13139 discarded for bad checksums 59 discarded for bad header offset fields 0 discarded because packet too short 671128 connection requests 24561962 connection accepts 828439 bad connection attempts 0 listen queue overflows 609 ignored RSTs in the windows 25177850 connections established (including accepts) 25237894 connections closed (including 2353820 drops) 10841887 connections updated cached RTT on close 11035299 connections updated cached RTT variance on close 6977595 connections updated cached ssthresh on close 1024 embryonic connections dropped 114982619 segments updated rtt (of 123166072 attempts) 8924001 retransmit timeouts 186887 connections dropped by rexmit timeout 68645 persist timeouts 50 connections dropped by persist timeout 30233 keepalive timeouts 27288 keepalive probes sent 2944 connections dropped by keepalive 25476087 correct ACK header predictions 47696448 correct data packet header predictions 24698120 syncache entries added 533954 retransmitted 697516 dupsyn 20 dropped 24561962 completed 0 bucket overflow 0 cache overflow 70399 reset 64950 stale 0 aborted 0 badack 808 unreach 0 zone failures 24698140 cookies sent 0 cookies received 3050778 SACK recovery episodes 4272923 segment rexmits in SACK recovery episodes 1699084548 byte rexmits in SACK recovery episodes 21226481 SACK options (SACK blocks) received 122811 SACK options (SACK blocks) sent 0 SACK scoreboard overflow ====================================== sysctl net.inet.ip.fw.*dyn* ====================================== counters (connection/stablished/open sockets/open files) 31237 187598 2405788 50 300 3950