Monitoring the CPU loads of all SER and Asterisk processes of SIP/VoIP B2B servers

Emin Gabrielyan

[1] [2] [3] [4] [5]

Monitoring the CPU loads of all SER and Asterisk processes of SIP/VoIP B2B servers. 1

2010-06-21: support of Linux and FreeBSD.. 1

2010-06-18: support of a well formatted password file. 3

2010-06-21: support of Linux and FreeBSD platforms

Currently the version a18.sh.txt is running. It supports now the CPU TOP command of Linux (in addition to FreeBSD), permitting us to monitor the load of new asterisk servers.

[zip]

The Bash shell and Perl scripts are zipped and available for download [zip] (previous versions of the shell script in the ZIP file are provided for a reference and can be removed).

openser3:~/folders/080212-portasip-ser-cpu#

openser3:~/folders/080212-portasip-ser-cpu# crontab -l

# m h dom mon dow command

#3 */3 * * * /usr/local/bin/serrestart

*/30 * * * * /usr/local/bin/callhistogram >> /var/log/callhistogram.log

*/5 * * * * /usr/local/bin/serprocmon >> /var/log/serprocmon.log

7 */3 * * * /usr/local/bin/googlerank >> /var/log/googlerank.log

0 0,12 * * * /root/folders/080212-portasip-ser-cpu/a18.sh.txt > /dev/null

27 * * * * /root/folders/090309-imap-stats/report.sh > /dev/null

51 11,20 * * * /root/folders/091017-support-numbers/report.sh support > /dev/null

51 2 * * * /root/folders/091017-support-numbers/report.sh oper > /dev/null

openser3:~/folders/080212-portasip-ser-cpu#

[1] [2] [3] [4] [5]

From the version a17.sh.txt to a18.sh.txt, the test host ‘ch1’ is removed as does not exist anymore. The two test asterisk nodes ‘astrad’ and ‘astrad3’ are added. The hostnames on the chart may appear with truncated characters in the middle if they length exceeds 5 positions. The processing of the TOP output is made compatible both for Linux and FreeBSD platforms. Now we are summing up the CPU loads of not only SER processes, but also of all processes containing ‘asterisk’ and ‘mysql’ keywords (the top consumers of the family of our ‘astrad’ servers).

openser3:~/folders/080212-portasip-ser-cpu# diff a17.sh.txt a18.sh.txt

5a6

> loop=72

21c22

< hosts="us1,ch1,fr1,fr2,fr3,fr4,fr5,fr6,fr7,dk1,astrad,astrad3"

---

> hosts="us1,fr1,fr2,fr3,fr4,fr5,fr6,fr7,dk1,astrad,astrad3"

23d23

< loop=72

39c39

< echo time,$hosts | tee $csv

---

> echo $hosts | perl -ne 's/[\r\n]//g; s/^/,/; s/$/,/; while(){s/(,[^,]{3})[^,]+([^,]{2},)/$1$2/; last unless $1} s/^,//; s/,$//; print "time,$_\n"' | tee $csv

88,89c88,89

< login=`echo $conf | perl -ne 'print "$1\@$2$3" if(/# +([^#]*)\@('$h')(\.[^# ]*) +([^# ][^#]*[^# ]) *#/)'`

< command=`echo $conf | perl -ne 'print "$4" if(/# +([^#]*)\@('$h')(\.[^# ]*) +([^# ][^#]*[^# ]) *#/)'`

---

> login=`echo $conf | perl -ne 'print "$1\@$2$3" if(/# *([^# ]*)\@('$h')(\.[^# ]*) +([^# ][^#]*[^# ]) *#/)'`

> command=`echo $conf | perl -ne 'print "$4" if(/# *([^# ]*)\@('$h')(\.[^# ]*) +([^# ][^#]*[^# ]) *#/)'`

97c97

< if(/ COMMAND$/)

---

> if(/ COMMAND *$/i)

101c101

< if($cols[$i] eq "WCPU" || $cols[$i] eq "%CPU")

---

> if($cols[$i]=~/^[W%]CPU$/)

109c109

< elsif(/ ser$/)

---

> elsif(/( ser *$|asterisk|mysql)/)

openser3:~/folders/080212-portasip-ser-cpu#

The version a17.sh.txt introduces a configuration string, containing the usernames, hostnames, domain names, and the remote shell commands of each host to be monitored. Such a configuration makes the system more flexible (we are not limited with the same domain name and user login). The value of ssh login string as well as the remote command is retrieved from the configuration string for each hostname (being currently processed in the loop). The output of the TOP command is now not restricted by the lines for SER process, but we tell ‘grep’ to show us also the processes containing ‘asterisk’ and ‘mysql’:

openser3:~/folders/080212-portasip-ser-cpu# diff a15.sh.txt a17.sh.txt

4,6d3

< user=sona

< domain=youroute.net

< hosts="us1,ch1,fr1,fr2,fr3,fr4,fr5,fr6,fr7,dk1"

8a6,22

> conf=\

> "# sona@us1.youroute.net top -b -d$topsamp -s$topdelay inf #"\

> "# sona@fr1.youroute.net top -b -d$topsamp -s$topdelay inf #"\

> "# sona@fr2.youroute.net top -b -d$topsamp -s$topdelay inf #"\

> "# sona@fr3.youroute.net top -b -d$topsamp -s$topdelay inf #"\

> "# sona@fr4.youroute.net top -b -d$topsamp -s$topdelay inf #"\

> "# sona@fr5.youroute.net top -b -d$topsamp -s$topdelay inf #"\

> "# sona@fr6.youroute.net top -b -d$topsamp -s$topdelay inf #"\

> "# sona@fr7.youroute.net top -b -d$topsamp -s$topdelay inf #"\

> "# sona@dk1.youroute.net top -b -d$topsamp -s$topdelay inf #"\

> "# root@astrad.switzernet.com top -b -d $topsamp -n $topdelay #"\

> "# root@astrad3.switzernet.com top -b -d $topsamp -n $topdelay #"\

> ""

> hosts="us1,ch1,fr1,fr2,fr3,fr4,fr5,fr6,fr7,dk1,astrad,astrad3"

12d25

< noerror=1

75,80c88,89

< perl -e '{if(rand()<='$noerror'){exit 0}{exit 1}}'

< if [ $? -ne 0 ]

< then

< h="$h-error"

< fi

< login="$user@$h.$domain"

---

> login=`echo $conf | perl -ne 'print "$1\@$2$3" if(/# +([^#]*)\@('$h')(\.[^# ]*) +([^# ][^#]*[^# ]) *#/)'`

> command=`echo $conf | perl -ne 'print "$4" if(/# +([^#]*)\@('$h')(\.[^# ]*) +([^# ][^#]*[^# ]) *#/)'`

82c91

< ssh -o ServerAliveInterval=5 -q $login "top -b -d$topsamp -s$topdelay inf | grep ' $ser\|COMMAND$$'" | perl -e '$h="'$h'";

---

92c101

< if($cols[$i] eq "WCPU")

---

> if($cols[$i] eq "WCPU" || $cols[$i] eq "%CPU")

openser3:~/folders/080212-portasip-ser-cpu#

A bug is fixed from the version a14.sh.txt to a15.sh.txt. In the new version the ftp uploading function executes its script in an isolated process.

openser3:~/folders/080212-portasip-ser-cpu# diff a14.sh.txt a15.sh.txt

47,48c47,48

< server=`eval echo \\$ftp_dst$i`

< set `egrep "^ftp +$server " $passfile`

---

> ftp_dst=`eval echo \\$ftp_dst$i`

> set `egrep "^ftp +$ftp_dst " $passfile`

53,62c53,64

< err=0

< for i in 1 2 3 4 5

< do

< ftp_dst=`eval echo \\$ftp_dst$i`

< ftp_usr=`eval echo \\$ftp_usr$i`

< ftp_cmd=`eval echo \\$ftp_cmd$i`

< lftp -e "$ftp_cmd" -u $ftp_usr $ftp_dst 2>&1

< err=$((err+$?))

< done

< return $err

---

> (

> err=0

> for i in 1 2 3 4 5

> do

> ftp_dst=`eval echo \\$ftp_dst$i`

> ftp_usr=`eval echo \\$ftp_usr$i`

> ftp_cmd=`eval echo \\$ftp_cmd$i`

> lftp -e "$ftp_cmd" -u $ftp_usr $ftp_dst 2>&1

> err=$((err+$?))

> done

> return $err

> )

openser3:~/folders/080212-portasip-ser-cpu#

2010-06-18: support of a well formatted password file

The CPU monitor script is running on 212.249.15.3:

$ ssh root@212.249.15.3

Last login: Tue Jun 15 13:10:51 2010 from 212.147.8.99

Linux openser3 2.6.18-6-686 #1 SMP Sat Dec 27 09:31:05 UTC 2008 i686

The programs included with the Debian GNU/Linux system are free software;

the exact distribution terms for each program are described in the

individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent

permitted by applicable law.

openser3:~#

openser3:~# crontab -l

# m h dom mon dow command

#3 */3 * * * /usr/local/bin/serrestart

*/30 * * * * /usr/local/bin/callhistogram >> /var/log/callhistogram.log

*/5 * * * * /usr/local/bin/serprocmon >> /var/log/serprocmon.log

7 */3 * * * /usr/local/bin/googlerank >> /var/log/googlerank.log

0 0,12 * * * /root/folders/080212-portasip-ser-cpu/a12.sh.txt > /dev/null

27 * * * * /root/folders/090309-imap-stats/report.sh > /dev/null

51 11,20 * * * /root/folders/091017-support-numbers/report.sh support > /dev/null

51 2 * * * /root/folders/091017-support-numbers/report.sh oper > /dev/null

openser3:~#

The home folder of the script is /root/folders/080212-portasip-ser-cpu/

openser3:~# cd /root/folders/080212-portasip-ser-cpu/

openser3:~/folders/080212-portasip-ser-cpu#

openser3:~/folders/080212-portasip-ser-cpu# ls

a10.sh.txt a12.sh.txt a6.sh.txt a8.sh.txt b7.pl.txt

a11.sh.txt a5.sh.txt a7.sh.txt a9.sh.txt cpu

openser3:~/folders/080212-portasip-ser-cpu#

openser3:~/folders/080212-portasip-ser-cpu# vi a12.sh.txt

A new version a14.sh.txt is designed. It supports a well formatted file of saved passwords. The bash shell and Perl scripts are zipped and available on the web [zip], and also are embedded below (for an MS Word version of this document [doc]):

The new version of the script deals with a well structured password file, having the following format (where the asterisks are to be replaced by password values):

ftp switzernet.com switzern ******

ftp www.unappel.ch unappel ******

ftp unappel.ch unappel ******

ftp intarnet.com intarnet ******

ftp parinternet.ch parinter ******

The differences between the old and new scripts show that a function ftp_upload is added, responsible for uploading the results to 5 different mirror FTP servers:

$ diff a12.sh.txt a14.sh.txt

2c2

---

18c18

< passfile=/root/files/070930-unappel-ftplogin.txt

---

> passfile=/root/files/100617-passwords.txt

29,31c29,64

< ftp_cmd="set ftp:passive-mode no; cd htdocs/public/080210-ser-cpu; put $gif; bye"

< ftp_usr="unappel,`cat $passfile`"

< ftp_dst=www.unappel.ch

---

> ftp_dst1=www.unappel.ch

> ftp_cmd1="set ftp:passive-mode no; cd htdocs/public/080210-ser-cpu; put $gif; bye"

> ftp_dst2=unappel.ch

> ftp_cmd2="set ftp:passive-mode no; cd htdocs/public/080210-ser-cpu; put $gif; bye"

> ftp_dst3=switzernet.com

> ftp_cmd3="set ftp:passive-mode no; cd public_html/2/public/100617-ser-cpu; put $gif; bye"

> ftp_dst4=intarnet.com

> ftp_cmd4="set ftp:passive-mode no; cd public_html/2/public/100617-ser-cpu; put $gif; bye"

> ftp_dst5=parinternet.ch

> ftp_cmd5="set ftp:passive-mode no; cd parinternet.ch/2/public/100617-ser-cpu; put $gif; bye"

> for i in 1 2 3 4 5

> do

> server=`eval echo \\$ftp_dst$i`

> set `egrep "^ftp +$server " $passfile`

> eval ftp_usr$i=\"$3,$4\"

> done

> function ftp_upload {

> err=0

> for i in 1 2 3 4 5

> do

> ftp_dst=`eval echo \\$ftp_dst$i`

> ftp_usr=`eval echo \\$ftp_usr$i`

> ftp_cmd=`eval echo \\$ftp_cmd$i`

> lftp -e "$ftp_cmd" -u $ftp_usr $ftp_dst 2>&1

> err=$((err+$?))

> done

> return $err

> }

102c135

< lftp -e "$ftp_cmd" -u $ftp_usr $ftp_dst 2>&1

---

> ftp_upload

114c147

< lftp -e "$ftp_cmd" -u $ftp_usr $ftp_dst 2>&1

---

> ftp_upload

The new version (a14.sh.txt) is currently executed twice per day by the cron:

openser3:~/folders/080212-portasip-ser-cpu#

openser3:~/folders/080212-portasip-ser-cpu# crontab -l

# m h dom mon dow command

#3 */3 * * * /usr/local/bin/serrestart

*/30 * * * * /usr/local/bin/callhistogram >> /var/log/callhistogram.log

*/5 * * * * /usr/local/bin/serprocmon >> /var/log/serprocmon.log

7 */3 * * * /usr/local/bin/googlerank >> /var/log/googlerank.log

0 0,12 * * * /root/folders/080212-portasip-ser-cpu/a14.sh.txt > /dev/null

27 * * * * /root/folders/090309-imap-stats/report.sh > /dev/null

51 11,20 * * * /root/folders/091017-support-numbers/report.sh support > /dev/null

51 2 * * * /root/folders/091017-support-numbers/report.sh oper > /dev/null

openser3:~/folders/080212-portasip-ser-cpu#

The uploaded results can be viewed on the following 5 mirror pages:

http://www.unappel.ch/public/080210-ser-cpu/

http://unappel.ch/public/080210-ser-cpu/

http://switzernet.com/2/public/100617-ser-cpu/

http://parinternet.ch/2/public/100617-ser-cpu/

http://intarnet.com/2/public/100617-ser-cpu/

References

The creation of this project:

http://switzernet.com/public/080212-remote-cpu-monitor/

http://www.unappel.ch/public/080212-remote-cpu-monitor/

http://unappel.ch/public/080212-remote-cpu-monitor/

http://4z.com/public/080212-remote-cpu-monitor/

Mirrors of this page:

http://switzernet.com/2/public/100618-remote-cpu-monitor/

http://www.unappel.ch/2/public/100618-remote-cpu-monitor/

http://unappel.ch/2/public/100618-remote-cpu-monitor/

http://intarnet.com/2/public/100618-remote-cpu-monitor/

http://parinternet.ch/2/public/100618-remote-cpu-monitor/

* * *