Internal support hints

Important note:

Following articles are designed to be used by internal support team. You can find hints usauseful but please use it at your own risk (use on testing servers first)

Testing regexp / country rules / billing

Regexp test

/usr/local/sbin/voipmonitor --config-file=/etc/voipmonitor-test.conf -k -v1 --test-regexp '^800880080[1-9][0-9][*]|80088008012345|' 

Country assignment test

/usr/local/sbin/voipmonitor --config-file=/etc/voipmonitor-test.conf -k -v1 --find-country-for-ip=102.140.68.113
/usr/local/sbin/voipmonitor --config-file=/etc/voipmonitor-test.conf -k -v1 --find-country-for-number 0034*12345

pricing / billing test

voipmonitor --config-file=config/voipmonitor.conf -k -v1 --test-billing test_data/billing

Example of test_data/billing file

Columns order
 - calldate 
 - connect_duration
 - caller
 - called
 - sipcallerip
 - sipcalledip
 - expected price operator - (voluntary option)
 - expected price customer - (voluntary option) 

2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151,800,8
2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151,800,8.1
2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151

License Hints

Write state to db (count of cc cchannels for license purposse)

php /var/www/html/php/run.php saveCallStatistics

License Details

detail on channels count usage

php /var/www/html/php/run.php checkLicense -v
php /var/www/html/php/run.php checkLicense -v 2

license file

cat /var/www/html/key.php

Get current license token from db

mysql> select content from voipmonitor.system where type='license token';

SCRIPTs for better run control

ALERTS

When alerts taking longer time to process then interval is schedulled it can cause out of memory or high load to CPU after some time because same alert is processed multipletimes for various time ranges. Following script allows manual run of disabled alert in GUI via crontab - its benefit is that it will run multiple alerts at same time unlike normal run where is processed one alert by one.

Follow this link script-for-multi-Instances-alerts-processing

Batch download for more then 1000CDRs

When you need to download lot of audio files - you use GUI->menu->cdr batch download,but it allows only 1000calls to process with one GUI's query.

Following script allows you to create audio from longer set of CDRs, also it process in multiple instances to get audio faster (beware of HDD IO/CPU) script-for-big-batchdownloads

RAM (alerts / dashboards cache)

When there is lot of CDRs then alerts or caching of dashboard data can take lot of RAM, folloowing script will control if some php thread will not consume over limit, when it does, its oom_adj_score is set to be killed first (prevent killing of voipmonitor sniffer od db when running on same server)

Follow link script-for-GUI-cron-control-oom

voipmonitor sniffer memory leaks detect and debug

When voipmonitor uses more and more RAM (RSS/VSZ stat in syslog) without processing more CALLs (regs/messages) following steps can tell you where the RAM ends (it can be leak or some thread overload)

compile je_malloc

Download jemalloc from https://github.com/jemalloc/jemalloc

./autogen.sh (requieres development tools Deb/ubu apt install build-essential, centos/rh yum groupinstall 'Development Tools')
./configure --enable-prof
make
make install

enable jemalloc for voipmonitor and compile it

./configure

edit config.h,to have there

#define HAVE_LIBTCMALLOC 0
#define HAVE_LIBJEMALLOC 1

edit Makefile and disable optimization

substitute there -O2 with -O0 (It is capital o followed by zero)

Finish compilation

make clean
make -j8

enable X88 compatible sources and recompile

Copy sniffer sources with compiled binary to new location and recreate binary with -X88 (This is necessary to get correct memory allocation for debuging usin jeprof)

cp -a /usr/src/voipmonitor-git /usr/src/sniffer-git-jemalloc-X88
./voipmonitor -k -v1 -c -X88
#It will list of source files modification,then re-make
make clean
make

Modify init file or run sniffer manually with extra env arguments and parameters

edit /etc/init.d/voipmonitor

ARGS="-v 1,force_log_sqlq,memory_stat,memory_stat_ignore_limit=100000"
#and in start section (2places) of a script
LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' /usr/src/sniffer-git-jemalloc-X88/voipmonitor --config-file $CONFIGFILE --pid-file $PIDFILE $ARGS

or run it from konsole / when sniffer service stopped

LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' /usr/src/sniffer-git-jemalloc-X88/voipmonitor --config-file=/etc/voipmonitor.conf -v 1,force_log_sqlq,memory_stat_ex,memory_stat_ignore_limit=100000,heapsafe

Check results

in some interval you should check the results of memory usage with

je_prof.pdf

yum install graphviz ghostscript
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029 > /tmp/je_prof; jeprof --show_bytes --pdf /usr/src/sniffer-git-jemalloc-X88/voipmonitor /tmp/je_prof > je_prof.pdf

sniffer_memory.txt

echo 'jemalloc_stat_full' | nc 127.0.0.1 5029

script for periodical collecting

#!/bin/bash
DATE=`date '+%d_%H:%M:%S'`
LOGDIR=/root/vm_debug/log/$DATE
mkdir -p $LOGDIR
pdftmpname=/tmp/je_prof
pdfname=$LOGDIR/je_prof.pdf
txtname=$LOGDIR/mem_stat.txt
senname=$LOGDIR/senlog.txt
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029 > $pdftmpname 2>&1
/usr/local/bin/jeprof --show_bytes --pdf /usr/src/sniffer-git-jemalloc-X88/voipmonitor $pdftmpname > $pdfname 2>&1
echo memory_stat | nc 127.0.0.1 5029 > $txtname 2>&1
echo sniffer_stats | nc 127.0.0.1 5029 > $senname 2>&1
rm $pdftmpname

testing customer pcaps

testing via basic upload aka GUI upload

voipmonitor --config-file=/etc/voipmonitor-customer.conf -r /tmp/file.pcap

testing via packetbuffer upload

time shift to now, increased speed of replaying

voipmonitor --config-file=/etc/voipmonitor-customer.conf -rpbsa9: /tmp/file.pcap

no time shift via pb

voipmonitor --config-file=/etc/voipmonitor-customer.conf -rpb: /tmp/file.pcap

tcpreplay to running service

you need to enable additional argument when starting the service in /etc/init.d/voipmonitor

ARGS="-v 1,dump_call_flags,tcpreplay"

then use tcpreplay

tcpreplay -i eth0 /tmp/l/x.pcap

when error appears like

sending out eth0
processing file: /tmp/l/x.pcap
Warning in tcpreplay.c:replay_file() line 227:
/tmp/l/x.pcap DLT (LINUX_SLL) does not match that of the outbound interface: eth0 (EN10MB)

probably captured on any interface and layer2 missing, here follow stpes on how to add it:

adding layer2 to pcap

after capture was done on any interface and you need to replay it to hw interface you need to add layer2 (at least in 3.4.3 v.tcpreplay)

tcprewrite --dlt=enet -i /tmp/l/x.pcap -o /tmp/l/x_enet-encaps.pcap
tcprewrite --enet-dmac=00:55:22:AF:C6:37 --enet-smac=00:44:66:FC:29:AF --infile=/tmp/l/x_enet-encaps.pcap --outfile=/tmp/l/x_enet-encaps-l2.pcap

rewrite IPv4 address in whole pcap

when you need to test alert based on IP adresses (also country continent alert based on IPs) you can find usefull to change IP address in pcap and use it multiple times with various addresses:

tcprewrite --pnat=8.7.6.5/32:1.2.3.4/32 -i ./7efd0e8b87a54f48ac04b7a49fdf5669.pcap -o ./7e-rew.pcap

Rename mysql database

It's not possible to do it via direct command. You need to use RENAME TABLE command. With this SELECT you generate the commands for the rename of the all tables in the database. (Replace OLD_DB_NAME and NEW_DB_NAME)

SELECT CONCAT('RENAME TABLE ',table_schema,'.',table_name,
    ' TO ','NEW_DB_NAME.',table_name,';')
FROM information_schema.TABLES
WHERE table_schema LIKE 'OLD_DB_NAME';

The output:

RENAME TABLE OLD_DB_NAME.table1 TO NEW_DB_NAME.table1;
RENAME TABLE OLD_DB_NAME.table2 TO NEW_DB_NAME.table2;
RENAME TABLE OLD_DB_NAME.tableX TO NEW_DB_NAME.tableX;

Run these commands a you are ready.

Attention! Sometimes you can have a problem with moving of the foreign keys.

disabling TCP sack

On buggy kernels (all before 2019/03) following prevents SACK issue sideback is that on lossy lines (packet lost or big delays) the tcp traffic will increase)

echo "0" > /proc/sys/net/ipv4/tcp_sack