Internal support hints

Important note:

Following articles are designed to be used by internal support team. You can find hints usauseful but please use it at your own risk (use on testing servers first)

Testing configure options

If you need to test configure options without need to modify the config file or database settings, you can use json_config option:

--json_config='[{"id_sensor":"11"},{"interface":"lo"},{"natalias","8.8.8.8 10.10.100.50;9.9.9.9 192.168.88.33"}]'

(The otpions defined using json_config command line option will overrides the configfile and database settings)

Examples

This will override options rtpip_find_endpoints,rrd,natalias no matter what is set in config file.

./voipmonitor --config-file=config/voipmonitor.conf -p XobmuJ -b voipmonitor_ipv6 -k -v1,pcap_stat_period=5,hash_rtp,rtp_streams -rpba:/totest/test.pcap --json_config='[{"rtpip_find_endpoints","yes"},{"rrd","no"},{"natalias","8.8.8.8 10.10.100.50;9.9.9.9 192.168.88.33"}]'

Testing regexp / country rules / billing

Regexp test

/usr/local/sbin/voipmonitor --config-file=/etc/voipmonitor-test.conf -k -v1 --test-regexp '^800880080[1-9][0-9][*]|80088008012345|' 

Country assignment test

/usr/local/sbin/voipmonitor --config-file=/etc/voipmonitor-test.conf -k -v1 --find-country-for-ip=102.140.68.113
/usr/local/sbin/voipmonitor --config-file=/etc/voipmonitor-test.conf -k -v1 --find-country-for-number 0034*12345

• gui version (obsolete)

root@vmweb:GUI_ROOT# php php/run.php testFindCountryForNumber -n 4909699999
number:           4909699999
country:          DE
is_international: Y

pricing / billing test

voipmonitor --config-file=config/voipmonitor.conf -k -v1 --test-billing test_data/billing

Example of test_data/billing file

Columns order
 - calldate 
 - connect_duration
 - caller
 - called
 - sipcallerip
 - sipcalledip
 - expected price operator - (voluntary option)
 - expected price customer - (voluntary option) 

2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151,800,8
2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151,800,8.1
2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151

License Hints

Write state to db (count of cc channels for license purposse)

php /var/www/html/php/run.php saveCallStatistics

keep the license updated without processing alerts reports

(put into /etc/crontab)

04 04   * * *   root php run.php runUpdateLicenseKey

License Details

detail on channels count usage

php /var/www/html/php/run.php checkLicense -v

Less details:

php /var/www/html/php/run.php checkLicense -v|grep max_peak -A3 -B1

license file

cat /var/www/html/key.php

Get current license token from db

mysql> select content from voipmonitor.system where type='license token';

SCRIPTs for better run control

Alerts and Report

When alerts taking longer time to process then interval is schedulled it can cause out of memory or high load to CPU after some time because same alert is processed multipletimes for various time ranges. Following script allows manual run of disabled alert in GUI via crontab - its benefit is that it will run multiple alerts at same time unlike normal run where is processed one alert by one.

Follow this link script-for-multi-Instances-alerts-processing (deprecated)

The GUI now supports parralel run of the alerts/report - enable it in system settings->Advanced->Number of parallel tasks

Batch download for more then 1000CDRs

When you need to download lot of audio files - you use GUI->menu->cdr batch download,but it allows only 1000calls to process with one GUI's query.

Following script allows you to create audio from longer set of CDRs, also it process in multiple instances to get audio faster (beware of HDD IO/CPU) script-for-big-batchdownloads

RAM (alerts / dashboards cache)

When there is lot of CDRs then alerts or caching of dashboard data can take lot of RAM, folloowing script will control if some php thread will not consume over limit, when it does, its oom_adj_score is set to be killed first (prevent killing of voipmonitor sniffer od db when running on same server)

Follow link script-for-GUI-cron-control-oom

Backup the settings from all probes and GUI

The script gets voipmonitor.conf from all probes and do GUI backup config tables from db. Requires ssh access without password, error redirect to /dev/null, run it using cron

Follow link script-for-probes-backup

charts binary crashes debug steps

When apachelog or dmesg reports crashes of vm's charts binary like

Aug 20 10:32:30 C17VOIPMONDB02 kernel: charts-x86_64[7321]: segfault at 10 ip 0000000000412553 sp 00007ffff2638430 error 4 in charts-x86_64[400000+6c6000]

create debug folder for charts

Create vm_charts_parameters in /tmp

after GUI will create some charts there files with arguments will be created.

test Arguments passed to charts binary

/var/www/html/bin/charts-x86_64 -i /tmp/vm_charts_parameters/files

voipmonitor sniffer memory leaks detect and debug

When voipmonitor uses more and more RAM (RSS/VSZ stat in syslog) without processing more CALLs (regs/messages) following steps can tell you where the RAM ends (it can be leak or some thread overload)

compile je_malloc

Download jemalloc from https://github.com/jemalloc/jemalloc

./autogen.sh
#requires development tools Deb/ubu apt install build-essential, centos/rh yum groupinstall 'Development Tools'
./configure --enable-prof
make
make install

enable jemalloc for voipmonitor and compile it

./configure

edit config.h,to have there

#define HAVE_LIBTCMALLOC 0
#define HAVE_LIBJEMALLOC 1

edit Makefile and disable optimization and add ljemalloc to libs

substitute there -O2 with -O0 (It is capital o followed by zero)
LIBS = ${SHARED_LIBS} -ljemalloc

Finish compilation

make clean
make -j8

enable X88 compatible sources and recompile

Copy sniffer sources with compiled binary to new location and recreate binary with -X88 (This is necessary to get correct memory allocation for debuging usin jeprof)

cp -a /usr/src/voipmonitor-git /usr/src/sniffer-git-jemalloc-X88
LD_PRELOAD=/usr/local/lib/libjemalloc.so ./voipmonitor -k -v1 -c -X88
#It will list of source files modification,then re-make
make clean
make

More detailed instructions on debuging using X88 X88-memoryConsumptionDebug

Modify init file or run sniffer manually with extra env arguments and parameters

edit /etc/init.d/voipmonitor

ARGS="-v 1,force_log_sqlq,memory_stat,memory_stat_ignore_limit=100000"
#and in start section (2places) of a script
LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' /usr/src/sniffer-git-jemalloc-X88/voipmonitor --config-file $CONFIGFILE --pid-file $PIDFILE $ARGS

or run it from konsole / when sniffer service stopped

LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' /usr/src/sniffer-git-jemalloc-X88/voipmonitor --config-file=/etc/voipmonitor.conf -v 1,force_log_sqlq,memory_stat_ex,memory_stat_ignore_limit=100000,heapsafe

Check results

in some interval you should check the results of memory usage with

je_prof.pdf

yum install graphviz ghostscript
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029 > /tmp/je_prof; jeprof --show_bytes --pdf /usr/src/sniffer-git-jemalloc-X88/voipmonitor /tmp/je_prof > je_prof.pdf

sniffer_memory.txt

echo 'jemalloc_stat_full' | nc 127.0.0.1 5029

script for periodical collecting

#!/bin/bash
DATE=`date '+%Y%m%d_%H:%M:%S'`
LOGDIR=/root/vm_debug/log/$DATE
mkdir -p $LOGDIR
pdftmpname=/tmp/je_prof
pdfname=$LOGDIR/je_prof.pdf
txtname=$LOGDIR/mem_stat.txt
senname=$LOGDIR/senlog.txt
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029 > $pdftmpname 2>&1
/usr/local/bin/jeprof --show_bytes --pdf /usr/src/sniffer-git-jemalloc-X88/voipmonitor $pdftmpname > $pdfname 2>&1
echo memory_stat | nc 127.0.0.1 5029 > $txtname 2>&1
echo sniffer_stats | nc 127.0.0.1 5029 > $senname 2>&1
rm $pdftmpname

voipmonitor sniffer compile notes ss7 (wireshark) module

(requires ws version 3.2):

modify config.h

#define HAVE_LIBWIRESHARK 1
#define LIBWIRESHARK_VERSION 30200

modify Makefile's INCLUDE

-I/usr/include/wireshark

modify Makefile's SHARED_LIBS

-lwiretap -lwireshark -lwsutil

internal build in JIRA:/opt/debian-7-64bit

disable optimize in Makefile

change -O2 to -O0

compile

./configure && make clean && make ss7

testing customer pcaps

testing special pcaps

ipfix

Run first instance of voipmonitor that will act as receiver for packets and will push CDRs to db in the settings enable ipfix options:

ipfix = yes
ipfix_bind_ip = 0.0.0.0
ipfix_bind_port = 12345

Then run scecond instance that will act as source for replay the ipfix encapsulated pcap

./voipmonitor --config-file=config/voipmonitor.conf -k -v1 --ipfix-client-emulation='/tmp/TCPdump_ipfix_stream.pcap;1.2.3.4;8.8.8.8;127.0.0.1;12345'

Params:

pcap file
2 IP address - 1.ip fix client,2.ip fix server
last IP : port telling to where to send the ipfix stream (IP:port of ipfix options set for the 1.st instance)

testing via basic upload aka GUI upload

voipmonitor --config-file=/etc/voipmonitor-customer.conf -r /tmp/file.pcap

testing via packetbuffer upload

time shift to now, increased speed of replaying

voipmonitor --config-file=/etc/voipmonitor-customer.conf -rpbsa9: /tmp/file.pcap

no time shift via pb

voipmonitor --config-file=/etc/voipmonitor-customer.conf -rpb: /tmp/file.pcap

tcpreplay to running service

you need to enable additional argument when starting the service in /etc/init.d/voipmonitor

ARGS="-v 1,dump_call_flags,tcpreplay"

then use tcpreplay

tcpreplay -i eth0 /tmp/l/x.pcap

when error appears like

sending out eth0
processing file: /tmp/l/x.pcap
Warning in tcpreplay.c:replay_file() line 227:
/tmp/l/x.pcap DLT (LINUX_SLL) does not match that of the outbound interface: eth0 (EN10MB)

probably captured on any interface and layer2 missing, here follow stpes on how to add it:

adding layer2 to pcap

after capture was done on any interface and you need to replay it to hw interface you need to add layer2 (at least in 3.4.3 v.tcpreplay)

tcprewrite --dlt=enet -i /tmp/l/x.pcap -o /tmp/l/x_enet-encaps.pcap
tcprewrite --enet-dmac=00:55:22:AF:C6:37 --enet-smac=00:44:66:FC:29:AF --infile=/tmp/l/x_enet-encaps.pcap --outfile=/tmp/l/x_enet-encaps-l2.pcap

rewrite IPv4 address in whole pcap

when you need to test alert based on IP adresses (also country continent alert based on IPs) you can find usefull to change IP address in pcap and use it multiple times with various addresses:

tcprewrite --pnat=8.7.6.5/32:1.2.3.4/32 -i ./7efd0e8b87a54f48ac04b7a49fdf5669.pcap -o ./7e-rew.pcap

Rename mysql database

It's not possible to do it via direct command. You need to use RENAME TABLE command. With this SELECT you generate the commands for the rename of the all tables in the database. (Replace OLD_DB_NAME and NEW_DB_NAME)

SELECT CONCAT('RENAME TABLE ',table_schema,'.',table_name,
    ' TO ','NEW_DB_NAME.',table_name,';')
FROM information_schema.TABLES
WHERE table_schema LIKE 'OLD_DB_NAME';

The output:

RENAME TABLE OLD_DB_NAME.table1 TO NEW_DB_NAME.table1;
RENAME TABLE OLD_DB_NAME.table2 TO NEW_DB_NAME.table2;
RENAME TABLE OLD_DB_NAME.tableX TO NEW_DB_NAME.tableX;

Run these commands a you are ready.

Attention! Sometimes you can have a problem with moving of the foreign keys.

disabling TCP sack

On buggy kernels (all before 2019/03) following prevents SACK issue sideback is that on lossy lines (packet lost or big delays) the tcp traffic will increase)

echo "0" > /proc/sys/net/ipv4/tcp_sack

MacOS high sierra in virtual box (on windows)

Download Image

from - https://drive.google.com/file/d/0B0EsZGPuNAPvVDlicGt1NUx0Y0E/view

Prepare ISO for install

Create High Sierra DMG Disk Image

hdiutil create -o /tmp/Highsierra -size 7900m -volname highsierra -layout SPUD -fs HFS+J

Mount DMG Image to your macOS

hdiutil attach /tmp/Highsierra.dmg -noverify -mountpoint /Volumes/Highsierra

Create a High Sierra ISO image

sudo /Applications/Install\ macOS\ High\ Sierra.app/Contents/Resources/createinstallmedia --volume /Volumes/Highsierra

Unmount Disk Image

hdiutil detach /volumes/Install\ macOS\ High\ Sierra

Convert DMG to ISO

hdiutil convert /tmp/Highsierra.dmg -format UDTO -o ~/Desktop/Highsierra

Rename and Move ISO Image to Desktop

mv ~/Desktop/Highsierra.cdr ~/Desktop/Highsierra.iso

Create new virtual Image

during preconfigure of new image set:

memmory 3-6GB
disk size 100GB
then edit it's settings and additionally modify:
system -> Motherboard -> Chipset: PIIX3
system -> Processor -> count (2), Enable PAE/NX
system -> Storage -> Attributes-> Optical drive->Sata port 1, for the newly created iso
display -> Video memory -> 128MB
display -> Graphics Controler -> VMSVGA

Add the VirtualBox macOS Code to CMD

cd "C:\Program Files\Oracle\VirtualBox\"
VBoxManage.exe modifyvm "Your VM Name" --cpuidset 00000001 000106e5 00100800 0098e3fd bfebfbff
VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/efi/0/Config/DmiSystemProduct" "iMac11,3"
VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/efi/0/Config/DmiSystemVersion" "1.0"
VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/efi/0/Config/DmiBoardProduct" "Iloveapple"
VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/smc/0/Config/DeviceKey" "ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc"
VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/smc/0/Config/GetKeyFromRealSMC" 1

Install macOS on virtulBox

1. Open VirtualBox and turn on your High Sierra, then Open "Disk Utility".
2. Format VirtualBox Virtual Disk with "Mac OS Extended". (GUID:Partition map)
3. Now, you can install macOS High Sierra on new virtual disk with your ISO Image.
4. After your installation is completed, turn off your macOS High Sierra. Open VirtualBox, and remove High Sierra.iso file.
5. Turn on your Virtual Machine. At UEFI Shell screen, boot with this commands:

>"fs1:\macOS Install Data\Locked Files\Boot Files\boot.efi"

Measurement

Memory speed

sysbench memory --threads=2 run

• should be over 5000 MiB/sec

SSL and keylogger

How to test it in voipmonitor

./voipmonitor --config-file /etc/test.conf -k -v1,pcap_stat_period=1,_ssl,_tls,_ssldecode,ssl_sessionkey,dump_sip_line -rpba:/tmp/keys.pcap@@/tmp/call.pcap --json_config='[{"sipport":"8089"},{"ssl":"yes"},{"ssl_ipport":"192.168.0.1:8089"},{"ssl_sessionkey_udp":"yes"},{"ssl_sessionkey_udp_port":"1234"},{"ssl_sessionkey_udp_maxwait_ms":10000},{"interface":"lo"},{"spooldir":"/var/spool/voipmonitorTLS"}]'

How to test it in wireshark/tshark

Get the keys from keyloger's dump

ngrep -I /tmp/keycapture.pcap |grep '^ ' |cut -d ' ' -f3- > /tmp/keys.txt

Decrypt with tshark

tshark -r '/tmp/tlscall.pcap' -o 'tls.keylog_file: /tmp/keys.txt' -o 'tls.debug_file: /dev/stdout'

Decrypt in wireshark

In edit->preferences->protocol->SSL->pre-master secret log filename and choose the keys.txt then find the client Hello HS packet, right click on it and follow stream - TLS

Spooldir testing

When there are problems with cleaning the spooldir or spooldir size not respects the settings you can use following command to test

voipmonitor --config-file /etc/voipmonitor.conf -k -v1 --test-cleanspool-load all

Manager API via server port

get list of the connected clients

echo '{"type_connection":"manager_command","command":"active"}'|nc 127.0.0.1 60024

send terminating manager command to a probe

echo '{"type_connection":"gui_command","sensor_id":1011,"command":"terminating"}'|nc 127.0.0.1 60024

HA proxy

sample HAproxy configuration

High CPS config for voipmonitor with hugepages

voipmonitor and hugepages

IOPS tests

--rw can be randread,randwrite,randrw

 fio --name=rootTest --ioengine=libaio --rw=randwrite --bs=8k --numjobs=16 --size=1G --runtime=600s --time_based --ramp_time=15s --norandommap --filename=/root/testW.dat --output=/root/fio_db_write.txt

Rocky 9

Rocky 9

Manager api of a sniffer

encryption_in_manager_api

Manager api of a sniffer for customers

encryption_in_manager_api_customer

TLS decode by Ribbon SBC 7k

Ribbon7k monitoring profiles

Email templates

common requests in supports common body of support emails

IPACCOUNT

How to enable ipaccount feature in the voipmonitor

DTLS

DTLS SRTP decryption using voipmonitor

ioncube is crashing for apache/https

wrong privileges

If following command is crashig under www-data used but not under root user:

su -s '/bin/bash' -c "php -r 'echo extension_loaded(\"ionCube Loader\")?\"yes\":\"no\";'" www-data

it can be caused by wrong mod for the ioncube.ini file or ioncube's module, you can test with strace (search for ioncube or denied)

strace -f  su -s '/bin/bash' -c "php -r 'echo extension_loaded(\"ionCube Loader\")?\"yes\":\"no\";'" www-data 2> /tmp/text.txt

crashes in libc

It was experieced in ubuntu 22.04 with php 7.4 that ioncube causes crash in ldap.so (displays following error on the login page)

Failed check Ioncube.com PHP Loader for php cli : PHP Warning: PHP Startup: Unable to load dynamic library 'ldap.so' (tried: /usr/lib/php/20190902/ldap.so (/usr/lib/php/20190902/ldap.so: undefined symbol: RETURN_THROWS), /usr/lib/php/20190902/ldap.so.so (/usr/lib/php/20190902/ldap.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0

The reason is new ldap library broken in ubuntu 22.04

https://ubuntu.com/security/CVE-2024-8932

https://askubuntu.com/questions/1535405/php7-4-ldap-broken-on-focal-after-13-dec-2024

Backup current ldap libs

cd /usr/lib/php/20190902
mkdir BKUP
cp -pv ldap.so* BKUP

Download and extract the package.

wget https://pl.archive.ubuntu.com/ubuntu/pool/main/p/php7.4/php7.4-ldap_7.4.3-4ubuntu2.24_amd64.deb
ar x php7.4-ldap_7.4.3-4ubuntu2.24_amd64.deb

Copy in the fixed lib from local tmp/ to prod.

cp -v ./usr/lib/php/20190902/ldap.so /usr/lib/php/20190902/ldap.so
'./usr/lib/php/20190902/ldap.so' -> '/usr/lib/php/20190902/ldap.so'

Restart apache and test.

systemctl restart apache2

thanks to Andrew from distributel.ca for this article