Internal support hints
Important note:
Following articles are designed to be used by internal support team. You can find hints usauseful but please use it at your own risk (use on testing servers first)
Testing configure options
If you need to test configure options without need to modify the config file or database settings, you can use json_config option:
--json_config='[{"id_sensor":"11"},{"interface":"lo"},{"natalias","8.8.8.8 10.10.100.50;9.9.9.9 192.168.88.33"}]'
(The otpions defined using json_config command line option will overrides the configfile and database settings)
Examples
This will override options rtpip_find_endpoints,rrd,natalias no matter what is set in config file.
./voipmonitor --config-file=config/voipmonitor.conf -p XobmuJ -b voipmonitor_ipv6 -k -v1,pcap_stat_period=5,hash_rtp,rtp_streams -rpba:/totest/test.pcap --json_config='[{"rtpip_find_endpoints","yes"},{"rrd","no"},{"natalias","8.8.8.8 10.10.100.50;9.9.9.9 192.168.88.33"}]'
Testing regexp / country rules / billing
Regexp test
/usr/local/sbin/voipmonitor --config-file=/etc/voipmonitor-test.conf -k -v1 --test-regexp '^800880080[1-9][0-9][*]|80088008012345|'
Country assignment test
/usr/local/sbin/voipmonitor --config-file=/etc/voipmonitor-test.conf -k -v1 --find-country-for-ip=102.140.68.113 /usr/local/sbin/voipmonitor --config-file=/etc/voipmonitor-test.conf -k -v1 --find-country-for-number 0034*12345
- gui version (obsolete)
root@vmweb:GUI_ROOT# php php/run.php testFindCountryForNumber -n 4909699999 number: 4909699999 country: DE is_international: Y
pricing / billing test
voipmonitor --config-file=config/voipmonitor.conf -k -v1 --test-billing test_data/billing
Example of test_data/billing file
Columns order - calldate - connect_duration - caller - called - sipcallerip - sipcalledip - expected price operator - (voluntary option) - expected price customer - (voluntary option)
2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151,800,8 2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151,800,8.1 2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151
License Hints
Write state to db (count of cc channels for license purposse)
php /var/www/html/php/run.php saveCallStatistics
keep the license updated without processing alerts reports
(put into /etc/crontab)
04 04 * * * root php run.php runUpdateLicenseKey
License Details
detail on channels count usage
php /var/www/html/php/run.php checkLicense -v
Less details:
php /var/www/html/php/run.php checkLicense -v|grep max_peak -A3 -B1
license file
cat /var/www/html/key.php
Get current license token from db
mysql> select content from voipmonitor.system where type='license token';
SCRIPTs for better run control
Alerts and Report
When alerts taking longer time to process then interval is schedulled it can cause out of memory or high load to CPU after some time because same alert is processed multipletimes for various time ranges. Following script allows manual run of disabled alert in GUI via crontab - its benefit is that it will run multiple alerts at same time unlike normal run where is processed one alert by one.
Follow this link script-for-multi-Instances-alerts-processing (deprecated)
The GUI now supports parralel run of the alerts/report - enable it in system settings->Advanced->Number of parallel tasks
Batch download for more then 1000CDRs
When you need to download lot of audio files - you use GUI->menu->cdr batch download,but it allows only 1000calls to process with one GUI's query.
Following script allows you to create audio from longer set of CDRs, also it process in multiple instances to get audio faster (beware of HDD IO/CPU) script-for-big-batchdownloads
RAM (alerts / dashboards cache)
When there is lot of CDRs then alerts or caching of dashboard data can take lot of RAM, folloowing script will control if some php thread will not consume over limit, when it does, its oom_adj_score is set to be killed first (prevent killing of voipmonitor sniffer od db when running on same server)
Follow link script-for-GUI-cron-control-oom
Backup the settings from all probes and GUI
The script gets voipmonitor.conf from all probes and do GUI backup config tables from db. Requires ssh access without password, error redirect to /dev/null, run it using cron
Follow link script-for-probes-backup
charts binary crashes debug steps
When apachelog or dmesg reports crashes of vm's charts binary like
Aug 20 10:32:30 C17VOIPMONDB02 kernel: charts-x86_64[7321]: segfault at 10 ip 0000000000412553 sp 00007ffff2638430 error 4 in charts-x86_64[400000+6c6000]
create debug folder for charts
Create vm_charts_parameters in /tmp
after GUI will create some charts there files with arguments will be created.
test Arguments passed to charts binary
/var/www/html/bin/charts-x86_64 -i /tmp/vm_charts_parameters/files
voipmonitor sniffer memory leaks detect and debug
When voipmonitor uses more and more RAM (RSS/VSZ stat in syslog) without processing more CALLs (regs/messages) following steps can tell you where the RAM ends (it can be leak or some thread overload)
compile je_malloc
Download jemalloc from https://github.com/jemalloc/jemalloc
./autogen.sh #requires development tools Deb/ubu apt install build-essential, centos/rh yum groupinstall 'Development Tools' ./configure --enable-prof make make install
enable jemalloc for voipmonitor and compile it
./configure
edit config.h,to have there
#define HAVE_LIBTCMALLOC 0 #define HAVE_LIBJEMALLOC 1
edit Makefile and disable optimization and add ljemalloc to libs
substitute there -O2 with -O0 (It is capital o followed by zero)
LIBS = ${SHARED_LIBS} -ljemalloc
Finish compilation
make clean make -j8
enable X88 compatible sources and recompile
Copy sniffer sources with compiled binary to new location and recreate binary with -X88 (This is necessary to get correct memory allocation for debuging usin jeprof)
cp -a /usr/src/voipmonitor-git /usr/src/sniffer-git-jemalloc-X88 LD_PRELOAD=/usr/local/lib/libjemalloc.so ./voipmonitor -k -v1 -c -X88 #It will list of source files modification,then re-make make clean make
More detailed instructions on debuging using X88 X88-memoryConsumptionDebug
Modify init file or run sniffer manually with extra env arguments and parameters
edit /etc/init.d/voipmonitor
ARGS="-v 1,force_log_sqlq,memory_stat,memory_stat_ignore_limit=100000" #and in start section (2places) of a script LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' /usr/src/sniffer-git-jemalloc-X88/voipmonitor --config-file $CONFIGFILE --pid-file $PIDFILE $ARGS
or run it from konsole / when sniffer service stopped
LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' /usr/src/sniffer-git-jemalloc-X88/voipmonitor --config-file=/etc/voipmonitor.conf -v 1,force_log_sqlq,memory_stat_ex,memory_stat_ignore_limit=100000,heapsafe
Check results
in some interval you should check the results of memory usage with
je_prof.pdf
yum install graphviz ghostscript echo 'jemalloc_stat_full' | nc 127.0.0.1 5029 > /tmp/je_prof; jeprof --show_bytes --pdf /usr/src/sniffer-git-jemalloc-X88/voipmonitor /tmp/je_prof > je_prof.pdf
sniffer_memory.txt
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029
script for periodical collecting
#!/bin/bash DATE=`date '+%Y%m%d_%H:%M:%S'` LOGDIR=/root/vm_debug/log/$DATE mkdir -p $LOGDIR pdftmpname=/tmp/je_prof pdfname=$LOGDIR/je_prof.pdf txtname=$LOGDIR/mem_stat.txt senname=$LOGDIR/senlog.txt echo 'jemalloc_stat_full' | nc 127.0.0.1 5029 > $pdftmpname 2>&1 /usr/local/bin/jeprof --show_bytes --pdf /usr/src/sniffer-git-jemalloc-X88/voipmonitor $pdftmpname > $pdfname 2>&1 echo memory_stat | nc 127.0.0.1 5029 > $txtname 2>&1 echo sniffer_stats | nc 127.0.0.1 5029 > $senname 2>&1 rm $pdftmpname
voipmonitor sniffer compile notes ss7 (wireshark) module
(requires ws version 3.2):
modify config.h
#define HAVE_LIBWIRESHARK 1 #define LIBWIRESHARK_VERSION 30200
modify Makefile's INCLUDE
-I/usr/include/wireshark
modify Makefile's SHARED_LIBS
-lwiretap -lwireshark -lwsutil
internal build in JIRA:/opt/debian-7-64bit
disable optimize in Makefile
change -O2 to -O0
compile
./configure && make clean && make ss7
testing customer pcaps
testing special pcaps
ipfix
Run first instance of voipmonitor that will act as receiver for packets and will push CDRs to db in the settings enable ipfix options:
ipfix = yes ipfix_bind_ip = 0.0.0.0 ipfix_bind_port = 12345
Then run scecond instance that will act as source for replay the ipfix encapsulated pcap
./voipmonitor --config-file=config/voipmonitor.conf -k -v1 --ipfix-client-emulation='/tmp/TCPdump_ipfix_stream.pcap;1.2.3.4;8.8.8.8;127.0.0.1;12345'
Params:
pcap file 2 IP address - 1.ip fix client,2.ip fix server last IP : port telling to where to send the ipfix stream (IP:port of ipfix options set for the 1.st instance)
testing via basic upload aka GUI upload
voipmonitor --config-file=/etc/voipmonitor-customer.conf -r /tmp/file.pcap
testing via packetbuffer upload
time shift to now, increased speed of replaying
voipmonitor --config-file=/etc/voipmonitor-customer.conf -rpbsa9: /tmp/file.pcap
no time shift via pb
voipmonitor --config-file=/etc/voipmonitor-customer.conf -rpb: /tmp/file.pcap
tcpreplay to running service
you need to enable additional argument when starting the service in /etc/init.d/voipmonitor
ARGS="-v 1,dump_call_flags,tcpreplay"
then use tcpreplay
tcpreplay -i eth0 /tmp/l/x.pcap
when error appears like
sending out eth0 processing file: /tmp/l/x.pcap Warning in tcpreplay.c:replay_file() line 227: /tmp/l/x.pcap DLT (LINUX_SLL) does not match that of the outbound interface: eth0 (EN10MB)
probably captured on any interface and layer2 missing, here follow stpes on how to add it:
adding layer2 to pcap
after capture was done on any interface and you need to replay it to hw interface you need to add layer2 (at least in 3.4.3 v.tcpreplay)
tcprewrite --dlt=enet -i /tmp/l/x.pcap -o /tmp/l/x_enet-encaps.pcap tcprewrite --enet-dmac=00:55:22:AF:C6:37 --enet-smac=00:44:66:FC:29:AF --infile=/tmp/l/x_enet-encaps.pcap --outfile=/tmp/l/x_enet-encaps-l2.pcap
rewrite IPv4 address in whole pcap
when you need to test alert based on IP adresses (also country continent alert based on IPs) you can find usefull to change IP address in pcap and use it multiple times with various addresses:
tcprewrite --pnat=8.7.6.5/32:1.2.3.4/32 -i ./7efd0e8b87a54f48ac04b7a49fdf5669.pcap -o ./7e-rew.pcap
Rename mysql database
It's not possible to do it via direct command. You need to use RENAME TABLE command. With this SELECT you generate the commands for the rename of the all tables in the database. (Replace OLD_DB_NAME and NEW_DB_NAME)
SELECT CONCAT('RENAME TABLE ',table_schema,'.',table_name,
' TO ','NEW_DB_NAME.',table_name,';')
FROM information_schema.TABLES
WHERE table_schema LIKE 'OLD_DB_NAME';
The output:
RENAME TABLE OLD_DB_NAME.table1 TO NEW_DB_NAME.table1; RENAME TABLE OLD_DB_NAME.table2 TO NEW_DB_NAME.table2; RENAME TABLE OLD_DB_NAME.tableX TO NEW_DB_NAME.tableX;
Run these commands a you are ready.
Attention! Sometimes you can have a problem with moving of the foreign keys.
disabling TCP sack
On buggy kernels (all before 2019/03) following prevents SACK issue sideback is that on lossy lines (packet lost or big delays) the tcp traffic will increase)
echo "0" > /proc/sys/net/ipv4/tcp_sack
MacOS high sierra in virtual box (on windows)
Download Image
from - https://drive.google.com/file/d/0B0EsZGPuNAPvVDlicGt1NUx0Y0E/view
Prepare ISO for install
Create High Sierra DMG Disk Image
hdiutil create -o /tmp/Highsierra -size 7900m -volname highsierra -layout SPUD -fs HFS+J
Mount DMG Image to your macOS
hdiutil attach /tmp/Highsierra.dmg -noverify -mountpoint /Volumes/Highsierra
Create a High Sierra ISO image
sudo /Applications/Install\ macOS\ High\ Sierra.app/Contents/Resources/createinstallmedia --volume /Volumes/Highsierra
Unmount Disk Image
hdiutil detach /volumes/Install\ macOS\ High\ Sierra
Convert DMG to ISO
hdiutil convert /tmp/Highsierra.dmg -format UDTO -o ~/Desktop/Highsierra
Rename and Move ISO Image to Desktop
mv ~/Desktop/Highsierra.cdr ~/Desktop/Highsierra.iso
Create new virtual Image
during preconfigure of new image set:
memmory 3-6GB disk size 100GB then edit it's settings and additionally modify: system -> Motherboard -> Chipset: PIIX3 system -> Processor -> count (2), Enable PAE/NX system -> Storage -> Attributes-> Optical drive->Sata port 1, for the newly created iso display -> Video memory -> 128MB display -> Graphics Controler -> VMSVGA
Add the VirtualBox macOS Code to CMD
cd "C:\Program Files\Oracle\VirtualBox\" VBoxManage.exe modifyvm "Your VM Name" --cpuidset 00000001 000106e5 00100800 0098e3fd bfebfbff VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/efi/0/Config/DmiSystemProduct" "iMac11,3" VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/efi/0/Config/DmiSystemVersion" "1.0" VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/efi/0/Config/DmiBoardProduct" "Iloveapple" VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/smc/0/Config/DeviceKey" "ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/smc/0/Config/GetKeyFromRealSMC" 1
Install macOS on virtulBox
- Open VirtualBox and turn on your High Sierra, then Open "Disk Utility".
- Format VirtualBox Virtual Disk with "Mac OS Extended". (GUID:Partition map)
- Now, you can install macOS High Sierra on new virtual disk with your ISO Image.
- After your installation is completed, turn off your macOS High Sierra. Open VirtualBox, and remove High Sierra.iso file.
- Turn on your Virtual Machine. At UEFI Shell screen, boot with this commands:
>"fs1:\macOS Install Data\Locked Files\Boot Files\boot.efi"
Start the High Sierra Virtual machine and Enjoy
Measurement
Memory speed
sysbench memory --threads=2 run
- should be over 5000 MiB/sec
SSL and keylogger
How to test it in voipmonitor
./voipmonitor --config-file /etc/test.conf -k -v1,pcap_stat_period=1,_ssl,_tls,_ssldecode,ssl_sessionkey,dump_sip_line -rpba:/tmp/keys.pcap@@/tmp/call.pcap --json_config='[{"sipport":"8089"},{"ssl":"yes"},{"ssl_ipport":"192.168.0.1:8089"},{"ssl_sessionkey_udp":"yes"},{"ssl_sessionkey_udp_port":"1234"},{"ssl_sessionkey_udp_maxwait_ms":10000},{"interface":"lo"},{"spooldir":"/var/spool/voipmonitorTLS"}]'
How to test it in wireshark/tshark
Get the keys from keyloger's dump
ngrep -I /tmp/keycapture.pcap |grep '^ ' |cut -d ' ' -f3- > /tmp/keys.txt
Decrypt with tshark
tshark -r '/tmp/tlscall.pcap' -o 'tls.keylog_file: /tmp/keys.txt' -o 'tls.debug_file: /dev/stdout'
Decrypt in wireshark
In edit->preferences->protocol->SSL->pre-master secret log filename and choose the keys.txt then find the client Hello HS packet, right click on it and follow stream - TLS
Spooldir testing
When there are problems with cleaning the spooldir or spooldir size not respects the settings you can use following command to test
voipmonitor --config-file /etc/voipmonitor.conf -k -v1 --test-cleanspool-load all
Manager API via server port
get list of the connected clients
echo '{"type_connection":"manager_command","command":"active"}'|nc 127.0.0.1 60024
send terminating manager command to a probe
echo '{"type_connection":"gui_command","sensor_id":1011,"command":"terminating"}'|nc 127.0.0.1 60024
HA proxy
High CPS config for voipmonitor with hugepages
IOPS tests
--rw can be randread,randwrite,randrw
fio --name=rootTest --ioengine=libaio --rw=randwrite --bs=8k --numjobs=16 --size=1G --runtime=600s --time_based --ramp_time=15s --norandommap --filename=/root/testW.dat --output=/root/fio_db_write.txt
Rocky 9
Manager api of a sniffer
Manager api of a sniffer for customers
encryption_in_manager_api_customer
TLS decode by Ribbon SBC 7k
Email templates
common requests in supports common body of support emails
IPACCOUNT
How to enable ipaccount feature in the voipmonitor