View source for Internal support hints


==Important note: ==
 Following articles are designed to be used by internal support team. You can find hints usauseful but please use it at your own risk (use on testing servers first)

= Testing configure options =
If you need to test configure options without need to modify the config file or database settings, you can use json_config option:
 --json_config='[{"id_sensor":"11"},{"interface":"lo"},{"natalias","8.8.8.8 10.10.100.50;9.9.9.9 192.168.88.33"}]'
(The otpions defined using '''json_config''' command line option will overrides the configfile and database settings)

== Examples ==
This will override options '''rtpip_find_endpoints,rrd,natalias''' no matter what is set in config file.
 ./voipmonitor --config-file=config/voipmonitor.conf -p XobmuJ -b voipmonitor_ipv6 -k -v1,pcap_stat_period=5,hash_rtp,rtp_streams -rpba:/totest/test.pcap --json_config='[{"rtpip_find_endpoints","yes"},{"rrd","no"},{"natalias","8.8.8.8 10.10.100.50;9.9.9.9 192.168.88.33"}]'

= Testing regexp / country rules / billing =
== Regexp test ==
 /usr/local/sbin/voipmonitor --config-file=/etc/voipmonitor-test.conf -k -v1 --test-regexp '^800880080[1-9][0-9][*]|80088008012345|' 

== Country assignment test ==
 /usr/local/sbin/voipmonitor --config-file=/etc/voipmonitor-test.conf -k -v1 --find-country-for-ip=102.140.68.113
 /usr/local/sbin/voipmonitor --config-file=/etc/voipmonitor-test.conf -k -v1 --find-country-for-number 0034*12345

* gui version (obsolete)

 root@vmweb:GUI_ROOT# php php/run.php testFindCountryForNumber -n 4909699999
 number:           4909699999
 country:          DE
 is_international: Y

== pricing / billing test ==
 voipmonitor --config-file=config/voipmonitor.conf -k -v1 --test-billing test_data/billing

=== Example of test_data/billing file===
 Columns order
  - calldate 
  - connect_duration
  - caller
  - called
  - sipcallerip
  - sipcalledip
  - expected price operator - (voluntary option)
  - expected price customer - (voluntary option) 

 2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151,800,8
 2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151,800,8.1
 2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151

= License Hints =
== Write state to db (count of cc cchannels for license purposse) ==
 php /var/www/html/php/run.php saveCallStatistics

== keep the license updated without processing alerts reports ==
(put into /etc/crontab)
 04 04   * * *   root php run.php runUpdateLicenseKey

== License Details ==
=== detail on channels count usage ===
 php /var/www/html/php/run.php checkLicense -v
Less details:
 php /var/www/html/php/run.php checkLicense -v|grep max_peak -A3 -B1

=== license file ===
 cat /var/www/html/key.php

=== Get current license token from db ===
 mysql> select content from voipmonitor.system where type='license token';

= SCRIPTs for better run control =
== Alerts and Report ==
When alerts taking longer time to process then interval is schedulled it can cause out of memory or high load to CPU after some time because same alert is processed multipletimes for various time ranges. Following script allows manual run of disabled alert in GUI via crontab - its benefit is that it will run multiple alerts at same time unlike normal run where is processed one alert by one.

Follow this link [[script-for-multi-Instances-alerts-processing]] (deprecated)

The GUI now supports parralel run of the alerts/report - enable it in system settings->Advanced->Number of parallel tasks

== Batch download for more then 1000CDRs ==
When you need to download lot of audio files - you use GUI->menu->cdr batch download,but it allows only 1000calls to process with one GUI's query.

Following script allows you to create audio from longer set of CDRs, also it process in multiple instances to get audio faster (beware of HDD IO/CPU) [[script-for-big-batchdownloads]]

== RAM (alerts / dashboards cache) ==
When there is lot of CDRs then alerts or caching of dashboard data can take lot of RAM, folloowing script will control if some php thread will not consume over limit, when it does, its oom_adj_score is set to be killed first (prevent killing of voipmonitor sniffer od db when running on same server)

Follow link [[script-for-GUI-cron-control-oom]]

== Backup the settings from all probes and GUI ==
The script gets voipmonitor.conf from all probes and do GUI backup config tables from db. Requires ssh access without password, error redirect to /dev/null, run it using cron

Follow link [[script-for-probes-backup]]

= charts binary crashes debug steps =
When apachelog or dmesg reports crashes of vm's charts binary like
 Aug 20 10:32:30 C17VOIPMONDB02 kernel: charts-x86_64[7321]: segfault at 10 ip 0000000000412553 sp 00007ffff2638430 error 4 in charts-x86_64[400000+6c6000]
== create debug folder for charts ==
Create vm_charts_parameters in /tmp

after GUI will create some charts there files with arguments will be created.
== test Arguments passed to charts binary ==
 /var/www/html/bin/charts-x86_64 -i /tmp/vm_charts_parameters/files

= voipmonitor sniffer memory leaks detect and debug =
When voipmonitor uses more and more RAM (RSS/VSZ stat in syslog) without processing more CALLs (regs/messages) following steps can tell you where the RAM ends (it can be leak or some thread overload)
== compile je_malloc ==
Download jemalloc from https://github.com/jemalloc/jemalloc
 ./autogen.sh
 #requires development tools Deb/ubu apt install build-essential, centos/rh yum groupinstall 'Development Tools'
 ./configure --enable-prof
 make
 make install

== enable jemalloc for voipmonitor and compile it ==
 ./configure
edit config.h,to have there
 #define HAVE_LIBTCMALLOC 0
 #define HAVE_LIBJEMALLOC 1
edit Makefile and disable optimization and add ljemalloc to libs
 substitute there -O2 with -O0 (It is capital o followed by zero)
 LIBS = ${SHARED_LIBS} -ljemalloc

Finish compilation
 make clean
 make -j8

== enable X88 compatible sources and recompile ==
Copy sniffer sources with compiled binary to new location and recreate binary with -X88 (This is necessary to get correct memory allocation for debuging usin jeprof)
 cp -a /usr/src/voipmonitor-git /usr/src/sniffer-git-jemalloc-X88
 LD_PRELOAD=/usr/local/lib/libjemalloc.so ./voipmonitor -k -v1 -c -X88
 #It will list of source files modification,then re-make
 make clean
 make

More detailed instructions on debuging using X88 [[X88-memoryConsumptionDebug]]

== Modify init file or run sniffer manually with extra env arguments and parameters ==
=== edit /etc/init.d/voipmonitor ===
 ARGS="-v 1,force_log_sqlq,memory_stat,memory_stat_ignore_limit=100000"
 #and in start section (2places) of a script
 LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' /usr/src/sniffer-git-jemalloc-X88/voipmonitor --config-file $CONFIGFILE --pid-file $PIDFILE $ARGS

=== or run it from konsole / when sniffer service stopped ===
 LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' /usr/src/sniffer-git-jemalloc-X88/voipmonitor --config-file=/etc/voipmonitor.conf -v 1,force_log_sqlq,memory_stat_ex,memory_stat_ignore_limit=100000,heapsafe

== Check results ==
in some interval you should check the results of memory usage with
=== je_prof.pdf ===
 yum install graphviz ghostscript
 echo 'jemalloc_stat_full' | nc 127.0.0.1 5029 > /tmp/je_prof; jeprof --show_bytes --pdf /usr/src/sniffer-git-jemalloc-X88/voipmonitor /tmp/je_prof > je_prof.pdf

=== sniffer_memory.txt ===
 echo 'jemalloc_stat_full' | nc 127.0.0.1 5029

=== script for periodical collecting ===

 #!/bin/bash
 DATE=`date '+%Y%m%d_%H:%M:%S'`
 LOGDIR=/root/vm_debug/log/$DATE
 mkdir -p $LOGDIR
 pdftmpname=/tmp/je_prof
 pdfname=$LOGDIR/je_prof.pdf
 txtname=$LOGDIR/mem_stat.txt
 senname=$LOGDIR/senlog.txt
 echo 'jemalloc_stat_full' | nc 127.0.0.1 5029 > $pdftmpname 2>&1
 /usr/local/bin/jeprof --show_bytes --pdf /usr/src/sniffer-git-jemalloc-X88/voipmonitor $pdftmpname > $pdfname 2>&1
 echo memory_stat | nc 127.0.0.1 5029 > $txtname 2>&1
 echo sniffer_stats | nc 127.0.0.1 5029 > $senname 2>&1
 rm $pdftmpname

= voipmonitor sniffer compile notes ss7 (wireshark) module =
(requires ws version 3.2):

== modify config.h ==
 #define HAVE_LIBWIRESHARK 1
 #define LIBWIRESHARK_VERSION 30200

== modify Makefile's INCLUDE ==
 -I/usr/include/wireshark
== modify Makefile's SHARED_LIBS==
 -lwiretap -lwireshark -lwsutil

internal build in JIRA:/opt/debian-7-64bit
== disable optimize in Makefile==
 change -O2 to -O0

== compile ==
 ./configure && make clean && make ss7

= testing customer pcaps =
== testing special pcaps ==
=== ipfix ===
Run first instance of voipmonitor that will act as receiver for packets and will push CDRs to db
in the settings enable ipfix options:
 ipfix = yes
 ipfix_bind_ip = 0.0.0.0
 ipfix_bind_port = 12345

Then run scecond instance that will act as source for replay the ipfix encapsulated pcap
 ./voipmonitor --config-file=config/voipmonitor.conf -k -v1 --ipfix-client-emulation='/tmp/TCPdump_ipfix_stream.pcap;1.2.3.4;8.8.8.8;127.0.0.1;12345'
Params:
 pcap file
 2 IP address - 1.ip fix client,2.ip fix server
 last IP : port telling to where to send the ipfix stream (IP:port of ipfix options set for the 1.st instance)

 


== testing via basic upload aka GUI upload ==
 voipmonitor --config-file=/etc/voipmonitor-customer.conf -r /tmp/file.pcap
== testing via packetbuffer upload ==
=== time shift to now, increased speed of replaying ===
 voipmonitor --config-file=/etc/voipmonitor-customer.conf -rpbsa9: /tmp/file.pcap
=== no time shift via pb===
 voipmonitor --config-file=/etc/voipmonitor-customer.conf -rpb: /tmp/file.pcap
=== tcpreplay to running service ===
you need to enable additional argument when starting the service in /etc/init.d/voipmonitor
 ARGS="-v 1,dump_call_flags,tcpreplay"
then use tcpreplay
 tcpreplay -i eth0 /tmp/l/x.pcap
when error appears like
 sending out eth0
 processing file: /tmp/l/x.pcap
 Warning in tcpreplay.c:replay_file() line 227:
 /tmp/l/x.pcap DLT (LINUX_SLL) does not match that of the outbound interface: eth0 (EN10MB)
probably captured on any interface and layer2 missing, here follow stpes on how to add it:

==== adding layer2 to pcap ====
after capture was done on any interface and you need to replay it to hw interface you need to add layer2 (at least in 3.4.3 v.tcpreplay)
 tcprewrite --dlt=enet -i /tmp/l/x.pcap -o /tmp/l/x_enet-encaps.pcap
 tcprewrite --enet-dmac=00:55:22:AF:C6:37 --enet-smac=00:44:66:FC:29:AF --infile=/tmp/l/x_enet-encaps.pcap --outfile=/tmp/l/x_enet-encaps-l2.pcap

==== rewrite IPv4 address in whole pcap ====
when you need to test alert based on IP adresses (also country continent alert based on IPs) you can find usefull to change IP address in pcap and use it multiple times with various addresses:
 tcprewrite --pnat=8.7.6.5/32:1.2.3.4/32 -i ./7efd0e8b87a54f48ac04b7a49fdf5669.pcap -o ./7e-rew.pcap

= Rename mysql database =
It's not possible to do it via direct command. You need to use '''RENAME TABLE''' command.
With this SELECT you generate the commands for the rename of the all tables in the database. (Replace OLD_DB_NAME and NEW_DB_NAME)

 SELECT CONCAT('RENAME TABLE ',table_schema,'.',table_name,
     ' TO ','NEW_DB_NAME.',table_name,';')
 FROM information_schema.TABLES
 WHERE table_schema LIKE 'OLD_DB_NAME';

The output:

 RENAME TABLE OLD_DB_NAME.table1 TO NEW_DB_NAME.table1;
 RENAME TABLE OLD_DB_NAME.table2 TO NEW_DB_NAME.table2;
 RENAME TABLE OLD_DB_NAME.tableX TO NEW_DB_NAME.tableX;

Run these commands a you are ready.

'''Attention! Sometimes you can have a problem with moving of the foreign keys.'''

= disabling TCP sack =
On buggy kernels (all before 2019/03) following prevents SACK issue
sideback is that on lossy lines (packet lost or big delays) the tcp traffic will increase)
 echo "0" > /proc/sys/net/ipv4/tcp_sack

= MacOS high sierra in virtual box (on windows) =
== Download Image ==
 from - https://drive.google.com/file/d/0B0EsZGPuNAPvVDlicGt1NUx0Y0E/view
== Prepare ISO for install ==
===Create High Sierra DMG Disk Image===
 hdiutil create -o /tmp/Highsierra -size 7900m -volname highsierra -layout SPUD -fs HFS+J

=== Mount DMG Image to your macOS ===
 hdiutil attach /tmp/Highsierra.dmg -noverify -mountpoint /Volumes/Highsierra

=== Create a High Sierra ISO image ===
 sudo /Applications/Install\ macOS\ High\ Sierra.app/Contents/Resources/createinstallmedia --volume /Volumes/Highsierra

=== Unmount Disk Image ===
 hdiutil detach /volumes/Install\ macOS\ High\ Sierra

=== Convert DMG to ISO ===
 hdiutil convert /tmp/Highsierra.dmg -format UDTO -o ~/Desktop/Highsierra

=== Rename and Move ISO Image to Desktop ===
 mv ~/Desktop/Highsierra.cdr ~/Desktop/Highsierra.iso

== Create new virtual Image ==
during preconfigure of new image set:
 memmory 3-6GB
 disk size 100GB
 then edit it's settings and additionally modify:
 system -> Motherboard -> Chipset: PIIX3
 system -> Processor -> count (2), Enable PAE/NX
 system -> Storage -> Attributes-> Optical drive->Sata port 1, for the newly created iso
 display -> Video memory -> 128MB
 display -> Graphics Controler -> VMSVGA

== Add the VirtualBox macOS Code to CMD ==
 cd "C:\Program Files\Oracle\VirtualBox\"
 VBoxManage.exe modifyvm "Your VM Name" --cpuidset 00000001 000106e5 00100800 0098e3fd bfebfbff
 VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/efi/0/Config/DmiSystemProduct" "iMac11,3"
 VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/efi/0/Config/DmiSystemVersion" "1.0"
 VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/efi/0/Config/DmiBoardProduct" "Iloveapple"
 VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/smc/0/Config/DeviceKey" "ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc"
 VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/smc/0/Config/GetKeyFromRealSMC" 1

== Install macOS on virtulBox ==
#Open VirtualBox and turn on your High Sierra, then Open "Disk Utility".
#Format VirtualBox Virtual Disk with "Mac OS Extended". (GUID:Partition map)
#Now, you can install macOS High Sierra on new virtual disk with your ISO Image.
#After your installation is completed, turn off your macOS High Sierra. Open VirtualBox, and remove High Sierra.iso file.
#Turn on your Virtual Machine. At UEFI Shell screen, boot with this commands:

 >"fs1:\macOS Install Data\Locked Files\Boot Files\boot.efi"


== Start the High Sierra Virtual machine and Enjoy ==

= Measurement =
== Memory speed ==

 sysbench memory --threads=2 run

* should be over 5000 MiB/sec

= SSL and keylogger =
== How to test it in voipmonitor ==
 ./voipmonitor --config-file /etc/test.conf -k -v1,pcap_stat_period=1,_ssl,_tls,_ssldecode,ssl_sessionkey,dump_sip_line -rpba:/tmp/keys.pcap@@/tmp/call.pcap --json_config='[{"sipport":"8089"},{"ssl":"yes"},{"ssl_ipport":"192.168.0.1:8089"},{"ssl_sessionkey_udp":"yes"},{"ssl_sessionkey_udp_port":"1234"},{"ssl_sessionkey_udp_maxwait_ms":10000},{"interface":"lo"},{"spooldir":"/var/spool/voipmonitorTLS"}]'

== How to test it in wireshark/tshark ==

=== Get the keys from keyloger's dump ===
 ngrep -I /tmp/keycapture.pcap |grep '^ ' |cut -d ' ' -f3- > /tmp/keys.txt

=== Decrypt with tshark ===
 tshark -r '/tmp/tlscall.pcap' -o 'tls.keylog_file: /tmp/keys.txt' -o 'tls.debug_file: /dev/stdout'

=== Decrypt in wireshark ===
In edit->preferences->protocol->SSL->pre-master secret log filename and choose the keys.txt
then find the client Hello HS packet, right click on it and follow stream - TLS

= Spooldir testing =
When there are problems with cleaning the spooldir or spooldir size not respects the settings you can use following command to test
 voipmonitor --config-file /etc/voipmonitor.conf -k -v1 --test-cleanspool-load all

= Manager API via server port =
== get list of the connected clients ==
 echo '{"type_connection":"manager_command","command":"active"}'|nc 127.0.0.1 60024
== send terminating manager command to a probe ==
 echo '{"type_connection":"gui_command","sensor_id":1011,"command":"terminating"}'|nc 127.0.0.1 60024

= HA proxy =
[[sample HAproxy configuration]]

== High CPS config for voipmonitor with hugepages ==
[[voipmonitor and hugepages]]

= IOPS tests =
--rw can be randread,randwrite,randrw
  fio --name=rootTest --ioengine=libaio --rw=randwrite --bs=8k --numjobs=16 --size=1G --runtime=600s --time_based --ramp_time=15s --norandommap --filename=/root/testW.dat --output=/root/fio_db_write.txt

= Rocky 9=
[[Rocky 9]]

= Email templates =
common requests in supports [[common body of support emails]]