Data Cleaning: Difference between revisions
(Created page with "= PCAP spool directory = By default sniffer stores all data to /var/spool/voipmonitor which can be changed in voipmonitor.conf - spooldir = ... Version >= 8.0 implements ne...") |
|||
| (5 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
= PCAP spool directory = | = PCAP spool directory = | ||
By default sniffer stores all data to /var/spool/voipmonitor which can be changed in voipmonitor.conf | By default sniffer stores all data to /var/spool/voipmonitor which can be changed in voipmonitor.conf by '''spooldir''' option. | ||
Cleaning procedure runs every 5 minutes and checks the size or days according to following options. If you set maxpoolsize it will wipe out the oldest data by minute until desired size is reached. '''maxpooldays''' keeps maximum number of data to keep in days. The same is for sip rtp and graph so you can keep sip pcaps longer than rtp pcaps. All following options can be activated at once and all sizes are in MB | |||
#set default maxpoolsize to 100 GB (102400 MB) | |||
maxpoolsize = 102400 | |||
maxpooldays = | |||
maxpoolsipsize = | |||
maxpoolsipdays = | |||
maxpoolrtpsize = | |||
maxpoolrtpdays = | |||
maxpoolgraphsize = | |||
maxpoolgraphdays = | |||
maxpoolaudiosize = | |||
maxpoolaudiodays = | |||
===Detailed description=== | |||
each created file is indexed in specific directory like /var/spool/voipmonitor/2024-12-10/00/.cleanspool_cache in hours interval and the file size is added to aggregation mysql table files. Cleaning procedure iterates through index files and unlink files without need to scan directories. | |||
If you accidentally remove /var/spool/voipmonitor/YYYY-MM-DD/HH/.cleanspool_cache or files table in database, you need to reindex the spool directory otherwise the old not indexed files will be never deleted by the cleaning procedure. To initiate reindexing by send reindexfiles command to a manager api of the sniffer. | |||
following commands requires netcat with support of an option -U for linux-based-socket | |||
reindexfiles [ | echo 'manager_file start /tmp/vmsck' | nc 127.0.0.1 5029 | ||
echo reindexfiles | nc -U /tmp/vmsck | |||
Also you can use other method to send command via secured api as described in doc [https://www.voipmonitor.org/doc/Encryption_in_manager_api_customer here] | |||
= Database cleaning = | = Database cleaning = | ||
Since version 7 the sniffer is partitioning tables by days which allows efficient cleaning of partitions. There is configuration option in voipmonitor.conf [Sniffer_configuration#cleandatabase]. | Since version 7 the sniffer is partitioning tables by days which allows efficient cleaning of partitions. There is configuration option in voipmonitor.conf [[Sniffer_configuration#cleandatabase]]. | ||
If you do not have partitions you cannot use this option and you need to create some script which will delete old data from cdr table. This method is very slow since the delete has to update giant index file if tables are not partitioned. | If you do not have partitions you cannot use this option and you need to create some script which will delete old data from cdr table. This method is very slow since the delete has to update giant index file if tables are not partitioned. | ||
Latest revision as of 15:01, 12 December 2024
PCAP spool directory
By default sniffer stores all data to /var/spool/voipmonitor which can be changed in voipmonitor.conf by spooldir option.
Cleaning procedure runs every 5 minutes and checks the size or days according to following options. If you set maxpoolsize it will wipe out the oldest data by minute until desired size is reached. maxpooldays keeps maximum number of data to keep in days. The same is for sip rtp and graph so you can keep sip pcaps longer than rtp pcaps. All following options can be activated at once and all sizes are in MB
#set default maxpoolsize to 100 GB (102400 MB) maxpoolsize = 102400 maxpooldays = maxpoolsipsize = maxpoolsipdays = maxpoolrtpsize = maxpoolrtpdays = maxpoolgraphsize = maxpoolgraphdays = maxpoolaudiosize = maxpoolaudiodays =
Detailed description
each created file is indexed in specific directory like /var/spool/voipmonitor/2024-12-10/00/.cleanspool_cache in hours interval and the file size is added to aggregation mysql table files. Cleaning procedure iterates through index files and unlink files without need to scan directories.
If you accidentally remove /var/spool/voipmonitor/YYYY-MM-DD/HH/.cleanspool_cache or files table in database, you need to reindex the spool directory otherwise the old not indexed files will be never deleted by the cleaning procedure. To initiate reindexing by send reindexfiles command to a manager api of the sniffer.
following commands requires netcat with support of an option -U for linux-based-socket
echo 'manager_file start /tmp/vmsck' | nc 127.0.0.1 5029 echo reindexfiles | nc -U /tmp/vmsck
Also you can use other method to send command via secured api as described in doc here
Database cleaning
Since version 7 the sniffer is partitioning tables by days which allows efficient cleaning of partitions. There is configuration option in voipmonitor.conf Sniffer_configuration#cleandatabase.
If you do not have partitions you cannot use this option and you need to create some script which will delete old data from cdr table. This method is very slow since the delete has to update giant index file if tables are not partitioned.