Author Topic: Hosts Programs and Mnemonice not inserting  (Read 756 times)

0 Members and 1 Guest are viewing this topic.

Offline cdukes

  • LogZilla
  • Administrator
  • Hero Member
  • *****
  • Posts: 890
    • View Profile
    • LogZilla
Hosts Programs and Mnemonice not inserting
« on: August 29, 2010, 11:08:30 pm »
Hi everyone,
I found a small bug that will be corrected in v3.0.92.
If you are on a small server that doesn't get very many messages, the cache doesn't have enough time to build before the program exits, thus leaving you with an empty hosts, programs and mnemonics (if any).
To remedy this, you can add the following to your db_insert.pl around line 290:

Existing Code:
Code: [Select]
if (eof()) { # check for end of last file
        open (DUMP, ">$dumpfile") or die "can't open $dumpfile: $!\n";
        print LOG "EOF - Flushing buffer\n" if ($debug > 0);
        print STDOUT "EOF - Flushing buffer\n" if ($debug > 0);
        print STDOUT "Importing $#dumparr messages into the database\n" if ($debug > 0);
        print LOG "Importing $#dumparr messages into the database\n" if ($debug > 0);
        print DUMP @dumparr;
        undef (@dumparr);
        close (DUMP);
        $db_load->execute();
        if ($db_load->errstr()) {
           print STDOUT "FATAL: Unable to execute SQL statement: ", $db_load->errstr(), "\n" if ($debug > 0);
        }

Add the next bit of code right after the } statement above:
Code: [Select]
# Added to insert cached hosts, progs and mnes
        my @hosts = keys %host_cache;
        foreach my $h (@hosts) {
            $db_insert_host->execute($h);
        }
        my @prgs = keys %program_cache;
        foreach my $p (@prgs) {
            $db_insert_prg->execute($p, $program_cache{$p});
        }
        my @mnes = keys %mne_cache;
        foreach my $m (@mnes) {
            $db_insert_mne->execute($m, $mne_cache{$m});
        }
        %host_cache = ();
        %program_cache = ();
        %mne_cache = ();
        # End add
Your Network is Your Business.  Be Proactive.  Go LogZilla.
Clayton Dukes
CTO, LogZilla, LLC
http://www.logzilla.pro

Offline pacrew

  • Newbie
  • *
  • Posts: 2
    • View Profile
Re: Hosts Programs and Mnemonice not inserting
« Reply #1 on: September 09, 2010, 05:24:03 am »
I don't know if I'm in the correct subject.

I get in trouble with 'db_insert.pl'.

When I execute:
Quote
cat /var/log/logzilla/syslog_dump.log | ./db_insert.pl -d20 -v

I receive the following message:
Quote
Importing 4 messages into the database
    -> execute for DBD::mysql::st (DBI::st=HASH(0x8434f20)~0x8434e9c) thr#817c008
 -> dbd_st_execute for 08434fec
        >- dbd_st_free_result_sets
        <- dbd_st_free_result_sets RC -1
        <- dbd_st_free_result_sets
mysql_st_internal_execute MYSQL_VERSION_ID 50150
>parse_params statement LOAD DATA LOCAL INFILE '/dev/shm/infile.txt' INTO TABLE logs2 FIELDS TERMINATED BY "\t" LINES TERMINATED BY "\n" (host,facility,severity,program,msg,mne,fo,lo)
 <- dbd_st_execute returning imp_sth->row_num 0
    <- execute= '0E0' at ./db_insert.pl line 299
    -> errstr in DBD::_::common for DBD::mysql::st (DBI::st=HASH(0x8434f20)~0x8434e9c) thr#817c008
    <- errstr= undef at ./db_insert.pl line 300
Ending insert: 12:01:15
    -> DESTROY for DBD::mysql::st (DBI::st=HASH(0x8434e9c)~INNER) thr#817c008
    <- DESTROY= undef

In Mysql:
Quote
mysql> select * from logs2;
Empty set (0.00 sec)

Offline cdukes

  • LogZilla
  • Administrator
  • Hero Member
  • *****
  • Posts: 890
    • View Profile
    • LogZilla
Re: Hosts Programs and Mnemonice not inserting
« Reply #2 on: September 09, 2010, 08:06:38 am »
It sounds like you are missing partitions, please see here:
http://forum.logzilla.info/index.php/topic,187.0.html
Your Network is Your Business.  Be Proactive.  Go LogZilla.
Clayton Dukes
CTO, LogZilla, LLC
http://www.logzilla.pro

Offline pacrew

  • Newbie
  • *
  • Posts: 2
    • View Profile
Re: Hosts Programs and Mnemonice not inserting
« Reply #3 on: September 10, 2010, 04:49:07 am »
At the step:
root@TSTVM33D/home1/logzilla/sphinx/src/sphinx-0.9.9$ ./configure --prefix `pwd`/../..

Quote
checking zlib.h usability... no
checking zlib.h presence... no
checking for zlib.h... no
checking sql.h usability... no
checking sql.h presence... no
checking for sql.h... no


Could it explain my problem concerning insertion in log's table?

(I wrote a simple script in perl to make insertion in my database and it worked: the problem is not comming from the DBI module.)

Offline cdukes

  • LogZilla
  • Administrator
  • Hero Member
  • *****
  • Posts: 890
    • View Profile
    • LogZilla
Re: Hosts Programs and Mnemonice not inserting
« Reply #4 on: September 10, 2010, 12:36:19 pm »
No, Sphinx doesn't have anything to do with message insertion.

Can you paste the output of a "show create table logs" for me please?
Your Network is Your Business.  Be Proactive.  Go LogZilla.
Clayton Dukes
CTO, LogZilla, LLC
http://www.logzilla.pro