« Previous Next »

Thread: Input Format FROM bug?

Last post 05-27-2007 4:41 PM by LogParser User : Former_BigIronGuy. 3 replies.

Average Rating Rate It (5)

RSS

Page 1 of 1 (4 items)

Sort Posts:

  • 05-17-2007, 6:09 AM

    Input Format FROM bug?

    Hi.

    I've been writing a few scripts to consolidate IIS logfiles from some sites running on a webfarm in to single IIS files that are to be migrated to a single server setup.  I'm doing this in a few steps as I had performance problems and crashes if I tried to process a few years of data in one go

    Step 1 is to consolidate the hourly logs for each server in to monthly (or yearly) depending on their size).

    Step 2 is to combine each log file of the same date range for each server in to one master logfile.

    Step 1 is where I am having the problem though, here is my script (run on LogParser 2.2):

    "logparser -i:IISW3C -e 1000 -o:W3C "SELECT date, time, cs-method, cs-uri-stem, cs-uri-query, cs-username, c-ip, cs-version, cs(User-Agent), cs(Referer), sc-status, sc-bytes INTO C:\LogParser\Test-109.log FROM C:\LogParser\109\ex99*.log ORDER BY date, time ASC"

    I would expect this script to only look for log files written in 1999 (The file names are standard IIS format - "exyymmddhh.log".  In this case my input folder contains log files from ex07mmddhh to ex0508ddhh but LogParser writes an output file with records it has retrieved from files it should not have touched?! It seems to have picked up 219 records from dates that are in logfiles that can't possible match the ex99*.log input format.  For example 2005-11-23 - logfile ex05112320.log

    I can try it with different years and still get data that should be there.  There are some years that give me nothing though.  It's something I could work around but it's making me lose faith in how reliable the combined of log files are for years where I actually do have data!

    Any comment would be much appreciated.

    Thanks. Phil

     

  • 05-18-2007, 7:21 AM In reply to

    I would suspect corrupt log files.
    You could try a quick test like this to start with:
    "logparser -i:IISW3C -e 1000 "SELECT DISTINCT LogFilename FROM C:\LogParser\109\ex99*.log"


    And then if that looks correct, you could try adding the LogFilename and LogRow fields to your normal output and see where LP thinks those bogus records are coming from.
  • 05-18-2007, 9:47 AM In reply to

    Cheers for the reply, but can it really be corrupt log files? 

    I mean with a FROM of "FROM C:\LogParser\109\ex99*.log" it should only be looking at logfiles that have names that begin with ex99, there are none of these in the specific folder in question which is why it's strange it's bringing back results.  It seems to be looking in some off the other logfiles in that folder that don't match the file filter in the FROM but only pulling in a few records from them - it shouldn't be looking in them at all should it?

    Many thanks, Phil

  • 05-27-2007, 4:41 PM In reply to

    Phil,
    It has been since v2.0 that I did a lot with LogParser. I *do* remember that there were some logfiles on our intra-net server that for whatever reason, were completely munged. At that time, the fix I resorted to was to list the IN_ROW_NUMBER of the output and then built a two part query to take every thing up to the bad record and one to pick up everything after.
    Howsoever, it appears to me that 2.2 has lots more robustness than what I remember, and appears to be able to successfully handle the exception rows...

    FBIG
Page 1 of 1 (4 items)
Microsoft Communities