I have a directory that contains many log files. It turns out that using the input format "TEXTLINE" will allow me to extract the date/time that I need, so my question is: how do I execute the same Logparser command for each of the log file in that directory?
Is there a recursive type of command?
The only tricky thing is that some of those log filenames have spaces in them, so I would have to replace those filenames with the /u0020 character. But maybe I can rename them manually before running Logparser.
Thanks Gabriele for the wildcard suggestion. I will play with it and see how that works.
Yes, for each input file, I need to create a different output file. I plan to reuse the input file name and append some timestamp to create the new output file name.
For example:
input filename: B1 Errors.csv
output filename: B1 Errors 020604 092300.csv
If LogParser can go through a whole directory and parse every single file in that directory, it would be so useful. I would appreciate your help in figuring this out. Thanks!
Well, you can do that using the MultiPlex feature.
Depending on what output format you are going to use, the whole thing would look like this:
"SELECT LogFilename, MY_FORMATTED_DATE_TIME(...), whatever_else INTO *_*.txt FROM D:\myfolder\my*.log WHERE..."
The query above will send results to files whose name pattern is "*_*"; the first wildcard is substituted with the first field (in this case, the full input filename - which is a field of most input formats), and the second wildcard is substituted with the
second field (in this case, the formatted date time).
For example, the output filenames might look like this:
D:\myfolder\myfile1.log_2004-02-02.12-23-03.txt
D:\myfolder\myfile1.log_2004-02-02.12-23-05.txt
D:\myfolder\myfile2.log_2004-02-02.12-20-45.txt
Watch out for the timestamp format: since these become part of the final filename, you can't use characters that are illegal for filenames (e.g. ':').
------------------------------
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm"
Thanks Gabriele. I will try out your example and let you know how it goes. I am also reading the Multiplex section in the Help file which has a couple of examples as well.
Hmm! I am running into this problem. It doesn't like the the formatted date parameter which I copied from your query:
C:\Program Files\LogParser2.1>logparser -i:TEXTLINE "SELECT LogFilename, MY_FORM
ATTED_DATE_TIME(020604), Text FROM C:\LogParser2.1\Logs\*.log TO *_*.txt"
WARNING: Output format not specified - using NAT output format.
ERROR: SELECT clause: Syntax Error: unknown field 'MY_FORMATTED_DATE_TIME(020604)'
To see valid fields for the TEXTLINE input format type:
LogParser -h -i:TEXTLINE
I also tried this and it works, but it writes out an ouput file for each successful result from the WHERE statement. I have 4 input log files in the directory so I expected 4 output log files, but instead I got 47 output files and each
one contains 1 successful result from the WHERE statement! How can I get it to write the successful results for each input file to only 1 output file. Here is my command:
C:\Program Files\LogParser2.1>logparser -i:TEXTLINE "SELECT LogFilename, Index,
Text FROM C:\Program\u0020Files\LogParser2.1\Logs\*.log TO *_*.txt WHERE Text LI
KE '%02:00%:%' or Text LIKE '%02:01%:%' " -o:CSV
Statistics:
-----------
Elements processed: 9664
Elements output: 47
Execution time: 0.61 seconds
Sorry, I don't understand where "your_date_field_name" comes from?
I can't just make it up? I have to use a field that LogParser recognizes for a TEXTLINE input format. Right?
I tried this: C:\Program Files\LogParser2.1>logparser -i:TEXTLINE "SELECT LogFilename, TO_STRI
NG(MyDate, 'yymmdd hh-mm-ss'), Text FROM C:\Program\u0020Files\LogParser2.1\Logs
\*.log TO *_*.txt WHERE Text LIKE '%02:00%:%' or Text LIKE '%02:01%:%' " -o:CSV
-filemode:0
ERROR: SELECT clause: Syntax Error: unknown field 'MyDate'
To see valid fields for the TEXTLINE input format type:
LogParser -h -i:TEXTLINE
If you have time, I would appreciate knowing what to do about the second problem I posted in the last posting with the multiple output files. Thanks.1
Ok, sorry, it seems I didn't understand your intentions exactly :-)
So, for each original file, you want 1 new output file, whose name is exactly like the original file, plus some tag of your choice, correct?
If so, yes, you can make up the second value in the SELECT clause, i.e. you can use a constant instead of a field coming from the input, something like this:
logparser -i:TEXTLINE "SELECT LogFilename, '040207_12-35-23', Text FROM C:\Program\u0020Files\LogParser2.1\Logs\*.log TO *_*.txt WHERE Text LIKE '%02:00%:%' or Text LIKE '%02:01%:%' " -o:CSV
If your files look like this:
mylog1.log
mylog2.log
mylog3.log
Then the output files will look like this:
mylog1.log_040207_12-35-23.txt
mylog2.log_040207_12-35-23.txt
mylog3.log_040207_12-35-23.txt
Is this what you're trying to do?
(sorry for the misunderstandings<g>
------------------------------
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm"
Anonymous
6623 Posts
How to Run the same LogParser command for all the log files in a directory?
Feb 05, 2004 10:28 PM|LINK
I have a directory that contains many log files. It turns out that using the input format "TEXTLINE" will allow me to extract the date/time that I need, so my question is: how do I execute the same Logparser command for each of the log file in that directory? Is there a recursive type of command?
The only tricky thing is that some of those log filenames have spaces in them, so I would have to replace those filenames with the /u0020 character. But maybe I can rename them manually before running Logparser.
Anonymous
6623 Posts
Re: How to Run the same LogParser command for all the log files in a directory?
Feb 06, 2004 03:56 PM|LINK
Using wildcards in the filename does not help ? i.e. "SELECT whatever FROM my*.log" ?
Do you need a different output file for each input file ?
------------------------------
This posting is provided "AS IS" with no warranties, and confers no rights.
Anonymous
6623 Posts
Re: How to Run the same LogParser command for all the log files in a directory?
Feb 06, 2004 04:14 PM|LINK
Thanks Gabriele for the wildcard suggestion. I will play with it and see how that works.
Yes, for each input file, I need to create a different output file. I plan to reuse the input file name and append some timestamp to create the new output file name.
For example:
input filename: B1 Errors.csv
output filename: B1 Errors 020604 092300.csv
If LogParser can go through a whole directory and parse every single file in that directory, it would be so useful. I would appreciate your help in figuring this out. Thanks!
Anonymous
6623 Posts
Re: How to Run the same LogParser command for all the log files in a directory?
Feb 06, 2004 04:54 PM|LINK
Well, you can do that using the MultiPlex feature.
Depending on what output format you are going to use, the whole thing would look like this:
"SELECT LogFilename, MY_FORMATTED_DATE_TIME(...), whatever_else INTO *_*.txt FROM D:\myfolder\my*.log WHERE..."
The query above will send results to files whose name pattern is "*_*"; the first wildcard is substituted with the first field (in this case, the full input filename - which is a field of most input formats), and the second wildcard is substituted with the second field (in this case, the formatted date time).
For example, the output filenames might look like this:
D:\myfolder\myfile1.log_2004-02-02.12-23-03.txt
D:\myfolder\myfile1.log_2004-02-02.12-23-05.txt
D:\myfolder\myfile2.log_2004-02-02.12-20-45.txt
Watch out for the timestamp format: since these become part of the final filename, you can't use characters that are illegal for filenames (e.g. ':').
------------------------------
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm"
Anonymous
6623 Posts
Re: How to Run the same LogParser command for all the log files in a directory?
Feb 06, 2004 05:24 PM|LINK
Thanks Gabriele. I will try out your example and let you know how it goes. I am also reading the Multiplex section in the Help file which has a couple of examples as well.
I appreciate your prompt help in this!
Anonymous
6623 Posts
Re: How to Run the same LogParser command for all the log files in a directory?
Feb 06, 2004 07:27 PM|LINK
Hmm! I am running into this problem. It doesn't like the the formatted date parameter which I copied from your query:
C:\Program Files\LogParser2.1>logparser -i:TEXTLINE "SELECT LogFilename, MY_FORM
ATTED_DATE_TIME(020604), Text FROM C:\LogParser2.1\Logs\*.log TO *_*.txt"
WARNING: Output format not specified - using NAT output format.
ERROR: SELECT clause: Syntax Error: unknown field 'MY_FORMATTED_DATE_TIME(020604)'
To see valid fields for the TEXTLINE input format type:
LogParser -h -i:TEXTLINE
I also tried this and it works, but it writes out an ouput file for each successful result from the WHERE statement. I have 4 input log files in the directory so I expected 4 output log files, but instead I got 47 output files and each one contains 1 successful result from the WHERE statement! How can I get it to write the successful results for each input file to only 1 output file. Here is my command:
C:\Program Files\LogParser2.1>logparser -i:TEXTLINE "SELECT LogFilename, Index,
Text FROM C:\Program\u0020Files\LogParser2.1\Logs\*.log TO *_*.txt WHERE Text LI
KE '%02:00%:%' or Text LIKE '%02:01%:%' " -o:CSV
Statistics:
-----------
Elements processed: 9664
Elements output: 47
Execution time: 0.61 seconds
For example:
input file name: access-ARANEAEQA2-2004-1-14-2342_771KB.log
output file names resulting from the WHERE clause:
access-ARANEAEQA2-2004-1-14-2342_771KB.log_1.txt
access-ARANEAEQA2-2004-1-14-2342_771KB.log_2.txt
and 32 other files similar to this one with the index name changed (i.e. log_3.txt)
Here is the content of one of the files (access-ARANEAEQA2-2004-1-14-2342_771KB.log_1.txt ):
Text
144.203.8.32 - - [13/Jan/2004:02:01:31 +0000] "GET / HTTP/1.0" 200 343
(As you can see, the content is just one result from the WHERE clause since the timestamp meets the LIKE criteria).
I tried adding the "-filemode:0" option but it didn't make any difference.
Anonymous
6623 Posts
Re: How to Run the same LogParser command for all the log files in a directory?
Feb 06, 2004 07:34 PM|LINK
Anonymous
6623 Posts
Re: How to Run the same LogParser command for all the log files in a directory?
Feb 06, 2004 09:41 PM|LINK
Sorry, I don't understand where "your_date_field_name" comes from?
I can't just make it up? I have to use a field that LogParser recognizes for a TEXTLINE input format. Right?
I tried this: C:\Program Files\LogParser2.1>logparser -i:TEXTLINE "SELECT LogFilename, TO_STRI
NG(MyDate, 'yymmdd hh-mm-ss'), Text FROM C:\Program\u0020Files\LogParser2.1\Logs
\*.log TO *_*.txt WHERE Text LIKE '%02:00%:%' or Text LIKE '%02:01%:%' " -o:CSV
-filemode:0
ERROR: SELECT clause: Syntax Error: unknown field 'MyDate'
To see valid fields for the TEXTLINE input format type:
LogParser -h -i:TEXTLINE
If you have time, I would appreciate knowing what to do about the second problem I posted in the last posting with the multiple output files. Thanks.1
Anonymous
6623 Posts
Re: How to Run the same LogParser command for all the log files in a directory?
Feb 07, 2004 03:00 PM|LINK
Ok, sorry, it seems I didn't understand your intentions exactly :-)
So, for each original file, you want 1 new output file, whose name is exactly like the original file, plus some tag of your choice, correct?
If so, yes, you can make up the second value in the SELECT clause, i.e. you can use a constant instead of a field coming from the input, something like this:
logparser -i:TEXTLINE "SELECT LogFilename, '040207_12-35-23', Text FROM C:\Program\u0020Files\LogParser2.1\Logs\*.log TO *_*.txt WHERE Text LIKE '%02:00%:%' or Text LIKE '%02:01%:%' " -o:CSV
If your files look like this:
mylog1.log
mylog2.log
mylog3.log
Then the output files will look like this:
mylog1.log_040207_12-35-23.txt
mylog2.log_040207_12-35-23.txt
mylog3.log_040207_12-35-23.txt
Is this what you're trying to do?
(sorry for the misunderstandings<g>
------------------------------
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm"
Anonymous
6623 Posts
Re: How to Run the same LogParser command for all the log files in a directory?
Feb 09, 2004 04:43 PM|LINK
Whew! I am so glad! I hope we can create a knowledge base and add all of these great queries in there! I'd be happy to contribute. Thanks again!