Search results matching tag 'NCSA input format'

Parse Errors in NSCA format -- can I do anything?

msdn_dave — Tue, 10 Nov 2009 18:01:17 GMT

The following line causes parse errors ("Log row terminates unexpectedly.") when the input format is NCSA. (IPs altered to protect the guilty.)

299.99.45.146 - - [25/Oct/2009:01:39:44 -0700] "GET /images/expert/spacer.gif\" HTTP/1.1" 404 11124 "http://www.website.com/lms/agentDashboard.page" "Akamai Edge"

I'm pretty sure the problem is the forward slash after spacer.gif.

I'm assuming there is nothing I can do, short of pre-parsing and editing the error out, right?

Also, I had to go to quite a length to discover the problem. I used the -e:9999 tag, but that simply give me line numbers (in some very large files, hard to deal with.) Is there a way to actually output the troublesome lines in the future to STDERR somehow?

Thanks,

RE: Replacing URL (request) with Friendly Names??

LogParser User : Greg C — Thu, 31 May 2007 16:03:00 GMT

FYI: Found the answer on SQL forums - This works...:

select remotehostname, datetime,
replace(
replace(
replace(
replace(
replace(
replace(
replace(
replace(request,'GET /portal/site/marketintel/menuitem.6dc3d475ebbc54fca120231020248a0c/ HTTP/1.1','Home')
, 'GET /portal/site/marketintel/menuitem.45dc4e7d2e1e54fca120231020248a0c/ HTTP/1.1','News')
, 'GET /portal/site/marketintel/menuitem.d6831dc3617f54fca120231020248a0c/ HTTP/1.1','Research')
, 'GET /portal/site/marketintel/menuitem.13d8b98ff8e164fca120231020248a0c/ HTTP/1.1','Case Studies')
, 'GET /portal/site/marketintel/template.PAGE/menuitem.45dc4e7d2e1e54fca120231020248a0c/?addCatFilter=','News -Filter:')
, 'GET /portal/site/marketintel/template.PAGE/menuitem.13d8b98ff8e164fca120231020248a0c/?addCatFilter=','Case Studies-Filter:')
, 'GET /portal/site/marketintel/template.PAGE/menuitem.d6831dc3617f54fca120231020248a0c/?addCatFilter=','Research-Filter:')
, '&removeCatFilter=&moreCatList=&removeSearchText=&pastDays=10000&goingBack=10000&sources=0&industries=0 HTTP/1.1','')
, statuscode, bytessent
from.....

Replacing URL (request) with Friendly Names??

LogParser User : Greg C — Wed, 30 May 2007 17:38:00 GMT

Hi,

I'm using LogParser to import an Apache web server log into an MS SQL server db. I can't figure out how to clean up the URL's at the same time...

My command:
LogParser "SELECT RemoteHostName, DateTime, Request, StatusCode, BytesSent INTO MyTable FROM D:\IBMHTTPServer6\logs\access.log" -i:NCSA -o:SQL -server:MyServer -database:MyDB -driver:"SQL Server" -username:xxxx -password:xxxx -createTableN -iCheckPoint:\web\weblog.lpc

The input referer field looks like:

GET /portal/site/marketintel/menuitem.13d8b98ff8e164fca120231020248a0c/ HTTP/1.1
GET /portal/site/marketintel/menuitem.d6831dc3617f54fca120231020248a0c/ HTTP/1.1
GET /portal/site/marketintel/menuitem.45dc4e7d2e1e54fca120231020248a0c/ HTTP/1.1
GET /portal/site/marketintel/menuitem.6dc3d475ebbc54fca120231020248a0c/ HTTP/1.1

I want to replace the first (...13d8b...) with "Home" and the next (...d683...) with "News Page", so that the referer field in the DB looks like:

Home
News Page
Reseach Page
About

I figured out how to use REPLACE_STR but only for one 'url/page'...

Thanks!

Greg

RE: NCSA Input format and the same as Output?

LogParser User : NickSoft — Thu, 16 Nov 2006 08:07:00 GMT

Hi

I did it....everything now is clear and the Logfile did it the way I want.

For information the way I did it

The sql-file:

select RemoteHostName, UserName, DateTime, Request, StatusCode, BytesSent into oktober.log from wiseaccess311006.log WHERE DateTime > TO_TIMESTAMP('2006-10-15 00:00:00', 'yyyy-MM-dd hh:mm:ss')

The Command Line:

LogParser -i:NCSA -o:TSV file:selall.sql

This is the way I wanna do it

Many Thanks for help Daniel !!!

Greetz and cu soon

NickSoft

RE: NCSA Input format and the same as Output?

LogParser User : DEinspanjer — Tue, 14 Nov 2006 14:13:00 GMT

Oops, NAT doesn't support multiplexing.
Try using the TSV output instead. As long as you are only outputting one field, it shouldn't put any delimiters in and you should be alright.

Are you sure you used the select statement I suggested instead of a *?

SELECT FileNum, Text USING ...

RE: NCSA Input format and the same as Output?

LogParser User : NickSoft — Tue, 14 Nov 2006 07:43:00 GMT

Sounds good but doesn´t work

Well I just try it and the only think is that LP is wrote a number in front of Line. The Line looks like this now:
1 C:\Programme\Log Parser 2.2\z2.log, 18547, 142.160.44.19, -, Anonymous, 11/14/2006, 5:45:01, GET /UNIFIL/gpers_img HTTP/1.0, 200, 479, http://www.flottenkdo.mar/UNIFIL, Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0), -,

I try it also in another input formats.

Anyone another idea?

I just need the part after and incl. "Anonymous".

Thx and Greetz

RE: NCSA Input format and the same as Output?

LogParser User : DEinspanjer — Mon, 13 Nov 2006 17:34:00 GMT

If you use SELECT *, LP is going to output the filename and row number in addition to the other fields defined as NCSA.

Try using this query and see if it works for you:

logparser -i:TEXTLINE -o:NAT -headersFF "SELECT FileNum, Text USING ADD(1, DIV(Index, 1000)) AS FileNum INTO outfile-*.log FROM infile.log"

RE: NCSA Input format and the same as Output?

LogParser User : NickSoft — Mon, 13 Nov 2006 08:40:00 GMT

Hi me again

I think I´ve found the problem why ma Web-Analyser isn´t able do evaluate the Information.
It doesn´t matter which Outputformat I choose the LogParser always write something at the beginning of each line.
The things he wrote in front of each line are:

C:\Programme\Log Parser 2.2\wiseaccess311006.log 7479960

The number seems to be an counter the number is rising.

Is there an possibility that the LogParser didn´t write this Line??

Then I´m thinking this is se easiest way to make it understandable for my Web-Analyser.

Thx and Greetz

Nicki

RE: NCSA Input format and the same as Output?

LogParser User : NickSoft — Mon, 13 Nov 2006 07:10:00 GMT

Thx this sounds good

Sometimes is it better then see it from another view....the new problem how I will give only the textline out? The Input data in textline is no problem also the text comparison but in which format I can make the output....the output formats are already the same then Input in NCSA.
Or is there any possibility to put the line as textformat in an file out?

For sure there is an posibility...but im too stupid to find it o_O

Well I will search for it today and will report in but if you have an solution I will be happy.

Thx and Greetz

Nicki

RE: NCSA Input format and the same as Output?

LogParser User : DEinspanjer — Fri, 10 Nov 2006 15:53:00 GMT

If you just want to cut the file into smaller chunks so your web-analyzer is able to understand them then I wouldn't even bother with LP. There are plenty of programs that can chunk data like that or you could just write a simple Perl/Python/Ruby/Bash script to do it.

If you want to use LP, I'd suggest actually treating the files as TEXTLINE instead of NCSA. Use string parsing to do the timestamp comparison you want to do and just write out the Text field to the new file. This will be easier than trying to craft a new output format and it will also let you use the multiplexing feature if you want to automatically break the big files into month or week or daily chunks. Good luck!