I think I've found a solution to my problem. I'm very close to nailing it. :)
In another thread ( http://forums.iis.net/t/1150819.aspx ) I tried the RegEx solution using Log Parser Lizard. I played with the application a bit until I deviated from what the author suggested.
Here what I did
1.) I prepared my config file and called it mylogconfig.xml
<?xml version="1.0" encoding="UTF-8"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="C:\src\LogParserCSWebServiceInputFormat\LogParserRegexInputFormat.xsd">
<regex>^(?<DateTime>(?:\d{2})\s\w{3}\s\d{4}\s\d{2}:\d{2}:\d{2},\d{3})\s+\[Current\sTime=(?<CurrentTime>.*?)\]\s+\[Process\sName=(?<ProcessName>.*?)\]\s+\[Root\sJob\sId=(?<RootJobId>.*?)\]\s+\[Job\sId=(?<JobId>.*?)\]\s+(?<Message>.*)$</regex>
<fields>
<field name="DateTime" type="Timestamp" format="dd MMM yyyy HH:mm:ss,fff" />
<field name="CurrentTime" type="String"/>
<field name="ProcessName" type="String"/>
<field name="RootJobId" type="String"/>
<field name="JobId" type="String"/>
<field name="Message" type="String"/>
</fields>
</config>
2) in Log Parser Lizard, I created a simple query.
3) from the drop down selection box for Input Format, (here's the killer) instead of using RegEx Input Format, I used Log4Net Input Format.
4) There's a query properties icon next to the Input Format drop down, so I clicked it. Double clicked on ConfigFile and typed in "mylogconfig.xml"
5). I hit on generate and lo and behold my custom log is parsed!
Now, my ReqEx above for "Process name" and "Job Id" will not handle nested "[ ]" but for some magical reasons, Log4Net Input Format did it at least of Job Id.
Another thing though is I have a Full Message column, instead of just Message. I dont mind this title as long as the log message is there, I'm ok with it.
And there's an extra column called "Exception" at the very end. It's empty but it doesn't bother me. :)
This is good progress.
I just need some interpretation on what happened to mylogconfig.xml - it's party used and the regex seem to work.
Any ideas?