« Previous Next »

Answered Thread: LogParser through Windows Service

Last post 10-27-2009 3:36 PM by phoenix522. 12 replies.

Average Rating Rate It (5)

RSS

Page 1 of 1 (13 items)

Sort Posts:

  • 04-07-2008, 12:38 AM

    • Ya Ya
    • Not Ranked
    • Joined on 06-02-2006, 1:15 AM
    • Posts 3

    LogParser through Windows Service

    Hi all, I am working on small project using C# (2005) to use LogParser to query a particular server to grab it's log and transfer the log to SQL2005.

    I am able to do so if I access the LogParser through IE (through interface like letting the user select the server, date/time to parse etc). But when I transfer the entire script into Windows Service, I cannot get anything at all.

    Below is part of the codes that I have. For the Windows Service part, if I were to comment off the line LogRecordSet oRecordSet = oLogQuery.Execute( query, oEVTInputFormat ); it works (as in if I were to .write the query string out, I am able to) but when I put that line in, nothing happens although the service is started. Hope some experts here can help me! Thanks in advance!

    LogQuery oLogQuery = new LogQuery();
    EventLogInputFormat oEVTInputFormat = new EventLogInputFormat();
    oEVTInputFormat.direction = "BW";
    string query = @"SELECT * FROM ";
    if ( _ip != null )
    {
    if ( _ip != "local" )
    {
    query = query + "\\\\" + _ip + "\\SYSTEM";
    }
    }
    query = query + " WHERE (1 = 1)";
    if ( _lastmessageon != null )
    {
    query = query + " AND (TO_DATE(TimeGenerated) > TO_DATE(TIMESTAMP( '" + Common.MyFunctions.FormatDateTime( _lastmessageon, "yyyy-MM-dd hh:mm:ss" ) + "', 'yyyy-MM-dd hh:mm:ss')))";
    }
    LogRecordSet oRecordSet = oLogQuery.Execute( query, oEVTInputFormat );

    .... (codes to insert into SQL2005)


    oRecordSet.close();

     

  • 04-08-2008, 3:26 PM In reply to

    Answered Re: LogParser through Windows Service

    You need to configure your service to run as an account that has access to the remote server. This can be an AD account or if you create an account with the same name and password on all of the servers then that will work too.

  • 04-08-2008, 8:23 PM In reply to

    • Ya Ya
    • Not Ranked
    • Joined on 06-02-2006, 1:15 AM
    • Posts 3

    Re: LogParser through Windows Service

    My system does has access to the remote server. As mentioned, when I tested the exact same code as an application, it works, but when I transferred it to windows service, it stops at the execute line without throwing any errors.

  • 04-11-2008, 6:00 AM In reply to

    Re: LogParser through Windows Service

    Hi Ya Ya,

    Although your system has access to the remote server, it doesn't mean windows service has sufficient permission to the remote server. By default, windows service is running with minimal user rights. Grant an account with the user rights that your code may require. You can find some description here:

    http://support.microsoft.com/kb/821794

     

    Zhao Ji Ma
    Sincerely,
    Microsoft Online Community Support

    “Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. ”
  • 04-15-2008, 8:39 PM In reply to

    • Ya Ya
    • Not Ranked
    • Joined on 06-02-2006, 1:15 AM
    • Posts 3

    Re: LogParser through Windows Service

    oh.... ok.... thanks but how do i go about doing it?

    Does that mean that I have to create a specific account in every server which my service need to LP?

  • 10-14-2009, 10:50 AM In reply to

    • Luke101
    • Not Ranked
    • Joined on 09-10-2006, 4:36 PM
    • Posts 3

    Re: LogParser through Windows Service

    Can someone please answer this question. I am having the same issue. The log parser will not work in a service but it works in a regular web page

    incomehunting.com
  • 10-14-2009, 10:58 AM In reply to

    Re: LogParser through Windows Service

    "not work".. what specifically does that mean? How do you know that it did not work? How is the service configured? Why do you want to run it as a service? Please provide details and I will try to help.

  • 10-14-2009, 11:13 AM In reply to

    • Luke101
    • Not Ranked
    • Joined on 09-10-2006, 4:36 PM
    • Posts 3

    Re: LogParser through Windows Service

    I am having the same exact issue as the original post. I am running Windows Vista (Home Premium) and i am sucking all of my IIS log information in sql server then i am processing it. In the serviceprocessinstaller I have the "Local system" account attached to the service. The service is suppose to update the database every 10 seconds but obviously it is not updating it. When I run the code in a web page it works fine. But when I insert the code in a service it does not work

     

     

    string sqlconn = @"Server=luke-pc\SQLEXPRESS;Database=test2;User ID=test;Password=test;";
    
                // Create a DataTable. 
                DataTable table = new DataTable();
    
                // Create a DataColumn and set various properties. 
                DataColumn column;
                /*
                DataColumn column = new DataColumn();
                column.DataType = System.Type.GetType("System.String");
                column.AllowDBNull = false;
                column.ColumnName = "LogFilename";
                table.Columns.Add(column);
    
                column = new DataColumn();
                column.DataType = System.Type.GetType("System.Int32");
                column.AllowDBNull = false;
                column.ColumnName = "RowNumber";
                table.Columns.Add(column);
                 
    
                column = new DataColumn();
                column.DataType = System.Type.GetType("System.DateTime");
                column.AllowDBNull = false;
                column.ColumnName = "date";
                table.Columns.Add(column);
                */
    
                column = new DataColumn();
                column.DataType = System.Type.GetType("System.DateTime");
                column.AllowDBNull = true;
                column.ColumnName = "time";
                table.Columns.Add(column);
    
                column = new DataColumn();
                column.DataType = System.Type.GetType("System.Int32");
                column.AllowDBNull = true;
                column.ColumnName = "count";
                table.Columns.Add(column);
    
                /*
               column = new DataColumn();
               column.DataType = System.Type.GetType("System.String");
               column.AllowDBNull = false;
               column.ColumnName = "s-ip";
               table.Columns.Add(column);
    
               
               column = new DataColumn();
               column.DataType = System.Type.GetType("System.String");
               column.AllowDBNull = false;
               column.ColumnName = "cs-method";
               table.Columns.Add(column);
                */
    
                column = new DataColumn();
                column.DataType = System.Type.GetType("System.String");
                column.AllowDBNull = true;
                column.ColumnName = "cs-uri-stem";
                table.Columns.Add(column);
    
                /*
               column = new DataColumn();
               column.DataType = System.Type.GetType("System.String");
               column.AllowDBNull = true;
               column.ColumnName = "cs-uri-query";
               table.Columns.Add(column);
                
    
               column = new DataColumn();
               column.DataType = System.Type.GetType("System.String");
               column.AllowDBNull = true;
               column.ColumnName = "s-port";
               table.Columns.Add(column);
                
               column = new DataColumn();
               column.DataType = System.Type.GetType("System.String");
               column.AllowDBNull = true;
               column.ColumnName = "cs-username";
               table.Columns.Add(column);
               */
                column = new DataColumn();
                column.DataType = System.Type.GetType("System.String");
                column.AllowDBNull = true;
                column.ColumnName = "c-ip";
                table.Columns.Add(column);
    
                /*
                column = new DataColumn();
                column.DataType = System.Type.GetType("System.String");
                column.AllowDBNull = true;
                column.ColumnName = "cs(User-Agent)";
                table.Columns.Add(column);
                 
    
                column = new DataColumn();
                column.DataType = System.Type.GetType("System.String");
                column.AllowDBNull = true;
                column.ColumnName = "sc-status";
                table.Columns.Add(column);
    
                column = new DataColumn();
                column.DataType = System.Type.GetType("System.String");
                column.AllowDBNull = true;
                column.ColumnName = "sc-substatus";
                table.Columns.Add(column);
                
    
                column = new DataColumn();
                column.DataType = System.Type.GetType("System.String");
                column.AllowDBNull = true;
                column.ColumnName = "sc-win32-status";
                table.Columns.Add(column);
    
                column = new DataColumn();
                column.DataType = System.Type.GetType("System.String");
                column.AllowDBNull = true;
                column.ColumnName = "time-taken";
                table.Columns.Add(column);
                 */
    
                column = new DataColumn();
                column.DataType = System.Type.GetType("System.Int32");
                column.AllowDBNull = true;
                column.ColumnName = "questionid";
                table.Columns.Add(column);
    
                //get all search eninge ips
                List IPAddressesToAvoid = new List();
    
                string queryq = string.Format("select ip from SearchEngineIPs UNION select ip from BotIPs");
                using (SqlConnection conn = new SqlConnection(sqlconn))
                {
                    using (SqlCommand cmd = new SqlCommand(queryq, conn))
                    {
                        conn.Open();
                        SqlDataReader r = cmd.ExecuteReader();
    
                        while (r.Read())
                            IPAddressesToAvoid.Add("'" + r.GetString(0) + "'");
    
                        if (r != null)
                        {
                            r.Close();
                        }
    
                        conn.Close();
                    }
                }
    
                //lets join the ip addresses
                string ListofIPstoAvoid = string.Join(";", IPAddressesToAvoid.ToArray());
    
    
    
    
    
                // prepare LogParser Recordset & Record objects
                ILogRecordset rsLP = null;
                ILogRecord row = null;
    
                LogQueryClassClass LogParser = null;
                COMW3CInputContextClassClass W3Clog = null;
    
                string strSQL = null;
    
                LogParser = new LogQueryClassClass();
                W3Clog = new COMW3CInputContextClassClass();
    
    
                strSQL = @"select * from C:\inetpub\logs\LogFiles\W3SVC1\*.log where date = '" + DateTime.Now.ToString("yyyy-MM-dd") + "' and sc-status = 200 " +
                            "and cs-method = 'GET' and cs-uri-stem LIKE '/q/%' and cs-username IS NULL " +
                            "and cs-uri-stem not like '%.png' and  cs-uri-stem not like '%.jpg' and cs-uri-stem not like '%.gif' and cs-uri-stem not like '%.css' and cs-uri-stem not like '%.js' " +
                            "and cs(User-Agent) not like '%bot%' and cs(User-Agent) not like '%slurp%' and cs(User-Agent) not like '%crawl%' " +
                            "and c-ip NOT IN (" + ListofIPstoAvoid + ")";
    
                // run the query against W3C log
                System.Diagnostics.Debugger.Launch();
                rsLP = LogParser.Execute(strSQL, W3Clog);
                
                DataRow dr;
                while (!rsLP.atEnd())
                {
                    dr = table.NewRow();
                    row = rsLP.getRecord();
                    string[] urlarr = row.getValue("cs-uri-stem").ToString().Split('/');
    
                    dr["time"] = (DateTime)row.getValue("time");
                    dr["cs-uri-stem"] = row.getValue("cs-uri-stem").ToString();
                    dr["c-ip"] = row.getValue("c-ip").ToString();
                    dr["questionid"] = int.Parse(urlarr[3].ToString());
    
                    table.Rows.Add(dr);
                    rsLP.moveNext();
                }
    
                //get unique users for each question
                DataTable dt = table.Clone();
                var query = from row1 in table.AsEnumerable()
                            let time = row1.Field("time")
                            let uri = row1.Field("cs-uri-stem")
                            let ip = row1.Field("c-ip")
                            let questionid = row1.Field("questionid")
                            select new
                            {
                                time,
                                uri,
                                ip,
                                questionid
                            };
    
                //get all distinct question IDs from query vairable
                var query2 = from row2 in query
                             group row2 by row2.questionid into g
                             select new
                             {
                                 time = g.Select(x => x.time).FirstOrDefault(),
                                 theip = g.Select(x => x.ip).FirstOrDefault(),
                                 theurl = g.Select(x => x.uri).FirstOrDefault(),
                                 question1 = g.Key
                             };
    
                //get all question information from distinct information
                var query3 = from row3 in query2
                             select new
                             {
                                 question2 = row3.question1,
                                 questionInfo = (from row3Sub in query
                                                 where row3Sub.questionid == row3.question1
                                                 group row3Sub by row3Sub.ip into g
                                                 select new
                                                 {
                                                     time = g.Select(x => x.time).FirstOrDefault(),
                                                     theip = g.Key,
                                                     theurl = g.Select(x => x.uri).FirstOrDefault(),
                                                     question3 = g.Select(x => x.questionid).FirstOrDefault()
    
                                                 })
    
                             };
    
                var query4 = from row4 in query3
                             select new
                             {
                                 question4 = row4.question2,
                                 count2 = row4.questionInfo.Count()
    
                             };
    
                //get rid of the DataTable
                table.Dispose();
    
                StringBuilder sb = new StringBuilder();
                DateTime CurrDate = DateTime.Now;
                foreach (var row4 in query4)
                {
                    sb.Append("INSERT INTO qstatistics (qstatistics_questionid, qstatistics_questionid_uniquehits, qstatistics_date) VALUES (");
                    sb.Append(row4.question4);
                    sb.Append(",");
                    sb.Append(row4.count2);
                    sb.Append(",'");
                    sb.Append(CurrDate);
                    sb.Append("');");
                }
    
                SqlConnection conn1 = new SqlConnection(sqlconn);
                SqlCommand myCommand = new SqlCommand(sb.ToString(), conn1);
    
                conn1.Open();
                myCommand.ExecuteNonQuery();
                conn1.Close();
    
    incomehunting.com
  • 10-14-2009, 11:33 AM In reply to

    Re: LogParser through Windows Service

    Put a try ...catch around your code and write out the error to the application eventlog. I suspect that it can't find "luke-pc". You can also use Visual Studio and select Tools and then "Attach to a process". You should then be able to set a breakpoint and walk thru your code to look at error variables.

    You could also try @"Server=localhost\SQLEXPRESS  or @"Server=.\SQLEXPRESS

    You need to figure out what error you are getting.

    Just a thought, 10 seconds might work on your desktop, but if you would ever put this on a server that has a lot of activity, that might be too short of an interval.

  • 10-14-2009, 11:45 AM In reply to

    • Luke101
    • Not Ranked
    • Joined on 09-10-2006, 4:36 PM
    • Posts 3

    Re: LogParser through Windows Service

    Oh yea..I have already tested the connection to the database. I can insert data in the database from the service but the service will not execute this:
    rsLP = LogParser.Execute(strSQL, W3Clog);

    Any ideas?

    incomehunting.com
  • 10-14-2009, 11:56 AM In reply to

    Re: LogParser through Windows Service

    Do you get any return code or exception?

    Try using Process Monitor to trace the service. http://technet.microsoft.com/en-us/sysinternals/default.aspx

    Look for "access denied" or "not found" conditions.

  • 10-15-2009, 10:10 PM In reply to

    Re: LogParser through Windows Service

    I agree with Madness80. Try using ProcMon to see what account is trying to access your server. For the record, I have a windows services doing exactly what you are asking for and it works (with a few minor steps in between like ungzipping old archive files)...but I have it running under a higher account.

    Check out my blog for other cool tips and tricks:

    http://joelangley.blogspot.com/
  • 10-27-2009, 3:36 PM In reply to

    Re: LogParser through Windows Service

    With a Windows Service,  you need to do two things. You need to open the service properties and enter a domain account and password to get the service to connect to the remote system and two, that same domain account has to have log on locally rights for the server you are running it on.

    If it is doing SQL queries then it also needs read and/or write to the database being used.

     

    Regards,

    Ron

Page 1 of 1 (13 items)
Microsoft Communities