Parse Apache log with Python 2.7

Question

I'm trying to read a log file from a github url, add some geographic info using the IP as a lookup key, and then write some log info and the geographic info to a file. I've got the reading from and writing to file from the log, but I'm not sure what lib to use for looking up coordinates and such from an IP address, nor how to really go about this part. I found the regex module, and by the time I started to understand it, I found out it's deprecated. Here's what I've, got, any help would be great.

import urllib2 
apacheLog = 'https://raw.githubusercontent.com/myAccessLog.log'

data = urllib2.urlopen(apacheLog)
for line in data:
    with open('C:\LogCopy.txt','a') as f:
        f.write(line)

So, you are now trying to parse 'C:\LogCopy.txt'? Show what you have tried. — Dr. Jan-Philip Gehrcke
– Dr. Jan-Philip Gehrcke, Commented Feb 7, 2015 at 19:27
I'm writing to C:\LogCopy.txt from the file on github. The manipulation will happen before I write to LogCopy. I don't know what to use to break the lines up, besides some messy slicing, maybe. It looks like file is in Common Log Format, and I think I can use %x to pull pieces out, but I don't know if that is just for use with regex or what. I'm just not sure where to start. I'm not asking for the answer, just a push in the right direction. — RagePwn
– RagePwn, Commented Feb 7, 2015 at 19:39
without knowing what output you expect it is pretty had to give any reasonable answer, there is a re module you can use. — Padraic Cunningham
– Padraic Cunningham, Commented Feb 7, 2015 at 19:47

Community · Accepted Answer · 2017-05-23 12:06:24Z

1

The re module isn't deprecated, and is part of the standard library. Edit: here's the link for the 2.7 module
Your for loop is opening and closing the file at each iteration. Probably not a big deal but it might be faster for large files to open the file once and write what needs to be written. Just swap the locations of the for and with lines.

So

data = urllib2.urlopen(apacheLog)
for line in data:
    with open('C:\LogCopy.txt','a') as f: # probably need a double backslash
        f.write(line)

becomes

data = urllib2.urlopen(apacheLog)
with open('C:\LogCopy.txt','a') as f: # probably need a double backslash
    for line in data.splitlines():
        f.write(line) # might need a newline character
        # f.write(line + '\n')

Similar question regarding geolocation Python library

Best of luck!

Edit: added the data.splitlines() call after reading Piotr Kempa's answer

edited May 23, 2017 at 12:06

CommunityBot

11 silver badge

answered Feb 7, 2015 at 19:48

Andy Kubiak

1696 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Community · Accepted Answer · 2017-05-23 11:50:50Z

1

Well the first part is simple. Just use for line in data.split('\n') assuming the lines end with a normal newline (they should).

Then you use the re module (import re) - I hope it was still in use in python 2.7... You can extract the IP address with something like re.search(r"\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}", line), look up the re.search() function for details how to use it.

As for locating the IP geographically, it was already asked I think, try this question: What python libraries can tell me approximate location and time zone given an IP address?

edited May 23, 2017 at 11:50

CommunityBot

11 silver badge

answered Feb 7, 2015 at 19:50

Piotr Kempa

4022 silver badges6 bronze badges

1 Comment

Piotr Kempa Over a year ago

Oops, we posted two similar answers :) The part about moving the open() outside of the loop is a great suggestion in the other answer, you should follow it too!

Collectives™ on Stack Overflow

Parse Apache log with Python 2.7

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related