I use the scrapy library for scraping and it handels all the logging for me. The log file is very big and I was hoping for another way than to read through the whole file. I'm only interested in the exception.
Is it posible to get the raw logging.exception and read from that without creating another log file?
Use the python logging facility which is what Scrapy uses. At the program entry point you need to customize this logging facility, it can be in the __init__.py file for example or anything else before you use any log related calls.
What you need to do is set different handlers for different levels (ERROR, WARNING, ...). Exception level do not exist, you might be mistaking with Error or Critical levels.
This is the "Logging How To", you should check it.
Your solution would look like :
import logging
#this is the logger Scrapy uses throughout the app.
logger = logging.getLogger()
my_exception_handler = logging.FileHandler()
my_exception_handler.setLevel(Logging.ERROR)
logger.addHandler(my_exception_handler)
Finally, if you are really talking about exception and not error or critical levels. You'll need to inherit from your Exception class or overwrite it so you can use the same logging approach I just explained.
emit() method, which would call pymongo/mongodb. The process stays the same as stated above. Please accept my answer if this has fix your problem.