📘 Logging: Basic Configuration

🎯 Introduction

Welcome to this exciting tutorial on Python logging configuration! 🎉 In this guide, we’ll explore how to properly set up and configure Python’s logging system to track what happens in your applications.

You’ll discover how logging can transform your debugging experience. Whether you’re building web applications 🌐, automation scripts 🤖, or data pipelines 📊, understanding logging configuration is essential for monitoring and troubleshooting your code.

By the end of this tutorial, you’ll feel confident setting up professional-grade logging in your Python projects! Let’s dive in! 🏊‍♂️

📚 Understanding Logging Configuration

🤔 What is Logging Configuration?

Logging configuration is like setting up a security camera system for your code 📹. Think of it as creating a detailed diary 📔 that automatically records what your program does, when it does it, and what went wrong if something breaks!

In Python terms, logging configuration means setting up:

• ✨ Where to write log messages (files, console, servers)
• 🚀 What information to include (timestamps, severity levels)
• 🛡️ Which messages to keep or ignore (filtering by importance)

💡 Why Configure Logging?

Here’s why developers love proper logging configuration:

1. Debugging Made Easy 🔍: Find bugs faster with detailed trace logs
2. Production Monitoring 📊: Track application health in real-time
3. Audit Trails 📝: Keep records of important operations
4. Performance Analysis ⚡: Identify bottlenecks and slow operations

Real-world example: Imagine running an online store 🛒. With logging, you can track every order, payment, and error - making troubleshooting a breeze!

🔧 Basic Syntax and Usage

📝 Simple Configuration

Let’s start with a friendly example:

import logging

# 👋 Hello, Logging!
logging.basicConfig(
    level=logging.INFO,  # 📊 Set minimum severity level
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'  # 🎨 Format template
)

# 🎯 Create a logger
logger = logging.getLogger('my_app')

# 📝 Log some messages
logger.info("Application started! 🚀")
logger.warning("This is a warning ⚠️")
logger.error("Oops, something went wrong! 😱")

💡 Explanation: The basicConfig() sets up logging for your entire application. The format string defines how each log message looks!

🎯 Common Configuration Patterns

Here are patterns you’ll use daily:

import logging
from logging.handlers import RotatingFileHandler
import sys

# 🏗️ Pattern 1: Console + File logging
def setup_logging():
    # 📝 Create formatter
    formatter = logging.Formatter(
        '%(asctime)s | %(name)s | %(levelname)s | %(message)s'
    )
    
    # 💻 Console handler
    console_handler = logging.StreamHandler(sys.stdout)
    console_handler.setLevel(logging.INFO)
    console_handler.setFormatter(formatter)
    
    # 📁 File handler (with rotation)
    file_handler = RotatingFileHandler(
        'app.log',
        maxBytes=10485760,  # 10MB
        backupCount=5
    )
    file_handler.setLevel(logging.DEBUG)
    file_handler.setFormatter(formatter)
    
    # 🎯 Configure root logger
    root_logger = logging.getLogger()
    root_logger.setLevel(logging.DEBUG)
    root_logger.addHandler(console_handler)
    root_logger.addHandler(file_handler)
    
    return root_logger

# 🎨 Pattern 2: Different loggers for modules
auth_logger = logging.getLogger('auth')
db_logger = logging.getLogger('database')
api_logger = logging.getLogger('api')

# 🔄 Pattern 3: Log levels
def process_data(data):
    logger = logging.getLogger(__name__)
    logger.debug(f"Processing {len(data)} items 🔍")
    logger.info(f"Data processing started 🚀")
    
    try:
        # Process data...
        logger.info("Data processed successfully! ✅")
    except Exception as e:
        logger.error(f"Processing failed: {e} ❌")

💡 Practical Examples

🛒 Example 1: E-commerce Application Logger

Let’s build a real-world logging system:

import logging
import json
from datetime import datetime
from typing import Dict, Any

# 🛍️ Configure e-commerce logger
class EcommerceLogger:
    def __init__(self, app_name: str = "ecommerce"):
        self.app_name = app_name
        self.logger = self._setup_logger()
    
    def _setup_logger(self) -> logging.Logger:
        # 🎨 Create custom formatter
        formatter = logging.Formatter(
            '%(asctime)s | %(name)s | %(levelname)s | %(funcName)s | %(message)s'
        )
        
        # 📁 Orders log file
        order_handler = logging.FileHandler('orders.log')
        order_handler.setLevel(logging.INFO)
        order_handler.setFormatter(formatter)
        
        # 🚨 Error log file
        error_handler = logging.FileHandler('errors.log')
        error_handler.setLevel(logging.ERROR)
        error_handler.setFormatter(formatter)
        
        # 💻 Console output
        console_handler = logging.StreamHandler()
        console_handler.setLevel(logging.DEBUG)
        console_handler.setFormatter(formatter)
        
        # 🔧 Configure logger
        logger = logging.getLogger(self.app_name)
        logger.setLevel(logging.DEBUG)
        logger.addHandler(order_handler)
        logger.addHandler(error_handler)
        logger.addHandler(console_handler)
        
        return logger
    
    # 🛒 Log order events
    def log_order(self, order_id: str, user_id: str, amount: float, items: list):
        order_data = {
            "order_id": order_id,
            "user_id": user_id,
            "amount": amount,
            "items": len(items),
            "timestamp": datetime.now().isoformat()
        }
        self.logger.info(f"📦 New order placed: {json.dumps(order_data)}")
    
    # 💳 Log payment events
    def log_payment(self, order_id: str, status: str, method: str):
        if status == "success":
            self.logger.info(f"💰 Payment successful for order {order_id} via {method}")
        else:
            self.logger.error(f"❌ Payment failed for order {order_id}: {status}")
    
    # 📊 Log inventory updates
    def log_inventory(self, product_id: str, action: str, quantity: int):
        self.logger.debug(f"📊 Inventory {action}: {product_id} x{quantity}")

# 🎮 Let's use it!
ecom_logger = EcommerceLogger()

# Simulate order flow
ecom_logger.log_order("ORD-001", "USER-123", 99.99, ["iPhone", "Case"])
ecom_logger.log_payment("ORD-001", "success", "credit_card")
ecom_logger.log_inventory("PROD-456", "decreased", 1)

🎯 Try it yourself: Add a method to log shipping updates and customer service interactions!

🎮 Example 2: Game Server Logger

Let’s make logging fun with a game server:

import logging
import logging.config
import yaml
from enum import Enum
from typing import Optional

# 🏆 Game event types
class GameEvent(Enum):
    PLAYER_JOIN = "player_join"
    PLAYER_LEAVE = "player_leave"
    LEVEL_UP = "level_up"
    ACHIEVEMENT = "achievement"
    BATTLE = "battle"
    ERROR = "error"

# 📝 Logger configuration from YAML
LOGGING_CONFIG = """
version: 1
disable_existing_loggers: false

formatters:
  detailed:
    format: '%(asctime)s | %(name)s | %(levelname)s | %(message)s'
  simple:
    format: '%(levelname)s | %(message)s'

handlers:
  console:
    class: logging.StreamHandler
    level: INFO
    formatter: simple
    stream: ext://sys.stdout
  
  game_file:
    class: logging.handlers.RotatingFileHandler
    level: DEBUG
    formatter: detailed
    filename: game_server.log
    maxBytes: 10485760
    backupCount: 3
  
  player_file:
    class: logging.FileHandler
    level: INFO
    formatter: detailed
    filename: player_activity.log

loggers:
  game_server:
    level: DEBUG
    handlers: [console, game_file]
    propagate: false
  
  player_tracker:
    level: INFO
    handlers: [player_file]
    propagate: false

root:
  level: WARNING
  handlers: [console]
"""

# 🎮 Game Server Logger
class GameServerLogger:
    def __init__(self):
        # Load configuration
        config = yaml.safe_load(LOGGING_CONFIG)
        logging.config.dictConfig(config)
        
        self.server_logger = logging.getLogger('game_server')
        self.player_logger = logging.getLogger('player_tracker')
    
    # 🎯 Log player events
    def log_player_event(self, player_id: str, event: GameEvent, details: Optional[Dict] = None):
        emoji_map = {
            GameEvent.PLAYER_JOIN: "🎮",
            GameEvent.PLAYER_LEAVE: "👋",
            GameEvent.LEVEL_UP: "🎉",
            GameEvent.ACHIEVEMENT: "🏆",
            GameEvent.BATTLE: "⚔️",
            GameEvent.ERROR: "❌"
        }
        
        emoji = emoji_map.get(event, "📝")
        message = f"{emoji} Player {player_id}: {event.value}"
        
        if details:
            message += f" - {details}"
        
        if event == GameEvent.ERROR:
            self.server_logger.error(message)
        else:
            self.server_logger.info(message)
            self.player_logger.info(message)
    
    # 📈 Log server stats
    def log_server_stats(self, players_online: int, cpu_usage: float, memory_usage: float):
        self.server_logger.info(
            f"📊 Server Stats: {players_online} players | "
            f"CPU: {cpu_usage:.1f}% | RAM: {memory_usage:.1f}%"
        )
    
    # 🎊 Log special events
    def log_special_event(self, event_name: str, participants: list):
        self.server_logger.info(
            f"🎊 Special Event '{event_name}' started with "
            f"{len(participants)} participants!"
        )

# 🎮 Test the game logger
game_logger = GameServerLogger()

# Simulate game events
game_logger.log_player_event("Player123", GameEvent.PLAYER_JOIN)
game_logger.log_player_event("Player123", GameEvent.LEVEL_UP, {"new_level": 10})
game_logger.log_player_event("Player123", GameEvent.ACHIEVEMENT, {"name": "Dragon Slayer"})
game_logger.log_server_stats(150, 45.2, 62.8)

🚀 Advanced Concepts

🧙‍♂️ Advanced Topic 1: Custom Log Filters

When you’re ready to level up, try custom filtering:

import logging
import re

# 🎯 Custom filter to mask sensitive data
class SensitiveDataFilter(logging.Filter):
    def __init__(self):
        super().__init__()
        # 🔒 Patterns to mask
        self.patterns = [
            (r'\b\d{4}-\d{4}-\d{4}-\d{4}\b', '****-****-****-****'),  # Credit cards
            (r'\b[\w.]+@[\w.]+\b', '***@***.com'),  # Emails
            (r'password["\']?\s*[:=]\s*["\']?[\w]+', 'password=***')  # Passwords
        ]
    
    def filter(self, record: logging.LogRecord) -> bool:
        # 🛡️ Mask sensitive data
        for pattern, replacement in self.patterns:
            record.msg = re.sub(pattern, replacement, str(record.msg), flags=re.IGNORECASE)
        return True

# 🪄 Using the filter
logger = logging.getLogger('secure_app')
logger.addFilter(SensitiveDataFilter())

# Test it
logger.info("User email: [email protected], card: 1234-5678-9012-3456")
# Output: User email: ***@***.com, card: ****-****-****-****

🏗️ Advanced Topic 2: Structured Logging

For the brave developers - JSON structured logging:

import logging
import json
from pythonjsonlogger import jsonlogger

# 🚀 JSON formatter for structured logs
class CustomJsonFormatter(jsonlogger.JsonFormatter):
    def add_fields(self, log_record, record, message_dict):
        super().add_fields(log_record, record, message_dict)
        # 🎨 Add custom fields
        log_record['timestamp'] = self.formatTime(record)
        log_record['app_name'] = 'my_awesome_app'
        log_record['environment'] = 'production'
        
        # 🏷️ Add emojis for fun!
        severity_emoji = {
            'DEBUG': '🔍',
            'INFO': '📘',
            'WARNING': '⚠️',
            'ERROR': '❌',
            'CRITICAL': '🚨'
        }
        log_record['emoji'] = severity_emoji.get(record.levelname, '📝')

# Configure structured logging
logHandler = logging.StreamHandler()
formatter = CustomJsonFormatter()
logHandler.setFormatter(formatter)

logger = logging.getLogger()
logger.addHandler(logHandler)
logger.setLevel(logging.INFO)

# Use it!
logger.info("User logged in", extra={"user_id": "123", "ip": "192.168.1.1"})

⚠️ Common Pitfalls and Solutions

😱 Pitfall 1: Not Setting Log Levels Properly

# ❌ Wrong way - logging everything at DEBUG level in production!
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger()
logger.debug("Sensitive data: " + str(user_passwords))  # 😰 Too much info!

# ✅ Correct way - use appropriate levels!
logging.basicConfig(level=logging.INFO)  # Production level
logger = logging.getLogger()

# Use different levels appropriately
logger.debug("Detailed trace info")  # Won't show in production
logger.info("User logged in successfully")  # ✅ Important info
logger.warning("Disk space running low")  # ⚠️ Needs attention
logger.error("Database connection failed")  # ❌ Problem occurred

🤯 Pitfall 2: Forgetting to Close File Handlers

# ❌ Dangerous - file handles leak!
def bad_logging():
    handler = logging.FileHandler('app.log')
    logger = logging.getLogger()
    logger.addHandler(handler)
    # Oops, handler never closed! 💥

# ✅ Safe - proper cleanup!
def good_logging():
    handler = logging.FileHandler('app.log')
    logger = logging.getLogger()
    try:
        logger.addHandler(handler)
        # Do logging...
    finally:
        handler.close()  # ✅ Always clean up!
        logger.removeHandler(handler)

# ✅ Even better - use context manager!
import contextlib

@contextlib.contextmanager
def setup_file_logger(filename):
    handler = logging.FileHandler(filename)
    logger = logging.getLogger()
    logger.addHandler(handler)
    try:
        yield logger
    finally:
        handler.close()
        logger.removeHandler(handler)

🛠️ Best Practices

1. 🎯 Use Logger Hierarchy: Create loggers with __name__ for better organization
2. 📝 Log Actionable Information: Include context, not just “Error occurred”
3. 🛡️ Never Log Sensitive Data: Passwords, tokens, credit cards are no-nos!
4. 🎨 Use Consistent Format: Same timestamp and structure across your app
5. ✨ Rotate Log Files: Don’t let logs fill your disk!

# 🏆 Best practice example
import logging
from logging.handlers import RotatingFileHandler, TimedRotatingFileHandler

def setup_production_logging():
    # 📝 Use module name for logger
    logger = logging.getLogger(__name__)
    
    # 🎨 Consistent formatter
    formatter = logging.Formatter(
        '%(asctime)s - %(name)s - %(levelname)s - '
        '%(filename)s:%(lineno)d - %(funcName)s() - %(message)s'
    )
    
    # 📁 Rotating file handler (by size)
    file_handler = RotatingFileHandler(
        'app.log',
        maxBytes=10*1024*1024,  # 10MB
        backupCount=5,
        encoding='utf-8'
    )
    file_handler.setFormatter(formatter)
    file_handler.setLevel(logging.INFO)
    
    # 📅 Time-based rotation
    daily_handler = TimedRotatingFileHandler(
        'daily.log',
        when='midnight',
        interval=1,
        backupCount=30
    )
    daily_handler.setFormatter(formatter)
    
    logger.addHandler(file_handler)
    logger.addHandler(daily_handler)
    
    return logger

🧪 Hands-On Exercise

🎯 Challenge: Build a Web Application Logger

Create a comprehensive logging system for a web application:

📋 Requirements:

• ✅ Different log files for access logs, error logs, and security events
• 🏷️ Include request ID for tracking across logs
• 👤 Log user actions with anonymized user IDs
• 📅 Daily log rotation with compression
• 🎨 Color-coded console output for development

🚀 Bonus Points:

• Add performance logging (response times)
• Implement log aggregation to a central server
• Create a log analysis dashboard

💡 Solution

import logging
import logging.handlers
import uuid
import time
import gzip
import os
from datetime import datetime
from typing import Optional
import colorlog  # pip install colorlog

# 🎯 Custom formatter with request ID
class RequestFormatter(logging.Formatter):
    def format(self, record):
        # Add request ID if available
        if not hasattr(record, 'request_id'):
            record.request_id = 'NO-REQUEST-ID'
        return super().format(record)

# 🎨 Web application logger setup
class WebAppLogger:
    def __init__(self, app_name: str = "webapp"):
        self.app_name = app_name
        self.loggers = self._setup_loggers()
    
    def _setup_loggers(self) -> dict:
        loggers = {}
        
        # 📝 Common formatter
        file_formatter = RequestFormatter(
            '%(asctime)s | %(request_id)s | %(name)s | '
            '%(levelname)s | %(message)s'
        )
        
        # 🎨 Colored console formatter for development
        console_formatter = colorlog.ColoredFormatter(
            '%(log_color)s%(levelname)s%(reset)s | '
            '%(blue)s%(name)s%(reset)s | %(message)s',
            log_colors={
                'DEBUG': 'cyan',
                'INFO': 'green',
                'WARNING': 'yellow',
                'ERROR': 'red',
                'CRITICAL': 'red,bg_white',
            }
        )
        
        # 📁 Access logger
        access_logger = logging.getLogger(f'{self.app_name}.access')
        access_handler = logging.handlers.TimedRotatingFileHandler(
            'logs/access.log',
            when='midnight',
            interval=1,
            backupCount=30
        )
        access_handler.setFormatter(file_formatter)
        access_logger.addHandler(access_handler)
        access_logger.setLevel(logging.INFO)
        loggers['access'] = access_logger
        
        # 🚨 Error logger
        error_logger = logging.getLogger(f'{self.app_name}.error')
        error_handler = logging.handlers.RotatingFileHandler(
            'logs/error.log',
            maxBytes=10*1024*1024,
            backupCount=5
        )
        error_handler.setFormatter(file_formatter)
        error_logger.addHandler(error_handler)
        error_logger.setLevel(logging.ERROR)
        loggers['error'] = error_logger
        
        # 🔒 Security logger
        security_logger = logging.getLogger(f'{self.app_name}.security')
        security_handler = logging.handlers.TimedRotatingFileHandler(
            'logs/security.log',
            when='midnight',
            interval=1,
            backupCount=90  # Keep 3 months
        )
        security_handler.setFormatter(file_formatter)
        security_logger.addHandler(security_handler)
        security_logger.setLevel(logging.WARNING)
        loggers['security'] = security_logger
        
        # 💻 Console handler for all loggers (development)
        console_handler = logging.StreamHandler()
        console_handler.setFormatter(console_formatter)
        console_handler.setLevel(logging.DEBUG)
        
        for logger in loggers.values():
            logger.addHandler(console_handler)
        
        return loggers
    
    # 🌐 Log HTTP requests
    def log_request(self, method: str, path: str, status: int, 
                   duration: float, user_id: Optional[str] = None):
        request_id = str(uuid.uuid4())
        user_id = self._anonymize_user_id(user_id) if user_id else 'anonymous'
        
        message = (f"{method} {path} | Status: {status} | "
                  f"Duration: {duration:.3f}s | User: {user_id}")
        
        extra = {'request_id': request_id}
        
        if status >= 500:
            self.loggers['error'].error(f"🔥 {message}", extra=extra)
        elif status >= 400:
            self.loggers['access'].warning(f"⚠️ {message}", extra=extra)
        else:
            self.loggers['access'].info(f"✅ {message}", extra=extra)
        
        # Log performance if slow
        if duration > 1.0:
            self.loggers['error'].warning(
                f"🐌 Slow request: {duration:.3f}s for {path}", 
                extra=extra
            )
        
        return request_id
    
    # 🔒 Log security events
    def log_security_event(self, event_type: str, user_id: Optional[str], 
                          details: dict, request_id: Optional[str] = None):
        user_id = self._anonymize_user_id(user_id) if user_id else 'anonymous'
        
        security_events = {
            'login_success': ('🔓', logging.INFO),
            'login_failed': ('🚫', logging.WARNING),
            'unauthorized': ('⛔', logging.WARNING),
            'suspicious': ('🚨', logging.ERROR),
        }
        
        emoji, level = security_events.get(event_type, ('🔒', logging.INFO))
        
        message = f"{emoji} Security Event: {event_type} | User: {user_id} | Details: {details}"
        
        extra = {'request_id': request_id or 'NO-REQUEST-ID'}
        self.loggers['security'].log(level, message, extra=extra)
    
    # 🔍 Anonymize user IDs
    def _anonymize_user_id(self, user_id: str) -> str:
        # Simple anonymization - in production, use proper hashing
        if len(user_id) > 4:
            return f"{user_id[:2]}***{user_id[-2:]}"
        return "****"
    
    # 📊 Log performance metrics
    def log_performance(self, metric_name: str, value: float, 
                       unit: str = "ms", request_id: Optional[str] = None):
        message = f"📊 Performance: {metric_name} = {value:.2f}{unit}"
        extra = {'request_id': request_id or 'NO-REQUEST-ID'}
        self.loggers['access'].info(message, extra=extra)

# 🎮 Test the web logger!
if __name__ == "__main__":
    # Create logs directory
    os.makedirs('logs', exist_ok=True)
    
    # Initialize logger
    web_logger = WebAppLogger()
    
    # Simulate web requests
    request_id = web_logger.log_request('GET', '/api/users', 200, 0.125, 'user123')
    web_logger.log_security_event('login_success', 'user123', 
                                 {'ip': '192.168.1.1'}, request_id)
    
    request_id = web_logger.log_request('POST', '/api/orders', 201, 2.5, 'user456')
    web_logger.log_performance('db_query', 2400, 'ms', request_id)
    
    # Simulate errors
    request_id = web_logger.log_request('GET', '/api/invalid', 404, 0.05)
    web_logger.log_request('GET', '/api/crash', 500, 0.001)
    
    # Security events
    web_logger.log_security_event('login_failed', 'hacker999', 
                                 {'ip': '10.0.0.1', 'attempts': 5})
    web_logger.log_security_event('suspicious', None, 
                                 {'reason': 'SQL injection attempt'})

🎓 Key Takeaways

You’ve learned so much! Here’s what you can now do:

• ✅ Configure Python logging with confidence 💪
• ✅ Create custom formatters and handlers for any need 🛡️
• ✅ Implement production-ready logging in real projects 🎯
• ✅ Debug issues faster with proper log levels 🐛
• ✅ Build secure logging systems that protect sensitive data! 🚀

Remember: Good logging is like having a time machine for your code - you can see exactly what happened when! 🕰️

🤝 Next Steps

Congratulations! 🎉 You’ve mastered Python logging configuration!

Here’s what to do next:

1. 💻 Practice with the exercises above
2. 🏗️ Add logging to your existing projects
3. 📚 Move on to our next tutorial: Advanced Logging Techniques
4. 🌟 Share your logging setup with your team!

Remember: Every debugging session becomes easier with good logging. Keep logging, keep learning, and most importantly, have fun! 🚀

Happy coding! 🎉🚀✨