Moving Beyond print(): Why You Need Python Logging

When I started out learning to write Python scripts, print() is the easiest way to debug. If you want to check if a variable updated or a loop actually ran, you just print it to the console.

But in most data engineering environments, or when code has gone into production, relying on print() doesn't really work anymore. When a pipeline is scheduled to run automatically on a server at 3:00 AM, nobody will be there to keep an eye on the terminal. Those print statements just disappear, and if the script fails because an API is down or a file is missing, you're left guessing what went wrong.

That’s why logging is essential. It essentially writes a running history of your script to a file, tracking exactly when things happened and how severe any issues were.

The 5 Logging Levels

Instead of just dumping raw text, logging lets you categorize messages by how important they are. Think of it like a volume knob for how bad a situation is:

DEBUG: Deep details for troubleshooting (e.g., "Variable x is currently 5").
INFO: Standard milestones (e.g., "Successfully connected to the database").
WARNING: Something was weird, but the code kept going (e.g., "Connection slow, retrying").
ERROR: Something broke, but the script didn't completely die (e.g., "Missing data.csv, skipping this step").
CRITICAL: Major failure, the script is shutting down (e.g., "Database completely unreachable").

Setting It Up

Configuring a logger is pretty straightforward. You usually set up the configuration right at the top of your script before any of your actual logic runs.

Here is a basic setup:

import logging
import os

# 1. SETUP ZONE: Configure the logger
# Make sure a 'logs' folder exists so the script doesn't error out
log_dir = 'logs'
os.makedirs(log_dir, exist_ok=True)

logging.basicConfig(
    filename="logs/my_script_log.txt",
    filemode="a",  # "a" appends to the file instead of overwriting it every run
    format="%(asctime)s - %(levelname)s - %(message)s",  # Adds a clean timestamp
    level=logging.INFO,  # Ignore DEBUG noise, capture INFO and worse
)

logger = logging.getLogger()


# 2. LOGIC ZONE: Use logging inside your functions
def divide_numbers(a, b):
    logger.debug(f"Attempting to divide {a} by {b}")  # Skipped because level is INFO

    if b == 0:
        logger.error(f"Tried to divide {a} by zero. Aborting.")
        return None

    return a / b


# 3. EXECUTION ZONE: Running the code
logger.info("--- Script Started ---")

result_1 = divide_numbers(10, 2)
logger.info(f"First division successful. Result: {result_1}")

# This triggers the error log
result_2 = divide_numbers(5, 0)

logger.info("--- Script Finished Successfully ---")

If you run this, your terminal stays completely empty. Instead, a text file is created at logs/my_script_log.txt that looks like this:

2026-07-08 10:15:32,124 - INFO - --- Script Started ---
2026-07-08 10:15:32,125 - INFO - First division successful. Result: 5.0
2026-07-08 10:15:32,125 - ERROR - Tried to divide 5 by zero. Aborting.
2026-07-08 10:15:32,126 - INFO - --- Script Finished Successfully ---

Notice that the logger.debug line didn't show up. Because the config was set to INFO, it only logs messages of equal or higher levels. If you ever need to heavily troubleshoot, you just change that setting to logging.DEBUG and the detailed lines will show up.

The Main Takeaway

The best use of logging is when it is paired with logger.error() with try/except blocks.

Instead of letting a whole pipeline crash because of one bad row of data, you catch the error, log the specific issue to your text file so you can check it later, and let the rest of the script finish running smoothly. It makes automated pipelines a lot more resilient and way easier to manage.

Author:

Joe O'Connor

View Profile