Setting Up Your Coding Project: .venv, .env, and .gitignore

When you first start coding, it’s easy to get into the habit of creating a script on your Desktop, running pip install for your required libraries, and hardcoding credentials into the file.

In reality, that approach quickly falls apart. If somebody wants to run your code on another computer - or if you deploy it to a server - you need to guarantee that the environment is identical to the one you built it in. Packages update regularly, and an unexpected change in a newer library version can quietly break a pipeline that was working fine before the update. On top of that, hardcoding database passwords or API keys into scripts that end up on GitHub is a massive security risk.

Here's a few tips for setting up a local environment before writing any actual pipeline logic which can be used to keep your project isolated, secure, and reproducible.

1. Isolating the Project (.venv)

The first step when setting up any repository locally is to make sure your project’s dependencies don't mix with your global computer setup or other projects. We can use a virtual environment for this. Imagine this as setting up a separate area of your machine where you can have all the required libraries for your code to work, which wont be impacted by global changes to your system.

Once you’ve cloned your GitHub repository to your machine and opened it in VS Code, open your terminal and run:

python -m venv .venv

This creates a hidden .venv directory in your project folder containing an isolated copy of Python. To start using it, activate it:

# On Windows
.venv\scripts\activate

# On Mac/Linux
source .venv/bin/activate

Once activated, you’ll see (.venv) at the start of your terminal line. Now, any package you install using pip stays strictly inside this project.

2. Pinning Dependencies (requirements.txt)

Your code relies on specific versions of libraries to run predictably. If a package updates down the line, function signatures might change, methods could be deprecated, or behaviors could shift—causing unintended consequences for your pipeline.

To prevent this, you define the exact packages and versions your project requires. First, install the package you need inside your active environment:

pip install python-dotenv

Then, freeze your environment's state into a requirements.txt file:

pip freeze > requirements.txt

This generates a text file detailing every installed package along with its exact version (e.g., python-dotenv==1.0.1). When a teammate pulls down your repository or you deploy your script to an orchestration tool, running pip install -r requirements.txt builds the exact same environment every single time, eliminating "it worked on my machine" bugs.

3. Keeping Secrets Secret (.env)

In data engineering, your scripts will constantly connect to external APIs, databases, or cloud storage providers like S3 and Snowflake. Never put your credentials directly into your Python scripts. If you push that file to GitHub, your secrets are exposed to the world.

Instead, create a file named .env in the root of your project:

secret=your_actual_api_key_or_password_here

Inside your Python script, use python-dotenv and the built-in os module to read that secret safely:

import os
from dotenv import load_dotenv

# Load the environment variables from the .env file
load_dotenv()

# Retrieve the secret
secret = os.getenv('secret')

# To test if this is working. Do not leave this in your script!
print(secret)

4. The Gatekeeper (.gitignore)

Having a .env file is useless if you accidentally push it to GitHub anyway. That’s where .gitignore comes in. It tells Git which files and folders to leave on your local computer and ignore completely during commits.

Your root .gitignore file should look something like this:

# Environment files (holding secrets)
.env

# Virtual environments
.venv/

# Local data files (avoid clogging GitHub with large data)
*.json
/data
data/
*.csv 

# Local log files
logs/
*.log

Pro-Tip: Nested .gitignore Files

A .gitignore file applies to the folder it sits in as well as any subfolders beneath it. But you don't have to put every rule in the root file—you can actually nest multiple .gitignore files inside specific subdirectories to scope rules locally.

For example, say you want .txt files (like documentation) to be tracked by Git in your main project folder, but you have a data/ subdirectory where raw .txt export dumps are generated:

• You can place a secondary .gitignore file inside the data/ folder containing just *.txt.
• Git will ignore text files inside data/, but will happily track text files everywhere else in your repository.

Nesting rules like this gives you flexibility to control specific directories.

5. Saving and Pushing Your Setup

Once your environment is set up, you can switch to a new branch, stage your configuration files, commit, and push them to GitHub:

git switch -c adding-files
git add .
git commit -m 'Set up initial environment files'
git push origin adding-files

And if you need to switch back to main to pull the latest updates from your team:

git checkout main
git pull

Summary

Taking five minutes at the start of a project to configure your virtual environment, .env file, and .gitignore prevents issues later. It ensures your credentials stay safe, your codebase remains clean, and your execution environment remains predictable wherever your code is run.