Pipeline Orchestration - Comparing Methods

Once you've created a data ingestion script and tested that it works, you need to be able to set it up to run automatically, rather than having to manually run it yourself every time (especially if you need to run at regular intervals or at a certain time of day in a production use case).

This is where orchestration comes in. During my training in Data Engineering School, we explored three methods of orchestration—starting with basic automation on your local machine, moving to GitHub Actions, and finally containerized orchestrators like Kestra with Docker.

Here I will go through the three methods - the extract and load scripts used are from my Bikepoint ELT project which can be found here: https://github.com/JoeOConnorTIL/Bikepoint-ELT

Local: Windows Task Scheduler

The fastest and most basic way to automate a Python script is using your operating system's built-in scheduler.

Setup

Set up an action in Windows Task Scheduler pointing directly to the Python binary stored inside your project's virtual environment, in this case passing the main script as an argument:

• Program/Script: C:\Users\JOE\Documents\GitHub\Bikepoint-ELT\.venv\Scripts\python.exe
• Arguments: main.py
• Start in: C:\Users\JOE\Documents\GitHub\Bikepoint-ELT

How does it know which packages to use?

Task Scheduler doesn't actually read your requirements.txt file at runtime. Instead, it relies on python's binary pathing:

1. The Isolated Executable: By pointing the task to .venv\Scripts\python.exe instead of your global Python installation, Python automatically sets its internal system prefix (sys.prefix) to your local .venv folder.
2. Automatic Package Resolution: Because it executes from that specific binary, Python strictly imports libraries from .venv\Lib\site-packages—where pip install -r requirements.txt placed your dependencies.
3. Working Directory Scope: Setting Start in to your project root guarantees relative file paths work, allowing your script to locate local .env credential files.

The Trade-Offs

• Pros: 100% free, takes two minutes to set up, and requires zero cloud infrastructure.
• Cons: If your computer goes to sleep, restarts for an update, or loses Wi-Fi, your pipeline fails silently. Essentially, if your computer isn't on, this won't execute.
• Best For: Personal utility scripts or local data dumps where 100% uptime isn't critical.

Remote: GitHub Actions

To get your script running off your laptop and into the cloud, you can turn to GitHub Actions. While primarily built for software CI/CD pipelines, it works great for lightweight data ingestion jobs.

Setup

Commit a YAML workflow file (.github/workflows/ingestion.yml) directly into your repository. You can do this on your GitHub account by clicking on 'Actions', followed by 'New Workflow', and you can either use one of the suggested configurations if applicable or click on 'set up a workflow yourself' to write your own YAML. For time scheduling, define a cron schedule to trigger the job automatically and store sensitive API keys inside GitHub Repository Secrets (In your repository, go to settings -> Environments -> Create an environment for your project, and add the secrets) so raw credentials never touch your codebase.

name: Bike Point Ingestion

on:
  schedule:
    - cron: '*/15 * * * *'  # Trigger every 15 minutes
  workflow_dispatch:        # Allows manual triggers from GitHub UI

jobs:
  build:
    runs-on: ubuntu-latest
    environment: Bike Point DES7
    
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run Ingestion Script
        env:
          AWS_ACCESS_KEY: ${{ secrets.AWS_ACCESS_KEY }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          AWS_BUCKET_NAME: ${{ secrets.AWS_BUCKET_NAME }}
        run: python main.py

In this case there are 4 defined steps:

• Checkout the repository - this locks in the scripts that will run in this particular execution, incase there are any changes made as this is running.
• Set up the defined python version in a virtual machine.
• Install the dependencies you have set in your repo, so that any packages your scripts use will be available.
• Run the defined script with the secrets you have saved into the environment.

The Trade-Offs

• Pros: Runs entirely in the cloud, free for public/standard repositories, keeps your code and workflow in one place, and handles secrets securely.
• Cons:
  • Unreliable Schedule Timings: Because GitHub Actions runs on shared public runners, scheduled cron jobs can experience significant queue delays—or get skipped altogether—when GitHub's servers are under heavy load (especially on exact top-of-the-hour marks).
  • Lack of Data Features: It doesn't offer native task dependency DAGs, automatic retry handling for failing database calls, or historical backfilling.
• Best For: Standalone extraction scripts that need to run in the cloud, provided exact minute-to-minute timing isn't strict, and skipped executions will not be critical.

Dedicated Data Orchestration: Kestra + Docker

Kestra is more of a purpose-built data orchestration tool. It has extra features distinguishing it from GitHub actions such as visual dependency graphs (DAGs), task-level Docker container isolation, native integrations for tools like Snowflake and dbt, and control over backfills. While GitHub Actions is great for triggering standalone scripts, provides better observability, retry handling, and state tracking making it more useful for production data pipelines.

Setup

Kestra uses declarative YAML flows to manage end-to-end task execution. Instead of relying on a generic virtual machine, Kestra spins up a fresh Docker container specifically for your execution. In my case I was using Kestra with docker containers on my local machine - which would have to be switched on each day. In order to use this effectively in practice you would likely set it up on Kestra cloud or on your own Docker/Virtual machine via a cloud provider like AWS or GCP.

You can configure recoverMissedSchedules: NONE. This way if your orchestrator server ever goes offline for maintenance, Kestra won't try to run hundreds of missed historical jobs all at once when it boots back up, which could result in hitting your API rate limits and using up warehouse compute credits unnecessarily.

Again, you need to store your credentials securely - in Kestra you can do this by going to Tenant -> KV store (Key Value) and storing them there. Reference the credential in the script by defining your variable such as AWS_BUCKET_NAME: "{{ kv('AWS_BUCKET_NAME') }}" .

To write your flow in YAML, go to flows, and click '+Create'. Here is the setup I used for a regular interval setup:

id: bikepoint_ingestion_flow
namespace: bike_point

inputs:
  - id: github_url
    type: STRING
    defaults: https://github.com/JoeOConnorTIL/Bikepoint-ELT

triggers:
  - id: every_15_mins
    type: io.kestra.plugin.core.trigger.Schedule
    cron: '*/15 * * * *'
    timezone: UTC
    recoverMissedSchedules: NONE

tasks:
  - id: setup_working_directory
    type: io.kestra.plugin.core.flow.WorkingDirectory
    tasks:
      - id: clone_the_repo
        type: io.kestra.plugin.git.Clone
        url: "{{ inputs.github_url }}"
        
      - id: python_ingestion
        type: io.kestra.plugin.scripts.python.Commands
        namespaceFiles:
          enabled: true
        taskRunner:
          type: io.kestra.plugin.scripts.runner.docker.Docker
        containerImage: python:3.12-slim
        beforeCommands:
          - pip install -r requirements.txt
        commands:
          - python main.py
        env:
          AWS_ACCESS_KEY: "{{ kv('AWS_ACCESS_KEY') }}"
          AWS_SECRET_ACCESS_KEY: "{{ kv('AWS_SECRET_ACCESS_KEY') }}"
          AWS_BUCKET_NAME: "{{ kv('AWS_BUCKET_NAME') }}"

Again this follows similar steps to the GitHub Actions - cloning the repo, spinning up a docker container with python 3.12, installing the requirements and running the main.py script referencing the keys securely stored in KV Store. You can also trigger executions based on other tasks completing as well.

Kestra gives you records of your runs, whether they executed successfully or not and returns logs useful for debugging.

The Trade-Offs

• Pros: Dedicated schedule reliability, containerized Docker isolation, built-in retry mechanisms, control over backfills, and full visibility over multi-step pipelines.
• Cons: Requires hosting and maintaining an orchestrator instance (or using a managed cloud tenant).
• Best For: Enterprise data engineering pipelines requiring strict dependencies, retries, and reproducible execution environments.

Tool Comparison

Starting with Windows Task Scheduler is a great way to learn how automated triggers work. Moving up to GitHub Actions gets your scripts off your machine, but switching to a dedicated orchestrator like Kestra is what provides the reliability, isolation, and control needed for real production pipelines.