Data Engineering Fundamentals, Part 2: Python Basics Every Data Engineer Should Know

 

This is Part 2 of a 5-part series on data engineering fundamentals. Part 1 covered SQL, . Part 3 moves into ELT/ETL tools and core data engineering concepts.

Why Python, on top of SQL

SQL is excellent at querying and transforming structured data that already lives in a database. But it can't call an API, loop through files in a folder, handle complex conditional logic cleanly, or glue together multiple systems. That's where Python comes in. You don't need to become a software engineer, but you do need to be comfortable enough to automate the parts SQL can't reach.

Core data structures you'll use constantly

Lists and dictionaries. Most of the data you'll manipulate in Python outside of a dataframe will move through lists and dicts, records as dictionaries, collections of records as lists of dictionaries. Get comfortable with this pattern early.

List and dict comprehensions. Instead of writing a for loop to filter or transform a list, a comprehension does it in one readable line. This is used constantly in real pipeline code.

Tuples and sets, know when each matters. Tuples for fixed, ordered groupings, sets when you need uniqueness or fast membership checks.

Functions and error handling

Write functions early, even for small logic. Reusable functions make pipelines testable and far easier to debug than one long script.

Learn try/except properly. Data pipelines fail in production, files go missing, APIs time out, schemas change unexpectedly. Catching specific exceptions and logging them clearly is what separates a script from something production-ready.

Working with APIs

Most modern data ingestion involves pulling data from a REST API at some point. Get comfortable with the requests library, understanding status codes, handling pagination, and dealing with rate limits and retries. This single skill, calling an API reliably and handling its failure modes, comes up in almost every real-world data engineering role.

Pandas, at least the basics

You don't need to master every pandas function, but you should be comfortable reading a CSV or API response into a dataframe, filtering rows, selecting columns, grouping and aggregating, and writing the result back out. Pandas is often your bridge between raw ingested data and a clean table ready to load into a warehouse.

Virtual environments and dependency management

Learn to use a virtual environment (venv or conda) from the start. Nothing derails a pipeline faster than a dependency conflict between projects. Keep a requirements.txt file for every project so environments are reproducible.

Just enough object-oriented programming

You don't need deep OOP mastery, but understanding classes, why you'd wrap related functions and data together, helps a lot when working with modern data tools, many of which are built around classes and objects under the hood, including Python-based dbt models and custom pipeline frameworks.

A simple example pattern

Here's the shape of a small, realistic script: fetch data from an API, transform it, and prepare it for loading.

import requests

import pandas as pd

def fetch_data(url):

    response = requests.get(url, timeout=10)

    response.raise_for_status()

    return response.json()

def transform(raw_data):

    df = pd.DataFrame(raw_data)

    df = df.dropna(subset=["id"])

    df["processed_at"] = pd.Timestamp.now()

    return df

try:

    raw = fetch_data("https://api.example.com/records")

    clean_df = transform(raw)

    clean_df.to_csv("output.csv", index=False)

except requests.exceptions.RequestException as e:

    print(f"Failed to fetch data: {e}")

Nothing exotic here, but this pattern, fetch, transform, handle failure, load, is the backbone of a huge amount of real data engineering work.

Where to go from here

Once SQL and Python feel comfortable, you're ready to see how they come together inside an actual ELT tool. Part 3 covers ETL/ELT concepts using a modern tool as the example.

[Continue to Part 3: ELT/ETL Concepts Every Data Engineer Should Understand]