pandas: Working with Data Tables

The workhorse of data science

If you learn one library well for data science, make it pandas. It's the tool data scientists use more than any other, because it makes working with tables of data — the form most real data takes — genuinely pleasant. Think of pandas as a programmable, supercharged spreadsheet inside Python. It's built on NumPy, so it's fast, but it adds labelled rows and columns and hundreds of convenient operations.

import pandas as pd   # the universal convention: import pandas as pd

Always import pandas as pd — it's the standard everyone uses. pandas has two core structures: the Series (a single column of data) and the DataFrame (a whole table). You'll work mostly with DataFrames.

The DataFrame: a table of data

A DataFrame is a table with labelled columns and rows. You can create one from a dictionary, where each key becomes a column:

data = {
    "name": ["Mehul", "Riya", "Arjun", "Priya"],
    "age": [19, 20, 18, 21],
    "city": ["Ahmedabad", "Mumbai", "Delhi", "Pune"],
    "marks": [88, 95, 72, 60]
}

df = pd.DataFrame(data)
print(df)
#     name  age       city  marks
# 0  Mehul   19  Ahmedabad     88
# 1   Riya   20     Mumbai     95
# 2  Arjun   18      Delhi     72
# 3  Priya   21       Pune     60

That's a DataFrame — neat rows and columns with labels, just like a spreadsheet. The numbers on the left (0, 1, 2, 3) are the index, pandas's automatic row labels. In real work, you usually load data from a file rather than typing it, which is just as easy.

Loading real data from files

The most common starting point is a CSV file (a simple comma-separated spreadsheet). pandas loads it in one line — this single function is the doorway to most data science work:

# Load data from a CSV file
df = pd.read_csv("students.csv")

# pandas can also read Excel, JSON, databases, and more
# df = pd.read_excel("data.xlsx")
# df = pd.read_json("data.json")

pd.read_csv() reads the whole file into a DataFrame, automatically detecting columns and types. Whatever the source — a downloaded dataset, an export from a database, data from the web — pandas can usually load it in one line. This is where real projects begin.

Inspecting your data

The first thing you always do with new data is look at it. pandas gives you essential methods to understand what you're working with before diving in:

df.head()        # first 5 rows (df.head(10) for 10)
df.tail()        # last 5 rows
df.shape         # (rows, columns) -- e.g. (1000, 5)
df.columns       # the column names
df.info()        # column types and how many non-empty values
df.describe()    # summary statistics for numeric columns

These are your reflexes for any new dataset. df.head() shows a preview, df.shape tells you how big it is, df.info() reveals the types and any missing data, and df.describe() instantly gives you count, mean, min, max, and more for every numeric column. Running these first is the habit of every good data scientist — you always look before you leap.

Selecting columns and rows

To work with data, you need to select pieces of it. Selecting a column is intuitive — use its name; selecting rows uses the iloc (by position) and loc (by label) accessors:

# Select a single column (returns a Series)
print(df["name"])

# Select several columns (returns a DataFrame)
print(df[["name", "marks"]])

# Select rows by position
print(df.iloc[0])      # the first row
print(df.iloc[0:3])    # the first three rows

# A single value: row 0, column "marks"
print(df.loc[0, "marks"])   # 88

Selecting columns by name (df["name"]) is the everyday operation. Note the double brackets df[["name", "marks"]] for multiple columns — the inner brackets are a list of column names. With these selection tools, you can grab exactly the part of a table you need.

Filtering rows by condition

Just like NumPy's boolean indexing, you can filter a DataFrame to keep only rows meeting a condition — one of the most useful operations in all of data science:

# Students who scored 80 or above
top = df[df["marks"] >= 80]
print(top)

# Combine conditions: from Mumbai AND scored above 90
result = df[(df["city"] == "Mumbai") & (df["marks"] > 90)]

# Create a new column from existing ones
df["passed"] = df["marks"] >= 60
print(df)

The pattern df[df["marks"] >= 80] reads as "the rows of df where marks is at least 80". For multiple conditions, wrap each in parentheses and join with & (and) or | (or). You can also create new columns on the fly, as with the passed column. These operations — select, filter, create columns — are the bread and butter of data analysis. Next, we'll tackle the unglamorous but crucial skill of cleaning messy real-world data.