This post is not a tutorial but rather a collection of notes that I have taken while learning pandas. I will try to keep it updated as I learn more about pandas. I don’t want to write a tutorial because I want to keep it short and simple and mention only the features that I use most.

What is pandas?

pandas is a software library written for the Python programming language for data manipulation and analysis. In particular, it offers data structures and operations for manipulating numerical tables and time series. It is free software released under the three-clause BSD license. Wikipedia

Pandas has become the de facto standard for data analysis in Python. It is a powerful library that provides a high-performance, easy-to-use data structures and data analysis tools. It is built on top of NumPy and Matplotlib.

Installation

A good practice is to create a virtual environment for each project. This will allow you to install the required packages without affecting the global Python installation. You can use the following command to create a conda environment:

conda create -n pandas_intro python=3.10
conda activate pandas_intro

Pandas is available on PyPI and can be installed using pip:

pip install pandas

Personally, I prefer using pandas in Jupyter Lab. The Jupyter Lab is a web application that allows you to create and run Jupyter notebooks.

To install Jupyter Lab, you need to install the latest version of JupyterLab from the official repository.

pip install jupyterlab

Or using the conda package:

conda install -c conda-forge jupyterlab

To launch Jupyter Lab, you need to start it from the command line:

jupyter-lab

After starting Jupyter Lab, you can open a new notebook and start writing your code your web browser.

Importing pandas

A common practice is to import pandas as pd :

import pandas as pd

Pandas data frames

Pandas data frames are a powerful data structure that allow you to work with tabular data. They are similar to the tables in relational databases.

Here is an example of a data frame created from a list of columns:

df = pd.DataFrame([
  ['A', 'B', 'C'],
  [1, 2, 3],
  [4, 5, 6]
])

The data frame has three columns and three rows.

df.shape

Here is an example of a data frame created from a dictionary:

df = pd.DataFrame({
  'A': [1, 4],
  'B': [2, 5],
  'C': [3, 6]
})

Pandas data frame can be created from an excel file:

df = pd.read_excel('data.xlsx')

The resulting data is a dictionary of data frames. The keys of the dictionary are the sheet names.

df['Sheet1']

for df_sheet in df:
  print(df_sheet.shape)

Data frame indexing and slicing

Data frames can be indexed by row and column.

df.iloc[0] # row 0
df.iloc[0, 0] # row 0, column 0
df.loc[0] # row 0
df.loc[0, 'A'] # row 0, column A

I personally prefer to use the loc method because it is more intuitive.

df.loc[0, 'A'] # row 0, column A
df.loc[0, ['A', 'B']] # row 0, columns A and B

A data frame can be sliced by row and column:

df.iloc[0:2] # rows 0 and 1
df.iloc[0:2, 0:2] # rows 0 and 1, columns 0 and 1
df.loc[0:2] # rows 0 and 1
df.loc[0:2, 'A'] # rows 0 and 1, column A
df.loc[0:2, ['A', 'B']] # rows 0 and 1, columns A and B

Data frame basic operations

In this section, we will see how to perform basic operations on data frames.

Adding rows and columns

To add a new row at the end of the data frame, you can use the append method:

df = df.append({
  'A': 7,
  'B': 8,
  'C': 9
}, ignore_index=True) # ignore_index=True to reset the index

To add a new column, you can use the assign method:

df = df.assign(D=[10, 11, 12])

To add a new row at a specific position, you can use the insert method:

df = df.insert(0, 'E', [13, 14, 15])

Deleting rows and columns

The drop method can be used to delete rows and columns:

df.drop(0) # delete row 0
df.drop(columns=['A'], axis=1) # delete column A

Sometimes you may want to delete rows and columns that contain missing values.

df.dropna(axis=0, how='any') # delete rows that contain missing values
df.dropna(axis=1, how='any') # delete columns that contain missing values

To drop a column than contains only missing values:

df.dropna(axis=1, how='all') # delete columns that do not contain missing values

To drop duplicate rows:

df.drop_duplicates(keep='first', inplace=True) # delete duplicate rows except the first one (inplace=True to modify the data frame)

Editing data

To edit a cell, you can use the at method:

df.at[0, 'A'] = 0 # replace data in row 0, column A with 0

Replacing data

To replace data in a data frame, you can use the replace method:

df.replace(to_replace=1, value=0) # replace 1 with 0

A simpler and more intuitive way to replace data is to use the map method:

df['A'].map({1: 0, 4: 7}) # replace 1 with 0 and 4 with 7

To replace data in a column, you can use the replace method:

df['A'] = df['A'].str.replace('A', 'B') # replace A with B

String operations

For string operations, you can use the str property methods:

df['A'] = df['A'].str.lower() # convert to lowercase
df['A'] = df['A'].str.upper() # convert to uppercase
df['A'] = df['A'].str.strip() # remove leading and trailing spaces
df['A'] = df['A'].str.split(' ') # split by space
df['A'] = df['A'].str.replace(' ', '_') # replace space with underscore
df['A'] = df['A'].str.capitalize() # capitalize first letter
df['A'] = df['A'].str.replace(r'(\d+)', r'(\1)', regex=True) # replace digits with parentheses using regex

Filtering rows and columns

A data frame can be filtered by a condition expressed as a boolean expression (a mask).

mask = df['A'] > 1
masked_df = df[mask]

Data frame sorting

df.sort_values(by='A') # sort by column A
df.sort_values(by=['A', 'B']) # sort by columns A and B

Data frame merging

df1 = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
df2 = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
df1.merge(df2, on='A') # merge on column A

Data frame aggregation

The groupby method can be used to group data by a column. In this example, the data is grouped by the first column and the mean of the second column is calculated:

df.groupby('A').mean() # group by column A and compute mean

Data frame concatenation

To be able to concatenate data frames, the data frames must have the same columns.

df1 = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
df2 = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
pd.concat([df1, df2]) # concatenate df1 and df2

Data frame statistics

Statistics

The describe method can be used to calculate basic statistics:

df.describe()

Correlation

Correlation is a measure of the strength of the relationship between two variables. intuitively, it is a measure of how much one variable is related to the other.

In this example, correlation is the correlation between the first and second column:

df.corr()

Data frame plotting

Pandas uses matplotlib to plot data frames. Pandas data frames can be plotted using the plot method:

df.plot(kind='scatter', x='A', y='B')

Data frame conversion

A data frame can be converted to a multitude of formats. Personally I convert data frames to dictionaries and lists.

Converting to a dictionary

To convert a data frame to a dictionary, you can use the to_dict method:

df.to_dict()

Exporting to a json file

To export a data frame to a json file, you can use the to_json method:

with open('data.json', 'w') as f:
  df.to_json(f)

If the data frame contains unicode characters, you can use the ensure_ascii parameter:

with open('data.json', 'w', encoding='utf-8') as f:
  df.to_json(f, ensure_ascii=False)

Converting to a list of dictionaries

To convert a data frame to a list of dictionaries, you can use the to_dict method setting the orient parameter to records :

df.to_dict(orient='records')

Converting to a list of lists

To convert a data frame to a list of lists, you can use the values method:

df.values.tolist()

References

From the pandas documentation:

Articles:

Tutorials:

Leave a comment