Introduction

First of all let’s start with some definitions:

Containers are a way to package an application and its dependencies together. Containers are isolated from each other and bundle their own software, libraries and configuration files; they can communicate with each other through well-defined channels. All containers are run by a single operating system kernel and are thus more lightweight than virtual machines since they do not need to emulate a full hardware stack.
Containers are created from images that specify their precise contents. Images are often created by combining and modifying standard images downloaded from public repositories.

Docker is a set of platform as a service (PaaS) products that use container virtualization to deliver software in packages called containers.
Docker is a tool designed to make it easier to create, deploy, and run applications by using containers. Containers allow a developer to package up an application with all of the parts it needs, such as libraries and other dependencies, and ship it all out as one package. By doing so, thanks to the container, the developer can rest assured that the application will run on any other Linux machine regardless of any customized settings that machine might have that could differ from the machine used for writing and testing the code.

To create an image from scratch using docker, you can use a Dockerfile. A Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image. Using docker build you can create an automated build that executes several command-line instructions in succession.

Docker Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.

Docker installation

To install docker on Fedora 36, you can use the following procedure:

• Install the docker repository:

sudo dnf -y install dnf-plugins-core
sudo dnf config-manager \
  --add-repo \
  https://download.docker.com/linux/fedora/docker-ce.repo

• Install docker:

sudo dnf install docker-ce docker-ce-cli containerd.io docker-compose-plugin

• Install docker-desktop:

sudo dnf install docker-desktop-XX.YY.ZZ.x86_64.rpm

First steps

To start docker, you can use the following command:

sudo systemctl start docker

To check if docker is running, you can use the following command:

sudo systemctl status docker

To enable docker at boot, you can use the following command:

sudo systemctl enable docker

To add your user to the docker group in order to run docker commands without sudo, you can use the following command:

sudo usermod -aG docker $USER

To run an example hello-world container, which is a simple container that prints a message, you can use the following command:

docker run hello-world

As you can notice, the container image is downloaded from the docker hub, than a container is created from the downloaded image and launched. The docker hub is a registry of docker images. You can find a lot of images on the docker hub. You can also create your own images and upload them to the docker hub.

To list all the containers, you can use the following command:

docker ps -a

To remove a container, you can use the following command:

docker rm <container_id>

Docker images

To list all the images, you can use the following command:

sudo docker images

To remove an image, you can use the following command:

sudo docker rmi <image_id>

Dockerfile

To create a Dockerfile, you can use the following command:

sudo docker build -t <image_name> .

To run a container from an image, you can use the following command:

sudo docker run -d -p 80:80 <image_name>

The -d option runs the container in detached mode (the background). The -p option maps the container port 80 to the host port 80.

Docker file example

In this example, we will create a docker image that runs a postgresql database. The docker image will be based on the official postgresql image. The docker image will be created using a Dockerfile.

The Dockerfile:

FROM postgres:latest
ENV POSTGRES_PASSWORD=postgres
ENV POSTGRES_USER=postgres
ENV POSTGRES_DB=postgres

The docker image can be created using the following command:

sudo docker build -t postgresql .

The -t option specifies the name of the image. The dot at the end of the command specifies the path to the Dockerfile.

The docker image can be run using the following command:

sudo docker run -d -p 5432:5432 postgresql

The -d option runs the container in detached mode (the background). The -p option maps the container port 5432 to the host port 5432.

Docker compose

The docker compose tool is used to run multiple containers. The docker compose tool uses a docker-compose.yml file to define the containers and their configuration.

Docker compose file example

In this example, we will create a docker-compose.yml file that runs a postgresql database and a pgadmin4 container. The postgresql container will be based on the official postgresql image. The pgadmin4 container will be based on the official pgadmin4 image.

The docker-compose.yml file:

version: '3.8'

services:
  postgresql:
    image: postgres:latest
    container_name: postgresql
    restart: always
    environment:
      POSTGRES_PASSWORD: postgres
      POSTGRES_USER: postgres
      POSTGRES_DB: postgres
    ports:
      - 5432:5432
    volumes:
      - postgresql_data:/var/lib/postgresql/data
    networks:
      - postgresql_network

  pgadmin4:
    image: dpage/pgadmin4:latest
    container_name: pgadmin4
    restart: always
    environment:
      PGADMIN_DEFAULT_EMAIL: admin@example.com
      PGADMIN_DEFAULT_PASSWORD: admin
    ports:
      - 80:80
    volumes:
      - pgadmin4_data:/var/lib/pgadmin
    networks:
      - postgresql_network
      
volumes:
  postgresql_data:
  pgadmin4_data:

networks:
  postgresql_network:

The docker-compose.yml file can be run using the following command:

sudo docker compose up -d --build

The -d option runs the containers in detached mode (the background). The –build option builds the containers before running them.

I usually use custom names for docker compose files. For example, I use the following command to run the docker-compose-CUSTOM.yml file:

  sudo docker compose -f docker-compose-CUSTOM.yml up -d

Docker compose networks

Networks are used to connect containers. The docker compose tool creates a default network for the containers. The containers can be connected to the default network or to a custom network. The docker compose tool can create a custom network.

A custom network can be created using the following command:

sudo docker network create <network_name>

The network can be removed using the following command:

sudo docker network rm <network_name>

The docker compose file can be used to create a custom network as shown in the postgresql and pgadmin4 example above.

Docker compose volumes

Volumes are used to store data. The docker compose tool can create volumes. The docker compose tool can also use existing volumes.

A volume can be created using the following command:

sudo docker volume create <volume_name>

The volume can be removed using the following command:

sudo docker volume rm <volume_name>

The docker compose file can be used to create a volume as shown in the postgresql and pgadmin4 example above.

Docker compose ports

Ports are used to expose the container ports to the host. The docker compose file can be used to expose the container ports as shown in the postgresql and pgadmin4 example above.

Docker compose environment variables

Environment variables are used to set the environment variables for the container. The docker compose file can be used to set the environment variables as shown in the postgresql and pgadmin4 example above.

Docker compose restart policy

The docker compose file can be used to set the restart policy for the container. The following restart policies are available:

• no - Do not restart the container if it stops or crashes
• on-failure - Restart the container if it stops or crashes, but only if the container exits with an error code
• always - Always restart the container if it stops or crashes
• unless-stopped - Always restart the container unless it is explicitly stopped

The docker compose file can be used to set the restart policy as shown in the postgresql and pgadmin4 example above.

Docker compose healthcheck

The docker compose file can be used to set the healthcheck for the container. The following healthcheck options are available:

• disable - Disable the healthcheck
• interval - The interval between healthchecks
• timeout - The timeout for the healthcheck
• retries - The number of retries before the container is considered unhealthy
• start_period - The time to wait before starting the healthchecks
• test - The command to run to check the health
• none - Do not perform any healthchecks
• cmd - Run a command inside the container

The following docker compose file can be used to set the healthcheck for the postgresql container:

healthcheck:
  test: ["CMD-SHELL", "pg_isready -U postgres"]
  interval: 10s
  timeout: 5s
  retries: 5
  start_period: 30s

The following command can be used to check if the postgresql container is healthy:

sudo docker inspect --format='' postgresql

If the container is healthy, the following output will be displayed:

{"Status":"healthy","FailingStreak":0,"Log":[{"Start":"2021-07-05T13:59:00.000000000Z","End":"2021-07-05T13:59:00.000000000Z","ExitCode":0,"Output":"2021-07-05 13:59:00.000 UTC [1] LOG:  listening on IPv4 address \"
...

Docker compose depends_on

Docker compose build

Docker compose file version

Docker compose file reference

Docker cleanup

In most cases after you have finished with experiments with a docker compose file, you will have a lot of images, containers, volumes and networks that you don’t need anymore. To remove all the unused images, containers, volumes and networks, you can use the following command:

sudo docker compose down --rmi all --volumes --network all --remove-orphans

The –network all option removes all the networks. The –rmi all option removes all the images. The –volumes option removes all the volumes. The –remove-orphans option removes all the orphan containers.

Unfortunately, the script had a small bug that I didn’t notice.

I was using the rm command to delete the ../site/*’ folder using the following command:

rm -rf ../site/*

Later I added a an argument to the script to specify the path to the site folder and changed the command to:

rm -rf $2/*

The $1 argument is the command to run (build, serve, clean, deploy) and the $2 argument is the path to the site folder. So the syntax of the script is:

./deploy.sh [command] [path]

The thing that I didn’t notice is that if I forget to specify the path argument, the script will delete file as if I had run the following command:

rm -rf /*

As you can imagine, this is a very dangerous command. It will delete all the files in the root folder. And guess what? I did it. I ran the script without specifying the path argument and it deleted all the files in the root folder!!

I than spent two days dealing with the btrfs restore and btrfs-find-roots command to try to save my home folder.

I was lucky that I use cloud storage and I had a recent backup of my home folder. I was able to restore it and recover my files.

I learned a lot from this experience. I will never forget to test the existence of the arguments of a script and set default values if an argument is missing. I will also never forget to use the --dry-run option of the rm command.

For my case I used the following commands for my script:

if [ -z "$2" ]; then
  echo "No path specified. Using default path: ../site"
  path="../site"
else
  path=$2
fi
...
echo "The following files will be deleted in $path:"
rm -rf --dry-run $path/*
read -p "Continue with file deletion? (y/n) " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]
then
  rm -rf $path/*
else
  echo "Aborting..."
  exit 1
fi

I hope this will help you avoid making the same mistake as me.

What is pandas?

pandas is a software library written for the Python programming language for data manipulation and analysis. In particular, it offers data structures and operations for manipulating numerical tables and time series. It is free software released under the three-clause BSD license. Wikipedia

Pandas has become the de facto standard for data analysis in Python. It is a powerful library that provides a high-performance, easy-to-use data structures and data analysis tools. It is built on top of NumPy and Matplotlib.

Installation

A good practice is to create a virtual environment for each project. This will allow you to install the required packages without affecting the global Python installation. You can use the following command to create a conda environment:

conda create -n pandas_intro python=3.10
conda activate pandas_intro

Pandas is available on PyPI and can be installed using pip:

pip install pandas

Personally, I prefer using pandas in Jupyter Lab. The Jupyter Lab is a web application that allows you to create and run Jupyter notebooks.

To install Jupyter Lab, you need to install the latest version of JupyterLab from the official repository.

pip install jupyterlab

Or using the conda package:

conda install -c conda-forge jupyterlab

To launch Jupyter Lab, you need to start it from the command line:

jupyter-lab

After starting Jupyter Lab, you can open a new notebook and start writing your code your web browser.

Importing pandas

A common practice is to import pandas as pd :

import pandas as pd

Pandas data frames

Pandas data frames are a powerful data structure that allow you to work with tabular data. They are similar to the tables in relational databases.

Here is an example of a data frame created from a list of columns:

df = pd.DataFrame([
  ['A', 'B', 'C'],
  [1, 2, 3],
  [4, 5, 6]
])

The data frame has three columns and three rows.

df.shape

Here is an example of a data frame created from a dictionary:

df = pd.DataFrame({
  'A': [1, 4],
  'B': [2, 5],
  'C': [3, 6]
})

Pandas data frame can be created from an excel file:

df = pd.read_excel('data.xlsx')

The resulting data is a dictionary of data frames. The keys of the dictionary are the sheet names.

df['Sheet1']

for df_sheet in df:
  print(df_sheet.shape)

Data frame indexing and slicing

Data frames can be indexed by row and column.

df.iloc[0] # row 0
df.iloc[0, 0] # row 0, column 0
df.loc[0] # row 0
df.loc[0, 'A'] # row 0, column A

I personally prefer to use the loc method because it is more intuitive.

df.loc[0, 'A'] # row 0, column A
df.loc[0, ['A', 'B']] # row 0, columns A and B

A data frame can be sliced by row and column:

df.iloc[0:2] # rows 0 and 1
df.iloc[0:2, 0:2] # rows 0 and 1, columns 0 and 1
df.loc[0:2] # rows 0 and 1
df.loc[0:2, 'A'] # rows 0 and 1, column A
df.loc[0:2, ['A', 'B']] # rows 0 and 1, columns A and B

Data frame basic operations

In this section, we will see how to perform basic operations on data frames.

Adding rows and columns

To add a new row at the end of the data frame, you can use the append method:

df = df.append({
  'A': 7,
  'B': 8,
  'C': 9
}, ignore_index=True) # ignore_index=True to reset the index

To add a new column, you can use the assign method:

df = df.assign(D=[10, 11, 12])

To add a new row at a specific position, you can use the insert method:

df = df.insert(0, 'E', [13, 14, 15])

Deleting rows and columns

The drop method can be used to delete rows and columns:

df.drop(0) # delete row 0
df.drop(columns=['A'], axis=1) # delete column A

Sometimes you may want to delete rows and columns that contain missing values.

df.dropna(axis=0, how='any') # delete rows that contain missing values
df.dropna(axis=1, how='any') # delete columns that contain missing values

To drop a column than contains only missing values:

df.dropna(axis=1, how='all') # delete columns that do not contain missing values

To drop duplicate rows:

df.drop_duplicates(keep='first', inplace=True) # delete duplicate rows except the first one (inplace=True to modify the data frame)

Editing data

To edit a cell, you can use the at method:

df.at[0, 'A'] = 0 # replace data in row 0, column A with 0

Replacing data

To replace data in a data frame, you can use the replace method:

df.replace(to_replace=1, value=0) # replace 1 with 0

A simpler and more intuitive way to replace data is to use the map method:

df['A'].map({1: 0, 4: 7}) # replace 1 with 0 and 4 with 7

To replace data in a column, you can use the replace method:

df['A'] = df['A'].str.replace('A', 'B') # replace A with B

String operations

For string operations, you can use the str property methods:

df['A'] = df['A'].str.lower() # convert to lowercase
df['A'] = df['A'].str.upper() # convert to uppercase
df['A'] = df['A'].str.strip() # remove leading and trailing spaces
df['A'] = df['A'].str.split(' ') # split by space
df['A'] = df['A'].str.replace(' ', '_') # replace space with underscore
df['A'] = df['A'].str.capitalize() # capitalize first letter
df['A'] = df['A'].str.replace(r'(\d+)', r'(\1)', regex=True) # replace digits with parentheses using regex

Filtering rows and columns

A data frame can be filtered by a condition expressed as a boolean expression (a mask).

mask = df['A'] > 1
masked_df = df[mask]

Data frame sorting

df.sort_values(by='A') # sort by column A
df.sort_values(by=['A', 'B']) # sort by columns A and B

Data frame merging

df1 = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
df2 = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
df1.merge(df2, on='A') # merge on column A

Data frame aggregation

The groupby method can be used to group data by a column. In this example, the data is grouped by the first column and the mean of the second column is calculated:

df.groupby('A').mean() # group by column A and compute mean

Data frame concatenation

To be able to concatenate data frames, the data frames must have the same columns.

df1 = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
df2 = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
pd.concat([df1, df2]) # concatenate df1 and df2

Data frame statistics

Statistics

The describe method can be used to calculate basic statistics:

df.describe()

Correlation

Correlation is a measure of the strength of the relationship between two variables. intuitively, it is a measure of how much one variable is related to the other.

In this example, correlation is the correlation between the first and second column:

df.corr()

Data frame plotting

Pandas uses matplotlib to plot data frames. Pandas data frames can be plotted using the plot method:

df.plot(kind='scatter', x='A', y='B')

Data frame conversion

A data frame can be converted to a multitude of formats. Personally I convert data frames to dictionaries and lists.

Converting to a dictionary

To convert a data frame to a dictionary, you can use the to_dict method:

df.to_dict()

Exporting to a json file

To export a data frame to a json file, you can use the to_json method:

with open('data.json', 'w') as f:
  df.to_json(f)

If the data frame contains unicode characters, you can use the ensure_ascii parameter:

with open('data.json', 'w', encoding='utf-8') as f:
  df.to_json(f, ensure_ascii=False)

Converting to a list of dictionaries

To convert a data frame to a list of dictionaries, you can use the to_dict method setting the orient parameter to records :

df.to_dict(orient='records')

Converting to a list of lists

To convert a data frame to a list of lists, you can use the values method:

df.values.tolist()

References

From the pandas documentation:

• Pandas documentation
• Pandas cheat sheet
• Pandas cookbook
• Pandas tutorial

Articles:

• Add new rows and columns to Pandas dataframe

Tutorials:

• Python Pandas Tutorial
• Pandas Tutorial: DataFrames in Python

The installation and configuration steps are deliberately given in the same order as during installation process on my computer.

First, what is Flutter and why should it be considered?

What is Flutter?

Flutter SDK is Google’s UI toolkit for crafting beautiful, natively compiled applications for mobile, web, and desktop from a single codebase. Flutter.dev

Flutter is an open-source UI software development kit created by Google. It is used to develop applications for Android, iOS, Linux, Mac, Windows, Google Fuchsia, and the web from a single codebase. The first version of Flutter was known as codename “Sky” and ran on the Android operating system. Wikipedia

Why I chose Flutter

Two years ago I started developing web-based applications, primarily for telemetry applications and local SMBs.

I didn’t face any major issues with backend development (mainly python-based) including but not limited to: database (SQL Alchemy and mongoDB), multi-threading and multiprocessing daemons, fault-tolerant protocol implementation, REST …

The main issue I encountered in my journey to become a full-stack developer was the frontend side of the game

I realized that the task of creating high-quality cross platform (web, android, iOS and PC) was tedious and difficult and required a large set of knowledge and skill sets that were far beyond what was possible to achieve as a solo-developer.

In my journey I tested and used several solutions, most of which were for python developers wishing to do some GUI development. Some of the solutions which I used include:

• PyQt – a Python binding of the cross-platform GUI toolkit Qt
• Plotly Dash – python framework for building web analytic applications built on top of Flask, Plotly.js, and React.js
• Anvil – a free Python-based drag-and-drop web app builder
• Jupyter notebooks – an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text.

However, none of the options fully met my goals or solved the problems I faced. A solution based on a single language and a set of tools had to be found in order to cover all the tasks related to frontend design and interaction with the backend.

That is why I decided to give Flutter a try and I was amazed by the quality of the tools and the fluidity of the workflow.

Prerequisites

snap

In order to install snap use the command line:

sudo dnf install snapd

Then snap classic support should be enabled:

sudo ln -s /var/lib/snapd/snap /snap

Snap Store application can also be installed from the command line:

sudo snap install snap-store

tools

As stated by Flutter documentation, the following tools should be installed:

• bash
• curl
• file
• git 2.x
• mkdir
• rm
• unzip
• which
• xz-utils
• zip

Also the libGLU.so.1 library should be installed. The package providing this can be found using the command line:

dnf provides libGLU.so.1

To install:

sudo dnf install mesa-libGLU

When using the locate libGLU.so command I realized that the libGLU.so.1 file had been installed by snap. If asked I’ll check which of the installed libraries is being used by FLutter

Flutter installation

In order to install Flutter use this in the command line:

sudo snap install flutter --classic

Executing the following command will initialize the Flutter framework (download, install and initialize Dart and Flutter):

flutter sdk-path

The command itself is used to display SDK’s path

flutter channel beta
flutter upgrade
flutter config --enable-web

Configuration

The first thing to do is to add the path to the SDK to your PATH environment variable. This can be achieved by adding the following lines at the bottom of the ~/.bashrc file:

# Flutter and Dart paths
export PATH="$PATH:PATH_TO_FLUTTER_SDK/bin"

Remember to reload the terminal for the changes to take effect.

~/.bashrc can be patched and the terminal’s environment reloaded as follows:

cat << EOF >> ~/.bashrc
# Flutter and Dart paths
export PATH="\$PATH:`flutter sdk-path`/bin"
EOF
. ~/.bashrc

The next big step of the configuration process is to run the flutter doctor command from the command line to verify dependencies.

Below is an example of the command output:

Several issues are to be noted:

• The Android SDK is not installed
• Android studio is not installed
• Visual studio code is detected but the Flutter extension is not installed
• No device is connected

Device configuration and connection will be addressed in a separate post

Additional tools installation

Visual studio code

To install VS code on a Fedora machine:

sudo snap install --classic code

Then the Flutter plugin should be installed:

• Open VS code
• Search for Flutter In the Extensions section
• Click install

As shown in the image below, VS code is reported to be correctly configured:

Android Studio and Android SDK

Android Studio is the official Android app IDE based on IntelliJ IDEA from JetBrains.

you can use snap to install Android Studio:

sudo snap install android-studio --classic

JetBrain’s Toolbox App is another way to install Android Studio

Android Studio configuration

When Android Studio is started for the first time the Android Studio Setup Wizard is automatically launched. Follow the instructions as shown in the following screenshots:

The Standard Install Type is similar to the Custom one:

I prefer the dark theme:

I left these options untouched:

Please take note of the paths used:

Depending on the Internet connection speed, the installation can take a while:

After successful configuration, the Android Studio Welcome window pops up:

The next step is to install the Flutter plugin for Android Studio. To do this, open the Plugins window by clicking on the Configure button, then click on plugins, then type flutter in the search bar, choose Flutter plugin from flutter.dev and hit Install:

The Flutter plugin requires the Dart plugin to be installed:

After successful installation of Dart and Flutter plugins, restart Android Studio:

Now that Android Studio is installed and configured, let’s check the output of flutter doctor:

So the next step is to accept Android licenses using flutter doctor

flutter doctor --android-licenses

Use the following command to point flutter doctor to the installation path of Android Studio:

flutter config --android-studio-dir=/var/lib/snapd/snap/android-studio/current/android-studio/

Now the flutter doctor output looks as follows:

You can be safely ignore the two errors stating that Flutter and Dart plugins are not detected.

What’s next?

The next steps may include configuring and testing the Android Emulator, configuring on-device debugging, developing basic Flutter apps to explore different aspects of the SDK, exploring the features of VS code and Android Studio and exploring debugging tools.

References

Further details could be found in the following web resources:

• How to install Android Studio on Fedora
• Installing snap on Fedora
• Flutter documentation
• Flutter documentation – Set up an editor
• Pointing the Android Studio installation directory in Flutter (GitHub issue)
• Building a web application with Flutter
• Configure VM acceleration on Linux