HANDS-ON

PYTHON

with Exercises, Projects, Assignments & Final Exam

ADVANCED

Musa Arda

Hands-On Python with Exercises, Projects, Assignments & Final Exam: Advanced

By Musa Arda

Copyright © 2022 Musa Arda.

All rights reserved. No portion of this book or its supplementary materials may be reproduced in any form without permission from the publisher, except as permitted by U.S. copyright law.

For permissions contact: python.hands.on.book@gmail.com

Contents

Preface

1.Introduction

2.IDE - PyCharm Basics

3.Collections

4.Iterators

5.Generators

6.Date and Time

7.Decorators

8.Context Managers

9.Functional Programming in Python

10.Project 1 – Sending Emails with Python

11.Assignment 1 – Sending Emails with Python

12.Regular Expressions

13.Database Operations

14.Concurrency

15.Project 2 – Web Scraping with Python

16.Assignment 2 – Web Scraping with Python

17.Project 3 – API Development with Flask

18.Assignment 3 – API Development with Flask

19.Final Exam

20.Conclusion

Preface

About This Book

This book is an in-depth and activity-based introduction to the advanced level topics of Python programming. It follows a step-by-step practical approach by combining the theory of the language with the hands-on coding exercises including quizzes, projects, assignments and exams.

We begin by introducing the Collections module in Python. Then we cover Iterators, Generators, Date and Time Operations, Decorators and Context Managers in Python. We move on with Functional Programming in Python and we have our first project which is about sending emails using Python. Then we learn Regular Expressions, Database Operations and Concurrency in Python. After these topics, we build two projects. The second project in this book is Web Scraping with Python and Scrapy. And the third project is Developing Web APIs with Flask. After each project, you have an assignment to complete. And finally, we have the Final Exam.

By the end of the book, you will learn almost all of the advanced level concepts of Python in great detail by writing thousands of lines of code. All the supplementary resources (code files, quizzes, assignments, final exam etc.) are available for download at the GitHub repository. The link for the repository is provided in the book.

This is the third book in Hands-On Python Series. And here is what you will find in this book:

Theory: In each topic, we will cover all the Theoretical Details with example coding.

Coding Exercises: At the end of each chapter, we will have Coding Exercise, Quizzes.

Projects: We will build projects in this book. You will learn how to apply Python concepts on real world problems.

Assignments: After each project, you will have an Assignment. These assignments will let you build the project from scratch on your own.

Final Exam: At the end of this book, you will have the Final Exam. It is a multiple-choice exam with 20 questions and a limited duration. The exam will let you to test your Python level.

About Hands-On Python Series

This is the third book in our Hands-On Python Series. And it covers the advanced level topics. So, we assume that you already know the introductory and intermediate concepts of Python programming like; Variables, Functions, Conditional Statements, Loops, Strings, Lists, Dictionaries, Tuples, Sets and Comprehensions, Exception Handling, File Operations and OOP. If you don’t feel comfortable with these topics, you are strongly recommended to finish the first two books in our series, which are the Beginner and Intermediate levels. Here, you can find them.[1]

About the Author

Musa Arda has Bachelor’s degree from Industrial Engineering in 2007, and he has been working as a Software Developer for more than 14 years. He uses many programming languages and platforms like; Python, C#, Unity, Java, JavaScript, SAP ABAP, SQL, React, Flutter and more.

He creates online learning content and writes books to share his experience and knowledge with his students. His main purpose is to combine theory and hands-on practice in his teaching resources. That's why there are hundreds of programming exercises, quizzes, tests, projects, exams and assignments in all of his courses and books. He is dedicated to help his students to learn programming concepts in depth via a practical and exiting way.

How to Contact Us

Please feel free to get in contact with the author. To comment or ask technical questions about this book, you can send email to python.hands.on.book@gmail.com.

1.      Introduction

Who Is This Book For?

The goal of this book is to help students to learn Python programming language in a hands-on and project-based manner.

With its unique style of combining theory and practice, this book is for:

people who want to learn and practice advanced concepts in Python programming

people who are already working with Python language

What Can You Expect to Learn?

The purpose of this book is to provide you a good introduction to the advanced topics of Python programming. In general, you will gain solid programming skills and grasp the main idea of software development. In particular, here are some highlights on what you can you expect to learn with this book.

You will:

learn & master advanced Python topics in a hands-on approach

practice your Python knowledge with Quizzes and Coding Exercises

build Real-World Project with Python and do Assignments related to these projects

take the Final Exam on Python with 20 questions to assess your learning

build Python applications with PyCharm and master it

gain solid and profound Python Programming skills needed for a Python career

Outline of This Book

In Chapter 1, you will get the basics of the book. You will learn about this books approach to Python programming and how to get most out of it.

In Chapter 2, you will meet PyCharm, our IDE (Integrated Development Environment) for this book. You will learn all the basics of PyCharm and how to do debugging with it. You will also learn the base interpreter and interpreter configuration.

In Chapter 3, you will learn the Collections module in Python. Collections are specialized container datatypes providing alternatives to Python’s general-purpose containers, dict, list, set, and tuple. You will learn; ChainMap, Counter, Deque, DefaultDict, NamedTuple, OrderedDict, UserDict, UserList, and UserString.

In Chapter 4, you will learn Iterables and Iterators in Python. You will see the details of Iterator Protocol, how to loop through an Iterator, how to define custom Iterators and Infinite Iterators.

Chapter 5 is on Generators in Python. Generators allow you to define Iterators more easily and efficiently. You will define custom Generators and learn the benefits of using them.

In Chapter 6, you will learn all the details of Date and Time operations in Python, which are crucial for robust application development. You will learn the difference between Aware and Naive Objects and get the details of the classes in datetime module.

In Chapter 7, you will meet Decorators. A very important concept in Python programming. You will learn how to define a decorator, how to chain them and how to use class syntax for creating new ones.

In Chapter 8, you will learn the details of Context Managers, which are very handy tools when you need to deal with resource management in your code. You will see how to define and use a Context Manager in both class form and function form.

Chapter 9 is about Functional Programming in Python. We will briefly talk about Functional Programming paradigm and then you will learn the basic building blocks of this paradigm in Python. You will learn how to use lambda, map(), filter() and reduce() functions.

Chapter 10 will be the first Project in this book. You will learn how to send emails using Python. You will set a local mail server to send emails on your local machine. Then you will set a fake remote mail server to send and test emails. Finally, you will learn how to use a Gmail account to send real emails.

In Chapter 11, you will have an Assignment on the Sending Email Project. You will have #TODO directives to develop the same project. You will be able to write your own code, to finish the tasks.

In Chapter 12, you will learn one of the fundamental concepts in programming, which is Regular Expressions (RE). You will learn how to define RE patterns and how to search a given text based on this pattern. You will learn Python re module, and see lots of use cases with coding exercises.

Chapter 13 is dedicated to the Database Operations in Python. You will learn how to install a local MySQL Server, how to create databases and tables, how to insert records in these tables. You will also see how to use some common SQL operations like Select, Insert, Update and Delete in Python code.

In Chapter 14, you will learn how to handle Concurrency in Python. You will learn about Multithreading and Multiprocessing and the classes that Python provides to support them. You will see how to create and start threads to get better performance in your applications.

Chapter 15, is the second project in this book which is Web Scraping with Python. You will learn how to scrap data out of web pages using Python and Scrapy.

In Chapter 16, you will have an Assignment on the Web Scraping Project. You will have #TODO directives to develop the same project. You will be able to write your own code, to finish the tasks.

In Chapter 17, we have the third project which is API Development with Flask. You will learn how to setup and a Flask project from scratch, create API endpoints using Blueprints, Views and Templates. We will create a beautiful movie review web app using Python and Flask.

In Chapter 18, you will have an Assignment on the API Development with Flask Project. You will have #TODO directives to develop the same project. You will be able to write your own code, to finish the tasks.

Chapter 19 is your Final Exam. Now it’s time to assess your Python level. You will have an exam of 20 questions on the topics you learned in this book. The Final Exam will give you the chance to see what you learned and which topics you should repeat.

Chapter 20 is the Conclusion. You will finalize this book and will see how you should proceed for the next step.

Conventions Used in This Book

The following typographical conventions are used in this book:

Italic

Indicates new URLs, email addresses, filenames, and file extensions.

Bold

Indicates new terms and important concepts to pay attention.

Constant width

Used for string literals and programming concepts within paragraphs.

Constant width bold

Used for program elements within paragraphs, such as variable or function names, data types, statements, and keywords.

We will write our code in the code cells. Here is an example code cell with the cell number as 7:

Figure 1-1: A code cell example used in this book

Using Code Examples

You can find all the supplementary resources for the book (code files, quizzes, assignments, final exam etc.) available for download at the GitHub Repository of the book which is https://github.com/musaarda/python-hands-on-book-advanced.

You & This Book

This book is designed in a way that, you can learn and practice Python. At each chapter you will learn the basic concepts and how to use them with examples. Then you will have a quiz the end of the chapter. First you will try to solve the quiz questions on your own, then I will provide the solutions in detail. You will have projects after each block of core concepts. And after every project you will an assignment to test your understanding. This will be your path to learn real Python.

Before we deep dive into Python, I want to give you some tips for how you can get most out of this book:

Read the topics carefully before you try to solve the quizzes

Try to code yourself while you are reading the concepts in the chapters

Try to solve quizzes by yourself, before checking the solutions

Read the quiz solutions and try to replicate them

Code the projects line by line with the book

Do the assignments (seriously)

Do not start a new chapter before finishing and solving quiz of the previous one

Repeat the topics you fail in the Final Exam

Learning takes time, so give yourself enough time digest the concepts

2.      IDE - PyCharm Basics

A Brief History of Python

Python is a widely used and general-purpose, high-level programming language[2]. Its design philosophy emphasizes code readability and its syntax allows programmers to express concepts in fewer lines of code. Python is dynamically-typed and garbage-collected. It supports multiple programming paradigms, including procedural, object-oriented and functional programming.

It was initially designed by Guido van Rossum  in the late 1980s and developed by Python Software Foundation. Python was named after the BBC TV show Monty Python's Flying Circus.

In 1991, Van Rossum published the code labeled version 0.9.0. Then in 1994, Python reached version 1.0.

Python 2.0 was released in 2000, which started the Python 2.x family that has been used widely for almost a decade. The final release, Python 2.7.18, occurred on 2020 and marked the end-of-life of Python 2. As of January 1st, 2020 no new bug reports, fixes, or changes will be made to Python 2, and Python 2 is no longer supported.

Python 3.0 (also called "Python 3000" or "Py3K") was released on December 3, 2008 and it was a major, backwards-incompatible release. Since Python 3.0 broke backward compatibility, much Python 2 code does not run unmodified on Python 3. Although some old projects still use Python 2, the preferred approach is using Python 3. As of this writing, the latest version is 3.10, but every code we write in this book should run on Python 3.6 or any later version.

Python Installation

To install Python, open the official Python web site, python.org, in your web browser and navigate to the Downloads tab.

Figure 2-1: Official Python web site

Here, you can download the current version for your operating system. Once you download the executable file then it is very straightforward to install Python.

Figure 2-2: The latest version on python.org

Run the executable installer to start installation. You will see the initial screen for setup. The recommended installation includes pip, IDLE and documentation. Pip is the package installer for Python. We will see it later in this book. IDLE is the built-in IDE for Python development, but we will not be using it. We will be using PyCharm.

Figure 2-3: Python setup wizard

Make sure you select the Install launcher for all users and Add Python 3.10 to PATH checkboxes. After selecting these checkboxes you can click Install Now button. It will initialize the setup process. It may take several minutes to finalize depending on your computer.

Figure 2-4: Python setup progress

When your computer finishes installation you will see the final screen saying Setup was successful.

Figure 2-5: Final setup screen

There is an important option on the final screen. It is about the path length limit. In Windows, the pathnames are limited to 260 characters length. We don’t have that limitation in macOS or Linux. Python has introduced an option for disabling path length limit to avoid path length related problems. You can select disabling the path length limit, which is recommended. It will not affect any other system settings. Disabling it will resolve potential path length issues that may arise with projects developed in Linux or macOS.

Verify Python installation:

After the last screen, your Python setup is completed and you can verify if it was installed successfully. To verify it, open the command line on your machine (CMD on Windows, Terminal on macOS and Linux) and type python --version. You should see the installed Python version on your machine as below:

Figure 2-6: python --version command on Windows CMD

See where Python is installed:

We have installed Python and we verified the installation by printing its version on command line. Now let’s see where it is installed. In other words, we want to see the installation folder or folders for Python. You might have different Python installations on the same machine. To see the locations, type where python on the command line as below:

Figure 2-7: Locations of separate Python installations

As you see in Figure 2-7 there are multiple versions of Python installed on my machine. We can select any of them while creating our projects in PyCharm. You will see how in the next sections.

Python Virtual Environments

Virtual Environment is an isolated runtime environment that allows Python users and applications to install and upgrade Python packages without interfering with other Python applications running on the same system.

Python applications will often use packages and modules that don’t come as part of the standard library. Applications will sometimes need a specific version of a library, because the application may require that a particular bug has been fixed or the application may be written using an obsolete version of the library’s interface.

This means it may not be possible for one Python installation to meet the requirements of every application. If application A needs version 1.0 of a particular module but application B needs version 2.0, then the requirements are in conflict and installing either version 1.0 or 2.0 will leave one application unable to run.

The solution for this problem is to create a virtual environment, a self-contained directory tree that contains a Python installation for a particular version of Python, plus a number of additional packages.

Different applications can then use different virtual environments. To resolve the earlier example of conflicting requirements, application A can have its own virtual environment with version 1.0 installed while application B has another virtual environment with version 2.0. If application B requires a library be upgraded to version 3.0, this will not affect application A’s environment.[3]

Figure 2-8: Different Virtual Environments on the same OS, completely isolated from each other

We will use separate virtual environments (venv for short) for each project we create in this book. That’s the recommended way for Python development. Don’t worry, you will see how to create virtual environments in the next sections.

About PyCharm

In this book we will be using PyCharm as our development environment. PyCharm is a dedicated Python Integrated Development Environment (IDE) providing a wide range of essential tools for Python developers, tightly integrated to create a convenient environment for productive Python, web, and data science development[4]. PyCharm is a cross-platform IDE that works on Windows, macOS, and Linux.

PyCharm is available in three editions:

Community (free and open-sourced): for individual Python development, including code assistance, refactorings, visual debugging, and version control integration.

Professional (paid): for professional Python, web, and data science development, including code assistance, refactorings, visual debugging, version control integration, remote configurations, deployment, support for popular web frameworks, such as Django and Flask, database support, scientific tools (including Jupyter notebook support), big data tools.

Edu (free and open-sourced): for learning programming languages and related technologies with integrated educational tools.

We will be using the Community Edition in this book and it is more than enough for small to medium Python projects. In the next sections we will install PyCharm and learn its basics.

PyCharm Installation

To install PyCharm, open the official PyCharm web site in your web browser and navigate to the Downloads page. You can select your operating system here. We will download the Community edition which is free.

Figure 2-9: PyCharm download page

It is very straight forward to install PyCharm. Just click Next and start installation. It is strongly recommended to accept the default destination folder. Then you can click Next and finalize the installation.

Figure 2-10: PyCharm setup

When you open PyCharm for the first time, you will see the default Welcome Screen. Here you have 3 options:

You can create a New Project

You can Open an existing project

You can get an existing project from a Version Control System (VCS) such as Git.

Figure 2-11: Welcome screen in PyCharm

On the Welcome Screen, you can customize the color theme of your PyCharm IDE. You can use the black theme for example. The default black theme is called Darcula in PyCharm. Here is how you can change your theme:

Figure 2-12: Changing the color theme in PyCharm

First Project in PyCharm

Now that PyCharm is installed on our machine, let’s create our first project. On the Welcome Screen, click on New Project button to start the project creation wizard. You see the New Project wizard in Figure 2-13. Here you select a location and a name for your project. We name our first project as My_Project which is the last part in the location string. 

Figure 2-13: New Project wizard in PyCharm

The second configuration is about the interpreter.

Python Interpreter:

Python Interpreter is the application that runs your Python code. More precisely, it is the python.exe file which we installed in the previous sections. We can either create a new virtual environment or we can use the base environment with the Base Interpreter. As stated earlier, working with isolated virtual environments is the recommended way and that’s what we generally do in this book. So, we will select the New environment using Virtualenv option.

Figure 2-14: New virtual environment option

When we select the New environment using Virtualenv option, PyCharm creates a default location path for our new virtual environment. Which is a new folder in the project as: My_Project\venv. PyCharm will create this folder automatically and it will put the python.exe file and other necessary files into it. It will use the Base Interpreter to copy the python.exe file. In other words, PyCharm will copy the python.exe file in the Base Interpreter and paste it into the venv folder. You can select the Python version for the Base Interpreter.

The third configuration is about the main.py file. You can tell PyCharm that you want a default main file in your project. In general, main.py files are the starting point of Python projects. If you select this option, PyCharm will create it automatically. If you do not select it, you can create it later in your project. Let’s select it and click on the Create button to initialize our first PyCharm project.

It may take some time, for PyCharm to create the new project with a new virtual environment. And after it finishes creation, you will see the main.py file and all the folders in your project.

Figure 2-15: New project in PyCharm

As you see in Figure 2-15, the venv folder is your virtual environment. And inside venv folder you see python.exe under the Scripts folder. That’s our Python Interpreter.

You see the main.py file is open on the right-hand side of PyCharm. This is the default file, PyCharm creates for us.

Let’s run our first project in PyCharm. On the top right of the PyCharm screen you see the Run button with a green play icon. Click on it, and you will see the output at the bottom of the screen:

Figure 2-16: The run button and the output window in PyCharm

In Figure 2-16 we run our project and it prints “Hi, PyCharm” in the output window.

If you could run your first project without getting any errors, this means your Python and PyCharm installations are successful and you can start development in PyCharm. We will learn PyCharm basics in the next section.

PyCharm Basics

PyCharm user interface is composed of four main parts: Main Menu, Project Explorer, Code Editor and Output Window. In this section, we will see the default PyCharm layout. You can customize it and change the layout as you want.

Figure 2-17: PyCharm User Interface

Main Menu:

On the top you see the Main Menu, where you have all the commands related to PyCharm IDE, configuration settings and file editing options. Here are the items in the main menu and some of the important functionalities they have:

File: Create a new project, open an existing one, save options and IDE settings

Edit: Copy, paste, delete functions, find and replace, line operations

View: Viewing options for tool windows and appearance

Navigate: Backward and forward navigation, navigate in files

Code: Code completion & inspection, folding and formatting options

Refactor: Refactoring and extracting code fragments

Run: Run and debug with configurations

Tools: Tasks and team configurations

VCS: Version Control System settings like Github and Space

Window: Layout and Tabs configuration

Help: Keymap Reference, Demos and Screencasts, Learn IDE Features

Project Explorer:

Project Explorer is on the left side and it’s where you manage project files, folders and virtual environments. For the current project you can switch between different views as Project, Project Files and Open Files.

Figure 2-18: Project Explorer options in PyCharm

Code Editor:

On the right-hand side, we have Code Editor. We use the Editor to read, write, and explore the source code. You can have multiple files open in the editor. On the top right corner of the editor window, you see the quick access buttons like Run, Debug, Run Configurations, Search and Settings.

Figure 2-19: Quick Access buttons on top of the editor window

Output Window:

On the bottom of the screen, we have the Output Window. It displays the output generated by our application and messages from the Python Interpreter.

Here are the tabs in the Output Window:

Run: The output when you run the code. You can rerun the code or edit run configuration

Figure 2-20: Run tab in the output window

TODO: Displays the TODO lines in the code. TODO’s are special comment lines that starts as ‘# TODO’. In the image below you see two TODO lines in the code and their listing in the TODO tab:

Figure 2-21: TODO statements in the code and in TODO tab

Problems: Displays the errors and problems in the code and the entire project

Figure 2-22: Problems tab

Terminal: An integrated terminal in PyCharm. You do not need to start a standalone terminal for Python. This terminal works the same as your operating system terminal.

Figure 2-23: Integrated Terminal in PyCharm

Python Packages: This tab displays all the packages you have in the current project. You can view package documentation and add new packages in this tab.

Figure 2-24: Python Packages tab

Python Console: Integrated Python console. Python console is the text editor where you can run Python code. See an example in the screenshot below:

Figure 2-25: Python Console

You are now familiar with the PyCharm user interface and I think it’s enough to start development in PyCharm IDE. If you want to learn more about PyCharm and how to customize its look and feel, here is the complete documentation for PyCharm user interface on its official web site[5].

Running Code in PyCharm

You have three options to run a project in PyCharm:

1-      The Run button in the Context Menu by right clicking in the editor.

Figure 2-26: Run button in the context menu of editor

2-      The Run button in quick access menu on the top right of the editor.

Figure 2-27: Run button in the quick access menu

3-      The Run button in the Context Menu by right clicking on the file name in the Project Explorer.

Figure 2-28: Run button in the context menu of project explorer

And also there is a keyboard shortcut for Run command:

Windows: Ctrl + Shift + F10

macOS: ⌃ ⌥ R

When you press any of these Run buttons or use the keyboard shortcuts, Python Interpreter compiles and executes the code. Keep in mind that, the interpreter does compilation and execution line by line at run time.

In our current project, My_Project, we only have one Python file which is main.py. In general, Python projects have a starting point which is called the main file. Here, our main file is the main.py file which is created by PyCharm when we initialized the project (Figure 2-13).

You can download the Project in the Github Repository of this book. Here is the content of the main.py file:

[1]:

1

# This is a sample Python script.

2

3

# Press Shift+F10 to execute it or replace it with your code.

4

# Press Double Shift to search everywhere for classes, files, tool windows, actions, and settings.

5

6

7

def print_hi(name):

8

    # Use a breakpoint in the code line below to debug your script.

9

    print(f'Hi, {name}')  # Press Ctrl+F8 to toggle the breakpoint.

10

11

12

# Press the green button in the gutter to run the script.

13

if __name__ == '__main__':

14

    print_hi('PyCharm')

15

16

# See PyCharm help at https://www.jetbrains.com/help/pycharm/

17

18

# TODO - Refactor this code

19

20

# TODO - Define a global variable

When we run the project, Python Interpreter executes the code in cell 1 starting from the first line.

We have a function definition in line 7, and the function name is print_hi. This function takes a parameter as name and prints it in an f-string.

In line 13, we have an if statement as: if __name__ == '__main__'. It checks whether the global variable __name__ is equal to text '__main__' or not. If it is True, then it calls the print_hi function as: print_hi('PyCharm').

The __name__ is a special variable which is defined by PyCharm when we first create the project. You can see it in the Python Console in the bottom of the screen. You can set a new value for it if you want by right clicking on it.

Figure 2-29: The special variable __name__ in the Python Console

You don’t have to have a main.py file or a variable called __name__. You don’t have to check its value as we did in line 13. Later on we will change all of these, but for the time being, I wanted to show you how the things work when PyCharm initializes a project and how it runs the main file.

Interpreter Configuration

When you run the project, PyCharm looks for the Python Interpreter and tells it that it should run the code. So the interpreter configuration is crucial. If PyCharm cannot find it you will get an Interpreter Error. As we stated earlier the interpreter is simply the python.exe file and it resides in a virtual environment. Now we will prove this.

We have a project named My_Project in this chapter. And it has its own virtual environment in the venv folder. Remember Figure 2-15, where you can see the python.exe file under the path: \My_Project\venv\Scripts. You can navigate to this path from PyCharm, by right clicking it and Open In > Explorer.

Figure 2-30: Open PyCharm files and folders in file explorer

All of your Python packages, libraries and Python Interpreter reside in venv folder which is your virtual environment. Now that we know the Python Interpreter is in the venv folder let’s delete it and see what happens. To do this, navigate to the top level folder of your project, namely the project folder and delete the venv folder. You may need to close the PyCharm first, because the folder will be in use when you try to delete it.

Figure 2-31: Delete venv folder in the project

Now reopen PyCharm after you deleted the venv folder. You will see that, it will display an error as Invalid Python Interpreter selected for the project. This means that you don’t have a Python Interpreter for this project.

Figure 2-32: Interpreter Error after deleting venv folder

Let’s fix this error and configure the interpreter. There are multiple ways to access the interpreter configuration screen. The first way is to click on the Configure Python Interpreter button which is at the right of the error text. The second way is to click on the file name (which is main here) at the quick access menu and then select Edit Configurations. See the image below:

Figure 2-33: Edit Configurations button in the quick access menu

And the third way is under File > Settings:

Figure 2-34: File > Settings

Let’s move on with File > Settings way. When you click on the Settings button under the File menu it will open the settings window for PyCharm. This is the main screen where we do almost any customization in PyCharm.

Figure 2-35: Settings Window and Python Interpreter configuration

On the Settings Window, under the Project section, you see Python Interpreter. And you will see the Invalid text in the Python Interpreter field on the right (Figure 2-35). Now we will configure a new interpreter.

To configure a new interpreter click on the gear (settings) icon on the right and then click on Add… button (Figure 2-36).

Figure 2-36: Add new Python Interpreter button

PyCharm will open the Add Python Interpreter window as below:

Figure 2-37: Add Python Interpreter window

In Figure 2-37, the Virtual Environment tab is selected. Here you can either create a new virtual environment or use an existing one. We will create a new virtual environment. As you see, the default folder name is venv. When we click the OK button, PyCharm will create this folder and a new virtual environment in it. We will do it in a minute but before that, let me show you the System Interpreter on this screen.

System Interpreter:

If you do not want to create a new virtual environment and a new interpreter, you can use the System Interpreter. It’s the base Interpreter you have when you install Python. To see it, click on the System Interpreter tab on the left.

Figure 2-38: System Interpreter

When you click on the System Interpreter tab, you will see all the Python Interpreters on your machine (Figure 2-38). If you click OK, your current project will be using the Base Interpreter instead of creating a new virtual environment and a new interpreter.

As we stated earlier we will create a new virtual environment for this project, so click on the Virtual Environment tab again and then click the OK button (Figure 2-37). PyCharm creates a new virtual environment for us in the venv folder. Now you can see that the Interpreter Error has been resolved and the new venv folder created in the project. See the final figure below:

Figure 2-39: New virtual environment (venv folder) in the project

Debugging in PyCharm

In computer programming, Debugging is the process of detecting and removing of existing and potential errors. These errors are also called as ‘bugs’. The bugs in a program can cause it to behave unexpectedly or crash. In this section we will see how to do debugging in PyCharm.

Debugging is done via breakpoints. Breakpoints are the special places in the code where you intentionally stop (pause) the execution of the program. This is a temporary stop and we do this to see what’s going on in our program.

The breakpoints are only operational in debug mode. They do not work in normal run mode. In other words, to be able to debug your code at breakpoints you need to run the application in debug mode. We will see it in a minute.

To place a breakpoint in PyCharm, you simply click on the gutter once. The gutter is the column where you see the line numbers. In the image below we put a breakpoint on line 9:

Figure 2-40: A breakpoint in the gutter

We have a breakpoint on line 9, which means PyCharm will pause execution on this line. And we will have look in our program state.

Now let’s run the project in debug mode. In Figure 2-40, you can see the debug run button with a bug icon on the quick access bar. Let’s click on this button and run our application in debug mode.

Figure 2-41: PyCharm in Debug Mode

When you run the application in debug mode, you will see a new window in the bottom, which is the Debug screen (Figure 2-41). There are two main tabs on the Debug window, Debugger and Console. Under the Debugger tab, you can see two parts which are Frames and Variables. Frames tab displays the execution order of your program and the Variables part is where you manage your variables. For example, we only have one variable in the print_hi function, which is name. The value for this variable is ‘PyCharm’.

Resume Program:

You can continue the execution with the Resume Program button (Figure 2-42). PyCharm will continue to execute and it will finish execution if it does not encounter another breakpoint. If it encounters another breakpoint, it will pause again and wait.

Stop:

If you want to stop debugging and exit the Debugger you can use the Stop button (Figure 2-42). It will terminate the current debug session and set the program to its initial state.

Figure 2-42: Resume Program and Stop buttons in Debugger

Now let’s see debugging in more detail. We will first stop the current debug session. Click on the stop button in the debugger window. Then create a new Python file (.py) in our project and name it as debug_example.py. To create a new file, in the Project Explorer pane, right click on the project name and select New > File as in Figure 2-43.  And name the file as debug_example.py.

Figure 2-43: Create a new Python file in project

We will define a function in this new file. Then we will call this function with some arguments. Here is the complete code for the debug_example.py file:

[2]:

1

# define a function to add numbers

2

def summation(x_1, x_2):

3

    sum_of_nums = x_1 + x_2

4

    return sum_of_nums

5

6

# define two variables

7

a = 2

8

b = 5

9

10

# call the function

11

summation(a, b)

We will set two breakpoints in this file, on lines 3 and 11. See the image below:

Figure 2-44: Breakpoints in debug_example.py file

Now our debug_example.py file is ready with breakpoints. Before running it in debug mode, there is a configuration we should change. The run configuration. Currently, it is set to run the main.py. You can see it in Figure 2-44. So if you click Run or Debug buttons it will run the main.py file, not our new file. To change this, right click in the code editor of debug_example.py file and select Debug ‘debug_example’.

Figure 2-45: Run debug_example.py file

When you click on Debug ‘debug_example’, PyCharm runs the debug_example.py file in debug mode. As you see in Figure 2-46, the run configuration is debug_example now. PyCharm pauses the execution on line 11 and launch the Debugger at the bottom. Line 11 is where we call the summation function.

Figure 2-46: debug_example.py file is in debug mode

Now the code execution waits on line 11. In Figure 2-46, you can see that we have to variables a and b and their values. If you click on the Resume Program button, which is the play like green icon, , the code execution will continue until it stops at line 3. Why? Because we have another breakpoint on line 3. See the image below:

Figure 2-47: Second breakpoint is in the summation function

Now in the Debugger we see the summation function on the left the frame and on the right x_1 and x_2 are the variables in this frame.

There is a set of important buttons on the top bar of the Debugger window. These are the stepping toolbar buttons for the debugger. Here they are:

Figure 2-48: Stepping toolbar buttons in Debugger

Here are the functionalities for the most common stepping buttons:

Step Over: Executes the current line.

Step Into: Steps into the method called at the current execution point.

Step Out: Steps out of the current method, to the line executed right after it.

Evaluate Expression: Evaluates Python expressions.

Now let’s evaluate an expression. Remember our debugger is waiting for execution on line 3. And we have an addition operation in this line as: sum_of_nums = x_1 + x_2. Before running this line, let’s evaluate it. Click on the Evaluate Expression button (calculator icon) in the stepping toolbar. It will pop up the evaluation window as follows:

Figure 2-49: Evaluate window

The Evaluate window is very useful and you will be using it most of the time in your projects. You can evaluate an expression, you can do a function call, and you can define variables on this window. It doesn’t affect your project execution. It’s like a separate Python environment which can use the current project. In Figure 2-49 we evaluate the expression of x_1 + x_2 and it returns 7.

Let’s click the Step Over (F8) button to run the current line, which is line 3. PyCharm runs the code in line 3 and it waits on line 4. Because Step Over button only runs one line of code. See the image below:

Figure 2-50: The new line of execution is 4

As you see in Figure 2-50, after we click the Step Over button, PyCharm executes line 3 and it creates a new variable which is sum_of_nums. You can see it in the Variables tab in Debugger. The current line is line 4 now. If we click on Step Over button one more time, PyCharm will finish the execution of the summation function and it will return to the function call in line 11. And if you click it one more time, PyCharm will end the running because there is no more lines to execute.

As a final example, you can remove the breakpoint on line 3 and try the Step Into button to be able to pause inside the summation function. Here is the image for it:

Figure 2-51: Step Into the function

I think that’s enough for debugging in PyCharm. You learned the basics of debugging, how to set breakpoints and how to run and evaluate the code line by line.

We are ready to start actual Python development now. In the next chapter you will learn Exception Handling in Python.

3.      Collections

What are Collections

Collections are specialized container datatypes providing alternatives to Python’s general purpose built-in containers, dict, list, set, and tuple. A Container is a special-purpose object which is used to store different objects. It provides a way to access the contained objects and iterate over them.

Python provides the collections module which implements container datatypes. In this chapter we will learn different classes in this module. You can find the PyCharm project for this chapter in the GitHub Repository of this book.

Chapter Outline:

ChainMap

Counter

Deque

DefaultDict

NamedTuple

OrderedDict

UserDict

UserList

UserString

QUIZ – Collections

SOLUTIONS - Collections

ChainMap

A ChainMap class is provided for quickly linking a number of mappings so they can be treated as a single unit. It is often much faster than creating a new dictionary and running multiple update() calls.

Syntax:

class collections.ChainMap(*maps)

A ChainMap groups multiple dicts or other mappings together to create a single, updateable view (list of dictionaries). If no maps are specified, a single empty dictionary is provided so that a new chain always has at least one mapping.

The underlying mappings are stored in a list. That list is public and can be accessed or updated using the maps attribute. There is no other state in the ChainMap.

A ChainMap incorporates the underlying mappings by reference. So, if one of the underlying mappings gets updated, those changes will be reflected in ChainMap.

All of the usual dictionary methods are supported. In addition, there is:

a maps attribute,

a method for creating new sub contexts

a property for accessing all but the first mapping

maps:

A user updateable list of mappings. The list is ordered from first-searched to last-searched. It is the only stored state and can be modified to change which mappings are searched. The list should always contain at least one mapping.

[1]:

1

# import ChainMap class from collections module

2

from collections import ChainMap

3

4

#--- Defining a ChainMap ---#

5

numbers = {'one': 1, 'two': 2}

6

letters = {'a': 'A', 'b': 'B'}

7

8

# Define the ChainMap

9

chain_map = ChainMap(numbers, letters)

10

11

print(chain_map)

[1]:

ChainMap({'one': 1, 'two': 2}, {'a': 'A', 'b': 'B'})

In cell 1, we define a ChainMap object (chain_map) with two dictionaries. Then we print the ChainMap. As you see in the output, the result is a view of these dicts.

Accessing Keys and Values from ChainMap:

We can access the keys and values of a ChainMap by using the keys() and values() methods.

[2]:

1

#--- Accessing Keys and Values from ChainMap ---#

2

print(chain_map.keys())

3

print(chain_map.values())

[2]:

KeysView(ChainMap({'one': 1, 'two': 2}, {'a': 'A', 'b': 'B'}))

ValuesView(ChainMap({'one': 1, 'two': 2}, {'a': 'A', 'b': 'B'}))

As you see in the output of cell 2, the result of chain_map.keys() is a KeysView and the result of chain_map.values() is a ValuesView.

Accessing Individual Values with Key Names:

We can access individual values from a ChainMap by using the key name. This is exactly the same way what we do with regular dictionaries.

[3]:

1

#--- Accessing Individual Values with Key Names ---#

2

print(chain_map['one'])

3

print(chain_map['b'])

[3]:

1

B

In cell 3, we access the values of the individual items in the underlying dictionaries of the ChainMap by using the key names as: chain_map['one'].

Adding a New Dictionary to ChainMap:

ChainMap can contain any number of dictionaries in it. We use the built-in new_child() method to add new dictionaries to the ChainMap. The new_child() method returns a new ChainMap containing a new map followed by all of the maps in the current instance. One point to note here is, the newly added dict will be placed at the beginning of the ChainMap.

[4]:

1

#--- Adding a New Dictionary to ChainMap ---#

2

variables = {'x': 0, 'y': 1}

3

new_chain_map = chain_map.new_child(variables)

4

print('Old:', chain_map)

5

print('New:', new_chain_map)

[4]:

Old: ChainMap({'one': 1, 'two': 2}, {'a': 'A', 'b': 'B'})

New: ChainMap({'x': 0, 'y': 1}, {'one': 1, 'two': 2}, {'a': 'A', 'b': 'B'})

Get the List of Mappings in ChainMap:

We use the maps attribute the get the list of all mappings in the ChainMap.

[5]:

1

#--- Get the List of Mappings in ChainMap ---#

2

print(chain_map.maps)

[5]:

[{'one': 1, 'two': 2}, {'a': 'A', 'b': 'B'}]

In cell 5, we get all the mappings (dictionaries) in the chain_map. As you see in the output, the maps attribute returns a list type object (a list of dictionaries).

Counter

A Counter is a dict subclass for counting hashable objects. It is a collection where elements are stored as dictionary keys and their counts are stored as dictionary values. Counts are allowed to be any integer value including zero or negative counts. The Counter class is similar to bags or multisets in other languages.

Elements are counted from an iterable or initialized from another mapping (or counter). Here are some ways we create Counter objects in Python:

[6]:

1

from collections import Counter

2

3

# a new, empty counter

4

c1 = Counter()

5

print(c1)

6

7

# a new counter from an iterable

8

c2 = Counter('aabbbcddeeee')

9

print(c2)

10

11

# a new counter from a mapping

12

c3 = Counter({'orange': 6, 'red': 3, 'green': 5})

13

print(c3)

14

15

# a new counter from keyword args

16

c4 = Counter(cats=4, dogs=8)

17

print(c4)

[6]:

Counter()

Counter({'e': 4, 'b': 3, 'a': 2, 'd': 2, 'c': 1})

Counter({'orange': 6, 'green': 5, 'red': 3})

Counter({'dogs': 8, 'cats': 4})

Counter objects have a dictionary interface except that they return a zero count for missing items instead of raising a KeyError:

[7]:

1

# count of existing element

2

c5 = Counter(['eggs', 'ham', 'jar', 'ham'])

3

print(c5['ham'])

4

5

# count of a missing element is zero

6

print(c5['bacon'])

[7]:

2

0

Delete Elements from a Counter:

To delete elements from a Counter, we use the del keyword. Please keep in mind that, setting a count to zero does not remove an element from a counter.

[8]:

1

# --- Delete Elements from a Counter --- #

2

# counter entry with a zero count

3

c5['sausage'] = 0

4

print(c5)

5

6

# del actually removes the entry

7

del c5['sausage']

8

print(c5)

[8]:

Counter({'ham': 2, 'eggs': 1, 'jar': 1, 'sausage': 0})

Counter({'ham': 2, 'eggs': 1, 'jar': 1})

As you see in cell 8, we set zero to an item which even doesn’t exist in the Counter. And Python adds that item to the Counter with zero value. In line 7, we remove the item entirely with the del keyword.

Counter Methods:

Counter objects support additional methods beyond those available for all dictionaries. Here are the most common methods:

elements():

Return an iterator over elements repeating each as many times as its count. Elements are returned in the order first encountered. If an element’s count is less than one, elements() will ignore it.

[9]:

1

# --- Counter Methods --- #

2

# elements()

3

counter = Counter(a=1, b=2, c=0, d=-2, e=4)

4

sorted_elements = sorted(counter.elements())

5

print(sorted_elements)

[9]:

['a', 'b', 'b', 'e', 'e', 'e', 'e']

most_common([n]):

Return a list of the n most common elements and their counts from the most common to the least. If n is omitted or None, most_common() returns all elements in the counter. Elements with equal counts are ordered in the order first encountered:

[10]:

1

# most_common()

2

most_common_3 = Counter('abracadabra').most_common(3)

3

print(most_common_3)

[10]:

[('a', 5), ('b', 2), ('r', 2)]

subtract([iterable-or-mapping]):

Elements are subtracted from an iterable or from another mapping (or counter). Like dict.update() but subtracts counts instead of replacing them. Both inputs and outputs may be zero or negative.

[11]:

1

# subtract()

2

c_1 = Counter(a=4, b=2, c=0, d=-2)

3

c_2 = Counter(a=1, b=2, c=3, d=4)

4

c_1.subtract(c_2)

5

print(c_1)

[11]:

Counter({'a': 3, 'b': 0, 'c': -3, 'd': -6})

The usual dictionary methods are available for Counter objects except for two which work differently for counters:

fromkeys(iterable):

This class method is not implemented for Counter objects.

update([iterable-or-mapping]):

Elements are counted from an iterable or added-in from another mapping (or counter). Like dict.update() but adds counts instead of replacing them. Also, the iterable is expected to be a sequence of elements, not a sequence of (key, value) pairs.

[12]:

1

# update()

2

d = Counter(a=3, b=1)

3

d.update({'a': 5, 'c': 4})

4

print(d)

[12]:

Counter({'a': 8, 'c': 4, 'b': 1})

Deque

Deques are a generalization of stacks and queues (the name is pronounced “deck” and is short for “double-ended queue”). Deques support thread-safe, memory efficient appends and pops from either side of the deque with approximately the same O(1) performance in either direction.

Though list objects support similar operations, they are optimized for fast fixed-length operations and incur O(n) memory movement costs for pop(0) and insert(0, v) operations which change both the size and position of the underlying data representation.

collections.deque([iterable[, maxlen]]): Returns a new deque object initialized left-to-right (using append()) with data from iterable. If iterable is not specified, the new deque is empty.

maxlen: Maximum size of a deque or None if unbounded.

If maxlen is not specified or is None, deques may grow to an arbitrary length. Otherwise, the deque is bounded to the specified maximum length. Once a bounded length deque is full, when new items are added, a corresponding number of items are discarded from the opposite end.

[13]:

1

### Deque

2

3

from collections import deque

4

5

# Declaring the deque

6

q = deque(['user', 'password', 'token'])

7

print(q)

[13]:

deque(['user', 'password', 'token'])

In cell 13, we define a deque object by passing a list as argument. Now let’s create another one but this time we will use a string:

[14]:

1

# make a new deque with three items

2

d = deque('dqi')

3

# iterate over the deque's elements

4

for elem in d:

5

    print(elem.upper())

[14]:

D

Q

I

Now let’s see the contents in the deque object:

[15]:

1

# list the contents of the deque

2

deque_contents = list(d)

3

print(deque_contents)

4

5

# peek at leftmost item

6

print(d[0])

7

8

# peek at rightmost item

9

print(d[-1])

[15]:

['d', 'q', 'i']

d

i

Here are some methods that deque objects support:

append(x):

Add x to the right side of the deque.

appendleft(x):

Add x to the left side of the deque.

[16]:

1

# add a new entry to the right side

2

d.append('j')

3

4

# add a new entry to the left side

5

d.appendleft('f')

6

7

# show the representation of the deque

8

print(d)

[16]:

deque(['f', 'd', 'q', 'i', 'j'])

pop():

Remove and return an element from the right side of the deque. If no elements are present, raises an IndexError.

popleft():

Remove and return an element from the left side of the deque. If no elements are present, raises an IndexError.

[17]:

1

# return and remove the rightmost item

2

rightmost = d.pop()

3

print(rightmost)

4

5

# return and remove the leftmost item

6

leftmost = d.popleft()

7

print(leftmost)

[17]:

j

f

clear():

Remove all elements from the deque leaving it with length 0.

copy():

Create a shallow copy of the deque.

count(x):

Count the number of deque elements equal to x.

extend(iterable):

Extend the right side of the deque by appending elements from the iterable argument.

[18]:

1

# add multiple elements at once

2

d.extend('jkl')

3

print(d)

[18]:

deque(['d', 'q', 'i', 'j', 'k', 'l'])

extendleft(iterable):

Extend the left side of the deque by appending elements from iterable. Note, the series of left appends results in reversing the order of elements in the iterable argument.

[19]:

1

# extendleft() reverses the input order

2

d.extendleft('xyz')

3

print(d)

[19]:

deque(['z', 'y', 'x', 'd', 'q', 'i', 'j', 'k', 'l'])

index(x[, start[, stop]]):

Return the position of x in the deque (at or after index start and before index stop). Returns the first match or raises ValueError if not found.

insert(i, x):

Insert x into the deque at position i. If the insertion would cause a bounded deque to grow beyond maxlen, an IndexError is raised.

remove(value):

Remove the first occurrence of value. If not found, raises a ValueError.

rotate(n=1):

Rotate the deque n steps to the right. If n is negative, rotate to the left.

[20]:

1

# deque at the beginning

2

print(d)

3

4

# right rotation

5

d.rotate(1)

6

print(d)

7

8

# left rotation

9

d.rotate(-1)

10

print(d)

[20]:

deque(['z', 'y', 'x', 'd', 'q', 'i', 'j', 'k', 'l'])

deque(['l', 'z', 'y', 'x', 'd', 'q', 'i', 'j', 'k'])

deque(['z', 'y', 'x', 'd', 'q', 'i', 'j', 'k', 'l'])

reverse():

Reverse the elements of the deque in-place and then return None.

[21]:

1

# deque at the beginning

2

print('old deque:', d)

3

4

# reverse the elements in the deque

5

new_deq = d.reverse()

6

print('new deque:', new_deq)

7

8

# original deque after reversed()

9

print('old deque:', d)

[21]:

old deque: deque(['z', 'y', 'x', 'd', 'q', 'i', 'j', 'k', 'l'])

new deque: None

old deque: deque(['l', 'k', 'j', 'i', 'q', 'd', 'x', 'y', 'z'])

As you see in the output of cell 21, the reverse() method reverses the elements of the deque in-place, which means our original deque object is modified. And it returns None.

DefaultDict

One of the common problems with the Dictionary class in Python is the missing keys. When you try to access a key that does not exist in the dictionary you will get a KeyError. So, you have to handle this case whenever you need to access an element in the dictionary. Fortunately, we have DefaultDict class in Python. It is used to provide some default values for a key that does not exist and does not raise a KeyError.

DefaultDict is a subclass of the built-in dict class. It overrides one method and adds one writable instance variable. The remaining functionality is the same as for the dict class and is not documented here.

collections.defaultdict(default_factory=None, /[, ...]): Return a new dictionary-like object, DefaultDict, which is a subclass of the built-in dict class.

The first argument provides the initial value for the default_factory attribute; it defaults to None. All remaining arguments are treated the same as if they were passed to the dict constructor, including keyword arguments.

DefaultDict objects support the following method in addition to the standard dict operations:

__missing__(key):

If the default_factory attribute is None, this raises a KeyError exception with the key as argument.

If default_factory is not None, it is called without arguments to provide a default value for the given key, this value is inserted in the dictionary for the key, and returned.

DefaultDict objects support the following instance variable:

default_factory:

This attribute is used by the __missing__() method; it is initialized from the first argument to the constructor, if present, or to None, if absent.

[22]:

1

from collections import defaultdict

2

3

s = [('yellow', 1), ('blue', 2), ('yellow', 3), ('blue', 4), ('red', 1)]

4

d = defaultdict(list)

5

for k, v in s:

6

    d[k].append(v)

7

8

sorted_items = sorted(d.items())

9

print(sorted_items)

[22]:

[('blue', [2, 4]), ('red', [1]), ('yellow', [1, 3])]

In cell 20, we use the list type as the default_factory, to make it easy to group a sequence of key-value pairs into a dictionary of lists. When each key is encountered for the first time, an entry is automatically created using the default_factory function which returns an empty list. The list.append() operation then attaches the value to the new list. When keys are encountered again, the look-up proceeds normally (returning the list for that key) and the list.append() operation adds another value to the list. This technique is simpler and faster than an equivalent technique using dict.setdefault().

[23]:

1

river = 'mississippi'

2

dd = defaultdict(int)

3

for r in river:

4

    dd[r] += 1

5

6

s_items = sorted(dd.items())

7

print(s_items)

[23]:

[('i', 4), ('m', 1), ('p', 2), ('s', 4)]

In cell 23, we set the default_factory to int. This makes the DefaultDict useful for counting (like a bag or multiset in other languages). When a letter is first encountered, it is missing from the mapping, so the default_factory function calls int() to supply a default count of zero. The increment operation then builds up the count for each letter.

NamedTuple

NamedTuples assign meaning to each position in a tuple and allow for more readable, self-documenting code. They can be used wherever regular tuples are used, and they add the ability to access fields by name instead of position index.

collections.namedtuple(typename, field_names):

Returns a new tuple subclass named typename. The new subclass is used to create tuple-like objects that have fields accessible by attribute lookup as well as being indexable and iterable. Instances of the subclass also have a helpful docstring (with typename and field_names) and a helpful __repr__() method which lists the tuple contents in a name=value format.

The field_names are a sequence of strings such as ['x', 'y']. Alternatively, field_names can be a single string with each fieldname separated by whitespace and/or commas, for example 'x y' or 'x, y'.

To understand how the NamedTuple works, let’s assume we have an Employee object. Employee has id, name and age attributes.

[24]:

1

from collections import namedtuple

2

3

# Declare the namedtuple

4

Employee = namedtuple('Employee', ['id', 'name', 'age'])

5

6

# Add some values to the tuple

7

E_1 = Employee('111', 'Peter Parker', '18')

8

E_2 = Employee('222', 'Clark Kent', '26')

9

10

# Access using index

11

print("Employee name by index is : ", end="")

12

print(E_1[1])

13

14

# Access using keys

15

print("Employee name using key is : ", end="")

16

print(E_2.name)

[24]:

Employee name by index is : Peter Parker

Employee name using key is : Clark Kent

In addition to the methods inherited from tuples, named tuples support three additional methods and two attributes. To prevent conflicts with field names, the method and attribute names start with an underscore.

_make(iterable):

Class method that makes a new instance from an existing sequence or iterable.

[25]:

1

# _make()

2

# initialize an iterable

3

bat_data = ['333', 'Batman', '28']

4

batman = Employee._make(bat_data)

5

print(batman)

[25]:

Employee(id='333', name='Batman', age='28')

_asdict():

Return a new dict which maps field names to their corresponding values:

[26]:

1

# _asdict()

2

bat_dict = batman._asdict()

3

print(bat_dict)

[26]:

{'id': '333', 'name': 'Batman', 'age': '28'}

_replace(**kwargs):

Return a new instance of the named tuple replacing specified fields with new values:

[27]:

1

# _replace()

2

batman = batman._replace(id='777', age='34')

3

print(batman)

[27]:

Employee(id='777', name='Batman', age='34')

_fields:

Tuple of strings listing the field names. Useful for introspection and for creating new named tuple types from existing named tuples.

[28]:

1

# _fields

2

print(batman._fields)

[28]:

('id', 'name', 'age')

We can use the _fields attribute to create new namedtuples from existing ones:

[29]:

1

# namedtuple fields from others

2

Point = namedtuple('Point', ['x', 'y'])

3

Color = namedtuple('Color', 'red green blue')

4

Pixel = namedtuple('Pixel', Point._fields + Color._fields)

5

p = Pixel(5, 8, 128, 255, 0)

6

print(p)

[29]:

Pixel(x=5, y=8, red=128, green=255, blue=0)

OrderedDict

Ordered Dictionaries are just like regular dictionaries but have some extra capabilities relating to ordering operations. OrderedDicts remember the order in which the keys were inserted. They have become less important now that the built-in dict class gained the ability to remember insertion order (this new behavior became guaranteed in Python 3.7).

collections.OrderedDict([items]):

Return an instance of a dict subclass that has methods specialized for rearranging dictionary order.

popitem(last=True):

The popitem() method for ordered dictionaries returns and removes a (key, value) pair. The pairs are returned in LIFO (last in first out) order if last is true or FIFO (first in first out) order if false.

move_to_end(key, last=True):

Move an existing key to either end of an ordered dictionary. The item is moved to the right end if last is true (the default) or to the beginning if last is false. Raises KeyError if the key does not exist:

[30]:

1

from collections import OrderedDict

2

3

od = OrderedDict.fromkeys('abcde')

4

od.move_to_end('b')

5

print(''.join(od))

6

# 'acdeb'

7

od.move_to_end('b', last=False)

8

print(''.join(od))

9

# 'bacde'

[30]:

acdeb

bacde

Let’s say we delete and re-insert the same key to an OrderedDict. It will push this key to the end to maintain the order of insertion of the keys.

[31]:

1

# delete and re-insert same key

2

d = OrderedDict()

3

d['x'] = 'X'

4

d['y'] = 'Y'

5

d['z'] = 'Z'

6

7

print('OrderedDict before deleting')

8

for key, value in d.items():

9

    print(key, value)

10

11

# delete the element

12

d.pop('x')

13

14

# re-insert the same key

15

d['x'] = 'X'

16

17

print('\nOrderedDict after insertion')

18

for key, value in d.items():

19

    print(key, value)

[31]:

OrderedDict before deleting

x X

y Y

z Z

OrderedDict after insertion

y Y

z Z

x X

UserDict

The class, UserDict acts as a wrapper around dictionary objects. The need for this class has been partially supplanted by the ability to subclass directly from dict; however, this class can be easier to work with because the underlying dictionary is accessible as an attribute. You can use UserDict when you want to create your own dictionary with some modified or new functionality.

collections.UserDict([initialdata]):

Class that simulates a dictionary. The instance’s contents are kept in a regular dictionary, which is accessible via the data attribute of UserDict instances. If initialdata is provided, data is initialized with its contents; note that a reference to initialdata will not be kept, allowing it to be used for other purposes.

In addition to supporting the methods and operations of mappings, UserDict instances provide the following attribute:

data:

A real dictionary used to store the contents of the UserDict class.

[32]:

1

from collections import UserDict

2

3

us = {'name': 'John Doe', 'age': 24}

4

5

# Create UserDict object

6

ud = UserDict(us)

7

print(ud.data)

[32]:

{'name': 'John Doe', 'age': 24}

Let’s say we want to define a custom dictionary object which supports addition operation. When we add two instances of our custom dictionary, we want to get a new dictionary with all of the elements in both dictionaries. Keep in mind that, you will get TypeError if you try to add to regular dicts in Python. Let’s implement this with the help of UserDict:

[33]:

1

# class for our custom dict

2

# inherit from UserDict

3

class AddEnabledDict(UserDict):

4

    # override the __add__ method

5

    def __add__(self, other):

6

        d = AddEnabledDict(self.data)

7

        d.update(other.data)

8

        return d

9

10

# create custom objects

11

d_1 = AddEnabledDict(x = 10)

12

d_2 = AddEnabledDict(y = 20)

13

total = d_1 + d_2

14

print(total)

[33]:

{'x': 10, 'y': 20}

UserList

The UserList class acts as a wrapper around list objects. It is a useful base class for your own list-like classes which can inherit from them and override existing methods or add new ones. In this way, one can add new behaviors to lists in Python.

The need for this class has been partially supplanted by the ability to subclass directly from list; however, this class can be easier to work with because the underlying list is accessible as an attribute.

collections.UserList([list]):

Class that simulates a list. The instance’s contents are kept in a regular list, which is accessible via the data attribute of UserList instances. The instance’s contents are initially set to a copy of list, defaulting to the empty list []. list parameter can be any iterable, for example a real Python list or a UserList object.

In addition to supporting the methods and operations of mutable sequences, UserList instances provide the following attribute:

data:

A real list object used to store the contents of the UserList class.

Let’s say we want to define a list which doesn’t allow deleting the items in it. We can easily define such a class by inheriting UserList:

[34]:

1

from collections import UserList

2

3

# define a custom class

4

# this class will inherit from UserList

5

# it will not allow its items to be deleted

6

# List class in Python has to methods for delete:

7

# remove() and pop()

8

class ListWithNoItemDelete(UserList):

9

    # override remove() method

10

    def remove(self, s=None):

11

        self.not_allowed()

12

13

    # override pop() method

14

    def pop(self, s=None):

15

        self.not_allowed()

16

17

    def not_allowed(self):

18

        raise RuntimeError("Deletion not allowed")

19

20

# custom list object

21

custom_list = ListWithNoItemDelete(['a', 'b', 'c'])

22

23

# try to delete an item

24

custom_list.remove('b')

[34]:

RuntimeError: Deletion not allowed

UserString

The class, UserString acts as a wrapper around string objects. The need for this class has been partially supplanted by the ability to subclass directly from str; however, this class can be easier to work with because the underlying string is accessible as an attribute.

collections.UserString(seq):

Class that simulates a string object. The instance’s content is kept in a regular string object, which is accessible via the data attribute of UserString instances. The instance’s contents are initially set to a copy of seq. The seq argument can be any object which can be converted into a string using the built-in str() function.

In addition to supporting the methods and operations of strings, UserString instances provide the following attribute:

data:

A real str object used to store the contents of the UserString class.

Let’s say we want to define a custom str class that have concatenate() method in it:

[35]:

1

# UserString

2

3

from collections import UserString

4

5

# define a custom class

6

# this class will inherit from UserString

7

class CustomStrClass(UserString):

8

    # define a new method

9

    def concatenate(self, other=None, delimiter=' '):

10

        self.data += delimiter + other

11

12

# custom string object

13

custom_str = CustomStrClass('My Custom')

14

custom_str.concatenate('String Class')

15

print(custom_str)

[35]:

My Custom String Class

QUIZ - Collections

Now it’s time to solve the quiz for this chapter. You can download the quiz file, QUIZ_Collections.zip, from the GitHub Repository of this book. You should put the quiz file in the Collections project we build in this chapter.

Here are the questions for this chapter:

Q1:

Here are three dictionaries:

letters = {'a': 'A', 'b': 'B', 'c': 'C'}

numbers = {1: 1000, 2: 2000, 3: 3000}

fruits = {'orange': 300, 'apple': 500}

Define a ChainMap by using letters and numbers dictionaries.

The name of the ChainMap will be my_map.

Print the maps in my_map.

Print all of the keys and values in my_map as: "Key : Value".

Add fruits dictionary to the my_map.

Print the maps in this new ChainMap.

Print all of the keys and values in this new ChainMap as: "Key : Value".

Hints:

* new_child()

* new_child() will return a new ChainMap

[1]:

1

# Q 1:

2

3

# import ChainMap

4

# ---- your solution here ---

5

6

# dictionaries

7

# ---- your solution here ---

8

9

# define the ChainMap

10

# ---- your solution here ---

11

12

# print maps

13

# ---- your solution here ---

14

15

# print key: value pairs

16

# ---- your solution here ---

17

18

# add fruits dictionary

19

# ---- your solution here ---

20

21

# print maps

22

# ---- your solution here ---

23

24

# print key: value pairs

25

# ---- your solution here ---

[1]:

Mappings in my_map:

[{'a': 'A', 'b': 'B'}, {1: 100, 2: 200}]

Key:Value pairs in my_map:

1: 100

2: 200

a: A

b: B

Mappings in new_map:

[{'orange': 33, 'apple': 55}, {'a': 'A', 'b': 'B'}, {1: 100, 2: 200}]

Key:Value pairs in new_map:

1: 100

2: 200

a: A

b: B

orange: 33

apple: 55

Q2:

Here is some text:

'''Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Quisque bibendum sodales ipsum in lacinia.

Morbi metus dui, venenatis ut molestie a, porta sit amet est.

Maecenas sagittis turpis nec nisl tempus consequat.'''

Define a Counter which will store all the characters

and their respective counts in this text.

Print the most common 5 characters in the text.

Hints:

* Counter

* most_common()

* we don't want to count the space character

[2]:

1

# Q 2:

2

3

# import Counter class

4

from collections import Counter

5

6

# define the text

7

# ---- your solution here ---

8

9

# remove the space character

10

# ---- your solution here ---

11

12

# define the Counter

13

# ---- your solution here ---

14

15

# print most common 5 characters

16

# ---- your solution here ---

[2]:

[('i', 21), ('e', 20), ('s', 20), ('t', 18), ('a', 13)]

Q3:

Create a deque object with name my_deque.

my_deque will have first four letters in the alphabet: a, b, c, d.

Extend this deque from the left by adding three more letters: e, f, g.

Print the items in my_deque.

Create a new deque (new_deque) using my_deque.

new_deque will be the reverse of my_deque.

Print the items in new_deque.

Please be careful that, my_deque object will not be modified.

Hints:

* deque

* extendleft()

* reverse()

[3]:

1

# Q 3:

2

3

# import Deque class

4

# ---- your solution here ---

5

6

# define a new deque

7

# ---- your solution here ---

8

9

# extend my_deque from left

10

# ---- your solution here ---

11

12

# print items in my_deque

13

# ---- your solution here ---

14

15

# copy my_deque

16

# ---- your solution here ---

17

18

# reverse my_deque

19

# ---- your solution here ---

20

21

# print items in new_deque

22

# ---- your solution here ---

[3]:

['g', 'f', 'e', 'a', 'b', 'c', 'd']

['d', 'c', 'b', 'a', 'e', 'f', 'g']

Q4:

We have a long word in English: incomprehensibilities.

We want to count occurrences of each letter in this word.

Print the most common three letters in this word.

Hints:

* use defaultdict

* default_factory

* do not use Counter

[4]:

1

# Q 4:

2

3

# import defaultdict

4

# ---- your solution here ---

5

6

# define the text

7

# ---- your solution here ---

8

9

# define the defaultdict

10

# ---- your solution here ---

11

12

# fill the defaultdict

13

# ---- your solution here ---

14

15

# sort items in the defualtdict

16

# by number of occurrences in decreasing order

17

# ---- your solution here ---

18

19

# print the most common 3 words

20

# ---- your solution here ---

[4]:

[('i', 5), ('e', 3), ('n', 2)]

Q5:

Define a namedtuple as SuperHero.

SuperHero has three attributes:

* hero_name

* real_name

* age

Create two SuperHero instance as follows:

* Spiderman, Peter Parker, 18

* Superman, Clark Kent, 26

Print the real names of SuperHeroes using keys as follows:

* Real name of Spiderman is: Peter Parker

* Real name of Superman is: Clark Kent

Hints:

* namedtuple

* attributes are keys as: namedtuple.key

[5]:

1

# Q 5:

2

3

# import namedtuple

4

# ---- your solution here ---

5

6

# Declare the namedtuple

7

# ---- your solution here ---

8

9

# Create SuperHero instances

10

# ---- your solution here ---

11

12

# Access using key

13

# ---- your solution here ---

[5]:

Real name of Spiderman is: Peter Parker

Real name of Superman is: Clark Kent

SOLUTIONS - Collections

Here are the solutions for the quiz for this chapter.

S1:

[1]:

1

# S 1:

2

3

# import ChainMap

4

from collections import ChainMap

5

6

# dictionaries

7

letters = {'a': 'A', 'b': 'B'}

8

numbers = {1: 100, 2: 200}

9

fruits = {'orange': 33, 'apple': 55}

10

11

# define the ChainMap

12

my_map = ChainMap(letters, numbers)

13

14

# print maps

15

print('Mappings in my_map:\n', my_map.maps)

16

17

# print key: value pairs

18

print('\nKey:Value pairs in my_map:')

19

for key, value in my_map.items():

20

    print(f'{key}: {value}')

21

22

# add fruits dictionary

23

new_map = my_map.new_child(fruits)

24

25

# print maps

26

print('Mappings in new_map:\n', new_map.maps)

27

28

# print key: value pairs

29

print('\nKey:Value pairs in new_map:')

30

for key, value in new_map.items():

31

    print(f'{key}: {value}')

[1]:

Mappings in my_map:

[{'a': 'A', 'b': 'B'}, {1: 100, 2: 200}]

Key:Value pairs in my_map:

1: 100

2: 200

a: A

b: B

Mappings in new_map:

[{'orange': 33, 'apple': 55}, {'a': 'A', 'b': 'B'}, {1: 100, 2: 200}]

Key:Value pairs in new_map:

1: 100

2: 200

a: A

b: B

orange: 33

apple: 55

S2:

[2]:

1

# S 2:

2

3

# import Counter class

4

from collections import Counter

5

6

# define the text

7

text = '''Lorem ipsum dolor sit amet, consectetur adipiscing elit.

8

Quisque bibendum sodales ipsum in lacinia.

9

Morbi metus dui, venenatis ut molestie a, porta sit amet est.

10

Maecenas sagittis turpis nec nisl tempus consequat.'''

11

12

# remove the space character

13

text = ''.join(text.split())

14

15

# define the Counter

16

text_counter = Counter(text)

17

18

# print most common 5 characters

19

most_common_5 = text_counter.most_common(5)

20

print(most_common_5)

[2]:

[('i', 21), ('e', 20), ('s', 20), ('t', 18), ('a', 13)]

S3:

[3]:

1

# S 3:

2

3

# import Deque class

4

from collections import deque

5

6

# define a new deque

7

my_deque = deque(['a', 'b', 'c', 'd'])

8

print(list(my_deque))

9

10

# extend my_deque from left

11

my_deque.extendleft(['e', 'f', 'g'])

12

13

# print items in my_deque

14

print(list(my_deque))

15

16

# copy my_deque

17

new_deque = my_deque.copy()

18

19

# reverse my_deque

20

new_deque.reverse()

21

22

# print items in new_deque

23

print(list(new_deque))

[3]:

['g', 'f', 'e', 'a', 'b', 'c', 'd']

['d', 'c', 'b', 'a', 'e', 'f', 'g']

S4:

[4]:

1

# S 4:

2

3

# import defaultdict

4

from collections import defaultdict

5

6

# define the text

7

word = "incomprehensibilities"

8

9

# define the defaultdict

10

def_dict = defaultdict(int)

11

12

# fill the defaultdict

13

for letter in word:

14

    def_dict[letter] += 1

15

16

# sort items in the defualtdict

17

# by number of occurrences in decreasing order

18

sorted_items = sorted(def_dict.items(), key=lambda k_v: k_v[1], reverse=True)

19

20

# print the most common 3 words

21

print(sorted_items[:3])

[4]:

[('i', 5), ('e', 3), ('n', 2)]

S5:

[5]:

1

# S 5:

2

3

# import namedtuple

4

from collections import namedtuple

5

6

# Declare the namedtuple

7

SuperHero = namedtuple('SuperHero', ['hero_name', 'real_name', 'age'])

8

9

# Add some values to the tuple

10

S_1 = SuperHero('Spiderman', 'Peter Parker', '18')

11

S_2 = SuperHero('Superman', 'Clark Kent', '26')

12

13

# Access using key

14

print(f"Real name of {S_1.hero_name} is: {S_1.real_name}")

15

print(f"Real name of {S_2.hero_name} is: {S_2.real_name}")

[5]:

Real name of Spiderman is: Peter Parker

Real name of Superman is: Clark Kent

4.      Iterators

Iterables and Iterators

Iterator:

An Iterator is an object representing a stream of data and can be iterated upon. In technical terms, a Python iterator is an object that implements the iterator protocol, which consist of two special methods: __iter__() and __next__().

Iterable:

An Iterable is an object capable of returning its members one at a time. Examples of iterables include all sequence types (such as list, str, and tuple) and some non-sequence types like dict, and file objects. Technically, iterables are objects of any classes with an __iter__() method or with a __getitem__() method.

iter():

The iter() function (which calls the __iter__() method behind the scenes) returns an iterator object. So, we can say that; an iterable is an object which returns an iterator.

In this chapter, we will learn how iterators work in Python and how we can define our own iterator classes. You can find the PyCharm project for this chapter in the GitHub Repository of this book.

Chapter Outline:

The Iterator Protocol

Looping Through an Iterator

Define a Custom Iterator

Infinite Iterators

Benefits of Iterators

QUIZ – Iterators

SOLUTIONS - Iterators

The Iterator Protocol

In Python, Iterator objects are required to support the following two methods, which together form the Iterator Protocol:

__iter__():

Returns the iterator object itself. This is required to allow both containers and iterators to be used with the for loops and the in statements. You can use built-in iter() function which calls the __iter__() method.

__next__():

Returns the next item from an iterator. If there are no further items, raise the StopIteration exception. You can use built-in next() function which calls the __next__() method.

As we learned in the previous section, lists, tuples, dictionaries, and sets are all iterable types. In other words, they are the types which you can get an iterator from. Let’s see some examples:

[1]:

1

# define a tuple (iterable)

2

tup = ("A", "B", "C")

3

# get an iterator from the iterable -> iter()

4

tup_iter = iter(tup)

5

6

# call the next() function

7

item_1 = next(tup_iter)

8

print(item_1)

9

10

# call the next() function

11

item_2 = next(tup_iter)

12

print(item_2)

13

14

# call the next() function

15

item_3 = next(tup_iter)

16

print(item_3)

[1]:

A

B

C

In cell 1, we define a tuple which is an iterable. Then in line 4, we call the iter() function on this iterable. The iter() function returns an iterator and we name it as tup_iter. In lines 7 to 16 we call the next() function several times. Each time the next() function executes, it returns the next item in the iterator.

[2]:

1

# define a string (iterable)

2

pyt = 'python'

3

# get an iterator from the iterable -> iter()

4

pyt_iter = pyt.__iter__()

5

6

# call the next() function

7

item_1 = pyt_iter.__next__()

8

print(item_1)

9

10

# call the next() function

11

item_2 = pyt_iter.__next__()

12

print(item_2)

[2]:

p

y

In cell 2, we call the __iter__() method on a string object. Strings are iterable objects that contain a sequence of characters. The __iter__() method returns an iterator in line 4. And we print the elements in this iterator one by one by calling the __next__() method.

Looping Through an Iterator

As we see in the previous sections, we use the next() function (or __next__() method) to manually iterate over the items of an iterator. When the next() function reaches the end of the iterator, then there is no more data to be returned and you will get an StopIteration exception.

[3]:

1

# define a list (iterable)

2

a_list = [10, 20, 30]

3

4

# get an iterator from the iterable -> iter()

5

list_iter = iter(a_list)

6

7

# call the next() function

8

print(next(list_iter))

9

print(next(list_iter))

10

print(next(list_iter))

11

12

# the following next() call will raise exception

13

print(next(list_iter))

[3]:

10

20

30

StopIteration

In cell 3, we call the next() function four times which is more than the number of items in the iterator. And in the last call we get StopIteration exception.

The for loop in Python, is able to iterate automatically through any object that can return an iterator. In other words, the for loop can iterate over any iterable object in Python.

[4]:

1

# for loop

2

for element in a_list:

3

    print(element)

[4]:

10

20

30

In cell 4, we use the for loop to iterate over the list we defined in cell 3. As you see, we do not use the next() function manually or we don’t get any StopIteration exception. That’s the beauty of the for loop in Python. It handles all of these for us behind the scenes.

Now let’s define our own version of the for loop. We will use the while loop and copy the behavior of the for loop. At this point, we have everything we need for this implementation. Let’s do it:

[5]:

1

# custom for loop implementation

2

# iterable

3

my_list = [1, 2, 3]

4

5

# iterator from this iterable

6

list_iter = my_list.__iter__()

7

8

# while loop

9

while True:

10

    try:

11

        # next item

12

        element = list_iter.__next__()

13

        print(element)

14

    except StopIteration:

15

        break

[5]:

1

2

3

In cell 5, we implement our own version of the for loop. We use an infinite while loop as: while True. We set a try-except block inside the loop. In the try block, we get the next element by calling the __next__() method on our iterator. If this call is successful then we print the element. If an error occurs, which is of type StopIteration, then we catch that exception in the except block. What we do inside the except block is very simple. We simply break the loop. Which means we have already reached the end of our iterator.

Define a Custom Iterator

Now that we know about iterators and iterator protocol (__iter__() and __next__() methods) we can define our own iterator classes from scratch.

Defining an iterator is quite easy in Python. All we have to do is to implement __iter__() and __next__() methods in our class definition. Here are the general rules that we must follow:

The __iter__() method must return the iterator object itself.

The __next__() method must return the next item in the sequence. Also, it has to raise a StopIteration exception when it reaches the end of the sequence.

Let’s define an iterator object which will generate a sequence of odd numbers like 1, 3, 5, 7, 9, … etc.

[6]:

1

# custom iterator

2

class Odd:

3

    # implement __init__ method

4

    def __init__(self, limit):

5

        self.current = 1

6

        self.limit = limit

7

8

    # implement __iter__ method

9

    # simply return the object itself

10

    def __iter__(self):

11

        return self

12

13

    # implement __next__ method

14

    def __next__(self):

15

        # check if limit reached

16

        if self.current <= self.limit:

17

            # get the current value

18

            current_value = self.current

19

            # increase the current

20

            self.current += 2

21

            return current_value

22

        # limit is reached so raise exception

23

        else:

24

            raise StopIteration

In cell 6, we define our own iterator class. It implements both __iter__() and __next__() methods:

In the __iter__() method, it returns the object itself as: return self.

In the __next__() method, it checks if the current value is smaller than or equal to the limit. If this is True, then it returns the existing value of self.current and increases the self.current value by 2. If the limit is exceeded than it will raise a StopIteration exception.

Now let’s call this class and get some odd numbers up to 20:

[7]:

1

# instantiate an Odd object

2

odd_numbers = Odd(20)

3

4

# get first 4 odd numbers

5

print(odd_numbers.__next__())

6

print(odd_numbers.__next__())

7

print(next(odd_numbers))

8

print(next(odd_numbers))

[7]:

1

3

5

7

In cell 7, we instantiate an object from our Odd class. This object will hold the odd numbers from 1 to 20. And we print the first four odd numbers by calling the next() method on this object four times.

Since our Odd class is an iterator, we can easily set a for loop to iterate over it. Let’s do it:

[8]:

1

# iterate over Odd class

2

for n in Odd(8):

3

    print(n)

[8]:

1

3

5

7

Infinite Iterators

Infinite Iterators are special type objects which has no terminating conditions in their __next__() methods. They can be useful when you need to set a counter that you do not know where it will finalize. Let’s define a custom infinite iterator that keeps increasing one by one.

[9]:

1

class InfiniteCounter:

2

    def __init__(self):

3

        self.n = 1

4

5

    def __iter__(self):

6

        return self

7

8

    def __next__(self):

9

        current = self.n

10

        self.n += 1

11

        return current

Now let’s call our InfiniteCounter class and get some numbers in ascending order:

[10]:

1

# get first four numbers

2

counter = InfiniteCounter()

3

print(counter.__next__())

4

print(counter.__next__())

5

print(next(counter))

6

print(next(counter))

[10]:

1

2

3

4

Benefits of Iterators

Use of an iterator simplifies the code and makes it more efficient instead of using a list. For small datasets, iterator and list-based approaches have similar performance. But for larger datasets, iterators save both time and memory.

Here are some primary benefits of using iterators:

Iterators provide cleaner code

Theoretically, iterators can work with infinite sequences.

Iterators save resources. An Iterator stores only one element in the memory, while list (or tuple) stores all the elements.

Iterator treats variables of all types, sizes, and shapes uniformly, whether they fit in memory or not.

Iterator makes recursion unnecessary in handling arrays of arbitrary dimensionality.

Iterator supports iterating over multiple variables concurrently, because each variable's iteration state is maintained in its own iterator structure.

QUIZ - Iterators

Now it’s time to solve the quiz for this chapter. You can download the quiz file, QUIZ_Iterators.zip, from the GitHub Repository of this book. You should put the quiz file in the Iterators project we build in this chapter.

Here are the questions for this chapter:

Q1:

Define an iterator class which will generate a sequence of even numbers like 0, 2, 4, 6, 8, … etc.

Class name will be Even.

Instantiate an object of this class with name even_numbers.

The even_numbers object will be an iterator that provides even numbers from 0 to 20.

Print the even numbers using this object.

Hints:

* __init__()

* __iter__()

* __next__()

[1]:

1

# Q 1:

2

3

# define the iterator class

4

# ---- your solution here ---

5

6

# instantiate an Odd object

7

# ---- your solution here ---

8

9

# print even numbers

10

# ---- your solution here ---

[1]:

0

2

4

6

8

10

12

14

16

18

20

Q2:

Define an infinite iterator named Counter.

Counter has no terminating condition.

It starts from 10 and keeps increasing by 10 at each iteration.

Instantiate an object of this class with name counter.

The counter object will be an iterator that provides numbers which are multiples of 10.

Print first 5 multiplies of 10 by using the counter object.

Hints:

* __init__()

* __iter__()

* __next__()

[2]:

1

# Q 2:

2

3

# define the Counter class

4

# ---- your solution here ---

5

6

# create an object

7

# ---- your solution here ---

8

9

# print first 5 numbers

10

# ---- your solution here ---

[2]:

10

20

30

40

50

Q3:

We have a tuple of fruits:

fruits = ("orange", "apple", "cherry")

Convert this tuple to an iterator and print the fruits in it.

Get the StopIteration exception by trying to call the iterator.

Hints:

* iter()

* do not use loops

[3]:

1

# Q 3:

2

3

# define the tuple

4

# ---- your solution here ---

5

6

# convert tuple to an iterator

7

# ---- your solution here ---

8

9

# print items in the iterator

10

# ---- your solution here ---

11

12

# get StopIteration exception

13

# ---- your solution here ---

[3]:

orange

apple

cherry

StopIteration

Q4:

We have a string object as:

movie = "Inception"

Convert this string to an iterator and print the letters in it.

Hints:

* iter()

* use for loop

[4]:

1

# Q 4:

2

3

# define the string

4

# ---- your solution here ---

5

6

# convert tuple to an iterator

7

# ---- your solution here ---

8

9

# print items in the iterator

10

# ---- your solution here ---

[4]:

I

n

c

e

p

t

i

o

n

Q5:

Define a class named Numbers that implements iterator protocol (__iter__ and __next__ methods).

It will return integers starting from 1 and will increase by one at each iteration:

1, 2, 3, 4, 5, ...

Instantiate an object of this class with name numbers.

Convert the numbers object to an iterator.

Print first 4 integers using the numbers object.

Hints:

* no __init__() method in the class

* iter()

* next()

* no for loops

[5]:

1

# Q 5:

2

3

# define the Numbers class

4

# ---- your solution here ---

5

6

# instantiate the numbers object

7

# ---- your solution here ---

8

9

# convert the numbers object to an iterator

10

# we have to do this explicitly because

11

# the Numbers has no __init__() method

12

# ---- your solution here ---

13

14

# print first 4 integers

15

# ---- your solution here ---

[5]:

1

2

3

4

SOLUTIONS - Iterators

Here are the solutions for the quiz for this chapter.

S1:

[1]:

1

# S 1:

2

3

# define the iterator class

4

class Even:

5

    # implement __init__ method

6

    def __init__(self, limit):

7

        self.current = 0

8

        self.limit = limit

9

10

    # implement __iter__ method

11

    # simply return the object itself

12

    def __iter__(self):

13

        return self

14

15

    # implement __next__ method

16

    def __next__(self):

17

        # check if limit reached

18

        if self.current <= self.limit:

19

            # get the current value

20

            current_value = self.current

21

            # increase the current

22

            self.current += 2

23

            return current_value

24

        # limit is reached so raise exception

25

        else:

26

            raise StopIteration

27

28

# instantiate an Odd object

29

even_numbers = Even(20)

30

31

# print even numbers

32

for e in even_numbers:

33

    print(e)

[1]:

0

2

4

6

8

10

12

14

16

18

20

S2:

[2]:

1

# S 2:

2

3

# define the Counter class

4

class Counter:

5

    def __init__(self):

6

        self.n = 10

7

8

    def __iter__(self):

9

        return self

10

11

    def __next__(self):

12

        current = self.n

13

        self.n += 10

14

        return current

15

16

# create an object

17

counter = Counter()

18

19

# print first 5 numbers

20

for i in range(5):

21

    print(next(counter))

[2]:

10

20

30

40

50

S3:

[3]:

1

# S 3:

2

3

# define the tuple

4

fruits = ("orange", "apple", "cherry")

5

6

# convert tuple to an iterator

7

fuits_iter = iter(fruits)

8

9

# print items in the iterator

10

print(next(fuits_iter))

11

print(next(fuits_iter))

12

print(next(fuits_iter))

13

14

# get StopIteration exception

15

print(next(fuits_iter))

[3]:

orange

apple

cherry

StopIteration

S4:

[4]:

1

# S 4:

2

3

# define the string

4

movie = "Inception"

5

6

# convert tuple to an iterator

7

movie_iter = iter(movie)

8

9

# print items in the iterator

10

for letter in movie_iter:

11

    print(letter)

[4]:

I

n

c

e

p

t

i

o

n

S5:

[5]:

1

# S 5:

2

3

# define the Numbers class

4

class Numbers:

5

    # no __init__() method

6

7

    def __iter__(self):

8

        self.current = 1

9

        return self

10

11

    def __next__(self):

12

        n = self.current

13

        self.current += 1

14

        return n

15

16

# instantiate the numbers object

17

numbers = Numbers()

18

19

# convert the numbers object to an iterator

20

# we have to do this explicitly because

21

# the Numbers has no __init__() method

22

numbers_iter = iter(numbers)

23

24

# print first 4 integers

25

print(next(numbers_iter))

26

print(next(numbers_iter))

27

print(next(numbers_iter))

28

print(next(numbers_iter))

[5]:

1

2

3

4

5.      Generators

What is a Generator?

In the previous chapter we learned about Iterators, which are great tools especially when you need to deal with large datasets. However, building an iterator in Python is a bit cumbersome and time consuming. You have to define a new class which implements the iterator protocol (__iter__() and __next__() methods). In this class, you need to manage internal state of the variables and update them. Moreover you need to raise StopIteration exception when there is no value to return back in the __next__() method.

Fortunately, we have an elegant solution for this in Python. Python provides Generators to help you easily create iterators. A Generator allows you to declare a function that behaves like an iterator, i.e., it can be used in a for loop. In simple terms, a Generator is a function which returns an iterator object. So it’s an easy way of creating iterators. You don’t need to think about all the work needed when you create an iterator, because the Generator will handle all of them.

In this chapter, we will learn how generators work in Python and how we can define them. You can find the PyCharm project for this chapter in the GitHub Repository of this book.

Chapter Outline:

Defining Generators

Generator Function vs. Normal Function

Generator Expression

Benefits of Generators

QUIZ – Generators

SOLUTIONS – Generators

Defining Generators

As stated in the first section, a generator is a special type of function in Python. This function does not return a single value, instead, it returns an iterator object. In the generator function, we use the yield statement instead of the return statement. Let’s define a simple generator function:

[1]:

1

# define a generator function

2

def first_generator():

3

    print('Yielding First item')

4

    yield 'A'

5

6

    print('Yielding Second item')

7

    yield 'B'

8

9

    print('Yielding Last item')

10

    yield 'C'

In cell 1, we define a generator function. The function executes the yield statement instead of the return keyword. The yield statement is what makes this function a generator. When we call this function it will return (yield) an iterator object. Let’s see it:

[2]:

1

# call the generator

2

iter_obj = first_generator()

3

4

# print first item

5

first_item = next(iter_obj)

6

print(first_item)

7

8

# print second item

9

second_item = next(iter_obj)

10

print(second_item)

11

12

# print third item

13

third_item = next(iter_obj)

14

print(third_item)

[2]:

Yielding First item

A

Yielding Second item

B

Yielding Last item

C

In cell 2, we call the first_generator() function which is a generator and returns an iterator object. We name this iterator as iter_obj. Then we call the next() function on this iterator object. In each next() call, the iterator executes the yield statement in respective order and returns an item.

As a general rule, the generator function should not include the return keyword. Because if it includes, then the return statement will terminate the function.

Now let’s define a more realistic generator by the help of a for loop. In this example, we want to define a generator which will keep track of the sequence of numbers starting from zero and up to a given maximum limit.

[3]:

1

# generator for sequence of numbers

2

def get_sequence_gen(max):

3

    for n in range(max):

4

        yield n

5

6

# call the function and get iterator

7

sequence_iter = get_sequence_gen(10)

8

# call the next() method

9

print(sequence_iter.__next__())

10

print(sequence_iter.__next__())

11

print(next(sequence_iter))

12

print(next(sequence_iter))

[3]:

0

1

2

3

In cell 3, we define a generator function which yields the integers from zero up to a given number. As you see, the yield statement is inside the for loop. Please be careful that, the value of n is stored internally during successive next() calls to the function.

Generator Function vs. Normal Function

A function is a generator function if it contains at least one yield statement. It may contain other yield or return statements if needed. Both yield and return keywords will return “something” from a function.

The difference between return and yield keywords is very crucial for generators. While the return statement terminates a function entirely, yield statement pauses the function saving all its states and later continues from there on successive calls.

We call a generator function in the same way we call a normal one. During its execution, the generator pauses when it encounters the yield keyword. It sends the current value of the iterator stream to the calling environment and wait for the next call. Meanwhile, it saves the local variables and their states internally.

Below are the key points where a generator function differs from a normal function:

A Generator function returns (yields) an iterator object. You don’t need to worry about creating this iterator object explicitly, the yield keywords does this for you.

A Generator function must contain at least one yield statement. It may include multiple yield keywords if needed.

A Generator function implements the iterator protocol (iter() and next() methods) internally.

A Generator function saves the local variables and their states automatically.

A Generator function pauses execution at the yield keyword and pass the control to the caller.

A Generator function raises the StopIteration exception automatically when the iterator stream has no value to return.

Let’s consider a simple example to demonstrate the difference between a normal function and a generator function. In this example, we want to calculate the sum of first n positive integers. To do this, we will define a function that gives us the list of first n positive numbers. We will implement this function in both ways, a normal function and a generator.

Here is the code for the normal function:

[4]:

1

# import the time module

2

from time import time

3

4

# Normal Function

5

def first_n_numbers(max):

6

    n, numbers = 1, []

7

    while n <= max:

8

        numbers.append(n)

9

        n += 1

10

    return numbers

11

12

# call the function

13

start = time()

14

first_n_list = first_n_numbers(99999999)

15

sum_of_first_numbers = sum(first_n_list)

16

print(sum_of_first_numbers)

17

# elapsed time

18

end = time()

19

print("Elapsed Time in seconds:", end - start)

[4]:

4999999950000000

Elapsed Time in seconds: 17.859

In cell 4, we define a normal function that returns the list of first n positive integers. When we call this function it takes a while to complete execution because the list it creates is huge. It also uses a considerable amount of memory to complete this task.

Now let’s define a generator function for the same operation:

[5]:

1

# Generator Function

2

def first_n_numbers_gen(max):

3

    n = 1

4

    while n <= max:

5

        yield n

6

        n += 1

7

8

# call the function

9

start = time()

10

first_n_list = first_n_numbers_gen(99999999)

11

sum_of_first_numbers = sum(first_n_list)

12

print(sum_of_first_numbers)

13

# elapsed time

14

end = time()

15

print("Elapsed Time in seconds:", end - start)

[5]:

4999999950000000

Elapsed Time in seconds: 15.302

As you see in cell 5, the generator finishes the same task in less time and it uses less memory resources. Because the generator yields items one by one instead of returning the complete list.

The main reason for performance improvement (when we use generators) is the lazy generation of values. This on demand value generation, results in lower memory usage. One more advantage of generators is, you do not need to wait until all the elements have been generated before you start to use them.

Generator Expression

There are times that you need simple generators for relatively simple tasks in your code. This is where the Generator Expression comes in. You can easily create simple generators on the fly using generator expressions.

Generator Expressions are similar to lambda functions in Python. Remember that, lambda’s are anonymous functions which let us create one-line functions on the fly. Just like a lambda function, a generator expression creates an anonymous generator function.

The syntax of a generator expression looks like a list comprehension. The difference is, we have parentheses instead of square brackets in a generator expression. Let’s see an example:

[6]:

1

# define a simple list

2

nums = [1, 2, 3, 4, 5]

3

4

# list comprehension

5

num_cubes = [i**3 for i in nums]

6

7

# generator expression

8

cubes_gen = (i**3 for i in nums)

9

10

# print both objects

11

print(num_cubes)

12

print(cubes_gen)

[6]:

[1, 8, 27, 64, 125]

<generator object <genexpr> at 0x103059d20>

In cell 6, line 8 we define a simple generator with the help of the generator expression. Here is the syntax: cubes_gen = (i**3 for i in nums). And you see the generator object in the output. As we already know, to be able to get the items in a generator we either need to call the next() method explicitly or use a for loop to iterate over the generator. Let’s print the items in the cubes_gen object:

[7]:

1

# loop over generator

2

for item in cubes_gen:

3

    print(item)

[7]:

1

8

27

64

125

Let’s do another example. We will define a generator that converts the letters of a string to uppercase. Then we will call the next() method to print first two letters.

[8]:

1

# generator for upper case

2

text = 'machine learning'

3

upper_gen = (l.upper() for l in text)

4

print(upper_gen.__next__())

5

print(next(upper_gen))

[8]:

M

A

Benefits of Generators

Benefits are great tools especially when you need to deal with large data in relatively limited memory. Here are some key benefits of using generators in Python:

Memory Efficiency:

Let’s assume, you have a normal function that returns a very large sequence. A list with millions of items, for example. You have to wait for this function to finish all the execution and return you the list as a whole. This is obviously not efficient in terms of time and memory resources. On the other hand, if you use a generator function, it will return you the items one by one, and you will have the chance to continue to execute the next lines of code. You don’t need to wait for all of the items in the list to be executed by the function. Because the generator will give (yield) you one item at a time.

Lazy Evaluation:

Generators provide the power of lazy evaluation. Lazy evaluation is computing a value when it is really needed, not when it is instantiated. Let’s assume you have a large dataset to compute; lazy evaluation allows you to start using the data immediately while the whole data set is still being computed. Because you do not need the whole data set if you are using a generator.

Implement and Readability:

Generators are very easy to implement and provide code readability. Remember that, you do not need to worry about the __iter__() and __next__() methods if you are using a generator. All you need is a simple yield statement in your function.

Dealing with Infinite Streams:

Generators are wonderful tools when you need to represent an infinite stream of data. An infinite counter, for example. In theory, you cannot store an infinite stream in the memory. You cannot be sure about how much size you will need to store an infinite stream. This is where a generator really shines, since it produces only one item at a time, it can represent an infinite stream of data. And it doesn’t have to store all the stream in the memory.

QUIZ - Generators

Now it’s time to solve the quiz for this chapter. You can download the quiz file, QUIZ_Generators.zip, from the GitHub Repository of this book. You should put the quiz file in the Generators project we build in this chapter.

Here are the questions for this chapter:

Q1:

Define a generator function that returns (yields) the list of first n positive and even integers.

Here is a sample list: 2, 4, 6, 8, 10, ...

Function name will be positive_evens and it will take one parameter, n.

This will be the upper limit in the list.

Call this function and print positive and even integers up to 20.

Hints:

* yield

* while loop

[1]:

1

# Q 1:

2

3

# define the function

4

# ---- your solution here ---

5

6

# get positive and even numbers up to 20

7

# ---- your solution here ---

8

9

# convert the generator to a list and print

10

# ---- your solution here ---

[1]:

[2, 4, 6, 8, 10, 12, 14, 16, 18, 20]

Q2:

The Fibonacci Sequence is the series of numbers:

1, 1, 2, 3, 5, 8, 13, 21, 34, ...

The next number is found by adding up the two numbers before it:

* the 2 is found by adding the two numbers before it (1+1),

* the 3 is found by adding the two numbers before it (1+2),

* the 5 is (2+3),

* and so on!

Define a generator function which yields the Fibonacci Sequence.

Function name will be fibonnaci_generator and it takes one parameter, n.

This parameter is used to get the first n numbers in the sequence.

For example the first 9 numbers are: 1, 1, 2, 3, 5, 8, 13, 21, 34

Print the first 9 numbers by using this function.

Hints:

* yield

* for loop

[2]:

1

# Q 2:

2

3

# define the generator

4

# ---- your solution here ---

5

6

# get the first 9 numbers

7

# ---- your solution here ---

8

9

# print the numbers

10

# ---- your solution here ---

[2]:

1

1

2

3

5

8

13

21

34

Q3:

Define a generator function named get_squares_gen.

This function will take a list of integers as the parameter.

And it will yield the squares of each element one by one.

Call this generator function with the following list:

[1, 2, 3, 4, 5, 6]

Print the squares of items in this list.

To do this, first convert the iterator to a list.

Hints:

* yield

* for loop

* generator returns an iterator

[3]:

1

# Q 3:

2

3

# define the generator function

4

# ---- your solution here ---

5

6

# define a list of numbers

7

# ---- your solution here ---

8

9

# call the generator function

10

# ---- your solution here ---

11

12

# convert the iterator to a list

13

# ---- your solution here ---

14

15

# print squares list

16

# ---- your solution here ---

[3]:

[1, 4, 9, 16, 25, 36]

Q4:

Define a generator expression.

This is going to be a one-line generator.

It will return an iterator which contains the cubes of odd elements in the numbers list.

Here is the numbers list:

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

Convert the resulting iterator to a list.

Then print the items in this list.

Hints:

* generator expressions are very similar to list comprehensions

* generator expression returns an iterator object

[4]:

1

# Q 4:

2

3

# define a list of numbers

4

# ---- your solution here ---

5

6

# define the generator expression

7

# ---- your solution here ---

8

9

# convert the iterator to a list

10

# ---- your solution here ---

11

12

# print the cubes of odd numbers

13

# ---- your solution here ---

[4]:

[1, 27, 125, 343, 729]

Q5:

What is the difference between return and yield statements?

SOLUTIONS - Generators

Here are the solutions for the quiz for this chapter.

S1:

[1]:

1

# S 1:

2

3

# define the function

4

def positive_evens(n):

5

    counter = 1

6

    while counter <= n:

7

        if counter % 2 == 0:

8

            yield counter

9

        counter += 1

10

11

# get positive and even numbers up to 20

12

first_evens = positive_evens(20)

13

14

# convert the generator to a list and print

15

print(list(first_evens))

[1]:

[2, 4, 6, 8, 10, 12, 14, 16, 18, 20]

S2:

[2]:

1

# S 2:

2

3

# define the generator

4

def fibonnaci_generator(n):

5

    first = 1

6

    second = 1

7

    for i in range(n):

8

        yield first

9

        first, second = second, first + second

10

11

# get the first 9 numbers

12

fibo_9 = fibonnaci_generator(9)

13

14

# print the numbers

15

for f in fibo_9:

16

    print(f)

[2]:

1

1

2

3

5

8

13

21

34

S3:

[3]:

1

# S 3:

2

3

# define the generator function

4

def get_squares_gen(nums):

5

    for num in nums:

6

        yield num**2

7

8

# define a list of numbers

9

numbers = [1, 2, 3, 4, 5, 6]

10

11

# call the generator function

12

squares = get_squares_gen(numbers)

13

14

# convert the iterator to a list

15

squares_list = list(squares)

16

17

# print squares list

18

print(squares_list)

[3]:

[1, 4, 9, 16, 25, 36]

S4:

[4]:

1

# S 4:

2

3

# define a list of numbers

4

numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

5

6

# define the generator expression

7

odd_cubes_gen = (i**3 for i in numbers if i % 2 == 1)

8

9

# convert the iterator to a list

10

odd_cubes_list = list(odd_cubes_gen)

11

12

# print the cubes of odd numbers

13

print(odd_cubes_list)

[4]:

[1, 27, 125, 343, 729]

S5:

The difference between return and yield statements:

return:

The return statement, returns a value from a function, and terminates the function entirely.

Which means, the function loses its local state.

So, the next time we call this function, it starts over from its initial state.

yield:

The yield statement pauses the function, saving all its states and later continues from there on successive calls.

So, it maintains the state between function calls, and resumes from where it left off when we call the next() method again.

6.      Date and Time

Date and Time in Python

Date and Time objects are very important in programming. In Python, we have a dedicated module for date and time operations. The datetime module provides classes for manipulating dates and times. While date and time arithmetic are supported, the focus of the implementation of datetime module is on efficient attribute extraction for output formatting and manipulation.

In this chapter, we will learn how to use datetime module in Python. You can find the PyCharm project for this chapter in the GitHub Repository of this book.

Chapter Outline:

Aware and Naive Objects

Constants and Main Classes

Determining if an Object is Aware or Naive

Timedelta Class

Date Class

Datetime Class

Time Class

Formatting Date and Time

QUIZ – Date and Time

SOLUTIONS – Date and Time

Aware and Naive Objects

Date and time objects may be categorized as “aware” or “naive” depending on whether or not they include timezone information.

With sufficient knowledge of applicable algorithmic and political time adjustments, such as time zone and daylight-saving time information, an aware object can locate itself relative to other aware objects. An aware object represents a specific moment in time that is not open to interpretation.

A naive object does not contain enough information to unambiguously locate itself relative to other date/time objects. Whether a naive object represents Coordinated Universal Time (UTC), local time, or time in some other timezone is purely up to the program, just like it is up to the program whether a particular number represents meters, miles, or mass. Naive objects are easy to understand and to work with, at the cost of ignoring some aspects of reality.

For applications requiring aware objects, datetime and time objects have an optional time zone information attribute, tzinfo, that can be set to an instance of a subclass of the abstract tzinfo class. These tzinfo objects capture information about the offset from UTC time, the time zone name, and whether daylight-saving time is in effect.

Only one concrete tzinfo class, the timezone class, is supplied by the datetime module. The timezone class can represent simple timezones with fixed offsets from UTC, such as UTC itself or North American EST and EDT timezones. Supporting timezones at deeper levels of detail is up to the application. The rules for time adjustment across the world are more political than rational, change frequently, and there is no standard suitable for every application aside from UTC.

Constants and Main Classes

Constants:

The datetime module exports the following constants:

datetime.MINYEAR: The smallest year number allowed in a date or datetime object. MINYEAR is 1.

datetime.MAXYEAR: The largest year number allowed in a date or datetime object. MAXYEAR is 9999.

Available Types:

Here are the built-in types in datetime module in Python. Keep in mind that, objects of these types are immutable.

class datetime.date: An idealized naive date, assuming the current Gregorian calendar always was, and always will be, in effect. Attributes: year, month, and day.

class datetime.time: An idealized time, independent of any particular day, assuming that every day has exactly 24*60*60 seconds. (There is no notion of “leap seconds” here.) Attributes: hour, minute, second, microsecond, and tzinfo.

class datetime.datetime: A combination of a date and a time. Attributes: year, month, day, hour, minute, second, microsecond, and tzinfo.

class datetime.timedelta: A duration expressing the difference between two date, time, or datetime instances to microsecond resolution.

class datetime.tzinfo: An abstract base class for time zone information objects. These are used by the datetime and time classes to provide a customizable notion of time adjustment (for example, to account for time zone and/or daylight-saving time).

class datetime.timezone: A class that implements the tzinfo abstract base class as a fixed offset from the UTC.

Common Properties:

The date, datetime, time, and timezone types share these common features:

Objects of these types are immutable.

Objects of these types are hashable, meaning that they can be used as dictionary keys.

Objects of these types support efficient pickling via the pickle module.

Determining if an Object is Aware or Naive

Naive and aware concepts are crucial in data and time operations. Here are the general rules:

Objects of the date type are always naive.

An object of type time or datetime may be aware or naive.

A datetime object d is aware if both of the following hold:

d.tzinfo is not None

d.tzinfo.utcoffset(d) does not return None

Otherwise, d is naive.

A time object t is aware if both of the following hold:

t.tzinfo is not None

t.tzinfo.utcoffset(None) does not return None.

Otherwise, t is naive.

The distinction between aware and naive doesn’t apply to timedelta objects.

Timedelta Class

A timedelta object represents a duration, the difference between two dates or times.

class datetime.timedelta(days=0, seconds=0, microseconds=0, milliseconds=0, minutes=0, hours=0, weeks=0):

All arguments are optional and default to 0. Arguments may be integers or floats, and may be positive or negative.

Only days, seconds and microseconds are stored internally. Arguments are converted to those units:

A millisecond is converted to 1000 microseconds.

A minute is converted to 60 seconds.

An hour is converted to 3600 seconds.

A week is converted to 7 days.

and days, seconds and microseconds are then normalized so that the representation is unique, with

0 <= microseconds < 1000000

0 <= seconds < 3600*24 (the number of seconds in one day)

-999999999 <= days <= 999999999

The following example illustrates how any arguments besides days, seconds and microseconds are “merged” and normalized into those three resulting attributes:

[1]:

1

from datetime import timedelta

2

3

delta = timedelta(

4

     days=50,

5

     seconds=27,

6

     microseconds=10,

7

     milliseconds=29000,

8

     minutes=5,

9

     hours=8,

10

     weeks=2

11

)

12

# Only days, seconds, and microseconds remain

13

print("Days:", delta.days)

14

print("Seconds:", delta.seconds)

15

print("Microseconds:", delta.microseconds)

[1]:

Days: 64

Seconds: 29156

Microseconds: 10

In the next example let’s add days two timedelta objects:

[2]:

1

# Add two timedelta objects

2

delta1 = timedelta(minutes=10, seconds=50)

3

delta2 = timedelta(hours=25, seconds=20)

4

delta_sum = delta1 + delta2

5

print(delta_sum)

[2]:

1 day, 1:11:10

In cell 2, we add two timedelta objects and print the result. Addition (+), Subtraction (-), Multiplication (*), Division (/), Floor Division (//) and Modulo (%) operations are supported by the timedelta class in Python.

Date Class

A date object represents a date (year, month and day) in an idealized calendar, the current Gregorian calendar indefinitely extended in both directions.

January 1 of year 1 is called day number 1, January 2 of year 1 is called day number 2, and so on. 2

class datetime.date(year, month, day):

All arguments are required. Arguments must be integers, in the following ranges:

MINYEAR <= year <= MAXYEAR

1 <= month <= 12

1 <= day <= number of days in the given month and year

If an argument outside those ranges is given, ValueError is raised.

Class Attributes:

date.min: The earliest representable date, date(MINYEAR, 1, 1).

date.max: The latest representable date, date(MAXYEAR, 12, 31).

date.resolution: The smallest possible difference between non-equal date objects, timedelta(days=1).

Instance Attributes (read-only):

date.year: Between MINYEAR and MAXYEAR inclusive.

date.month: Between 1 and 12 inclusive.

date.day: Between 1 and the number of days in the given month of the given year.

[3]:

1

from datetime import date

2

3

# instantiate a date object: year, month, day

4

a_valid_date = date(2021, 3, 26)

5

print("This is a valid date:", a_valid_date)

6

7

an_ivalid_date = date(2021, 3, 48)

8

print("This is an invalid date:", an_ivalid_date)

[3]:

This is a valid date: 2021-03-26

ValueError: day is out of range for month

date.today():

Return the current local date. Let’s get the current date:

[4]:

1

# current date

2

current_date = date.today()

3

print("Current date is:", current_date)

4

print("Current year:", current_date.year)

5

print("Current month:", current_date.month)

6

print("Current day:", current_date.day)

[4]:

Current date is: 2022-03-25

Current year: 2022

Current month: 3

Current day: 25

date.fromtimestamp(timestamp):

Return the local date corresponding to the POSIX timestamp.

POSIX timestamp is the time expressed as the number of seconds that have passed since January 1, 1970. That zero moment, known as the epoch, is simply the start of the decade in which the Unix operating system (which first used this time representation) was invented.

[5]:

1

# fromtimestamp: get datetime from timestamp

2

date_time_from_timestamp = date.fromtimestamp(1527635439)

3

print("Timestamp to datetime:", date_time_from_timestamp)

[5]:

Timestamp to datetime: 2018-05-30

date.isoformat():

Return a string representing the date in ISO 8601 format, YYYY-MM-DD. It is equivalent to date.__str__().

[6]:

1

# isoformat() -> YYYY-MM-DD

2

iso_formatted_date = date(2002, 12, 4).isoformat()

3

print(iso_formatted_date)

[6]:

2002-12-04

date.ctime():

Return a string representing the date:

[7]:

1

# isoformat() -> YYYY-MM-DD

2

iso_formatted_date = date(2002, 12, 4).isoformat()

3

print(iso_formatted_date)

[7]:

Wed Dec  4 00:00:00 2002

date.fromisoformat(date_string):

Return a date corresponding to a date_string given in the format YYYY-MM-DD:

[8]:

1

# fromisoformat

2

date_from_iso_str = date.fromisoformat('2019-12-04')

3

print(date_from_iso_str)

[8]:

2019-12-04

date.fromordinal(ordinal):

Return the date corresponding to the proleptic Gregorian ordinal, where January 1 of year 1 has ordinal 1. ValueError is raised unless 1 <= ordinal <= date.max.toordinal().

For any date d, date.fromordinal(d.toordinal()) == d.

date.strftime(format):

Return a string representing the date, controlled by an explicit format string. Format codes referring to hours, minutes or seconds will see 0 values. date.__format__(format) dunder (double underscore) method also works the same.

[9]:

1

# fromordinal() and strftime()

2

# 730920th day after 1. 1. 0001

3

d = date.fromordinal(730920)

4

print(d)

5

6

# Methods related to formatting string output

7

print(d.isoformat())

8

print(d.strftime("%d/%m/%y"))

9

print(d.strftime("%A %d. %B %Y"))

[9]:

2002-03-11

2002-03-11

11/03/02

Monday 11. March 2002

date.isocalendar():

Return a named tuple object with three components: year, week and weekday. The ISO calendar is a widely used variant of the Gregorian calendar.

The ISO year consists of 52 or 53 full weeks, and where a week starts on a Monday and ends on a Sunday. The first week of an ISO year is the first (Gregorian) calendar week of a year containing a Thursday. This is called week number 1, and the ISO year of that Thursday is the same as its Gregorian year.

For example, 2004 begins on a Thursday, so the first week of ISO year 2004 begins on Monday, 29 Dec 2003 and ends on Sunday, 4 Jan 2004:

[10]:

1

# isocalendar()

2

iso_date_year_end = date(2003, 12, 29).isocalendar()

3

print(iso_date_year_end)

4

5

iso_date_year_start = date(2004, 1, 4).isocalendar()

6

print(iso_date_year_start)

[10]:

datetime.IsoCalendarDate(year=2004, week=1, weekday=1)

datetime.IsoCalendarDate(year=2004, week=1, weekday=7)

date.replace(year=self.year, month=self.month, day=self.day):

Return a date with the same value, except for those parameters given new values by whichever keyword arguments are specified.

[11]:

1

# replace()

2

d = date(2002, 12, 31)

3

d_new = d.replace(day=26)

4

print("d:", d)

5

print("d_new:", d_new)

[11]:

d: 2002-12-31

d_new: 2002-12-26

date.fromisocalendar(year, week, day):

Return a date corresponding to the ISO calendar date specified by year, week and day. This is the inverse of the function date.isocalendar().

date.toordinal():

Return the proleptic Gregorian ordinal of the date, where January 1 of year 1 has ordinal 1. For any date object d, date.fromordinal(d.toordinal()) == d.

date.weekday():

Return the day of the week as an integer, where Monday is 0 and Sunday is 6. For example, date(2002, 12, 4).weekday() == 2, a Wednesday.

date.isoweekday():

Return the day of the week as an integer, where Monday is 1 and Sunday is 7. For example, date(2002, 12, 4).isoweekday() == 3, a Wednesday.

date.timetuple():

Return a time.struct_time such as returned by time.localtime(), which we will see later.

Datetime Class

A datetime object is a single object containing all the information from a date object and a time object. Like a date object, datetime assumes the current Gregorian calendar extended in both directions; like a time object, datetime assumes there are exactly 3600*24 seconds in every day.

datetime.datetime(year, month, day, hour=0, minute=0, second=0, microsecond=0, tzinfo=None, *, fold=0):

The year, month and day arguments are required. tzinfo may be None, or an instance of a tzinfo subclass. The remaining arguments must be integers in the following ranges:

MINYEAR <= year <= MAXYEAR,

1 <= month <= 12,

1 <= day <= number of days in the given month and year,

0 <= hour < 24,

0 <= minute < 60,

0 <= second < 60,

0 <= microsecond < 1000000,

fold in [0, 1].

If an argument outside those ranges is given, ValueError is raised.

Let’s create a datetime object with different parameter sets:

[12]:

1

from datetime import datetime

2

3

# call the constructor

4

datetime_obj = datetime(2020, 5, 29)

5

print(datetime_obj)

6

7

# call the constructor with time parameters

8

datetime_obj_with_time = datetime(2020, 5, 29, 8, 45, 52, 162420)

9

print(datetime_obj_with_time)

[12]:

2020-05-29 00:00:00

2020-05-29 08:45:52.162420

Now let’s get the year, month, hour, minute, seconds, and timestamp attributes from a datetime object:

[13]:

1

# get attributes

2

dt = datetime(2019, 8, 17, 23, 38, 54)

3

print("Year:", dt.year)

4

print("Month:", dt.month)

5

print("Day:", dt.day)

6

print("Hour:", dt.hour)

7

print("Minute:", dt.minute)

8

print("Seconds:", dt.second)

9

print("Timestamp:", dt.timestamp())

[13]:

Year: 2019

Month: 8

Day: 17

Hour: 23

Minute: 38

Seconds: 54

Timestamp: 1566074334.0

datetime.today():

Return the current local datetime, with tzinfo None.

datetime.now(tz=None):

Return the current local date and time. If optional argument tz is None or not specified, this is like today(). If tz is not None, it must be an instance of a tzinfo subclass, and the current date and time are converted to tz’s time zone.

[14]:

1

# now()

2

current_date_time = datetime.now()

3

print("Current date & time:", current_date_time)

[14]:

Current date & time: 2022-03-26 12:51:39.045639

Class Attributes:

datetime.min: The earliest representable datetime, datetime(MINYEAR, 1, 1, tzinfo=None).

datetime.max: The latest representable datetime, datetime(MAXYEAR, 12, 31, 23, 59, 59, 999999, tzinfo=None).

datetime.resolution: The smallest possible difference between non-equal datetime objects, timedelta(microseconds=1).

Instance Attributes (read-only):

datetime.year: Between MINYEAR and MAXYEAR inclusive.

datetime.month: Between 1 and 12 inclusive.

datetime.day: Between 1 and the number of days in the given month of the given year.

datetime.hour: In range(24).

datetime.minute: In range(60).

datetime.second: In range(60).

datetime.microsecond: In range(1000000).

datetime.tzinfo: The object passed as the tzinfo argument to the datetime constructor, or None if none was passed.

datetime.fold: In [0, 1]. Used to disambiguate wall times during a repeated interval. (A repeated interval occurs when clocks are rolled back at the end of daylight-saving time or when the UTC offset for the current zone is decreased for political reasons.) The value 0 (1) represents the earlier (later) of the two moments with the same wall time representation.

Datetime class has the same methods with date class. Here is the list of datetime class methods:

astimezone():

Returns the datetime object with timezone (tz) information.

combine():

Combines the date and time objects and return a single datetime object.

ctime():

Returns a string representation of date and time.

date():

Return date object with same year, month and day.

fromisoformat():

Returns a datetime object from the string representation of the date and time.

fromordinal():

Returns a date object from the proleptic Gregorian ordinal, where January 1 of year 1 has ordinal 1. The hour, minute, second, and microsecond are 0.

fromtimestamp():

Returns the local date and time corresponding to the POSIX timestamp.

isocalendar():

Returns a named tuple with three components: year, week and weekday.

isoformat():

Returns a string representing the date and time in ISO 8601 format:

YYYY-MM-DDTHH:MM:SS.ffffff, if microsecond is not 0

YYYY-MM-DDTHH:MM:SS, if microsecond is 0

isoweekday():

Returns the day of the week as an integer, where Monday is 1 and Sunday is 7.

replace():

Returns a new datetime with the same attributes, except for those attributes given new values by whichever keyword arguments are specified.

strftime():

Returns a string representing the date and time, controlled by an explicit format string.

strptime():

Returns a datetime object corresponding to date string provided as parameter.

time():

Returns time object with same hour, minute, second, microsecond and fold. tzinfo is None.

timetuple():

Returns an object of type time.struct_time.

timetz():

Returns time object with same hour, minute, second, microsecond, fold, and tzinfo attributes.

toordinal():

Returns the proleptic Gregorian ordinal of the date. The same as self.date().toordinal().

tzname():

Returns the name of the timezone if tzinfo is not None. If tzinfo is None, returns None.

utcfromtimestamp():

Returns the UTC datetime corresponding to the POSIX timestamp, with tzinfo None. (The resulting object is naive.)

utcoffset():

Returns the UTC offset if tzinfo is not None. If tzinfo is None, returns None.

utcnow():

Returns the current UTC date and time, with tzinfo None. This is like now(), but returns the current UTC date and time, as a naive datetime object. An aware current UTC datetime can be obtained by calling datetime.now(timezone.utc).

weekday():

Returns the day of the week as an integer, where Monday is 0 and Sunday is 6.

Let’s see some examples of working with datetime objects:

[15]:

1

# Using datetime.combine()

2

d = date(2005, 7, 14)

3

t = time(12, 30)

4

combined_dt = datetime.combine(d, t)

5

print(combined_dt)

[15]:

2005-07-14 12:30:00

In cell 15, we instantiate two objects with date() and time() constructors. Then we combine them by using the datetime.combine() method.

[16]:

1

# Using datetime.now()

2

# GMT +1

3

now = datetime.now()

4

print(now)

5

6

# with timezone info

7

now_tz = datetime.now(timezone.utc)

8

print(now_tz)

[16]:

2022-03-26 14:42:10.279281

2022-03-26 11:42:10.279293+00:00

In cell 16, we call the datetime.now() method with and without timezone information.

[17]:

1

# Using datetime.strptime()

2

dt = datetime.strptime("21/11/06 16:30", "%d/%m/%y %H:%M")

3

print(dt)

[17]:

2006-11-21 16:30:00

In cell 17, we create a datetime object by calling the datetime.strptime(date_string, format) method. We pass the date_string and the format parameters.

[18]:

1

# Using datetime.timetuple() to get tuple of all attributes

2

tt = dt.timetuple()

3

for it in tt:

4

     print(it)

[18]:

2006    # year

11      # month

21      # day

16      # hour

30      # minute

0       # second

1       # weekday (0 = Monday)

325     # number of days since 1st January

-1      # dst - method tzinfo.dst() returned None

In cell 18, we print all of the attributes of the dt object which we defined in cell 17.

[19]:

1

# Date in ISO format

2

ic = dt.isocalendar()

3

for it in ic:

4

     print(it)

[19]:

2006    # ISO year

47      # ISO week

2       # ISO weekday

In cell 19, we print the ISO calendar information of our dt object. We get this data by calling the isocalendar() method on the datetime object.

[20]:

1

# Formatting a datetime

2

formatted_dt = dt.strftime("%A, %d. %B %Y %I:%M%p")

3

print(formatted_dt)

4

5

dt_str = 'The {1} is {0:%d}, the {2} is {0:%B}, the {3} is {0:%I:%M%p}.'.format(dt, "day", "month", "time")

6

print(dt_str)

[20]:

Tuesday, 21. November 2006 04:30PM

The day is 21, the month is November, the time is 04:30PM.

In cell 20, we print the formatted date in two different ways. The first one uses the datetime.strftime() method and the second one is the str.format() method.

Time Class

A time object represents a (local) time of day, independent of any particular day, and subject to adjustment via a tzinfo object.

datetime.time(hour=0, minute=0, second=0, microsecond=0, tzinfo=None, *, fold=0):

All arguments are optional. tzinfo may be None, or an instance of a tzinfo subclass. The remaining arguments must be integers in the following ranges:

0 <= hour < 24,

0 <= minute < 60,

0 <= second < 60,

0 <= microsecond < 1000000,

fold in [0, 1].

If an argument outside those ranges is given, ValueError is raised. All default to 0 except tzinfo, which defaults to None.

Class Attributes:

time.min: The earliest representable time, time(0, 0, 0, 0).

time.max:  The latest representable time, time(23, 59, 59, 999999).

time.resolution: The smallest possible difference between non-equal time objects, timedelta(microseconds=1), although note that arithmetic on time objects is not supported.

Instance Attributes (read-only):

time.hour: In range(24).

time.minute: In range(60).

time.second: In range(60).

time.microsecond: In range(1000000).

time.tzinfo: The object passed as the tzinfo argument to the time constructor, or None if none was passed.

time.fold: In [0, 1]. Used to disambiguate wall times during a repeated interval. (A repeated interval occurs when clocks are rolled back at the end of daylight-saving time or when the UTC offset for the current zone is decreased for political reasons.) The value 0 (1) represents the earlier (later) of the two moments with the same wall time representation.

time.fromisoformat(time_string):

Returns a time corresponding to a time_string in one of the formats emitted by time.isoformat().

time.replace():

Returns a time with the same value, except for those attributes given new values by whichever keyword arguments are specified.

time.isoformat():

Returns a string representing the time in ISO 8601 format.

time.__str__():

For a time t, str(t) is equivalent to t.isoformat().

time.strftime(format):

Returns a string representing the time, controlled by an explicit format string.

time.__format__(format):

Same as time.strftime(). This makes it possible to specify a format string for a time object in formatted string literals and when using str.format().

time.utcoffset():

If tzinfo is None, returns None, else returns self.tzinfo.utcoffset(None), and raises an exception if the latter doesn’t return None or a timedelta object with magnitude less than one day.

time.dst():

If tzinfo is None, returns None, else returns self.tzinfo.dst(None), and raises an exception if the latter doesn’t return None, or a timedelta object with magnitude less than one day.

time.tzname():

If tzinfo is None, returns None, else returns self.tzinfo.tzname(None), or raises an exception if the latter doesn’t return None or a string object.

Here are some examples of working with a time object:

[21]:

1

from datetime import time, tzinfo, timedelta

2

3

# define a custom class

4

class TZ1(tzinfo):

5

     def utcoffset(self, dt):

6

         return timedelta(hours=1)

7

     def dst(self, dt):

8

         return timedelta(0)

9

     def tzname(self,dt):

10

         return "+01:00"

11

     def  __repr__(self):

12

         return f"{self.__class__.__name__}()"

In cell 21, we define a custom class, TZ1, which inherits from tzinfo class. And we override some methods in its parent. Let’s use this class for creating a time object and print some attributes:

[22]:

1

# create a time object

2

t = time(12, 10, 30, tzinfo=TZ1())

3

print("t:",t)

4

print("isoformat:", t.isoformat())

5

print("dst:", t.dst())

6

print("tzname:", t.tzname())

7

print("strftime:", t.strftime("%H:%M:%S %Z"))

8

print("format:", 'The {} is {:%H:%M}.'.format("time", t))

[22]:

t: 12:10:30+01:00

isoformat: 12:10:30+01:00

dst: 0:00:00

tzname: +01:00

strftime: 12:10:30 +01:00

format: The time is 12:10.

Formatting Date and Time

We already covered the strftime() method that exist in the datetime class. This is the main method which is used for date and time formatting.

Here is the reference of all legal format codes that you can use for date and time formatting:

Directive

Description

Example

%a

Weekday, short version

Wed

%A

Weekday, full version

Wednesday

%w

Weekday as a number 0-6, 0 is Sunday

2

%d

Day of month 01-31

28

%b

Month name, short version

Dec

%B

Month name, full version

December

%m

Month as a number 01-12

12

%y

Year, short version, without century

18

%Y

Year, full version

2021

%H

Hour 00-23

17

%I

Hour 00-12

5

%p

AM/PM

PM

%M

Minute 00-59

41

%S

Second 00-59

8

%f

Microsecond 000000-999999

548513

%z

UTC offset

100

%Z

Timezone

CST

%j

Day number of year 001-366

365

%U

Week number of year, Sunday as the first day of week, 00-53

52

%W

Week number of year, Monday as the first day of week, 00-53

52

%c

Local version of date and time

Mon Dec 31 17:41:00 2018

%C

Century

20

%x

Local version of date

12/31/18

%X

Local version of time

17:41:00

%%

A % character

%

%G

ISO 8601 year

2018

%u

ISO 8601 weekday (1-7)

1

%V

ISO 8601 weeknumber (01-53)

1

Let’s see some examples for these formats:

[23]:

1

# Formatting Examples

2

from datetime import datetime

3

4

# current date & time

5

current = datetime.now()

6

print("No formatting", current)

7

8

# Weekday short version

9

weekday_short = current.strftime("%a %-m %y")

10

print('Weekday Short Version:', weekday_short)

11

12

# Weekday

13

weekday = current.strftime("%A %m %-Y")

14

print('Weekday:', weekday)

15

16

# Hour with PM

17

pm = current.strftime("%-I %p %S")

18

print('Hour with PM:', pm)

19

20

# Time

21

common_time = current.strftime("%H:%M:%S")

22

print('Common Time Representation:', common_time)

[23]:

No formatting 2022-03-26 18:17:54.351495

Weekday Short Version: Sat 3 22

Weekday: Saturday 03 2022

Hour with PM: 6 PM 54

Common Time Representation: 18:17:54

QUIZ – Date and Time

Now it’s time to solve the quiz for this chapter. You can download the quiz file, QUIZ_DateAndTime.zip, from the GitHub Repository of this book. You should put the quiz file in the DateAndTime project we build in this chapter.

Here are the questions for this chapter:

Q1:

Define two timedelta objects with following attributes:

* object 1: 23 days, 8 hours, 35 minutes

* object 2: 15 days, 6 hours, 50 minutes

Find the difference between these two objects (object 1 - object 2).

Print the result.

[1]:

1

# Q 1:

2

3

# import timedelta class

4

# ---- your solution here ---

5

6

# define objects

7

# ---- your solution here ---

8

9

# find the difference

10

# ---- your solution here ---

11

12

# print the result

13

# ---- your solution here ---

[1]:

8 days, 1:45:00

Q2:

Instantiate a date object using the date() constructor.

The date will be:

* year: 2017

* month: 6

* day: 5

Replace the day in this date with 24.

Print both dates (the old one and the new one).

Hints:

* date()

* replace()

* replace() will return a new date

[2]:

1

# Q 2:

2

3

# import date class

4

# ---- your solution here ---

5

6

# create a date object

7

# ---- your solution here ---

8

9

# replace the day

10

# ---- your solution here ---

11

12

# print both dates

13

# ---- your solution here ---

[2]:

2017-06-05

2017-06-24

Q3:

Get the following two information from datetime module:

* today's date (today)

* current date and time (now)

Assign them to two variables.

Print the today variable in formats below:

* dd/mm/YY

* mm/dd/y

* month (in text), day and year

* month (abbreviated), day and year

Hints:

* datetime (now)

* date (today)

* strftime()

[3]:

1

# Q 3:

2

3

# import datetime and date classes

4

# ---- your solution here ---

5

6

# get today

7

# ---- your solution here ---

8

9

# get current date and time

10

# ---- your solution here ---

11

12

# formatting

13

# dd/mm/YY

14

# ---- your solution here ---

15

16

# mm/dd/y

17

# ---- your solution here ---

18

19

# month (in text), day and year

20

# ---- your solution here ---

21

22

# month (abbreviated), day and year

23

# ---- your solution here ---

[3]:

dd/mm/YY: 06/04/2022

mm/dd/y: 04/06/22

month (in text), day and year: April 06, 2022

month (abbreviated), day and year: Apr 06, 2022

Q4:

Create a datetime object corresponding to date string of "%d/%m/%y %H:%M".

Here are the date values:

* year: 2020

* month: 10

* day: 26

* hour: 18

* minutes: 47

Then print the following text by formatting this datetime object:

"Monday, 26. October 2020 06:47PM"

Hints:

* strptime()

* strftime()

[4]:

1

# Q 4:

2

3

# import datetime class

4

# ---- your solution here ---

5

6

# create a datetime object with strptime()

7

# ---- your solution here ---

8

9

# Format the datetime with strftime()

10

# ---- your solution here ---

11

12

# print the formatted date

13

# ---- your solution here ---

[4]:

Monday, 26. October 2020 06:47PM

Q5:

Create a date object from ISO format.

The date will be: 2018-11-23

Then print the date in following formats:

* 2018-11-23

* 23/11/18

* Friday 23. November 201

Hints:

* fromisoformat()

* isoformat()

* strftime()

[5]:

1

# Q 5:

2

3

# import date class

4

# ---- your solution here ---

5

6

# create a date from ISO format

7

# ---- your solution here ---

8

9

# formatted print

10

# ---- your solution here ---

[5]:

2018-11-23

23/11/18

Friday 23. November 2018

SOLUTIONS – Date and Time

Here are the solutions for the quiz for this chapter.

S1:

[1]:

1

# S 1:

2

3

# import timedelta class

4

from datetime import timedelta

5

6

# define objects

7

object_1 = timedelta(days=23, hours=8, minutes=35)

8

object_2 = timedelta(days=15, hours=6, minutes=50)

9

10

# find the difference

11

difference = object_1 - object_2

12

13

# print the result

14

print(difference)

[1]:

8 days, 1:45:00

S2:

[2]:

1

# S 2:

2

3

# import date class

4

from datetime import date

5

6

# create a date object

7

date_1 = date(2017, 6, 5)

8

9

# replace the day

10

date_2 = date_1.replace(day=24)

11

12

# print both dates

13

print(date_1)

14

print(date_2)

[2]:

2017-06-05

2017-06-24

S3:

[3]:

1

# S 3:

2

3

# import datetime and date classes

4

from datetime import datetime, date

5

6

# get today

7

today = date.today()

8

9

# get current date and time

10

now = datetime.now()

11

12

# formatting

13

# dd/mm/YY

14

format_1 = today.strftime("%d/%m/%Y")

15

print("dd/mm/YY:", format_1)

16

17

# mm/dd/y

18

format_2 = today.strftime("%m/%d/%y")

19

print("mm/dd/y:", format_2)

20

21

# month (in text), day and year

22

format_3 = today.strftime("%B %d, %Y")

23

print("month (in text), day and year:", format_3)

24

25

# month (abbreviated), day and year

26

format_4 = today.strftime("%b %d, %Y")

27

print("month (abbreviated), day and year:", format_4)

[3]:

dd/mm/YY: 06/04/2022

mm/dd/y: 04/06/22

month (in text), day and year: April 06, 2022

month (abbreviated), day and year: Apr 06, 2022

S4:

[4]:

1

# S 4:

2

3

# import datetime class

4

from datetime import datetime

5

6

# create a datetime object with strptime()

7

dt = datetime.strptime("26/10/20 18:47", "%d/%m/%y %H:%M")

8

9

# Format the datetime with strftime()

10

formatted_dt = dt.strftime("%A, %d. %B %Y %I:%M%p")

11

12

# print the formatted date

13

print(formatted_dt)

[4]:

Monday, 26. October 2020 06:47PM

S5:

[5]:

1

# S 5:

2

3

# import date class

4

from datetime import date

5

6

# create a date from ISO format

7

d = date.fromisoformat('2018-11-23')

8

9

# formatted print

10

print(d.isoformat())

11

print(d.strftime("%d/%m/%y"))

12

print(d.strftime("%A %d. %B %Y"))

[5]:

2018-11-23

23/11/18

Friday 23. November 2018

7.      Decorators

Decorators in Python

Decorators are extremely useful tools in Python. A decorator is a function that takes another function as the parameter and extends its functionality without explicitly modifying it. It allows us to modify the behavior of a function or a class without touching its source code.

In other words, a decorator wraps a function in order to extend its behaviors, without permanently modifying it.

In this chapter, we will learn how decorators work in Python. You can find the PyCharm project for this chapter in the GitHub Repository of this book.

Chapter Outline:

Functions are First-Class Citizens

Defining a Decorator

Decorators with Parameters

General Decorators

Decorator Changes the Function Name

Chaining Decorators

Class Decorators

QUIZ – Decorators

SOLUTIONS – Decorators

Functions are First-Class Citizens

In order to understand how decorators work, we need to revisit some important concepts about functions in Python.

In Python, functions are first-class citizens. By this, we mean:

Functions can be assigned as regular variables

Functions can be passed as arguments to other functions

Functions can return functions

Functions can have other functions (inner functions or nested functions) in their function body

Now let’s see some examples of these points on functions:

Example 1: Functions can be assigned as regular variables.

[1]:

1

# Example 1:

2

# assign function to a variable

3

def say_hi(user_name):

4

    return 'Hi ' + user_name

5

6

# assign the function

7

hi_name = say_hi

8

print(hi_name('Bruce Wayne'))

[1]:

Hi Bruce Wayne

In cell 1, we define a function as say_hi. Then we assign this function to a local variable named hi_name. Now this hi_name variable is a function too. And in line 8, we call the hi_name as: hi_name('Bruce Wayne').

Example 2: Functions can be passed as arguments to other functions.

[2]:

1

# Example 2:

2

# Functions can be passed as arguments to other functions.

3

def print_hello(user):

4

    print('Hello', user)

5

6

def hi_with_function(func, user_name):

7

    func(user_name)

8

9

# call the function

10

hi_with_function(print_hello, 'Clark Kent')

[2]:

Hello Clark Kent

In cell 2, we define two functions; print_hello and hi_with_function. The second one takes a function as an argument: hi_with_function(func, user_name). And it calls this function in its function body in line 7 as: func(user_name).

Example 3: Functions can return functions.

[3]:

1

# Example 3:

2

# Functions can return functions

3

def return_hi_function():

4

    return say_hi

5

6

# call the function

7

hi = return_hi_function()

8

print(hi('Spiderman'))

[3]:

Hi Spiderman

In cell 3, we define a function named return_hi_function. This function simply returns another function, which is the say_hi function that we defined in cell 1. In line 7, we assign the returning function to a variable called hi. Now this hi variable is also a function. Then we call it in line 8.

Example 4: Functions can have other functions (inner functions) in their function body.

[4]:

1

# Example 4:

2

# Functions can have other functions in their body

3

def outer_func(msg):

4

    """Outer function"""

5

6

    # define a nested function

7

    def inner_func():

8

        """Inner function"""

9

        print(msg, 'from nested function.')

10

11

    # call the nested function

12

    inner_func()

13

14

# call the outer function

15

outer_func('The Batman')

[4]:

The Batman from nested function.

In cell 4 we define an outer function as outer_func. Inside this function we define a nested function named inner_func. And in line 12, we call the inner function.

When we call the outer the function in line 15, we pass the text of ‘The Batman’ for the value of the msg parameter. And the output is ‘The Batman from nested function.’. This text is printed by the inner_func. Be careful here, the inner_func uses the msg variable which is not defined in its own body. In other words, it uses a variable which belongs to its parent’s scope. This is the idea behind the Closures in Python.

Python Closures: A Closure is a function object that remembers the values in the parent’s scope even if they are not present in memory.

Defining a Decorator

A decorator takes in a function as an argument, adds some functionality to it and returns it back to the caller. Sometimes, people call this action as metaprogramming because a part of the program tries to modify another part of the program at compile time.

[5]:

1

# define a simple decorator

2

def first_decorator(func):

3

    def wrapper():

4

        print("Before running {0} function".format(func.__name__))

5

        func()

6

        print("After running {0} function".format(func.__name__))

7

    return wrapper

8

9

def greet():

10

    print("Hi there")

11

12

# call the first_decorator function

13

greet = first_decorator(greet)

14

15

# call the greet function now

16

greet()

[5]:

Before running greet function

Hi there

After running greet function

In cell 5, we define a simple decorator function. The decorator’s name is first_decorator and it has a nested function in it. The nested function is called wrapper. The first_decorator function simply returns this wrapper function.

The wrapper function prints some text then calls the function which is the parameter value (func) as: func(). And finally, it prints another text.

In line 13, we call the first_decorator function by passing the greet() function as the argument. And we re-assign the returning value to the greet variable. Here is the code: greet = first_decorator(greet). Remember that first_decorator returns a function (wrapper).

In line 16, we call the greet() function. And in the output you see that the behavior of greet() function is modified. In its original form, it was printing just one line. But now, it prints three lines.

To be sure about the final form of the greet() function, let’s print its object data:

[6]:

1

# greet function object data

2

print(greet)

[6]:

<function first_decorator.<locals>.wrapper at 0x100f569e0>

In cell 6, we print the greet() function object data. And as you see in the output, the function name is now wrapper in the memory. Why? Because we re-assign it from the returning value from the first_decorator function, which is the wrapper function.

Now we are sure that, we have decorated our greet() function. In other words, we didn’t modify its source code, but we added some new functionality.

Decorator Syntax:

Line 13 in cell 5 is the way we decorate the greet() function. It is: greet = first_decorator(greet). What we do is, we pass the function to the decorator and re-assign the resulting value to the same function (actually a variable with the same name).

In Python, we have a better and more readable syntax for decorating functions. Here it is:

[7]:

1

# Decorator Syntax:

2

3

# Long Way

4

# def greet():

5

#     print("Hi there")

6

7

# greet = first_decorator(greet)

8

# greet()

9

10

# Pythonic Way

11

@first_decorator

12

def greet():

13

    print("Hi there")

14

15

greet()

[7]:

Before running greet function

Hi there

After running greet function

In cell 7 line 11, you see the syntax for using a decorator. We simply put the decorator’s name with an @ symbol on the function definition line. Here it is:

[8]:

1

@first_decorator

2

def greet():

3

     …

So, saying @first_decorator is the simple way of saying greet = first_decorator(greet). This is how we apply a decorator to a function.

Decorators with Parameters

In the previous section, we learned how we define and use a decorator function. Now let’s see what happens if the function which we want to decorate accepts some parameters. How will we decorate such functions?

Let’s assume we want to define a function which takes two numbers, performs the division operation and returns the result. Here is the function:

[9]:

1

# division function

2

def division(x, y):

3

    return x / y

4

5

# call the function

6

result_1 = division(20, 5)

7

print(result_1)

8

9

result_2 = division(8, 0)

10

print(result_2)

[9]:

4.0

ZeroDivisionError: division by zero

In cell 9, you see the definition of the division() function. It simply returns the division result. And we call it with two parameters (20, 5) and it returns 4.0. But there is a problem here. What if the second number, the divisor, is 0? We will get a ZeroDivisionError if the second parameter is zero. We can implement a try-except block inside our division function to overcome this problem. However, we don’t want to modify the code in the function body. So, we need another way.

The solution is to decorate this function with a decorator. The decorator will be responsible of checking if the second parameter is zero or not. Let’s define this decorator function:

[10]:

1

# decorator

2

def division_decorator(f):

3

    # define the wrapper function

4

    def wrapper(a, b):

5

        if b == 0:

6

            print("Division by zero is not possible.")

7

            return

8

        else:

9

            return f(a, b)

10

11

    # return the wrapper function

12

    return wrapper

In cell 10, we define a decorator. The name is division_decorator and it is responsible for checking the ZeroDivisionError. In the wrapper function, it checks if b is equal to zero or not. If it is, then it simply prints an error message and returns. If b is not zero, then it calls the function f and returns it: return f(a, b). Finally, the division_decorator function returns the wrapper function. Remember that this is the idea behind the decorators in Python.

Now let’s decorate the division function with the division_decorator:

[11]:

1

# decorate division function

2

@division_decorator

3

def division(x, y):

4

    return x / y

Now that we use the decorator for our division() function let’s call it with zero for the divisor value:

[12]:

1

# call the function now

2

result_1 = division(20, 5)

3

print(result_1)

4

5

result_2 = division(8, 0)

6

print(result_2)

[12]:

4.0

Division by zero is not possible.

None

As you see in the output of cell 12, we handle the case where the caller may pass a zero for the divisor. And we managed this by the help of a decorator. The division_decorator function implements all the logic which is necessary for this case. More importantly, we didn’t modify our division() function body. It’s still the same function, but decorated now.

General Decorators

In the previous section we saw that the parameters of the division() function and the wrapper() function inside the decorator must match. Why? Because we wrapper() function will replace the division() function after decoration. So, their parameters should match. But this brings another problem. What if we want to use the same decorator with multiple functions? And what if these functions have different numbers of parameters? Let’s answer this question now.

To start with, let’s assume we want to print our users’ names in full capital letters. Some users may have just their first names, while the others may have first names and last names. So, we will define two separate functions as first_name and full_name. And we will define a decorator function which will convert the names into upper case. Here is the decorator:

[13]:

1

# define a general decorator

2

def upper_decorator(func):

3

    # wrapper function

4

    def wrapper(*args):

5

        # modify the items in *args

6

        new_args = []

7

        for i, arg in enumerate(args):

8

            new_args.append(arg.upper())

9

        new_args = tuple(new_args)

10

11

        # return the call to the func

12

        return func(*new_args)

13

14

    # return wrapper function

15

    return wrapper

In cell 13, we define a general decorator. Why is it general? Because, in its wrapper function, the parameter is *args. This enables the wrapper function to take any number of parameters. In other words, the wrapper function will be able to present any function which is decorated with this decorator.

In the wrapper function, it modifies the items in the args tuple. It converts each item to the upper case and appends it to a list. In line 9, it converts this list to a tuple. And finally, in line 12, it returns the call to the func parameter by passing *new_args tuple. Here is the line: return func(*new_args).

Now let’s define the first_name and full_name functions. We want both of them to be decorated with the upper_decorator. Here they are:

[14]:

1

@upper_decorator

2

def first_name(name):

3

    print(name)

4

5

# call first_name function

6

first_name('john')

7

8

@upper_decorator

9

def full_name(firt, last):

10

    print(firt, last)

11

12

# call the full_name function

13

full_name('john', 'doe')

[14]:

JOHN

JOHN DOE

In cell 14, we define two functions, first_name and full_name, and we decorate both with the upper_decorator. Be careful that, the functions have different number of parameters. Then we call both functions with different set of parameters. And in the output, you see that the names have been capitalized for both. That’s the way we use the same generator for functions with any number of parameters.

Here is a more general decorator syntax with both *args and **kwargs:

[15]:

1

# generator with *args and **kwargs

2

def most_general_decorator(func):

3

    def wrapper(*args, **kwargs):

4

        # implement some logic here

5

        return func(*args, **kwargs)

6

7

    return wrapper

Decorator Changes the Function Name

There is an important point which you should always keep in mind when you work with decorators. The function name will be changed after you decorate it. How is that possible? Let’s see with an example:

[16]:

1

# function without decorator

2

def last_name(last):

3

    print(last)

4

5

# print function name

6

print("Function name before decorator:", last_name.__name__)

7

8

# function with decoratoe

9

@upper_decorator

10

def last_name(last):

11

    print(last)

12

13

# print function name

14

print("Function name after decorator:", last_name.__name__)

[16]:

Function name before decorator: last_name

Function name after decorator: wrapper

In cell 16, we use the upper_decorator which we defined in cell 13. We define a new function as last_name.

In the first definition, we didn’t decorate it. And we print its name in line 6. The function name is ‘last_name’ as expected.

Then we redefine this function with a decorator this time. And in line 14, we print its name one more time. Surprisingly this time its name is ‘wrapper’.

Why does the name changes from ‘last_name’ to ‘wrapper’? Because when we decorate it, the decorator returns the wrapper function. Remember that decorating is exactly the same as this line:

last_name = upper_decorator(last_name)

We reassign the returning value from the decorator to our function. Since the decorator simply returns the wrapper function, the name of our original function changes to ‘wrapper’ now.

To fix this issue, Python provides a very simple solution which is the functools module. Let’s see how we can use this module to keep the original function name unchanged:

[17]:

1

# functools

2

import functools

3

4

# define a decorator with functools

5

def upper_decorator(func):

6

    @functools.wraps(func)

7

    def wrapper(*args):

8

        # modify the items in *args

9

        new_args = []

10

        for i, arg in enumerate(args):

11

            new_args.append(arg.upper())

12

        new_args = tuple(new_args)

13

14

        # return the call to the func

15

        return func(*new_args)

16

17

    # return wrapper function

18

    return wrapper

In cell 17 line 6, we use the functools.wraps() method. This method itself is a decorator. So, we use it on the wrapper function with the @ symbol. Here is the syntax: @functools.wraps(func). The original function (func) is the parameter to this method. This method preserves information about the original function.

Now let’s print the name of the last_name function one more time:

[18]:

1

# function without decorator

2

def last_name(last):

3

    print(last)

4

5

# print function name

6

print("Function name before decorator:", last_name.__name__)

7

8

# function with decoratoe

9

@upper_decorator

10

def last_name(last):

11

    print(last)

12

13

# print function name

14

print("Function name after decorator:", last_name.__name__)

[18]:

Function name before decorator: last_name

Function name after decorator: last_name

As you see in the output of cell 18, now the name of our original function is the same after we decorate it.

Chaining Decorators

Most of the time you will need to use more than one decorator for a function. This is called chaining decorators on the same function. Let’s see how we can use multiple decorators (or the same decorator multiple times).

[19]:

1

# function for printing symbols

2

def print_symbol(symbol, times):

3

    print(symbol * times)

4

5

# decorator 1

6

def plus_sign(f):

7

    def wrapper(*args, **kwargs):

8

        print_symbol('+', 20)

9

        f(*args, **kwargs)

10

        print_symbol('+', 20)

11

    return wrapper

12

13

# decorator 2

14

def minus_sign(f):

15

    def wrapper(*args, **kwargs):

16

        print_symbol('-', 20)

17

        f(*args, **kwargs)

18

        print_symbol('-', 20)

19

    return wrapper

In cell 19, we define two decorators. Each one prints a different symbol before and after calling the original function. Now let’s use both of them on the same function:

[20]:

1

# chain decorators

2

@plus_sign

3

@minus_sign

4

def say_hi(msg):

5

    print(msg)

6

7

# call the decorated function

8

say_hi("Hi Python Developer")

[20]:

++++++++++++++++++++

--------------------

Hi Python Developer

--------------------

++++++++++++++++++++

In cell 20, we chain two decorators on the say_hi() function. And in the output, you see the order of execution of the decorators. The one which is closer to the function definition executes before the others.

Now let’s reverse the order of decorators and see the output one more time:

[21]:

1

# change the order of chaining

2

@minus_sign

3

@plus_sign

4

def say_hi(msg):

5

    print(msg)

[21]:

--------------------

++++++++++++++++++++

Hi Python Developer

++++++++++++++++++++

--------------------

As you see in the output of cell 21, the order of execution changes when we change the order of decorators.

Class Decorators

Decorators can be either functions or classes in Python. In the previous sections we worked with function decorators. Now, we will learn how to define class decorators.

We will define custom classes that acts as a decorator. When a function is decorated with a class, that function becomes an instance of the class. Let’s see how:

[22]:

1

# define a class decorator

2

class ClassDecorator:

3

    # init method takes the function

4

    def __init__(self, func):

5

        self.func = func

6

7

    # implement __call__ method

8

    def __call__(self):

9

        # some logic before func call

10

        print('__call__ method before func')

11

        self.func()

12

        # Some logic after func call

13

        print('__call__ method after func')

In cell 22, we have a simple class decorator. For any class to be a decorator, it needs to implement the __call__() method. The __call__() method acts the same way as the wrapper function in the function decorators.

Now let’s use this class to decorate a function:

[23]:

1

# add class decorator to func

2

@ClassDecorator

3

def say_hi():

4

    print("Hi Python")

5

6

# call the decorated function

7

say_hi()

[23]:

__call__ method before func

Hi Python

__call__ method after func

Class Decorator with *args and **kwargs :

In order to use a class decorator with *args and **kwargs arguments, we need to implement the __call__() method with these arguments and pass them to the decorated function.

[24]:

1

# class decorator with *args & **kwargs

2

class ClassDecorator:

3

    def __init__(self, func):

4

        self.func = func

5

6

    def __call__(self, *args, **kwargs):

7

        # some logic before func call

8

        self.func(*args, **kwargs)

9

        # Some logic after func call

In cell 24, the __call__() method of the class decorator takes *args and **kwargs arguments. And in line 8, it passes them to the decorated function as: self.func(*args, **kwargs).

Let’s decorate a function with this class decorator now:

[25]:

1

# add class decorator to func

2

@ClassDecorator

3

def say_hi(first, last, msg='Hi'):

4

    print("{0} {1} {2}".format(msg, first, last))

5

6

# call the decorated function

7

say_hi("Bruce", "Wayne", "Hi")

[25]:

Hi Bruce Wayne

Class Decorator with the return statement:

Remember that, in the wrapper function of a function decorator we use the return keyword to return the decorated function. We will do the same thing here, but inside the __call__() method this time.

[26]:

1

# define a class decorator

2

class UpperDecorator:

3

    def __init__(self, func):

4

        self.func = func

5

6

    def __call__(self, *args):

7

        # modify the items in *args

8

        new_args = []

9

        for i, arg in enumerate(args):

10

            new_args.append(arg.upper())

11

        new_args = tuple(new_args)

12

13

        # return the call to the func

14

        return self.func(*new_args)

15

16

@UpperDecorator

17

def full_name(first, last):

18

    print(first, last)

19

20

# call decorated function

21

full_name('jane', 'doe')

[26]:

JANE DOE

In cell 26 line 14, in the __call__() method we return the decorated function as: return self.func(*new_args).

QUIZ – Decorators

Now it’s time to solve the quiz for this chapter. You can download the quiz file, QUIZ_Decorators.zip, from the GitHub Repository of this book. You should put the quiz file in the Decorators project we build in this chapter.

Here are the questions for this chapter:

Q1:

We have a basic function which takes one argument and simply returns it.

Here is this function:

def some_text(text):

    return text

We want this function to always return the text in uppercase letters.

For example when we call it as:

some_text('this is a lowercase text...')

We want it to return 'THIS IS A LOWERCASE TEXT...'.

But we do not want to modify the function body.

Function definition will not be changed.

What we need is a decorator to do the uppercase conversion for our function.

You need to define this decorator.

Also make sure that, if the function parameter is not a string (str) object, you shouldn't convert it to upper case.

Hints:

* @decorator

* type()

[1]:

1

# Q 1:

2

3

# define the decorator

4

# ---- your solution here ---

5

6

# decorate the function

7

# ---- your solution here ---

8

9

# call some_text function

10

# ---- your solution here ---

11

12

# print the final text

13

# ---- your solution here ---

[1]:

THIS IS A LOWERCASE TEXT...

Q2:

We want to track the execution time for our functions.

We will use the time module for time tracking.

Remember that time.time() returns the current time in seconds.

So, if we get the start and end times of execution, we can measure how long the function takes.

We want to define a decorator for this operation.

The decorator’s name will be time_tracker.

Since it is going to be a general-purpose decorator, we don't know the number of parameters in advance.

We want it to print something like this (after executing the function):

"Function executed in (seconds): 2.0050909519195557"

To test your decorator, simply define a function which sleeps for some seconds.

Here is the function:

def sleeper(seconds):

    time.sleep(seconds)

The sleeper() function just waits for the given number of seconds.

Decorate this function with your decorator and print its execution time.   

Hints:

* @decorator

* *args

* time.time()

[2]:

1

# Q 2:

2

3

# import time module

4

# ---- your solution here ---

5

6

# define the decorator

7

# ---- your solution here ---

8

9

# decorate function

10

# ---- your solution here ---

11

12

# call the sleeper function to sleep 2 seconds

13

# ---- your solution here ---

[2]:

Function executed in (seconds): 2.005089044570923

Q3:

Define a class decorator named FunctionDetails.

This decorator will print the function details as follows:

* Function Name

* Parameters

* Function Result

Here is the function which we want to decorate:

def full_name(first, last):

    print(first, last)

Call the function after you decorate it with FunctionDetails.

Hints:

* __init__

* __call__

* *args

[3]:

1

# Q 3:

2

3

# define a class decorator

4

# ---- your solution here ---

5

6

# decorate the function

7

# ---- your solution here ---

8

9

# call decorated function

10

# ---- your solution here ---

[3]:

Function Name: full_name

Parameters:

(0, 'john')

(1, 'doe')

Function Result:

john doe

Q4:

We have a function which simply adds two numbers:

def add_and_multiply(n_1, n_2):

    return n_1 + n_2

This function returns the summation of its parameters.

We want to multiply the result of this function with some numbers.

For this, we want to use a decorator.

Define a decorator which multiplies the function result by the given factor.

Decorator name will be multiply_by and it will take one parameter.

Here is the parameter:

* func: function to decorate

The wrapper (inner) function in the decorator will take 3 arguments:

* num_1: first parameter from the function (n_1)

* num_2: second parameter from the function (n_2)

* factor: the number which we want to multiply with

[4]:

1

# Q 4:

2

3

# define a decorator

4

# ---- your solution here ---

5

6

# decorate function

7

# ---- your solution here ---

8

9

# call the function with n_1, n_2, and factor

10

add_and_multiply_by_100 = add_and_multiply(4, 7, 100)

11

12

# print the result

13

print(add_and_multiply_by_100)

[4]:

1100

Q5:

Here is a simple decorator:

1

# functools

2

import functools

3

4

# define a decorator with functools

5

def upper_decorator(func):

6

    @functools.wraps(func)

7

    def wrapper(*args):

8

        # modify the items in *args

9

        new_args = []

10

        for i, arg in enumerate(args):

11

            new_args.append(arg.upper())

12

        new_args = tuple(new_args)

13

14

        # return the call to the func

15

        return func(*new_args)

16

17

    # return wrapper function

18

    return wrapper

Why do we need to use @functools.wraps(func) in line 6, to decorate the wrapper function?

What does it do?

SOLUTIONS – Decorators

Here are the solutions for the quiz for this chapter.

S1:

[1]:

1

# S 1:

2

3

# define the decorator

4

def uppercase_convert(func):

5

    # define the wrapper function

6

    def wrapper(text):

7

        func_result = func(text)

8

        # check if func result is str type

9

        if type(func_result) == str:

10

            return func_result.upper()

11

        else:

12

            return func_result

13

14

    return wrapper

15

16

# decorate the function

17

@uppercase_convert

18

def some_text(text):

19

    return text

20

21

# call some_text function

22

upper_text = some_text("this is a lowercase text...")

23

24

# print the final text

25

print(upper_text)

[1]:

THIS IS A LOWERCASE TEXT...

S2:

[2]:

1

# S 2:

2

3

# import time module

4

import time

5

6

# define the decorator

7

def time_tracker(func):

8

    # define the wrapper function

9

    def wrapper(*arg):

10

        # get start time

11

        t_start = time.time()

12

        # call the function

13

        func_result = func(*arg)

14

        # get end time

15

        t_end = time.time()

16

        print("Function executed in (seconds):", str(t_end - t_start))

17

        return func_result

18

19

    return wrapper

20

21

# decorate function

22

@time_tracker

23

def sleeper(seconds):

24

    time.sleep(seconds)

25

26

# call the sleeper function to sleep 2 seconds

27

sleeper(2)

[2]:

Function executed in (seconds): 2.005089044570923

S3:

[3]:

1

# S 3:

2

3

# define a class decorator

4

class FunctionDetails:

5

    def __init__(self, func):

6

        self.func = func

7

8

    def __call__(self, *args):

9

        print("Function Name:", self.func.__name__)

10

        print("Parameters:")

11

        for arg in enumerate(args):

12

            print(arg)

13

14

        # execute the function

15

        print("Function Result:")

16

        return self.func(*args)

17

18

# decorate the function

19

@FunctionDetails

20

def full_name(first, last):

21

    print(first, last)

22

23

# call decorated function

24

full_name('john', 'doe')

[3]:

Function Name: full_name

Parameters:

(0, 'john')

(1, 'doe')

Function Result:

john doe

S4:

[4]:

1

# S 4:

2

3

# define a decorator

4

def multiply_by(func):

5

    def wrapper(num_1, num_2, factor):

6

        return func(num_1, num_2) * factor

7

    return wrapper

8

9

# decorate function

10

@multiply_by

11

def add_and_multiply(n_1, n_2):

12

    return n_1 + n_2

13

14

# call the function with n_1, n_2, and factor

15

add_and_multiply_by_100 = add_and_multiply(4, 7, 100)

16

17

# print the result

18

print(add_and_multiply_by_100)

[4]:

1100

S5:

functools.wraps() function is a decorator which is applied to the wrapper function of a decorator.

It updates the wrapper function to look like decorated function.

So, it preserves information about the original function, such as __name__, __doc__ etc.

8.      Context Managers

What is a Context Manager?

A Context Manager is an object that defines the runtime context to be established when executing a with statement. The context manager handles the entry into, and the exit from, the desired runtime context for the execution of the block of code. Context managers are normally invoked using the with statement, but can also be used by directly invoking their methods.

Typical uses of context managers include saving and restoring various kinds of global state, locking and unlocking resources, closing opened files, etc...

In this chapter, we will learn how to work with context managers in Python and how to define custom context managers. You can find the PyCharm project for this chapter in the GitHub Repository of this book.

Chapter Outline:

The with Statement

Context Manager Protocol

Creating a Context Manager in Class Form

Creating a Context Manager in Function Form

QUIZ – Context Managers

SOLUTIONS – Context Managers

The width Statement

The with statement is used to wrap the execution of a block with methods defined by a context manager. This allows common try…except…finally usage patterns to be encapsulated for convenient reuse. Compared to traditional try…except…finally blocks, the with statement provide a shorter and reusable code.

In Python Standard Library many classes support with statement. A very common example is the built-in open() function which provides tools to work with the file objects using the with statement.

Here is the general syntax of the with statement:

[1]:

1

with expression as target:

2

    # do something

3

    # using the target

Let’s see an example with the open() function. We have a text file in the files folder in our current project. The file name is color_names.txt and it includes some color names. We want to open and print the content in this file by using the open() function and with statement. Here is the code.

[2]:

1

# define the file path

2

path = 'files/color_names.txt'

3

4

# with statement

5

with open(path, mode='r') as file:

6

    # read the file content

7

    print(file.read())

[2]:

red

orange

yellow

green

blue

white

black

In cell 2, you see a common use case of the with statement. We use the open() function to open the file at the given path. And the open() function returns the file object in read mode. We use this file object in line 7 to read and print its content as: print(file.read()).

Context Manager Protocol

Python’s with statement supports the concept of a runtime context defined by a context manager. This is implemented using a pair of methods that allow user-defined classes to define a runtime context that is entered before the statement body is executed and exited when the statement ends.

These methods are called the Context Manager Protocol. Here they are:

__enter__(self):

This method is called by the with statement to enter the runtime context related to current object. The with statement will bind this method’s return value to the target specified in the as clause of the statement, if any. See cell 1.

An example of a context manager that returns itself is a file object. File objects return themselves from __enter__() to allow open() to be used as the context expression in a with statement. See cell 2.

__exit__(self, exc_type, exc_value, traceback):

This method is called when the execution leaves the with code block. It exits the runtime context related to this object. The parameters describe the exception that caused the context to be exited. If the context was exited without an exception, all three arguments will be None.

If an exception is supplied, and the method wishes to suppress the exception (i.e., prevent it from being propagated), it should return a true value. Otherwise, the exception will be processed normally upon exit from this method.

The __exit__() method returns a Boolean value, either True or False.

The execution of the with statement with the methods in context manager protocol proceeds as follows:

[3]:

1

with EXPRESSION as TARGET:

2

    SUITE

The context expression is evaluated to obtain a context manager.

The context manager’s __enter__() is loaded for later use.

The context manager’s __exit__() is loaded for later use.

The context manager’s __enter__() method is invoked.

If a target was included in the with statement, the return value from __enter__() is assigned to it.

The suite (code block in the with statement scope) is executed.

The context manager’s __exit__() method is invoked. If an exception caused the suite to be exited, its type, value, and traceback are passed as arguments to __exit__(). Otherwise, three None arguments are supplied.

If the suite was exited for any reason other than an exception, the return value from __exit__() is ignored, and execution proceeds at the normal location for the kind of exit that was taken.

Creating a Context Manager in Class Form

Now that we know the basic idea behind the context manager protocol let’s implement it in a class. This class will be our context manager and we will use it later with the with statement.

[4]:

1

# custom context manager class

2

class CustomContextManager:

3

    # init method -> define variables

4

    def __init__(self, path, mode):

5

        self.path = path

6

        self.mode = mode

7

        self.file = None

8

9

    # __enter__ method -> open the file

10

    def __enter__(self):

11

        self.file = open(self.path, self.mode)

12

        return self.file

13

14

    # exit method to close the file

15

    def __exit__(self, exc_type, exc_value, exc_traceback):

16

        self.file.close()

Our CustomContextManager class implements the necessary methods to become a context manager: __enter__ and __exit__.

In its __init__ method, it defines three instance variables to store the path, mode and file objects.

In the __enter__ method it uses the built-in open() function to open the file in the specified path. Since the open() function returns a file object, we assign it to the self.file attribute.

In the __exit__ method we close the file as: self.file.close(). The __exit__ method accepts three arguments, which are required by context manager protocol.

We can use our custom context manager in a with statement now. Let’s do it:

[5]:

1

# custom context manager in with statement

2

file_path = 'files/color_names.txt'

3

4

with CustomContextManager(path=file_path, mode='r') as file:

5

    # print the file content

6

    print(file.read())

[5]:

red

orange

yellow

green

blue

white

black

In cell 5, we use our CustomContextManager class in the with statement. We read file content and print it.

Here is what happens behind the scenes:

The line 4 calls the __enter__ method of the CustomContextManager class.

The __enter__ method opens the file and returns it.

We name the opened file simply as file.

In the suite of with statement, we read the file content and print it.

The with statement calls the __exit__ method.

The __exit__ method closes the file.

Let’s define another context manager class. This time we want to print the list of files in the specified folder.

[6]:

1

# context manager for listing files

2

import os

3

4

class ContentList:

5

    '''Returns the content of a directory'''

6

7

    def __init__(self, directory):

8

        self.directory = directory

9

10

    def __enter__(self):

11

        return os.listdir(self.directory)

12

13

    def __exit__(self, exc_type, exc_val, exc_tb):

14

        if exc_type is not None:

15

            print("Error getting directory list.")

16

        return True

17

18

# print the contents of the project directory

19

project_directory = '.'

20

with ContentList(project_directory) as directory_list:

21

    print(directory_list)

[6]:

['ClassContextManager.py', 'TheWithStatement.py', 'files', '.idea']

In cell 6, we define a new context manager. The name of the class is ContentList. Why is it a context manager? Because it implements context manager protocol (__enter__ and __exit__ methods).

We take the directory path as the parameter in the class constructor, which is the __init__ method.

In the __enter__ method we get the list of the contents in this directory simply by calling the listdir() method in the os module as: os.listdir(self.directory). And we return this list. Please be careful that, our __enter__ method returns a list in this context manager.

In the __exit__ method, we check if there exist any errors. The exc_type, exc_val, exc_tb parameter values will not be None if there is an error in our context manager. So, we check if exc_type is not None to print an error text.

In line 20, we use our context manager in the with statement. Since it returns a list object, we simple assign the returning value to the directory_list variable. And in the body of the with statement, we print this list. In the cell output, you see the list of the contents in the project directory. Remember that, ‘.’ means the current directory, which is the project directory in our case.

Creating a Context Manager in Function Form

In the previous section we learned how to define context managers by using the class syntax. However, it is a bit cumbersome and lengthy. You need to implement __enter__ and __exit__ methods explicitly and you need to handle possible exceptions. Hopefully there is a better way of creating context managers in Python: function-based context managers.

Function-Based Context Managers are special functions which use generators and contextlib.contextmanager decorator. The contextlib.contextmanager decorator is responsible for implementing the context manager protocol.

Let’s define a context manager function:

[7]:

1

from contextlib import contextmanager

2

3

# define the context manager function

4

@contextmanager

5

def function_based_context_manager():

6

    print("Entering the context: __enter__")

7

    yield "This is a function based context manager"

8

    print("Leaving the context: __exit__")

9

10

# use context manager function in with statement

11

with function_based_context_manager() as yield_text:

12

    print(yield_text)

[7]:

Entering the context: __enter__

This is a function based context manager

Leaving the context: __exit__

In cell 7, we define a custom function which acts as a context manager. The contextmanager decorator is what turns a regular function into a full-stack context manager. You don’t need to worry about implementing the __enter__ and __exit__ functions if you implement @contextmanager decorator.

The yield statement in line 7, acts as the return statement in the __enter__ method in a class-based context manager. Since we have a yield statement, function-based context managers are also generator functions.

Let’s define a new context manager. This time it will open a file in write mode and append some text to it. Here it is:

[8]:

1

# context manager for write operations

2

from contextlib import contextmanager

3

4

@contextmanager

5

def writer_context_manager(path, mode='w'):

6

    file_object = None

7

    try:

8

        file_object = open(path, mode=mode)

9

        yield file_object

10

    finally:

11

        if file_object:

12

            file_object.close()

13

14

# context manager in with statement

15

with writer_context_manager("function_based_context_managers.txt") as file:

16

    file.write("The contextlib.contextmanager decorator \n"

17

               "is responsible for implementing the\n"

18

                "context manager protocol.")

In cell 8, we define a function-based context manager. In the try block it tries to open the file in the specified path. If it opens the file successfully, then it yields (returns) the file_object. In the finally block we check if we have a file_object to close. And we close the file_object if it is not None.

In the with statement in line 15, we call our context manager with a file name of function_based_context_managers.txt. The context manager opens this file in write mode and returns the file object which we simply name as file. And in line 16, we write some text in this file. Remember that ‘w’ mode will create an empty file, if such a file doesn’t exist.

QUIZ – Context Managers

Now it’s time to solve the quiz for this chapter. You can download the quiz file, QUIZ_ContextManagers.zip, from the GitHub Repository of this book. You should put the quiz file in the ContextManagers project we build in this chapter.

Here are the questions for this chapter:

Q1:

Define a custom context manager class as MyContextManager.

Define the __init__, __enter__ and __exit__ methods.

Your context manager should open and return the file object at specified path.

Also it should close the file on exit.

The parameters to the __init__ method should be:

* path: file path for the file to read

* mode: operation mode (read, write, append etc.)

Read and print the file content at "files/color_names.txt".

Hints:

* __init__

* __enter__

* __exit__

[1]:

1

# Q 1:

2

3

# define custom context manager class

4

# ---- your solution here ---

5

6

# define the file path as "files/color_names.txt"

7

file_path = 'files/color_names.txt'

8

9

# read and print the content of the file at this path

10

# ---- your solution here ---

[1]:

red

orange

yellow

green

blue

white

black

Q2:

Define a context manager class to get the list of folders in a directory.

The name will be FoldersInDirectory and it will return only the folder names in a specified directory.

The class constructor will take the directory path as the parameter.

Make sure, you implement __enter__ and __exit__ methods properly.

Print the folder list by using a with statement.

Hints:

* __init__

* __enter__

* __exit__

[2]:

1

# Q 2:

2

3

# import os module

4

# ---- your solution here ---

5

6

# context manager for listing folders

7

# ---- your solution here ---

8

9

# define the directory path as current directory

10

# ---- your solution here ---

11

12

# print the folders in the project directory

13

# using a with statement

14

# ---- your solution here ---

[2]:

files

.idea

Q3:

Define a function based context manager which creates a file and writes some text in it.

The name will be file_writer and it will take two parameters:

* path: file path for the file to read

* mode: operation mode (read, write, append etc.)

Use this context manager in a with statement to create a file and add the text below in this file.

File name should be: "quiz_text_file_for_writing.txt".

And the text is:

"The file is created by quiz question number 3.

It is a function-based context manager,

Decorated with contextlib.contextmanager."

You should see "quiz_text_file_for_writing.txt" file after you run the code in this question.

Hints:

* contextlib.contextmanager

* try-finally

[3]:

1

# Q 3:

2

3

# import contextmanager class

4

# ---- your solution here ---

5

6

# context manager for write operations

7

# define and decorate

8

# ---- your solution here ---

9

10

11

# context manager in with statement

12

# ---- your solution here ---

Q4:

Define a function-based context manager named sleeper_context_manager.

It will print the following data:

* Start time: Time at start of its execution

* End time: Time at end of its execution

* Execution time in seconds: Difference between end time and start time

Use this context manager in a with statement.

You should wait for 2 seconds in the with statement body.

And you should print something similar to this:

"Start time: 1649345704.337794

End time: 1649345706.342916

Execution time in seconds: 2.005121946334839"

The start time and end time will change when you run the code.

But the execution time should be around 2.00 seconds.

Hints:

* contextlib.contextmanager

* yield nothing

* time.sleep()

[4]:

1

# Q 4:

2

3

# import contextmanager class and time module

4

# ---- your solution here ---

5

6

# define and decorate the context manager

7

# ---- your solution here ---

8

9

# use the context manager in a with statement

10

# ---- your solution here ---

[4]:

Start time: 1649346091.427541

End time: 1649346093.432682

Execution time in seconds: 2.005141019821167

Q5:

What are the methods in the Context Manager Protocol?

And what are their functionalities?

SOLUTIONS – Context Managers

Here are the solutions for the quiz for this chapter.

S1:

[1]:

1

# S 1:

2

3

# define custom context manager class

4

class MyContexManager:

5

    # init method -> define variables

6

    def __init__(self, path, mode):

7

        self.path = path

8

        self.mode = mode

9

        self.file = None

10

11

    # __enter__ method -> open the file

12

    def __enter__(self):

13

        self.file = open(self.path, self.mode)

14

        return self.file

15

16

    # exit method to close the file

17

    def __exit__(self, exc_type, exc_value, exc_traceback):

18

        self.file.close()

19

20

# define the file path as "files/color_names.txt"

21

file_path = 'files/color_names.txt'

22

23

# read and print the content of the file at this path

24

with MyContexManager(path=file_path, mode='r') as file:

25

    # print the file content

26

    print(file.read())

[1]:

red

orange

yellow

green

blue

white

black

S2:

[2]:

1

# S 2:

2

3

# import os module

4

import os

5

6

# context manager for listing folders

7

class FoldersInDirectory:

8

    '''Returns the folder list in the given directory'''

9

10

    def __init__(self, directory):

11

        self.directory = directory

12

13

    def __enter__(self):

14

        # return folder names only (no file name)

15

        return [name for name in os.listdir(self.directory) if os.path.isdir(name)]

16

17

    def __exit__(self, exc_type, exc_val, exc_tb):

18

        if exc_type is not None:

19

            print("Error getting directory list.")

20

        return True

21

22

# define the directory path as current directory

23

current_directory = '.'

24

25

# print the folders in the project directory

26

# using a with statement

27

with FoldersInDirectory(current_directory) as folder_list:

28

    for folder in folder_list:

29

        print(folder)

[2]:

files

.idea

S3:

[3]:

1

# S 3:

2

3

# import contextmanager class

4

from contextlib import contextmanager

5

6

# context manager for write operations

7

@contextmanager

8

def file_writer(path, mode='w'):

9

    file_object = None

10

    try:

11

        file_object = open(path, mode=mode)

12

        yield file_object

13

    finally:

14

        if file_object:

15

            file_object.close()

16

17

18

# context manager in with statement

19

with file_writer("quiz_text_file_for_writing.txt") as file:

20

    file.write("The file is created by quiz question number 3.\n"

21

               "It is a function based context manager,\n"

22

               "Decorated with contextlib.contextmanager.")

S4:

[4]:

1

# S 4:

2

3

# import contextmanager class

4

from contextlib import contextmanager

5

import time

6

7

# define and decorate the context manager

8

@contextmanager

9

def sleeper_context_manager():

10

    start_time = time.time()

11

    print(f"Start time: {start_time}")

12

    yield

13

    end_time = time.time()

14

    print(f"End time: {end_time}")

15

    print(f"Execution time in seconds: {end_time - start_time}")

16

17

# use the context manager in a with statement

18

with sleeper_context_manager():

19

    # sleep for 2 seconds

20

    time.sleep(2)

[4]:

Start time: 1649346091.427541

End time: 1649346093.432682

Execution time in seconds: 2.005141019821167

S5:

The methods in the Context Manager Protocol are __enter__ and __exit__.

__enter__(self):

This method is called by the with statement to enter the runtime context related to current object.

The with statement will bind this method’s return value to the target specified in the as clause of the statement, if any.

__exit__(self, exc_type, exc_value, traceback):

This method is called when the execution leaves the with code block.

It exits the runtime context related to this object.

The parameters describe the exception that caused the context to be exited.

If the context was exited without an exception, all three arguments will be None.

9.      Functional Programming in Python

What is Functional Programming?

Functional Programming is a programming paradigm where computer programs are constructed by applying and composing functions. It is a declarative programming paradigm in which programs are created by sequential functions rather than statements. The main focus in Functional Programming is “what to solve” in contrast to an imperative style where the main focus is “how to solve”. Functional Programming uses expressions instead of statements.

Declarative vs. Imperative:

Declarative programming expresses the logic of a computation without describing its control flow. It is like asking your friend to fix your coffee machine. You don’t care how your friend fixes it.

Imperative programming, on the other hand, uses statements that change a program’s state. It is like giving your friend a list of steps that shows how to fix the coffee machine.

Functional Programming is declarative. So, the function definitions are trees of expressions that map values to other values, rather than a sequence of imperative statements which update the running state of the program.

In this chapter, we will learn the concepts of Functional Programming in Python. You can find the PyCharm project for this chapter in the GitHub Repository of this book.

Chapter Outline:

Core Concepts of Functional Programming

Functional Programming in Python

Higher-Order Functions

lambda

map()

filter()

reduce()

QUIZ – Functional Programming in Python

SOLUTIONS – Functional Programming in Python

Core Concepts of Functional Programming

Functional Programming has some concepts that any functional programming language needs to follow. Here are some of these concepts:

Functions are First-Class Citizens:

In functional programming, functions are treated as first-class citizens, meaning that they can be bound to names (including local identifiers), passed as arguments, and returned from other functions, just as any other data type can. This allows programs to be written in a declarative and composable style, where small functions are combined in a modular manner.

Pure Functions:

In Functional Programming functions need to be pure functions.

A pure function is a function that has the following properties:

The return values are identical for identical arguments (no variation with local static variables, non-local variables, mutable reference arguments or input streams).

The function has no side effects (no mutation of local static variables, non-local variables, mutable reference arguments or input/output streams).

Functions can be Higher-Order:

A higher-order function is a function that has at least one of the following properties:

takes one or more functions as arguments

returns a function as its result

If a function has one of these two, then we call it higher-order function and Functional Programming is mainly based on higher-order functions.

Immutability:

An immutable object is an object whose state cannot be modified after it is created. In other words, it is unchangeable. In Functional Programming you are not allowed to modify an object after it has been initialized.

Immutability supports referential integrity, which leads to function purity and reduction of side effects.

Recursion:

Recursion is the case when a function calls itself. Recursion allows us to break down a problem into smaller pieces.

In theory, there shouldn’t exist any loop structure (for or while loops) in Functional Languages. You should use recursion for iteration instead of loops. Recursion helps to remove some side effects that might occur while writing looping structures. And also, it makes your code more expressive, readable and maintainable.

There are more concepts in Functional Programming but these will suffice for the purpose of this chapter.

Functional Programming in Python

Python is not primarily a functional language. It is as a multi-paradigm programming language which can be used for both object-oriented and functional programming. However, Python has many constructs that enable a programmer to follow functional programming approach in application development.

In Python, everything is an object, functions are first-class citizens and can be higher-order. Python has built-in data structures which support immutability.

In the next sections we will explore built-in higher-order functions in Python which are very important for Functional Programming.

lambda

lambda function allows us to create function definitions in a declarative way. It is a one-line, anonymous function. It can take any number of arguments, but can only have one expression. Here is the syntax:

[1]:

1

lambda arguments : expression

Let’s say we want to split the given text into its words. Python has built-in split() function for this purpose. First, let’s see how the split() function works:

[2]:

1

text = 'Python Hands-On Advanced'

2

words = text.split()

3

print(words)

[2]:

['Python', 'Hands-On', 'Advanced']

In cell 2, we call the split() function on a string and it returns a list of the words in it. Actually, a list of items which are separated with space character.

Now let’s define a lambda function which uses the split() function:

[3]:

1

# lambda function

2

lambda x: x.split

In cell 3, we define a lambda function. Here, lambda is the keyword, x is the parameter and the code after the colon (:) is the function body. The function splits the parameter x as: x.split().

How will we call this lambda function? To be able to call a lambda expression we need to assign it to a variable. Let’s do it now:

[4]:

1

# assign lambda function to a variable

2

my_splitter_function = lambda x: x.split()

3

4

# call the function variable

5

words = my_splitter_function(text)

6

print(words)

[4]:

['Python', 'Hands-On', 'Advanced']

In cell 4, we assign our lambda function to a variable named my_splitter_function.  Since lambda is a function that takes an argument named x, so is the my_splitter_function.  In line 5, we call this function by passing some text argument. And as expected, the my_splitter_function function returns a list of words in the given text. Because behind the scenes, this is what the lambda function returns.

Let’s define a new lambda function. This time we want to define a function for multiplying two numbers. So, the lambda will have two parameters, x and y. And it will return the multiplication result of them.

[5]:

1

# multiplication with lambda function

2

multiply = lambda x, y: x * y

3

result = multiply(5, 8)

4

print(result)

[5]:

40

In cell 5, we define a lambda function and assign it to a variable called multiply. This is a function and we call it in line 3 as: result = multiply(5, 8). And we print the result in the next line.

Let’s define another lambda function. This time for the power operation. The name will be exponential and it will take two parameters:

[6]:

1

# power function with lambda

2

exponential = lambda num, pow: num**pow

3

print(exponential(2, 7))

[6]:

128

Important Note: We do not need an explicit return statement in the lambda function. Python executes lambda function body and returns the result automatically.

map()

The map() function is one of the most useful tools for Functional Programming in Python. It is a higher-order function which takes a function and an iterable as arguments. Here is the syntax:

[7]:

1

map(function, iterable)

Parameters:

function: It is the given function to which map() passes each element of the given iterable.

iterable: It is an iterable which is to be mapped.

The map() function returns an iterator after applying the given function to each item of a given iterable.

Let’s see an example on how to use it:

[8]:

1

# define a function

2

def capitalize(text):

3

    return text.upper()

4

5

# define a list

6

heroes = ['batman', 'superman', 'spiderman']

7

8

# call the map() function

9

capital_heroes = map(capitalize, heroes)

10

print(capital_heroes)

[8]:

<map object at 0x1012fecb0>

In cell 8, we define a regular function first. The function name is capitalize and it takes some string as the argument. It returns the capitalized form of this string. This is the function which we will pass to the map() function.

In line 6, we define a list of superheroes. The items are all-in lower-case letters. This is the iterable for the map() function.

In line 9, we call the map() function. It takes two arguments; the capitalize function and the heroes list. And we assign the returning value to a variable called capital_heroes. As you see in the cell output, capital_heroes is a map object (which is an iterator).

Now let’s print the items in this map object iterator. We can easily convert it to a list and print it:

[9]:

1

# print the items in map object

2

print(list(capital_heroes))

[9]:

['BATMAN', 'SUPERMAN', 'SPIDERMAN']

In the output of cell 9, you see the resulting list after we apply the map() function to the heroes list. All the letters are in capital form now. The map() function passes each item in the iterable(heroes) one by one to the given function(capitalize).

Let’s see another example. In this example we want to use a lambda function in the map() function call.

[10]:

1

# map() function with lambda

2

nums = [1, 2, 3, 4, 5]

3

4

# call the map() function

5

cubes = map(lambda x: x**3, nums)

6

7

# print the result

8

print(list(cubes))

[10]:

[1, 8, 27, 64, 125]

In cell 10, line 5, we call the map() function as: map(lambda x: x**3, nums). Here, the first parameter is a lambda function. Lambda takes in a number and returns its cube. The second parameter to the map() function is the nums list. And as you see in the output, the result is the cubes of all items in this list.

Passing more than one iterable:

It is possible to pass multiple iterables to the same map() function. Here is the syntax for this:

[11]:

1

map(function, iterable_1, iterable_2, iterable_3, ...)

The number of iterable arguments passed to the map() must match the number of arguments that function parameter expects. Because the function will act on all of the iterables at the same time. Let’s see an example:

[12]:

1

# define two list

2

list_1 = [1, 2, 3, 4, 5]

3

list_2 = ['a', 'b', 'c', 'd', 'e']

4

5

# define a concat function

6

def concat(x, y):

7

    return str(x) + str(y)

8

9

# call the map function on them

10

combined_list = map(concat, list_1, list_2)

11

print(list(combined_list))

[12]:

['1a', '2b', '3c', '4d', '5e']

In cell 12, we call the map() function with two iterables, list_1 and list_2. And the function we pass to the map is the concat function which simply returns the string concatenation of its parameters. As you see in the cell output, the final list is the concatenation of respective items from both lists. Please keep in mind that, the number of parameters of the concat function is 2, which is the same as the number of the lists we pass to the map.

filter()

The filter() function is another important tool for Functional Programming in Python. Before the definition let’s see its syntax first:

filter(function, iterable)

Parameters:

function: The function that tests each element of the iterable. It either returns True or False.

iterable: The sequence which needs to be filtered. It can be a container of any iterator type (list, tuple, set etc.).

Returns:

It returns an iterator which contains filtered elements.

The filter() function constructs an iterator from those elements of iterable for which function returns True. iterable may be either a sequence, a container which supports iteration, or an iterator. If function is None, the identity function is assumed, that is, all elements of iterable that are False are removed.

Let’s see an example to understand it better. In our example we will define a function which checks if the given number is an odd number. This function will return True if the number is an odd number, False otherwise. Then we will use this function in the filter() function. The filter() function will use it to filter the odd numbers from a list of integers.

[13]:

1

# define a function

2

def is_odd(number):

3

    """checks if number is odd"""

4

    if number % 2 == 1:

5

        return True

6

    else:

7

        return False

8

9

# define a list to filter

10

integers = [-4, -3, -2, -1, 0, 1, 2, 3, 4]

11

12

# call the filter() function

13

odd_integers = filter(is_odd, integers)

14

15

# print the filter object

16

print(odd_integers)

[13]:

<filter object at 0x103012c80>

In cell 13, we define a function named is_odd. This function returns True if the given number is an odd number. In line 7, we define a list of integer numbers. And in line 10 we call the filter() function as: filter(is_odd, integers).

The parameters of the filter() function is the is_odd (function) and the integers list (iterable). It returns a filter object as you see in the output.

We have to loop over this object or convert to a list to see its items.

[14]:

1

# print the items

2

print(list(odd_integers))

[14]:

[-3, -1, 1, 3]

In cell 14, we convert the odd_integers object to a list and print it. And you see the items in the cell output, which are the odd numbers from the integers list.

Here is how the filter() function works in Python:

1-      Each element in the integers list is passed to the is_odd() function one by one.

2-      If is_odd() returns True, that element is selected, otherwise it will be eliminated.

In the next example, let’s use a lambda function to filter some words from a given text. We want to filter the words which contain the letter ‘a’ (either in lowercase or uppercase).

[15]:

1

# some dummy text

2

text = 'Lorem ipsum dolor sit amet consectetur adipiscing elit.'

3

4

# filter() function with lambda

5

words_with_a = filter(lambda w: 'a' in w.lower(), text.split())

6

7

# print the words with a in it

8

print(list(words_with_a))

[15]:

['amet', 'adipiscing']

In cell 15, we use the filter() function with the lambda function. The lambda, takes a word (w) as an argument and returns True if this word (in lowercase) contains the letter ‘a’. The iterable to the filter() function is text.split(), which is the list of the words in the text.

reduce()

The reduce() function is the last one which we will cover in this chapter for Functional Programming in Python. It performs a rolling-computation on a given sequence object. It is defined in functools module. Here is the syntax:

functools.reduce(function, iterable[, initializer]):

Parameters:

function: the function to apply, also known as the predicate of reduce function

iterable: The sequence which needs to be reduced. It can be a container of any iterator type (list, tuple, set etc.).

initializer: the value to start with.

Notice that the arguments in square brackets [ ] are optional.

The reduce() function applies function of two arguments cumulatively to the items of iterable, from left to right, so as to reduce the iterable to a single value.

For example, reduce(lambda x, y: x+y, [1, 2, 3, 4, 5]) calculates ((((1+2)+3)+4)+5).

The left argument, x, is the accumulated value and the right argument, y, is the update value from the iterable. If the optional initializer is present, it is placed before the items of the iterable in the calculation, and serves as a default when the iterable is empty. If initializer is not given and iterable contains only one item, the first item is returned.

Let’s use the reduce() function which adds all the items in a list and returns the final result:

[16]:

1

# import reduce() function

2

from functools import reduce

3

4

# define a summation function

5

def custom_sum(n1, n2):

6

    return n1 + n2

7

8

# define a list

9

numbers = [1, 2, 3, 4, 5]

10

11

# call reduce() function

12

summation_result = reduce(custom_sum, numbers)

13

14

# print the result

15

print(summation_result)

[16]:

15

In cell 16, line 12, we call the reduce function with custom_sum and numbers arguments. The custom_sum function takes two arguments and returns the sum of them. Here is the mathematical representation of the reduce() function in this example: (((1 + 2) + 3) + 4) + 5) => 15

Here are the steps behind the scenes:

The reduce() function passes the first two items (1 and 2) of the numbers list to the custom_sum function.

The custom_sum function returns the result of 1 + 2 as 3.

The reduce() function takes this 3 and passes with the next item in the list which is 3.

The custom_sum function returns the result of 3 + 3 as 6.

The reduce() function takes this 6 and passes with the next item in the list which is 4.

The custom_sum function returns the result of 6 + 4 as 10.

The reduce() function takes this 10 and passes with the next item in the list which is 5.

The custom_sum function returns the result of 10 + 5 as 15.

Since there is no items left in the numbers list, the reduce() function returns 15 as the final result.

Let’s do another example. This time we want to find the largest number in a list. We will define a custom function that takes two parameters. It will return the largest of these numbers. Then we will use this function in the reduce() function. Let’s see:

[17]:

1

# get the largest number in a list

2

from functools import reduce

3

4

nums = [5, 3, 12, 7, 15, 64, 19, 10]

5

6

def get_max(n_1, n_2):

7

    if n_1 <= n_2:

8

        return n_2

9

    else:

10

        return n_1

11

12

largest = reduce(get_max, nums)

13

print(f"The Largest is: {largest}")

[16]:

The Largest is: 64

In cell 16, we define the get_max() function which returns the maximum of two parameters. Then in line 12, we pass this function to the reduce() function. The second argument of the reduce() function is the nums list. The reduce() function will pass the item pairs in the nums list to the get_max() function and each time it will return a new value for the next pair. Finally, it will return the largest value in the list, which is 64 in this case.

QUIZ – Functional Programming in Python

Now it’s time to solve the quiz for this chapter. You can download the quiz file, QUIZ_FunctionalProgrammingInPython.zip, from the GitHub Repository of this book. You should put the quiz file in the FunctionalProgrammingInPython project we build in this chapter.

Here are the questions for this chapter:

Q1:

Please explain the main concepts in Functional Programming paradigm:

* Functions are First-Class Citizens

* Pure Functions

* Functions can be Higher-Order

* Immutability

* Recursion

Q2:

Define a lambda function which takes one numeric parameter and checks if the given number is even or not.

Assign it to a local variable.

Finally call this variable and print the results for 17 and 18.

[2]:

1

# Q 2:

2

3

# define and assign the lambda function

4

# ---- your solution here ---

5

6

# call and print the result for 17 and 18.

7

# ---- your solution here ---

[2]:

False

True

Q3:

Define a map() function which takes a lambda expression and a list as arguments.

This map() will use the lambda function which we define in Q2.

We want it to return a map object that keeps True and False values for each item in the list.

If the item is even, the respective value should be True, otherwise it should be False.

The list is:

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

And the expected result (which should be a new list):

[False, True, False, True, False, True, False, True, False, True]

[3]:

1

# Q 3:

2

3

# define the list

4

# ---- your solution here ---

5

6

# call the map() function

7

# ---- your solution here ---

8

9

# print the resulting list

10

# ---- your solution here ---

[3]:

[False, True, False, True, False, True, False, True, False, True]

Q4:

We have a list of numbers which we want to filter the negative ones.

Here is the list:

[4, -3, 0, -12, 9, 1, 2, -7, 8]

Define a filter() function which gives us the negative numbers in this list.

It should take a lambda function and this list as the parameters.

The filter function will return a filter object.

You should convert this filter object to a list.

Finally you should sort the list in ascending order in-place.

The resulting list should be:

[-12, -7, -3]

Hints:

* lambda

* filter()

* list()

* sort()

[4]:

1

# Q 4:

2

3

# define the list

4

# ---- your solution here ---

5

6

# define the filter function

7

# ---- your solution here ---

8

9

# convert filter object to list

10

# ---- your solution here ---

11

12

# sort the list

13

# ---- your solution here ---

14

15

# print the result

16

# ---- your solution here ---

[4]:

[-12, -7, -3]

Q5:

Define a reduce() function which multiplies all the items in the given list.

The initial value will be 1000, which means it will start to multiply the first item with 1000.

Here is the list to multiply:

[1, 2, 3, 4]

The expected result is: 24000

Hints:

* functools.reduce()

* lambda

[5]:

1

# Q 5:

2

3

# import reduce function

4

# ---- your solution here ---

5

6

# define the list

7

# ---- your solution here ---

8

9

# define the reduce function

10

# ---- your solution here ---

11

12

# print the final result

13

# ---- your solution here ---

[5]:

24000

Q6:

Define a function named str_to_list.

It will take a list of strings as the parameter.

It will use the map() function to convert each string item into a list.

And it will return a list of lists.

Expected Output:

List of strings:

['ABC', '123', 'xyz']

List of lists after function call:

[['A', 'B', 'C'], ['1', '2', '3'], ['x', 'y', 'z']]

[6]:

1

# Q 6:

2

3

# define the function

4

# ---- your solution here ---

5

6

# define the list

7

txt_list = ["ABC", "123", "xyz"]

8

9

# print the string list

10

print(f"List of strings:\n{txt_list}")

11

12

# call the function

13

list_of_lists = str_to_list(txt_list)

14

15

# print list of lists

16

print(f"List of lists:\n{list_of_lists}")

[6]:

List of strings:

['ABC', '123', 'xyz']

List of lists:

[['A', 'B', 'C'], ['1', '2', '3'], ['x', 'y', 'z']]

Q7:

We have to lists as:

list_1 = [1,2,3,4,5,6,7,10]

list_2 = [0,2,1,4,7,8,9,10]

From these lists, we want to filter the same elements which are at the same indices.

We will define a function named get_common_items_at_same_indices.

It will take two lists and return a new list that contains the common elements.

Here is the expected output:

Common items at same indices: [2, 4, 10]

We will use two Python functions in our get_common_items_at_same_indices function:

1- map()

2. itertools.compress()

itertools.compress(test_list, bool_list):

compress() function filters out all the elements from the test_list based on the True values in the bool_list.

[7]:

1

# Q 7:

2

3

# import itertools.compress

4

from itertools import compress

5

# import eq from operator

6

from operator import eq

7

8

# define the function

9

# ---- your solution here ---

10

11

# define the lists

12

list_1 = [1,2,3,4,5,6,7,10]

13

list_2 = [0,2,1,4,7,8,9,10]

14

15

# print the lists

16

print(f"Lists:\n{list_1}\n{list_2}\n")

17

18

# call function and get common items

19

common_items = get_common_items_at_same_indices(list_1, list_2)

20

21

# print common items

22

print(f"Common items at same indices:\n{common_items}")

[7]:

Lists:

[1, 2, 3, 4, 5, 6, 7, 10]

[0, 2, 1, 4, 7, 8, 9, 10]

Common items at same indices:

[2, 4, 10]

SOLUTIONS – Functional Programming in Python

Here are the solutions for the quiz for this chapter.

S1:

The definitions of some core concepts in Functional Programming:

Functions are First-Class Citizens:

In functional programming, functions are treated as first-class citizens, meaning that they can be bound to names (including local identifiers), passed as arguments, and returned from other functions, just as any other data type can. This allows programs to be written in a declarative and composable style, where small functions are combined in a modular manner.

Pure Functions:

In Functional Programming functions need to be pure functions.

A pure function is a function that has the following properties:

The return values are identical for identical arguments (no variation with local static variables, non-local variables, mutable reference arguments or input streams).

The function has no side effects (no mutation of local static variables, non-local variables, mutable reference arguments or input/output streams).

Functions can be Higher-Order:

A higher-order function is a function that has at least one of the following properties:

takes one or more functions as arguments

returns a function as its result

If a function has one of these two, then we call it higher-order function and Functional Programming is mainly based on higher-order functions.

Immutability:

An immutable object is an object whose state cannot be modified after it is created. In other words, it is unchangeable. In Functional Programming you are not allowed to modify an object after it has been initialized.

Immutability supports referential integrity, which leads to function purity and reduction of side effects.

Recursion:

Recursion is the case when a function calls itself. Recursion allows us to break down a problem into smaller pieces.

In theory, there shouldn’t exist any loop structure (for or while loops) in Functional Languages. You should use recursion for iteration instead of loops. Recursion helps to remove some side effects that might occur while writing looping structures. And also, it makes your code more expressive, readable and maintainable.

S2:

[2]:

1

# S 2:

2

3

# define and assign the lambda function

4

is_even = lambda n: n % 2 == 0

5

6

# call and print the result for 17 and 18.

7

print(is_even(17))

8

print(is_even(18))

[2]:

False

True

S3:

[3]:

1

# S 3:

2

3

# define the list

4

a_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

5

6

# call the map() function

7

evens = map(lambda n: n % 2 == 0, a_list)

8

9

# print the resulting list

10

print(list(evens))

[3]:

[False, True, False, True, False, True, False, True, False, True]

S4:

[4]:

1

# S 4:

2

3

# define the list

4

all_numbers = [4, -3, 0, -12, 9, 1, 2, -7, 8]

5

6

# define the filter function

7

negative_numbers_filtered = filter(lambda x: x < 0, all_numbers)

8

9

# convert filter object to list

10

negative_numbers = list(negative_numbers_filtered)

11

12

# sort the list

13

negative_numbers.sort()

14

15

# print the result

16

print(negative_numbers)

[4]:

[-12, -7, -3]

S5:

[5]:

1

# S 5:

2

3

# import reduce function

4

from functools import reduce

5

6

# define the list

7

list_mult = [1, 2, 3, 4]

8

9

# define the reduce function

10

multiplication = reduce(lambda x, y: x * y, list_mult, 1000)

11

12

# print the final result

13

print(multiplication)

[5]:

24000

S6:

[6]:

1

# S 6:

2

3

# define the function

4

def str_to_list(str_list):

5

    result = map(list, str_list)

6

    return list(result)

7

8

# define the list

9

txt_list = ["ABC", "123", "xyz"]

10

11

# print the string list

12

print(f"List of strings:\n{txt_list}")

13

14

# call the function

15

list_of_lists = str_to_list(txt_list)

16

17

# print list of lists

18

print(f"List of lists:\n{list_of_lists}")

[6]:

List of strings:

['ABC', '123', 'xyz']

List of lists:

[['A', 'B', 'C'], ['1', '2', '3'], ['x', 'y', 'z']]

S7:

[7]:

1

# S 7:

2

3

# import itertools.compress

4

from itertools import compress

5

# import eq from operator

6

from operator import eq

7

8

# define the function

9

def get_common_items_at_same_indices(list1, list2):

10

    result = map(eq, list1, list2)

11

    commons = compress(list1, result)

12

    return list(commons)

13

14

# define the lists

15

list_1 = [1,2,3,4,5,6,7,10]

16

list_2 = [0,2,1,4,7,8,9,10]

17

18

# print the lists

19

print(f"Lists:\n{list_1}\n{list_2}\n")

20

21

# call function and get common items

22

common_items = get_common_items_at_same_indices(list_1, list_2)

23

24

# print common items

25

print(f"Common items at same indices:\n{common_items}")

[7]:

Lists:

[1, 2, 3, 4, 5, 6, 7, 10]

[0, 2, 1, 4, 7, 8, 9, 10]

Common items at same indices:

[2, 4, 10]

10.        Project 1 – Sending Emails with Python

Project Setup

In this project, we will learn how to send emails using Python. You can find the PyCharm project for this chapter in the GitHub Repository of this book.

Chapter Outline:

What is SMTP?

The smtplib Module

Setting Up a Local SMTP Server

Configure a Fake Email Server

Sending HTML Emails

Sending Emails with Attachments

Sending Emails with Images

Sending Emails via Gmail

What is SMTP?

Simple Mail Transfer Protocol (SMTP) is a communication protocol for electronic mail transmission. It is a universal standard, which was first defined in 1982 by RFC 821, and updated in 2008 by RFC 5321 to Extended SMTP additions. Mail servers and other message transfer agents use SMTP to send and receive mail messages.

The smtplib Module

In Python, email sending operations are mainly done by the help of the built-in smtplib module. The smtplib module defines an SMTP client session object that can be used to send mail to any internet machine with an SMTP or ESMTP listener daemon.

We also have a separate package called email in Python. The email package is a library for managing email messages. It is specifically not designed to do any sending of email messages to SMTP (RFC 2821), NNTP, or other servers; those are functions of the smtplib module. The email package attempts to be as RFC-compliant as possible, supporting RFC 5322 and RFC 6532 (Internet standards for SMTP).

Setting Up a Local SMTP Server

To be able to work in our local environment to send emails, we need to set up the local SMTP debugging server. Python offers the smtpd module for this purpose. It has a DebuggingServer class, which will discard messages you are sending out and will print them to the console.

class smtpd.DebuggingServer(localaddr, remoteaddr):

Creates a new debugging server. Arguments are as per SMTPServer. Messages will be discarded, and printed on stdout.

Now, let us set a local SMTP server on localhost:1025. PyCharm has a built-in terminal in it. So, we will use this feature to run Python commands without leaving PyCharm. The command below will set an SMTP server on port 1025. Either use “python” or “python3” at start depending on your operating system.

python -m smtpd -n -c DebuggingServer localhost:1025

 or

python3 -m smtpd -n -c DebuggingServer localhost:1025

If you run this command successfully, now you have a local SMTP server listening on port 1025. Let’s test it:

LocalSMTPServer.py file:

[1]:

1

# import modules and packages

2

import smtplib

3

from email.message import EmailMessage

4

5

# define email attributes

6

subject = "Test Email Subject"

7

sender = "sender@example.com"

8

receiver = "receiver@example.com"

9

10

# define email object

11

msg = EmailMessage()

12

13

# set headers

14

msg['Subject'] = subject

15

msg['From'] = sender

16

msg['To'] = receiver

17

18

# set body text

19

msg.set_content("This is test email body on local server...")

20

21

# define port

22

port = 1025

23

24

# call smtplib.SMTP in the with context manager

25

with smtplib.SMTP('localhost', port) as server:

26

    # send message

27

    server.send_message(msg)

28

    # print result to run console

29

    print("Test email sent successfully!")

[1]:

---------- MESSAGE FOLLOWS ----------

b'Subject: Test Email Subject'

b'From: sender@example.com'

b'To: receiver@example.com'

b'Content-Type: text/plain; charset="utf-8"'

b'Content-Transfer-Encoding: 7bit'

b'MIME-Version: 1.0'

b'X-Peer: ::1'

b''

b'This is test email body on local server...'

------------ END MESSAGE ------------

In cell 1, we send a simple email using the smtplib module, email package and a local development server. Before getting into the details of this code, here is an image of PyCharm showing the result in the terminal window:

Figure 10-1: Email output on local SMTP Server

In cell 1, we import the necessary smtplib module and the EmailMessage class from the email package.

In lines 6 to 8, we define the subject, sender and receiver for our email.

In line 11, we instantiate an EmailMessage object named msg. This object will contain the email header and the body.

In lines 14 to 16 we set some attributes of the msg object, which are ‘Subject’, ‘From’ and ‘To’.

In line 19 we set the content of the email as: msg.set_content("This is test email body on local server..."). The set_content() method is responsible for creating the email body. Here, we just pass a simple text.

In line 22, we set the port for our local server, which is 1025 in our case.

In line 25, we have a with context manager. In which we call the SMTP class constructor as: smtplib.SMTP('localhost', port). We pass two parameters: localhost and port. The constructor returns an active local server listening on the specified port.

In line 27, inside the with context manager scope, we call the send_message() on our local server as: server.send_message(msg).

And as you see in the Terminal output, our email is sent successfully on our local server.

Configure a Fake Email Server

An email server is an application which is used to send and receive emails, by using the SMTP (for Outgoing Messages) and POP3 (for Incoming Messages). These servers are provided globally by email providers.

Here are some public mail servers:

Google mail - Gmail

Server

Authentication

Port

SMTP Server (Outgoing Messages)

smtp.gmail.com

SSL

465

smtp.gmail.com

StartTLS

587

POP3 Server (Incoming Messages)

pop.gmail.com

SSL

995

Outlook.com

Server:

Authentication:

Port:

SMTP Server (Outgoing Messages)

smtp.live.com

StartTLS

587

POP3 Server (Incoming Messages)

pop3.live.com

SSL

995

Yahoo Mail

Server:

Authentication:

Port:

SMTP Server (Outgoing Messages)

smtp.mail.yahoo.com

SSL

465

POP3 Server (Incoming Messages)

pop.mail.yahoo.com

SSL

995

To set up a real mail server, there are some security settings which you have to change in your account to make them work. We will see these settings for a Gmail account later in this chapter. For the sake of simplicity, in this project, we will mainly work with a fake SMTP server.

Fake SMTP Server:

Fake SMTP server imitates the work of a real 3rd party web server. In this project, we will use Mailtrap. Mailtrap is a platform for safe and quick email testing using SMTP. Beyond testing email sending, it will let us check how the email will be rendered and displayed, review the message raw data as well as will provide us with a spam report. Mailtrap is very easy to set up: all you need is just to copy the credentials generated by the app and paste them into your code.

Mailtrap offers a free plan which enables you to send 500 mails per month with no cost. You can use your Google, GitHub of Office 365 accounts to set up Mailtrap easily. Or you can sign up with your email if you don’t have any of these accounts. Once you signed up, you can see the necessary credentials on your settings page.

To copy the credentials, navigate to My Inbox. And under the SMTP Settings tab, you will see a dropdown as Integrations. Select smtplib under Python and you will see all the information you need to send emails to Mailtrap. See the image below, showing my settings:

Figure 10-2: Login Credentials and sample email on Mailtrap

Once you set up your account on Mailtrap and copy the smtplib integration details, it is very easy to use it in PyCharm project. Create a new Python file named Mailtrap.py and paste the code you copied. Here it is:

Mailtrap.py file:

[2]:

1

# send email to Mailtrap fake server

2

3

import smtplib

4

5

sender = "Private Person <from@example.com>"

6

receiver = "A Test User <to@example.com>"

7

8

message = f"""\

9

Subject: Hi from Hands-On Python Advanced

10

To: {receiver}

11

From: {sender}

12

13

# This is a test e-mail message from PyCharm,

14

# developed with smtplib with Mailtrap server.

15

Python is really awesome :))

16

"""

17

18

with smtplib.SMTP("smtp.mailtrap.io", 2525) as server:

19

    try:

20

        server.login("<your user_id>", "<your password>")

21

        server.sendmail(sender, receiver, message)

22

    except Exception as ex:

23

        print(ex)

24

    else:

25

        print("Message sent successfully to Mailtrap.")

[2]:

Message sent successfully to Mailtrap.

In cell 2, we paste the code we copied from the Mailtrap SMTP Settings. All we add is a simple try-except-else block in the with context manager. You have to replace your user_id and password in line 20.

Here is what they are:

user_id: your login generated by Mailtrap

password: your password generated by Mailtrap

When we run this code, a simple text email is sent successfully to the Mailtrap server. See the image below:

Figure 10-3: A text email sent to the Mailtrap server

Sending HTML Emails

In the previous section we sent simple text emails. Which is not what you need in real world. In an actual email, you need to add some formatting, links, or images. The common way of doing this is to put all of them into an HTML structure. Python’s built-in email package is the tool for any kind of email formatting.

MIME: Multipurpose Internet Mail Extensions (MIME) is an Internet standard that extends the format of email messages to support text in character sets other than ASCII, as well as attachments of audio, video, images, and application programs. It is the way we combine HTML and plain text in emails. In Python, we have email.mime module for handling MIME-specific operations.

The common approach is to write two separate versions of the same email. A plain text version and an HTML version. Then we merge them with an instance of MIMEMultipart class. In that way, our message has two rendering options. In case the HTML part isn’t be rendered successfully for some reason, a text version will still be available. You thank think of the text version as a fallback option.

SendingHTMLEmail.py file:

[3]:

1

# Sending HTML Email

2

3

# import smtplib and email.mime classes

4

import smtplib

5

from email.mime.text import MIMEText

6

from email.mime.multipart import MIMEMultipart

7

8

# define the port

9

port = 2525

10

11

# SMTP server

12

smtp_server = "smtp.mailtrap.io"

13

14

# login credentials

15

# paste your login generated by Mailtrap

16

login = "<your user_id>"

17

# paste your password generated by Mailtrap

18

password = "<your password>"

19

20

# sender and receiver emails

21

sender = "sender@example.com"

22

receiver = "receiver@example.com"

23

24

# message object

25

message = MIMEMultipart("alternative")

26

message["Subject"] = "HTML Mail Test with multipart"

27

message["From"] = sender

28

message["To"] = receiver

29

30

# --- PLAIN TEXT PART ---

31

with open("files/plain_text_part.txt") as plain_text_part:

32

    text = plain_text_part.read()

33

34

# --- HTML PART ---

35

with open("files/html_part.txt") as html_part:

36

    html = html_part.read()

37

38

# convert both parts to MIMEText objects

39

part1 = MIMEText(text, "plain")

40

part2 = MIMEText(html, "html")

41

42

# add the parts to the MIMEMultipart message

43

message.attach(part1)

44

message.attach(part2)

45

46

# send your email

47

with smtplib.SMTP(smtp_server, port) as server:

48

    try:

49

        server.login(login, password)

50

        server.sendmail(sender, receiver, message.as_string())

51

    except Exception as ex:

52

        print(ex)

53

    else:

54

        print("Multipart Message sent successfully...")

[3]:

Multipart Message sent successfully...

For plain text and HTML options, we create two separate .txt files under the “files/” folder in our PyCharm project. Their names are “plain_text_part.txt” and “html_part.txt”. We read their contents and assign them to local variables in lines 31 and 35.

In lines 39 and 40, we call the MIMEText class to pass these two parts; “plain” and “html”.

In lines 43 and 44, we attach the parts to the MIMEMultipart message.

In line 47, we set a with context manager to login and send the message to our SMTP server.

Here is the mail which we send with this code:

Figure 10-4: An HTML email sent to the Mailtrap server

Here are the contents of “plain_text_part.txt” and “html_part.txt” files:

plain_text_part.txt:

Hello Developer,

We are glad to see you here.

https://www.amazon.com/dp/B09WVB49W8

Hands-On Python Advanced is great for learning Python in-depth.

Feel free to contact us if you need any help!

Musa Arda

html_part.txt:

<html>

  <body>

    <p>Hello Developer,<br>

       We are glad to see you here.</p>

    <p>

        <a href="https://www.amazon.com/dp/B09WVB49W8">

            Hands-On Python Advanced

        </a>

        is great for learning Python in-depth.

    </p>

    <br>

    <p>Feel free to <strong>contact us</strong> if you need any help!</p>

    <br>

    <p><b>Musa Arda</b></p>

  </body>

</html>

Sending Emails with Attachments

Attachments are MIME objects but they need to be encoded by using the base64 module before sending. You can attach text files, images, audio files, and even applications in Python. Every object type has its own email class. For example audio files are email.mime.audio.MIMEAudio or images are email.mime.image.MIMEImage.

We will use a PDF attachment for the next example. The PDF file which we will send is the Keymap Reference of PyCharm IDE. To view the keymap reference as PDF, select Help | Keyboard Shortcuts PDF from the main menu in PyCharm. You can also find the Mac version under the “files” folder in the current PyCharm project. File name is ReferenceCardForMac.pdf.

Here is the complete code:

SedingAttachments.py file:

[4]:

1

# Sending Attachments with Email

2

# import modules and classes

3

import smtplib

4

from email import encoders

5

from email.mime.base import MIMEBase

6

from email.mime.multipart import MIMEMultipart

7

from email.mime.text import MIMEText

8

9

# server and port

10

smtp_server = "smtp.mailtrap.io"

11

port = 2525

12

13

# your login credentials generated by Mailtrap

14

login = "<your user_id>"

15

password = "<your password>"

16

17

# sender and receiver emails

18

sender = "sender@example.com"

19

receiver = "receiver@example.com"

20

21

# message object

22

message = MIMEMultipart()

23

message["Subject"] = "PyCharm Keymap Reference"

24

message["From"] = sender

25

message["To"] = receiver

26

27

# email body

28

email_body = "You may find the Keymap Reference of PyCharm IDE in the attachments."

29

message.attach(MIMEText(email_body, "plain"))

30

31

# file info

32

filepath = "files/ReferenceCardForMac.pdf"

33

filename = "ReferenceCardForMac.pdf"

34

35

# open the PDF file in read-binary mode

36

with open(filepath, "rb") as attachment:

37

    # content type "application/octet-stream" means

38

    # a MIME attachment is a binary file

39

    part = MIMEBase("application", "octet-stream")

40

    part.set_payload(attachment.read())

41

42

# Encoding => to base64

43

encoders.encode_base64(part)

44

45

# Add mail header

46

part.add_header(

47

    "Content-Disposition",

48

    f"attachment; filename= {filename}",

49

)

50

51

# Add attachment to the message and convert it to string

52

message.attach(part)

53

text = message.as_string()

54

55

# send the email with attachment

56

with smtplib.SMTP(smtp_server, port) as server:

57

    try:

58

        server.login(login, password)

59

        server.sendmail(sender, receiver, text)

60

    except Exception as ex:

61

        print(ex)

62

    else:

63

        print("Mail with attachment sent successfully...")

[4]:

Mail with attachment sent successfully...

In cell 4, line 39, we set the application type as "octet-stream" which means binary files, including PDF. Then in line 40, we set the payload by reading the file in the specified path.

In line 43, we encode the file content to base64 with the help of the encoders module in the email package.

In lines 46 to 48, we set the header.

In line 52, we attach the encoded attachment to the message object as: message.attach(part). Then we convert the message object to string as: text = message.as_string().

And finally, inside the with context manager we login to the server and send the email. Here is the mail output after we run this code:

Figure 10-5: An email sent with attachment to the Mailtrap server

Sending Emails with Images

It is very easy to send emails with images in it. All we need is some HTML document with some image content. Keep in mind that, we need to encode the images to base64. Here is a simple example:

MailsWithImages.py file:

[5]:

1

# Sending Emails with Images

2

# import the necessary components first

3

import smtplib

4

from email.mime.text import MIMEText

5

from email.mime.multipart import MIMEMultipart

6

import base64

7

8

# server and port

9

smtp_server = "smtp.mailtrap.io"

10

port = 2525

11

12

# your login credentials generated by Mailtrap

13

login = "<your user_id>"

14

password = "<your password>"

15

16

# sender and receiver emails

17

sender = "sender@example.com"

18

receiver = "receiver@example.com"

19

20

# message object

21

message = MIMEMultipart("alternative")

22

message["Subject"] = "Python Logo in Email"

23

message["From"] = sender

24

message["To"] = receiver

25

26

# read and encode the image file

27

encoded_image = base64.b64encode(open("images/python_logo.jpeg", "rb").read()).decode()

28

29

html = f"""\

30

<html>

31

<body>

32

    <h1>This is an email with image in it</h1> <br>

33

   <img src="data:image/jpg;base64,{encoded_image}">

34

</body>

35

</html>

36

"""

37

38

# set MIME type and attach it to the message

39

part = MIMEText(html, "html")

40

message.attach(part)

41

42

# send the email

43

with smtplib.SMTP(smtp_server, port) as server:

44

    try:

45

        server.login(login, password)

46

        server.sendmail(sender, receiver, message.as_string())

47

    except Exception as ex:

48

        print(ex)

49

    else:

50

        print("Email with Image sent successfully...")

[5]:

Email with Image sent successfully…

And here is the email on Mailtrap server after we run the code in cell 5:

Figure 10-6: An email sent with image in it to the Mailtrap server

Sending Emails via Gmail

In this section, we will see how we can send email with attachments via Gmail. We choose Gmail, because it is one the most popular mail servers and the process is very similar to any other commercial server.

To be able to send emails via your Gmail account, there are some settings you need to do on Gmail side. First you need to disable 2-Step Verification (which is not recommend). Second you need to allow less secure apps (which is not recommend either).

Figure 10-7: Allowing less secure apps on Gmail Account

The best practice is to set up OAuth2 authorization protocol for Gmail API. OAuth2 is more difficult but recommended due to obvious security reasons.

If you have finished the necessary security settings, then you are ready to use Gmail server for sending emails with Python. In a test account, I allowed less secure apps for the sake of simplicity in this project.

Here is some basic information which you should know about Gmail:

server name: smtp.gmail.com

port: 465 for SSL/TLS connection (preferred)

or port: 587 for STARTTLS connection

username: your Gmail email address

password: your password

Let’s send a simple email using our Gmail account:

SendViaGmail.py file:

[6]:

1

# Sending Emails via Gmail

2

# import libraries

3

import smtplib

4

from email.mime.multipart import MIMEMultipart

5

from email.mime.text import MIMEText

6

7

# server and port

8

smtp_server = "smtp.gmail.com"

9

port = 587  # for STARTTLS connection

10

11

content = '''Hi Gmail,

12

This is a simple email created with Python.

13

Sending emails with Python is very fun :)

14

15

Best Regards,

16

Musa Arda'''

17

18

# mail addresses and password

19

sender = '<your_gmail_address>@gmail.com'

20

password = '<your_gmail_password>'

21

receiver = 'receiver@example.com'

22

23

# set MIME data

24

message = MIMEMultipart()

25

message['From'] = sender

26

message['To'] = receiver

27

message['Subject'] = 'Email sent from Python to Gmail.'

28

29

# set the body

30

message.attach(MIMEText(content, 'plain'))

31

32

# create SMTP session

33

session = smtplib.SMTP('smtp.gmail.com', port)

34

35

# enable security

36

session.starttls()

37

38

# login with credentials

39

session.login(sender, password)

40

41

# convert message to string before sending

42

text = message.as_string()

43

44

# send the email

45

session.sendmail(sender, receiver, text)

46

47

# close the session

48

session.quit()

In cell 6, we send an email using Gmail server.

In line 33, we start an SMTP session.

And in line 36, we enabled security by calling the starttls() method. starttls() puts the SMTP connection in TLS (Transport Layer Security) mode. All SMTP commands that follow will be encrypted.

In line 39, we login on the current session as: session.login(sender, password).

In line 45, we send the email and in line 48, we quit the session.

Here is the email in Gmail account inbox after running this code:

Figure 10-8: An email sent via Gmail server

This finalizes our project on sending emails in Python. In the next chapter, you will have an assignment related to project. Good luck with your assignment.

11.        Assignment 1 – Sending Emails with Python

Assignment Setup

We finished our first project in this book which is Project 1 - Sending Emails with Python. Now it’s your turn to recreate the same project. This chapter is the assignment on the Project 1.

In this assignment you will start setting up a local development server to send emails. Then you will configure a fake email server to be able to send emails to a remote server. Then you will send html emails, emails with attachments and images. Finally, you will send emails using a real email server like Gmail.

To complete the assignment, follow the instructions where you see a #TODO statement. You can find the solutions in the previous chapter.

We will create a new PyCharm project. The project name is Assignment_1. You should download the project from the GitHub Repository of this book before starting to code.

Chapter Outline:

What is SMTP?

The smtplib Module

Setting Up a Local SMTP Server

Configure a Fake Email Server

Sending HTML Emails

Sending Emails with Attachments

Sending Emails with Images

Sending Emails via Gmail

What is SMTP?

Simple Mail Transfer Protocol (SMTP) is a communication protocol for electronic mail transmission. It is a universal standard, which was first defined in 1982 by RFC 821, and updated in 2008 by RFC 5321 to Extended SMTP additions. Mail servers and other message transfer agents use SMTP to send and receive mail messages.

The smtplib Module

In Python, email sending operations are mainly done by the help of the built-in smtplib module. The smtplib module defines an SMTP client session object that can be used to send mail to any internet machine with an SMTP or ESMTP listener daemon.

We also have a separate package called email in Python. The email package is a library for managing email messages. It is specifically not designed to do any sending of email messages to SMTP (RFC 2821), NNTP, or other servers; those are functions of the smtplib module. The email package attempts to be as RFC-compliant as possible, supporting RFC 5322 and RFC 6532 (Internet standards for SMTP).

Setting Up a Local SMTP Server

To be able to work in our local environment to send emails, we need to set up the local SMTP debugging server. Python offers the smtpd module for this purpose. It has a DebuggingServer class, which will discard messages you are sending out and will print them to the console.

class smtpd.DebuggingServer(localaddr, remoteaddr):

Creates a new debugging server. Arguments are as per SMTPServer. Messages will be discarded, and printed on stdout.

Now, let us set a local SMTP server on localhost:1025. PyCharm has a built-in terminal in it. So, we will use this feature to run Python commands. The command below will set an SMTP server on port 1025. Either use “python” or “python3” at start depending on your operating system.

python -m smtpd -n -c DebuggingServer localhost:1025

 or

python3 -m smtpd -n -c DebuggingServer localhost:1025

If you run this command successfully, now you have a local SMTP server listening on port 1025. Let’s test it.

LocalSMTPServer.py file:

[1]:

1

# import modules and packages

2

# TODO - import smtplib module

3

# TODO - import EmailMessage class from email.message

4

5

# define email attributes

6

# TODO - define subject variable

7

# TODO - define sender variable

8

# TODO - define receiver variable

9

10

# define email object

11

# TODO - instantiate an EmailMessage object

12

13

# set headers

14

# TODO - set headers for EmailMessage object

15

# - Subject

16

# - From

17

# - receiver

18

19

# set body text

20

# TODO - set content for EmailMessage object

21

22

# define port

23

# TODO - define the port as 1025

24

25

# call smtplib.SMTP in the with context manager

26

# TODO - set a with context manager for local server

27

#      by calling the related class in the smtplib module

28

# TODO - send the message

29

# TODO - print the final text for success

[1]:

---------- MESSAGE FOLLOWS ----------

b'Subject: Test Email Subject'

b'From: sender@example.com'

b'To: receiver@example.com'

b'Content-Type: text/plain; charset="utf-8"'

b'Content-Transfer-Encoding: 7bit'

b'MIME-Version: 1.0'

b'X-Peer: ::1'

b''

b'This is test email body on local server...'

------------ END MESSAGE ------------

In cell 1, we send a simple email using the smtplib module, email package and a local development server. Before getting into the details of this code, here is an image of PyCharm showing the result in the terminal window:

Figure 11-1: Email output on local SMTP Server

In cell 1, we import the necessary smtplib module and the EmailMessage class from the email package.

In lines 6 to 8, we define the subject, sender and receiver for our email.

In line 11, we instantiate an EmailMessage object named msg. This object will contain the email header and body.

In lines 14 to 16 we set some attributes of the msg object, which are ‘Subject’, ‘From’ and ‘To’.

In line 19 we set the content of the email as: msg.set_content("This is test email body on local server..."). The set_content() method is responsible for setting the email body. Here, we just pass a simple text.

In line 22, we set the port for our local server, which is 1025 in our case.

In line 25, we have a with context manager. In which we call the SMTP class constructor as: smtplib.SMTP('localhost', port). We pass two parameters: localhost and port. The constructor returns an active local server listening on the specified port.

In line 27, inside the with context manager scope, we call the send_message() on our local server as: server.send_message(msg).

And as you see in the Terminal output, our email is sent successfully on the local server.

Configure a Fake Email Server

An email server is an application which used to send and receive emails, by using the SMTP (for Outgoing Messages) and POP3 (for Incoming Messages). These servers are provided globally by email providers. For example, here are some public mail servers:

Google mail - Gmail

Server

Authentication

Port

SMTP Server (Outgoing Messages)

smtp.gmail.com

SSL

465

smtp.gmail.com

StartTLS

587

POP3 Server (Incoming Messages)

pop.gmail.com

SSL

995

Outlook.com

Server:

Authentication:

Port:

SMTP Server (Outgoing Messages)

smtp.live.com

StartTLS

587

POP3 Server (Incoming Messages)

pop3.live.com

SSL

995

Yahoo Mail

Server:

Authentication:

Port:

SMTP Server (Outgoing Messages)

smtp.mail.yahoo.com

SSL

465

POP3 Server (Incoming Messages)

pop.mail.yahoo.com

SSL

995

To set up a real mail server, there are some security settings which you have to change in your account to make them work. We will see these settings for a Gmail account later in this chapter. For the sake of simplicity, in this project, we will mainly work with a fake SMTP server.

Fake SMTP Server:

Fake SMTP server imitates the work of a real 3rd party web server. In this project, we will use Mailtrap. Mailtrap is a platform for safe and quick email testing using SMTP. Beyond testing email sending, it will let us check how the email will be rendered and displayed, review the message raw data as well as will provide us with a spam report. Mailtrap is very easy to set up: all you need is just to copy the credentials generated by the app and paste them into your code.

Mailtrap offers a free plan which enables you to send 500 mails per month with no cost. You can use your Google, GitHub of Office 365 accounts to set up Mailtrap easily. Or you can sign up with your email if you don’t have any of these accounts. Once you signed up, you can see the necessary credentials on your settings page.

To copy the credentials, navigate to My Inbox. And under the SMTP Settings tab, you will see a dropdown as Integrations. Select smtplib under Python and you will see all the information you need to send emails to Mailtrap. See the image below, showing my settings:

Figure 11-2: Login Credentials and sample email on Mailtrap

Once you set up your account on Mailtrap and copy the smtplib integration details, it is very easy to use it in PyCharm project. Create a new Python file named Mailtrap.py and paste the code you copied. Here it is:

Mailtrap.py file:

[2]:

1

# send email to Mailtrap fake server

2

3

# TODO - import smtplib module

4

5

# TODO - define sender and receiver variables

6

7

# TODO - set the message object with an f-string

8

9

# TODO - set a with context manager to call the SMTP class

10

#        with mailtrap server and port

11

# TODO - write a try-except-else block,

12

#        login to the server and send the email

13

#        print a success message in the end

[2]:

Message sent successfully to Mailtrap.

In cell 2, we paste the code we copied from the Mailtrap SMTP Settings. All we add is a simple try-except-else block in the with context manager. You have to replace your user_id and password in line 20.

Here is what they are:

user_id: your login generated by Mailtrap

password: your password generated by Mailtrap

When we run this code, a simple text email is sent successfully to the Mailtrap server. See the image below:

Figure 11-3: A text email sent to the Mailtrap server

Sending HTML Emails

In the previous section we sent simple text emails. Which is not what you need in real world. In an actual email, you need to add some formatting, links, or images. The common way of doing this is to put all of them into an HTML structure. Python’s built-in email package is the tool for any kind of email formatting.

MIME: Multipurpose Internet Mail Extensions (MIME) is an Internet standard that extends the format of email messages to support text in character sets other than ASCII, as well as attachments of audio, video, images, and application programs. It is the way we combine HTML and plain text in emails. In Python, we have email.mime module for handling MIME-specific operations.

The common approach is to write two separate versions of the same email. A text version and an HTML version. Then we merge them with an instance of MIMEMultipart class. In that way, our message has two rendering options. In case the HTML part isn’t be rendered successfully for some reason, a text version will still be available. You thank think of the text version as a fallback option.

SendingHTMLEmail.py file:

[3]:

1

# Sending HTML Email

2

3

# TODO - import smtplib and email.mime classes

4

#        MIMEText and MIMEMultipart

5

6

# TODO - define the port on 2525

7

8

# TODO - define SMTP server for mailtrap

9

10

# TODO - set login credentials

11

12

# TODO - set sender and receiver emails

13

14

# TODO - create a message object from MIMEMultipart

15

16

# TODO - set Subject, From, To attributes of message object

17

18

# --- PLAIN TEXT PART ---

19

# TODO - set a with context manager to read "files/plain_text_part.txt"

20

#        assign the content to a variable called text

21

22

# --- HTML PART ---

23

# TODO - set a with context manager to read "files/html_part.txt"

24

#        assign the content to a variable called html

25

26

# TODO - convert both parts to MIMEText objects

27

#       "plain" and "html"

28

29

# TODO - add the parts to the MIMEMultipart message

30

31

# TODO - set a with context manager to call the SMTP class

32

#        with mailtrap server and port

33

# TODO - write a try-except-else block,

34

#        login to the server and send the email

35

#        print a success message in the end

[3]:

Multipart Message sent successfully...

For plain text and html options, we create two separate txt files under the “files/” folder in our PyCharm project. Their names are “plain_text_part.txt” and “html_part.txt”. We read their contents and assign them to local variables in lines 31 and 35.

In lines 39 and 40, we call the MIMEText class to pass these two parts; “plain” and “html”.

In lines 43 and 44, we attach the parts to the MIMEMultipart message.

In line 47, we set a with context manager to login and send the message to our SMTP server.

Here is the mail which we send with this code:

Figure 11-4: An HTML email sent to the Mailtrap server

Here are the contents of “plain_text_part.txt” and “html_part.txt” files:

plain_text_part.txt:

Hello Developer,

We are glad to see you here.

https://www.amazon.com/dp/B09WVB49W8

Hands-On Python Advanced is great for learning Python in-depth.

Feel free to contact us if you need any help!

Musa Arda

html_part.txt:

<html>

  <body>

    <p>Hello Developer,<br>

       We are glad to see you here.</p>

    <p>

        <a href="https://www.amazon.com/dp/B09WVB49W8">

            Hands-On Python Advanced

        </a>

        is great for learning Python in-depth.

    </p>

    <br>

    <p>Feel free to <strong>contact us</strong> if you need any help!</p>

    <br>

    <p><b>Musa Arda</b></p>

  </body>

</html>

Sending Emails with Attachments

Attachments are MIME objects but they need to be encoded by using the base64 module before sending. You can attach text files, images, audio files, and even applications in Python. Every object type has its own email class. For example audio files are email.mime.audio.MIMEAudio or images are email.mime.image.MIMEImage.

We will use a PDF attachment for the next example. The PDF file which we will send is the Keymap Reference of PyCharm IDE. To view the keymap reference as PDF, select Help | Keyboard Shortcuts PDF from the main menu in PyCharm. You can also find the Mac version under the “files” folder in the current PyCharm project. File name is ReferenceCardForMac.pdf.

Here is the complete code:

SedingAttachments.py file:

[4]:

1

# Sending Attachments with Email

2

# TODO - import modules and classes

3

#       - smtplib, encoders, MIMEBase, MIMEMultipart, MIMEText

4

5

# TODO - set server and port

6

7

# TODO - set login credentials by Mailtrap

8

9

# TODO - set sender and receiver emails

10

11

# TODO - create a message object with MIMEMultipart

12

13

# TODO - set Subject, From, To attributes of message object

14

15

# TODO - set email body

16

17

# TODO - attach email body to the message

18

19

# TODO - set file info (file path and file name)

20

21

# TODO - open the PDF file in read-binary mode

22

# TODO - set application of the MIMEBase class

23

# TODO - set payload to attachment content

24

25

# TODO - encode the message part to base64

26

27

# TODO - Add mail header

28

#       - Content-Disposition,

29

#       - attachment; filename=

30

31

# TODO - Add attachment to the message

32

33

# TODO - convert message it to string

34

35

# TODO - set a with context manager to call the SMTP class

36

#        with mailtrap server and port

37

# TODO - write a try-except-else block,

38

#        login to the server and send the email

39

#        print a success message in the end

[4]:

Mail with attachment sent successfully...

In cell 4, line 39, we read set the application type as "octet-stream" which means binary files, including PDF. Then in line 40, we set the payload by reading the file in the specified path.

In line 43, we encode the file content to base64 with the help of the encoders module in the email package.

In lines 46 to 48, we set the header.

In line 52, we attach the encoded attachment to the message object as: message.attach(part). Then we convert the message object to string as: text = message.as_string().

And finally, inside the with context manager we login to the server and send the email. Here is the mail output after we run this code:

Figure 11-5: An email sent with attachment to the Mailtrap server

Sending Emails with Images

It is very easy to send emails with images in it. All we need is some HTML document with some image content. Keep in mind that, we need to encode the images to base64. Here is a simple example:

MailsWithImages.py file:

[5]:

1

# Sending Emails with Images

2

# TODO - import the necessary components first

3

#       - smtplib module

4

#       - MIMEText, MIMEMultipart

5

#       - base64

6

7

# TODO - set server and port

8

9

# TODO - set login credentials by Mailtrap

10

11

# TODO - set sender and receiver emails

12

13

# TODO - create a message object with MIMEMultipart

14

15

# TODO - set Subject, From, To attributes of message object

16

17

# TODO - read and encode the image file at "images/python_logo.jpeg"

18

19

# TODO - define html content with an f-string

20

#       - having an img tag in it with src as your image

21

22

# TODO - create a MIMEText part with MIME type as html

23

24

# TODO - attach the part to the message

25

26

# TODO - set a with context manager to call the SMTP class

27

#        with mailtrap server and port

28

# TODO - write a try-except-else block,

29

#        login to the server and send the email

30

#        print a success message in the end

[5]:

Email with Image sent successfully…

And here is the email on Mailtrap server after we run the code in cell 5:

Figure 11-6: An email sent with image in it to the Mailtrap server

Sending Emails via Gmail

In this section, we will see how we can send email with attachments via Gmail. We choose Gmail, because it is one the most popular mail servers and the process is very similar to any other commercial server.

To be able to send emails via your Gmail account, there are some settings you need to do on Gmail side. First you need to disable 2-Step Verification (which is not recommend). Second you need to allow less secure apps (which is not recommend either).

Figure 11-7: Allowing less secure apps on Gmail Account

The best practice is to set up OAuth2 authorization protocol for Gmail API. OAuth2 is more difficult but recommended due to obvious security reasons.

If you finished the necessary security settings, then you are ready to use Gmail server for sending emails with Python. In a test account, I allowed less secure apps for the sake of simplicity in this project.

Here is some basic information which you should know about Gmail:

server name: smtp.gmail.com

port: 465 for SSL/TLS connection (preferred)

or port: 587 for STARTTLS connection

username: your Gmail email address

password: your password

Let’s send a simple email using our Gmail account:

SendViaGmail.py file:

[6]:

1

# Sending Emails via Gmail

2

# TODO - import modules and classes

3

#       - smtplib module

4

#       - MIMEMultipart, MIMEText

5

6

# TODO - set server and port

7

#       - "smtp.gmail.com"

8

#       - 587 (for STARTTLS connection)

9

10

# TODO - set some content

11

12

# TODO - set mail addresses and password

13

#       - sender, password, receiver

14

15

# TODO - create a message object with MIMEMultipart

16

17

# TODO - set Subject, From, To attributes of message object

18

19

# TODO - set the body by attaching the content to message

20

21

# TODO - create SMTP session

22

23

# TODO - enable security (starttls)

24

25

# TODO - login to the session with credentials

26

27

# TODO - convert message to string before sending

28

29

# TODO - send the email on the session

30

31

# TODO - quit the session

In cell 6, we send an email using Gmail server.

In line 33, we start an SMTP session.

And in line 36, we enabled security by calling the starttls() method. starttls() puts the SMTP connection in TLS (Transport Layer Security) mode. All SMTP commands that follow will be encrypted.

In line 39, we login on the current session as: session.login(sender, password).

In line 45, we send the email and in line 48, we quit the session.

Here is the email in Gmail account inbox after running this code:

Figure 11-8: An email sent via Gmail server

This finalizes our project on sending emails in Python.

12.        Regular Expressions

Regular Expressions (RE) are a powerful language concepts for matching text patterns. A Regular Expression is a sequence of characters that forms a search pattern. It specifies a set of strings that matches it.

Python has the re module for handling regular expressions. The functions in this module let you check if a particular string matches a given regular expression (or if a given regular expression matches a particular string, which comes down to the same thing).

This chapter gives an introduction to regular expressions and shows how they work in Python. You can find the PyCharm project for this chapter in the GitHub Repository of this book.

Chapter Outline:

The re Module

Metacharacters in RE

Sequences in RE

Sets in RE

RE Functions

Match Object

QUIZ – Regular Expressions

SOLUTIONS – Regular Expressions

The re Module

The re module in Python, provides regular expression matching operations. Both patterns and strings to be searched can be Unicode strings (str) as well as 8-bit strings (bytes).

Regular expressions use the backslash character ('\') to indicate special forms or to allow special characters to be used without invoking their special meaning. This collides with Python’s usage of the same character for the same purpose in string literals; for example, to match a literal backslash, one might have to write '\\\\' as the pattern string, because the regular expression must be \\, and each backslash must be expressed as \\ inside a regular Python string literal.

Raw String (r): The 'r' at the start of the pattern string designates a Python “raw string” which passes through backslashes without change. This is very handy when you are dealing with regular expressions. So, for regular expressions, it is recommended to write pattern strings with the 'r' in front of them.

Let’s do a basic example to see a simple regular expression. We want to search some characters in the given text. The code for this section is in reModule.py file. Here is the first example:

[1]:

1

# re Module

2

3

import re

4

5

text = 'Hands-On Python Advanced with Coding Exercises'

6

7

# call the search method

8

match = re.search(r'on', text)

9

10

# print start and end indices

11

print('Start Index:', match.start())

12

print('End Index:', match.end())

[1]:

Start Index: 13

End Index: 15

In cell 1, we see a simple regular expression search by using the re.search() method. We will see this method in detail later in this chapter. The parameters are the pattern we want to search and the text. The pattern here is r’on’, which is the raw string of the ‘on’ text. And the result of the search() method is a match object.

In lines 11 and 12, we print the start and end indices of the match object. As you see the start index is 13, because the search is case-sensitive and it is the first index which ‘on’ text resides. And the end index is 15.

Metacharacters in RE

Metacharacters are one of the main the building blocks of regular expressions. They are special-purpose characters which define how the regular expressions are interpreted. They indicate rules of regular expressions. For example, the ^ (caret) metacharacter is used to match a pattern only at the beginning of the string.

Here is the table of metacharacters in regular expressions:

Character

Description

Example

[]

A set of characters

"[c-y]"

\

Signals a special sequence (can also be used to escape special characters)

"\d"

.

Any character (except newline character)

"py..on"

python

py12on

py#_on

^

Starts with

"^python"

$

Ends with

"code$"

*

0 or more occurrences (for the preceding character)

"tre*"

tr

tre

tree

+

1 or more occurrences (for the preceding character)

"tre+"

tre

tree

?

0 or 1 occurrences (for the preceding character)

"colou?r"

color

colour

{n}

Exactly the specified number of occurrences

(for the preceding character)

{min, max} -> at least min, at most max

"py{2}"

pyy

|

Either or

"good|nice"

()

Capture and group.

Groups multiple tokens together and creates a capture group for extracting a substring.

Table 12-1: Metacharacters in Regular Expressions

Sequences in RE

Regular Expressions mainly work with sequences. A sequence is a special-purpose character following the backslash character ('\'). For example, \A is a sequence. Below is the list of sequences in RE and their meanings:

Sequence

Description

Example

\A

Returns a match if the specified characters are at the beginning of the string

"\AThis"

\b

Matches a word boundary (the position between a word and a space). Returns a match where the specified characters are at the beginning or at the end of a word. For example, rt\b matches the rt in start but not the rt in party.

"rt\b"

\B

Matches a nonword boundary. Returns a match where the specified characters are present, but NOT at the beginning (or at the end) of a word.

"\Brt"

\d

Returns a match where the string contains digits (numbers from 0-9)

"\d"

\D

Returns a match where the string DOES NOT contain digits

"\D"

\s

Returns a match where the string contains a white space character (spaces, tabs, form-feed characters, etc...)

"\s"

\S

Matches any non-white space character.

"\S"

\w

Returns a match where the string contains any word characters (from a to Z, digits from 0-9, and the underscore _). This expression is equivalent to [A-Za-z0-9_].

"\w"

\W

Matches any non-word character. This expression is equivalent to [^A-Za-z0-9_].

"\W"

\Z

Matches only the end of a string, or before a newline character at the end.

"world\Z"

Table 12-2: Sequences in Regular Expressions

Sets in RE

A set in Regular Expressions terminology, is a combination of characters inside a pair of square brackets [] with denotes a special meaning. Here are the sets we use in Regular Expressions in Python:

Set

Description

[adz]

Returns a match where one of the specified characters (a, d, or z) are present

[a-z]

Returns a match for any lower-case character, alphabetically between a and z

[^bdh]

Returns a match for any character EXCEPT b, d, and h

[012]

Returns a match where any of the specified digits (0, 1 or 2) are present

[0-9]

Returns a match for any digit between 0 and 9

[2-7][0-4]

Returns a match for any two-digit numbers from 20 and 74

[a-zA-Z]

Returns a match for any character alphabetically between a and z,

lower case OR upper case

[+]

In sets, +, *, ., |, (), $,{} has no special meaning, so [+] means:

return a match for any + character in the string

Table 12-3: Sets in Regular Expressions

RE Functions

The re module defines several functions, constants, and an exception. Some of the functions are simplified versions of the full featured methods for compiled regular expressions. Most non-trivial applications always use the compiled form.

re.compile(pattern, flags=0):

Compile a regular expression pattern into a regular expression object, which can be used for matching using its match(), search() and other methods, described below.

The expression’s behavior can be modified by specifying a flags value. Values can be any of the following variables, combined using bitwise OR (the | operator).

The sequence:

compiled_re = re.compile(pattern)

result = compiled_re.match(string)

is equivalent to:

result = re.match(pattern, string)

But using re.compile() and saving the resulting regular expression object for reuse is more efficient when the expression will be used several times in a single program.

Let’s rewrite the code in cell 1, using the compile() function this time:

[2]:

1

# compile()

2

import re

3

4

text = 'Hands-On Python Advanced with Coding Exercises'

5

6

# define the pattern

7

pattern = r'on'

8

9

# compile the regular expression

10

compiled_re = re.compile(pattern)

11

12

# call the search method on the compiled pattern

13

match = compiled_re.search(text)

14

15

# print start and end indices

16

print('Start Index:', match.start())

17

print('End Index:', match.end())

[2]:

Start Index: 13

End Index: 15

In cell 2, we first define the pattern variable as: pattern = r'on'. Keep in mind that, we use the ‘r’ character in front of the text to make it a raw string.

Then in line 10, we create a compiled regular expression object by calling the compile() function as: compiled_re = re.compile(pattern).

Now that we have a compiled RE object we can do the actual search as: match = compiled_re.search(text). As you see in the output, the result is exactly the same as the result of cell 1.

re.search(pattern, string, flags=0):

Scan through string looking for the first location where the regular expression pattern produces a match, and return a corresponding match object. Return None if no position in the string matches the pattern.

re.match(pattern, string, flags=0):

If zero or more characters at the beginning of string match the regular expression pattern, return a corresponding match object. Return None if the string does not match the pattern. If you want to locate a match anywhere in string, use search() instead.

[3]:

1

# match()

2

text = 'Hands-On Python Advanced with Coding Exercises'

3

4

# define the pattern

5

pattern = r'on'

6

7

# compile the regular expression

8

compiled_re = re.compile(pattern)

9

10

# call the search method on the compiled pattern

11

match = compiled_re.match(text)

12

13

# print start and end indices

14

try:

15

    print('Start Index:', match.start())

16

    print('End Index:', match.end())

17

except:

18

    print(f"No match found at the beginning of the string for {pattern}")

[3]:

No match found at the beginning of the string for on

re.fullmatch(pattern, string, flags=0):

If the whole string matches the regular expression pattern, return a corresponding match object. Return None if the string does not match the pattern.

Difference between re.match() and re.fullmatch():

Both of the re.match() and re.fullmatch() functions attempt to match at the beginning of the string.  The difference is that re.match() matches only at the beginning but re.fullmatch() tries to match at the end as well.

re.split(pattern, string, maxsplit=0, flags=0):

Split string by the occurrences of pattern. If capturing parentheses are used in pattern, then the text of all groups in the pattern are also returned as part of the resulting list. If maxsplit is nonzero, at most maxsplit splits occur, and the remainder of the string is returned as the final element of the list.

[4]:

1

# split()

2

text_to_split = 'Words, words, words.'

3

4

# split from non-word characters (space, comma, period etc.)

5

print(re.split(r'\W+', text_to_split))

6

7

# split from non-word characters with maxsplit = 1

8

print(re.split(r'\W+', text_to_split, 1))

9

# ['Words', 'words, words.']

10

11

# split from letters a to f and ignore case

12

print(re.split('[a-f]+', '0a3B9', flags=re.IGNORECASE))

[4]:

['Words', 'words', 'words', '']

['Words', 'words, words.']

['0', '3', '9']

re.findall(pattern, string, flags=0):

Return all non-overlapping matches of pattern in string, as a list of strings or tuples. The string is scanned left-to-right, and matches are returned in the order found. Empty matches are included in the result.

[5]:

1

# findall()

2

# find words starting with f and have any number of letters (a to z)

3

print(re.findall(r'\bf[a-z]*', 'which foot or hand fell fastest'))

[5]:

['foot', 'fell', 'fastest']

In cell 5, we search for the pattern of \bf[a-z]*. Here is how we read this pattern from left to right:

\bf match the letter f which are at the beginning of the words (word boundary)

[a-z]* matches any lower-case character, alphabetically between a and z and any number of occurrences (zero or many).

So, the expression of \bf[a-z]* actually means the words which starts with the letter f. And in the output, you see a list of these words as: ['foot', 'fell', 'fastest'].

[6]:

1

# find groups with equal sign between (words and digits)

2

print(re.findall(r'(\w+)=(\d+)', 'set width=20 and height=10'))

[6]:

[('width', '20'), ('height', '10')]

In cell 6, we search for the pattern of (\w+)=(\d+). Here is how we read this pattern:

The parentheses, (), mean capturing groups.

\w means any word characters

\d means any digits

= is the equal sign

+ means 1 or more occurrences of the preceding character

So, the patter of (\w+)=(\d+) is looking for something like this: width=20. A word on the left of equal sign and some digits on the right. And in the output, you see a list of tuples.

re.finditer(pattern, string, flags=0):

Return an iterator yielding match objects over all non-overlapping matches for the RE pattern in string. The string is scanned left-to-right, and matches are returned in the order found. Empty matches are included in the result.

[7]:

1

# finditer()

2

text = 'python machine leaning is incredible.'

3

pattern = r'(in)'

4

5

# call finditer()

6

matches = re.finditer(pattern, text)

7

8

# print matches

9

for match in matches:

10

    print(match)

[7]:

<re.Match object; span=(11, 13), match='in'>

<re.Match object; span=(19, 21), match='in'>

<re.Match object; span=(26, 28), match='in'>

re.sub(pattern, repl, string, count=0, flags=0):

Return the string obtained by replacing the leftmost non-overlapping occurrences of pattern in string by the replacement repl. If the pattern isn’t found, string is returned unchanged. repl can be a string or a function; if it is a string, any backslash escapes in it are processed. That is, \n is converted to a single newline character, \r is converted to a carriage return, and so forth.

[8]:

1

# sub()

2

text = "Data Science with Python"

3

replaced_text = re.sub("\s", "_", text)

4

print(replaced_text)

[8]:

Data_Science_with_Python

In cell 8, we replaced the white space characters, “\s”, with the underscores. As you see in the cell output.

Match Object

In regular expressions, a Match object is an object which contains information about the RE search and the its result. If there is no match after the RE search, then Python Interpreter will return None, instead of a Match object.

[9]:

1

# Match Object

2

3

import re

4

5

text = "Python 3.0 Deep Learning"

6

match_obj = re.search(r"\d", text)

7

print(match_obj)

[9]:

<re.Match object; span=(7, 8), match='3'>

In cell 9, we search for any digit, \d, in the given text. And you see the resulting Match object in the cell output.

The Match object is very useful, when you need information about the RE search and the search result. Here are some of its attributes:

Match.group([group1, ...]):

Returns one or more subgroups of the match. If there is a single argument, the result is a single string; if there are multiple arguments, the result is a tuple with one item per argument.

[10]:

1

# Match.group()

2

text = "Isaac Newton, physicist"

3

match_obj = re.search(r"\bN\w+", text)

4

print(match_obj.group())

[10]:

Newton

In cell 10, we search for a pattern which is "\bN\w+". This pattern means the capital letter N at the beginning of a word (\b) and any word character (\w) after it. The match is letter N in the word “Newton”. We print the group attribute of the Match object and you see the whole word of the search result.

Match.span([group]):

For a match m, return the 2-tuple (m.start, m.end). So, it gives the start and end indices of the match.

[11]:

1

# span()

2

text = "Isaac Newton, physicist"

3

match_obj = re.search(r"\bN\w+", text)

4

print(match_obj.span())

[11]:

(6, 12)

Match.string:

This attribute gives the string passed to match() or search().

[12]:

1

# Match.string

2

text = "Isaac Newton, physicist"

3

match_obj = re.search(r"\bN\w+", text)

4

print(match_obj.string)

[12]:

Isaac Newton, physicist

This finalizes the chapter on Regular Expression in Python. Next section is the quiz on the topics in this chapter.

QUIZ – Regular Expressions

Now it’s time to solve the quiz for this chapter. You can download the quiz file, QUIZ_RegularExpressions.zip, from the GitHub Repository of this book. You should put the quiz file in the RegularExpressions project we build in this chapter.

Here are the questions for this chapter:

Q1:

Define a regular expression pattern in the following way.

It should:

* start with 3 digits

* first dash (-) after 3 digits

* 4 alphabetical lower case letters (a to z)

* second dash (-)

* one or more word characters after the second dash

Here are some examples:

* "247-abcd-5555"

* "416-dfuy-t"

[1]:

1

# Q 1:

2

3

# import re module

4

# ---- your solution here ---

5

6

# define the pattern

7

# ---- your solution here ---

8

9

# check match for both examples

10

# ---- your solution here ---

11

12

# print the groups of match objects

13

# ---- your solution here ---

[1]:

247-abcd-5555

416-dfuy-t

Q2:

Here is the general format for emails:

(user_name)@(domain_name).(top_level_domain)

Write a function named email_validator.

The function will take one parameter which is the email.

It will use a regular expression to validate the given email.

It will return the match object (either an object or None) depending on the email being valid.

Hints:

Some valid email addresses:

* name.lastname@somemail.com

* name_lastname@outlook.com

* johndoe123@yahoo.co.uk

Invalid examples:

* abc_example.com

* xyz123@yahoo

[2]:

1

# Q 2:

2

3

# import re module

4

# ---- your solution here ---

5

6

# define the function

7

# ---- your solution here ---

8

9

# call the function with valid examples

10

email = "name.lastname@somemail.com"

11

print(email_validator(email))

12

13

email = "name_lastname@outlook.com"

14

print(email_validator(email))

15

16

email = "johndoe123@yahoo.co.uk"

17

print(email_validator(email))

18

19

# call the function with invalid examples

20

email = "abc_example.com"

21

print(email_validator(email))

22

23

email = "xyz123@yahoo"

24

print(email_validator(email))

[2]:

<re.Match object; span=(0, 26), match='name.lastname@somemail.com'>

<re.Match object; span=(0, 25), match='name_lastname@outlook.com'>

<re.Match object; span=(0, 22), match='johndoe123@yahoo.co.uk'>

None

None

Q3:

Define a function named find_all_with_regex.

The function will take a regular expression and a text as parameters.

Ant will return the list of all possible matches of this regex in the text.

The regex will be: all words starting with 'a' or 'c' (a word must include at least two characters).

The text is:

"Lorem ipsum dolor sit amet, a consectetur adipiscing elit. Sed id cid tempor risus.

Quisque imperdiet, neque c pulvinar sollicitudin, augue nisl varius nibh, a suscipit cerat non lectus."

[3]:

1

# Q 3:

2

3

# import re module

4

# ---- your solution here ---

5

6

# define the function

7

# ---- your solution here ---

8

9

# define the text

10

# ---- your solution here ---

11

12

# define regex: find all the words starting with 'a' or 'c'

13

# ---- your solution here ---

14

15

# call the function

16

# ---- your solution here ---

17

18

# print result

19

# ---- your solution here ---

[3]:

['amet', 'consectetur', 'adipiscing', 'cid', 'augue', 'cerat']

Q4:

Define a function named make_snake_case.

It will take a text as the parameter and will return the snake case form of this text.

Snake case means; replacing all occurrences of space, comma, or dot with an underscore.

Normal text: this is, a text.

Snake case: this_is__a_text_

Hints:

* sub()

[4]:

1

# Q 4:

2

3

# import re module

4

# ---- your solution here ---

5

6

# define the function

7

# ---- your solution here ---

8

9

# call the function

10

# ---- your solution here ---

11

12

# print the result

13

# ---- your solution here ---

[4]:

this_is__a_text_

Q5:

Define a function named remove_non_alphanumeric.

It will take a text as the parameter and will return a new modified text.

The function will remove everything except alphanumeric characters from the text.

[5]:

1

# Q 5:

2

3

# import re module

4

# ---- your solution here ---

5

6

# define the function

7

# ---- your solution here ---

8

9

# call the function

10

# ---- your solution here ---

11

12

# print the result

13

# ---- your solution here ---

[5]:

HandsOnPythonAdvanced333

SOLUTIONS – Regular Expressions

Here are the solutions for the quiz for this chapter.

S1:

[1]:

1

# S 1:

2

3

# import re module

4

import re

5

6

# define the pattern

7

pattern = re.compile(r"\d{3}-[a-z]{4}-\w+")

8

9

# check match for both examples

10

m = pattern.search("247-abcd-5555")

11

m_2 = pattern.search("416-dfuy-t")

12

13

# print the groups of match objects

14

print(m.group())

15

print(m_2.group())

[1]:

247-abcd-5555

416-dfuy-t

S2:

[2]:

1

# S 2:

2

3

# import re module

4

import re

5

6

# define the function

7

def email_validator(email):

8

    # define the regular expression

9

    regex = re.compile(r'[A-Za-z0-9_\.-]+@[A-Za-z0-9-]+(\.[A-Z|a-z]{2,})+')

10

    return regex.search(email)

11

12

# call the function with valid examples

13

email = "name.lastname@somemail.com"

14

print(email_validator(email))

15

16

email = "name_lastname@outlook.com"

17

print(email_validator(email))

18

19

email = "johndoe123@yahoo.co.uk"

20

print(email_validator(email))

21

22

# call the function with invalid examples

23

email = "abc_example.com"

24

print(email_validator(email))

25

26

email = "xyz123@yahoo"

27

print(email_validator(email))

[2]:

<re.Match object; span=(0, 26), match='name.lastname@somemail.com'>

<re.Match object; span=(0, 25), match='name_lastname@outlook.com'>

<re.Match object; span=(0, 22), match='johndoe123@yahoo.co.uk'>

None

None

S3:

[3]:

1

# S 3:

2

3

# import re module

4

import re

5

6

# define the function

7

def find_all_with_regex(regex, text):

8

    return re.findall(regex, text)

9

10

# define the text

11

text = """Lorem ipsum dolor sit amet, a consectetur adipiscing elit. Sed id cid tempor risus.

12

Quisque imperdiet, neque c pulvinar sollicitudin, augue nisl varius nibh, a suscipit cerat non lectus."""

13

14

# define regex: find all the words starting with 'a' or 'c'

15

regex = r"\b[ac]\w+"

16

17

# call the function

18

match_list = find_all_with_regex(regex, text)

19

20

# print result

21

print(match_list)

[3]:

['amet', 'consectetur', 'adipiscing', 'cid', 'augue', 'cerat']

S4:

[4]:

1

# S 4:

2

3

# import re module

4

import re

5

6

# define the function

7

def make_snake_case(text):

8

    return re.sub("[ ,.]", "_", text)

9

10

# call the function

11

text = 'this is, a text.'

12

snake_case = make_snake_case(text)

13

14

# print the result

15

print(snake_case)

[4]:

this_is__a_text_

S5:

[5]:

1

# S 5:

2

3

# import re module

4

import re

5

6

# define the function

7

def remove_non_alphanumeric(text):

8

    # define the pattern

9

    pattern = re.compile('[\W_]+')

10

    # call and return the sub() function

11

    return pattern.sub('', text)

12

13

# call the function

14

s = '_*!/Hands-On Python - Advanced\+++ 333...'

15

s_new = remove_non_alphanumeric(s)

16

17

# print the result

18

print(s_new)

[5]:

HandsOnPythonAdvanced333

13.        Database Operations

Database Operations are one of the fundamental parts in application development. In this chapter you will learn how to set up a local database server and create a database instance with tables in it. Then you will learn the basic SQL commands and see how they work with Python.

You can find the PyCharm project for this chapter in the GitHub Repository of this book.

Chapter Outline:

SQL

Install MySQL Server

Install MySQL Connector

Connect to MySQL Server

Create a Database

Creating Tables

Inserting Records

Querying Records

Updating Records

Deleting Records

QUIZ – Database Operations

SOLUTIONS – Database Operations

SQL

SQL (Structured Query Language) is a standard language used in programming and designed for managing data held in a relational database management system (RDBMS).

SQL is designed for a specific purpose: to query and manage the data contained in a relational database. It is a set-based, declarative programming language, not an imperative programming language like C or BASIC. However, extensions to Standard SQL add procedural programming language functionality, such as control-of-flow constructs.

Python has powerful features for database operations. It supports various databases like MySQL, Oracle, Microsoft SQL Server, Sybase, PostgreSQL, IBM DB2, etc. It also supports Data Definition Language (DDL), Data Manipulation Language (DML) and Data Query Statements.

This chapter is not about SQL language syntax, so we assume that you are familiar with SQL basic commands like SELECT, INSERT, UPDATE, DELETE etc...

Install MySQL Server

MySQL is a free and open-source relational database management system (RDBMS). A relational database, organizes data into one or more data tables in which data types may be related to each other; these relations help structure the data. MySQL implements a relational database in a computer's storage system, manages users, allows for network access and facilitates testing database integrity and creation of backups.

At the time of this writing, MySQL is used by many of the major tech companies, like IBM, Google, Facebook, Sony, LinkedIn, Uber, Netflix, Twitter, and more.

In this chapter, we will use MySQL Community Server as it is free and widely used in the industry. You can skip this section if you already have MySQL on your development environment.

Here are the installation guides for major operating systems:

Windows: https://dev.mysql.com/doc/refman/8.0/en/windows-installation.html

macOS: https://dev.mysql.com/doc/refman/8.0/en/macos-installation.html

Linux: https://dev.mysql.com/doc/refman/8.0/en/linux-installation.html

For Microsoft Windows, the simplest and recommended method is to download MySQL Installer (for Windows) and let it install and configure a specific version of MySQL Server.

I am using macOS, so here are the steps to install MySQL on macOS using the package installer:

Download the disk image (.dmg) file (the community version is available here) that contains the MySQL package installer. Double-click the file to mount the disk image and see its contents.

Double-click the MySQL installer package from the disk. It is named according to the version of MySQL you have downloaded. For example, for MySQL server 8.0.28 it might be named mysql-8.0.28-macos-10.13-x86_64.pkg.

The initial wizard introduction screen references the MySQL server version to install. Click Continue to begin the installation.

The MySQL community edition shows a copy of the relevant GNU General Public License. Click Continue and then Agree to continue.

From the Installation Type page, you can either click Install to execute the installation wizard using all defaults, click Customize to alter which components to install (MySQL server, MySQL Test, Preference Pane, Launchd Support -- all but MySQL Test are enabled by default).

Note: Although the Change Install Location option is visible, the installation location cannot be changed.

Figure 13-1: MySQL Package Installer Wizard: Installation Type

Figure 13-2: MySQL Package Installer Wizard: Customize

Click Install to install MySQL Server. The installation process ends here if upgrading a current MySQL Server installation, otherwise follow the wizard's additional configuration steps for your new MySQL Server installation.

After a successful new MySQL Server installation, complete the configuration steps by choosing the default encryption type for passwords, define the root password, and also enable (or disable) MySQL server at startup.

The default MySQL 8.0 password mechanism is caching_sha2_password (Strong), and this step allows you to change it to mysql_native_password (Legacy).

Figure 13-3: MySQL Package Installer Wizard: Choose a Password Encryption Type

Choosing the legacy password mechanism alters the generated launchd file to set --default_authentication_plugin=mysql_native_password under ProgramArguments. Choosing strong password encryption does not set --default_authentication_plugin because the default MySQL Server value is used, which is caching_sha2_password.

Define a password for the root user, and also toggle whether MySQL Server should start after the configuration step is complete.

Figure 13-4: MySQL Package Installer Wizard: Define Root Password

Summary is the final step and references a successful and complete MySQL Server installation. Close the wizard.

Figure 13-5: MySQL Package Installer Wizard: Summary

MySQL server is now installed on your machine. If you chose to not start MySQL, then use either launchctl from the command line or start MySQL by clicking Start MySQL Server button using the MySQL Preference Pane. It is located under the System Preferences. See the images below:

Figure 13-6: MySQL in System Preferences

Figure 13-7: MySQL in Open Spotlight

Figure 13-8: MySQL Preference Pane

When installing using the package installer, the files are installed into a directory within /usr/local matching the name of the installation version and platform. For example, the installer file mysql-8.0.28-macos10.15-x86_64.dmg installs MySQL into /usr/local/mysql-8.0.28-macos10.15-x86_64/ with a symlink to /usr/local/mysql.

Install MySQL Connector

MySQL Connector is a self-contained Python driver for communicating with MySQL servers, and is used it to develop database applications.

MySQL Connector enables Python programs to access MySQL databases, using an API that is compliant with the Python Database API Specification v2.0 (PEP 249).

MySQL Connector/Python includes support for:

Almost all features provided by MySQL Server up to and including MySQL Server version 8.0.

MySQL Connector 8.0 also supports X DevAPI.

Converting parameter values back and forth between Python and MySQL data types, for example Python datetime and MySQL DATETIME. You can turn automatic conversion on for convenience, or off for optimal performance.

All MySQL extensions to standard SQL syntax.

Protocol compression, which enables compressing the data stream between the client and server.

Connections using TCP/IP sockets and on Unix using Unix sockets.

Secure TCP/IP connections using SSL.

Self-contained driver. MySQL Connector does not require the MySQL client library or any Python modules outside the standard library.

In this chapter, we will use MySQL Connector as the driver to access the MySQL database. Since we use, PyCharm IDE, we will install MySQL Connector using PyCharm package manager.

In PyCharm navigate to Preferences, then under the Project section you will Python Interpreter. This is where we install Python packages. Click on the plus (+) icon to search for the package. Official MySQL Connector for Python is: mysql-connector-python. Search for this name and install the package which is from “Oracle and/or its affiliates”. See the image below:

Figure 13-9: Installing mysql-connector-python package in PyCharm

To test MySQL Connector, simply try to import it in the project as: import mysql.connector. If you do not get any errors, then you the connector is installed successfully.

Connect to MySQL Server

Now that we installed MySQL Connector/Python we are ready for database operations. Let’s start by creating our first connection to MySQL Server. Make sure the server is up and running on your local machine. We will start by creating two Python files. Here are the files; Credentials.py and MySQLConnector.py.

Credentials.py:

[1]:

1

# Credentials for local MySQL Server

2

3

mysql_host = "localhost"

4

mysql_user = "root"

5

mysql_password = "<your_root_password>"

In cell 1, we have the credentials for connection. We will import them whenever needed in this chapter.

MySQLConnector.py:

[2]:

1

# Create a MySQL Connection

2

3

# import MySQL Connector

4

import mysql.connector

5

6

# import credentials from Credentials.py file

7

from Credentials import mysql_host, mysql_user, mysql_password

8

9

# define a connection function

10

def connect_to_mysql(host=mysql_host,

11

                     user=mysql_user,

12

                     password=mysql_password):

13

    connection = None

14

    try:

15

        # start the connection

16

        connection = mysql.connector.connect(

17

          host=host,

18

          user=user,

19

          password=password

20

        )

21

        print("Successfully connected to MySQL Server.")

22

    except mysql.connector.Error as e:

23

        print(f"An Error occurred: '{e}'")

24

    # else:

25

    #     connection.close()

26

27

    return connection

28

29

# call the function and get connection

30

con = connect_to_mysql()

[2]:

Successfully connected to MySQL Server.

In cell 2, we establish our first connection to the MySQL server on our local machine. We import the mysql.connector class first.

In line 7, we import the credentials (mysql_host, mysql_user, mysql_password) from Credentials.py file. This file will keep the host, user name and password for this chapter.

In line 10, we define the connect_to_mysql() function. The parameters are host, user and password. The default values are our saved credentials which we import here. In the try block, we instantiate a connection by calling the mysql.connector.connect() function. We pass the host, user and password arguments to it. In the except block, we simply catch the mysql.connector.Error and print it. In the else block, we normally close the connection by calling the close() method. But we will need an open connection for the next sections, so we comment this out. Finally, in line 25, we return an active connection object.

You should see the success message in the output. Which shows that, your connection is successful and you can move on to the next section.

Create a Database

In this section we will create a database (db) for keeping the data for superheroes. The database name will be HEROES and it will have three tables in it; Superhero, Movie, and HeroMovie. Let’s start by creating the db first:

[3]:

1

# Create a Database

2

3

# import connector

4

import mysql.connector

5

6

# import custom connection function

7

from MySQLConnector import connect_to_mysql

8

9

# function for create the db

10

def create_database(db_name):

11

    try:

12

        # connect to server

13

        connection = connect_to_mysql()

14

        if connection is not None:

15

            # set the cursor

16

            cursor = connection.cursor()

17

            # create the database

18

            cursor.execute(f"CREATE DATABASE {db_name}")

19

    except mysql.connector.Error as e:

20

        print(f"An Error occurred in DB Creation:\n'{e}'")

21

    else:

22

        if connection is not None:

23

            # close the cursor and the connection

24

            cursor.close()

25

            connection.close()

26

            print("Database created successfully.")

27

28

# call the function and create the db

29

create_database("heroes")

[3]:

Successfully connected to MySQL Server.

Database created successfully.

In cell 3, we create a database named heroes. We define a function for this operation, create_database.

Inside the try block, we call the connect_to_mysql() function which we defined in the MySQLConnector.py file. This function returns a connection object if it can connects successfully. We check this object in line 14, to see if it is not None.

In line 16, we set a cursor object by calling the cursor() method on our connection object. We execute DDL (Data Definition Language) statements using a structure known as the cursor.

In line 18, we call the execute() method on the cursor object, by passing the SQL statement to create a new database. Here is the syntax: cursor.execute(f"CREATE DATABASE {db_name}").

In the else block, we again check for the connection object. If it is not None, then we close the cursor and the connection in respective order.

When we run the code in cell 3, we will have a new database instance on our local MySQL server. Let’s check if it really exists or not:

[4]:

1

# check database

2

def print_databases():

3

    try:

4

        # connect to server

5

        connection = connect_to_mysql()

6

        if connection is not None:

7

            # set the cursor

8

            cursor = connection.cursor()

9

            # execute query to show dbs

10

            cursor.execute("SHOW DATABASES")

11

            # print dbs

12

            for db in cursor:

13

                print(db)

14

    except mysql.connector.Error as e:

15

        print(f"An Error occurred in DB Creation:\n'{e}'")

16

    else:

17

        if connection is not None:

18

            # close the cursor and the connection

19

            cursor.close()

20

            connection.close()

21

22

print_databases()

[4]:

Successfully connected to MySQL Server.

('heroes',)

('information_schema',)

('mysql',)

('performance_schema',)

('sys',)

In cell 4, we define a function to print database names on our MySQL server. The SQL command for this is "SHOW DATABASES". Then we print the name of each database by using a for loop on the cursor object.

As you see, we have a new database called 'heroes', which means we are ready to create the some tables in it.

Creating Tables

Now that we have an active database, we can create our tables. We will create three tables in our heroes db. The tables will be:

Superhero: We will store superhero details like; hero name, real name, birth date and city.

Movie: We will store movie details like; movie_name and release_year.

HeroMovie: This table will store the relation between a superhero and a movie. hero_id and movie_id will be the foreign keys to the respective tables.

Here are the tables which we will create in this section:

Superhero

Movie

HeroMovie

hero_id

movie_id

hero_id

hero_name

movie_name

movie_id

real_name

release_year

birth_date

city

Table 13-1: The tables in HEROES database

[5]:

1

# Creating Tables

2

3

# import mysql classes

4

import mysql.connector

5

from mysql.connector import Error, errorcode

6

7

# import custom connection function

8

from MySQLConnector import connect_to_mysql

9

10

# set db name

11

db_name = 'heroes'

12

13

# define a dictionary for tables

14

tables = {}

15

16

# define superhero table

17

tables['superhero'] = (

18

    "CREATE TABLE `superhero` ("

19

    "  `hero_id` int(11) NOT NULL AUTO_INCREMENT,"

20

    "  `hero_name` varchar(40) NOT NULL,"

21

    "  `real_name` varchar(40) NOT NULL,"

22

    "  `birth_date` date NOT NULL,"

23

    "  `city` varchar(20) NOT NULL,"

24

    "  PRIMARY KEY (`hero_id`)"

25

    ") ENGINE=InnoDB")

26

27

# define movie table

28

tables['movie'] = (

29

    "CREATE TABLE `movie` ("

30

    "  `movie_id` int(11) NOT NULL AUTO_INCREMENT,"

31

    "  `movie_name` varchar(1000) NOT NULL,"

32

    "  `release_year` int(4) NOT NULL,"

33

    "  PRIMARY KEY (`movie_id`)"

34

    ") ENGINE=InnoDB")

35

36

# define heromovie table

37

tables['heromovie'] = (

38

    "CREATE TABLE `heromovie` ("

39

    "  `hero_id` int(11) NOT NULL,"

40

    "  `movie_id` int(11) NOT NULL,"

41

    "  `release_year` int(4) NOT NULL,"

42

    "  PRIMARY KEY (`hero_id`,`movie_id`),"

43

    "  CONSTRAINT `hero_id_fk` FOREIGN KEY (`hero_id`) "

44

    "     REFERENCES `superhero` (`hero_id`) ON DELETE CASCADE,"

45

    "  CONSTRAINT `movie_id_fk` FOREIGN KEY (`movie_id`) "

46

    "     REFERENCES `movie` (`movie_id`) ON DELETE CASCADE"

47

    ") ENGINE=InnoDB")

In cell 5, define a dictionary to keep our tables data. The keys are table names and the values are SQL commands which we need to execute to create the tables. You see MySQL syntax for creating tables, table fields, primary keys and foreign keys in the examples.

The final lines, which are ENGINE=InnoDB", are necessary to set the storage engine for MySQL server. InnoDB is a storage engine for the database management system in MySQL.

Now that we have the syntax to create our tables let’s define functions for this operation. We will define two separate functions. The first one will set the connection to the MySQL server and the second one will execute the SQL commands to create the tables in our database.

[6]:

1

# connect and start creating tables

2

def start_tables_creation():

3

    try:

4

        # connect to server

5

        connection = connect_to_mysql()

6

        if connection is not None:

7

            # set the cursor

8

            cursor = connection.cursor()

9

            # set use db for cursor

10

            cursor.execute(f"USE {db_name}")

11

            # create tables

12

            create_tables_in_db(connection, cursor)

13

    except Error as e:

14

        print(f"An Error occurred in Table Creation:\n'{e}'")

In cell 6, we have the start_tables_creation() function. It is responsible for starting an active connection by calling the connect_to_mysql() function and setting the cursor. In line 10, we execute a special syntax on the cursor to declare the database which we want to work on. Here is the syntax: USE heroes. This code selects the heroes db for the next tasks which we will execute using this cursor.

In line 12, we call another function for actual creation operation. The function name is create_tables_in_db() and we pass the connection and the cursor objects to it.

[7]:

1

# define the create function

2

def create_tables_in_db(connection, cursor):

3

    # create tables one by one

4

    for table in tables:

5

        table_creation_code = tables[table]

6

        try:

7

            print(f"Creation started for: {table}")

8

            cursor.execute(table_creation_code)

9

        except Error as e:

10

            if e.errno == errorcode.ER_TABLE_EXISTS_ERROR:

11

                print(f"Table {table} already exists.")

12

            else:

13

                print(e.msg)

14

        else:

15

            print(f"Table {table} created successfully.")

16

17

    # close active connections

18

    cursor.close()

19

    connection.close()

20

21

# call the function to start creating tables

22

start_tables_creation()

[7]:

Successfully connected to MySQL Server.

Creation started for: superhero

Table superhero created successfully.

Creation started for: movie

Table movie created successfully.

Creation started for: heromovie

Table heromovie created successfully.

In cell 7, we define the create_tables_in_db() function. This function uses a for loop to iterate over the tables dictionary. In line 8, it executes the SQL command to create the current table in the loop. We also check the error codes to be able handle possible exceptions. Finally, we close the active connection in line 18.

In line 22, we call start_tables_creation() function to start creation. And in the output, you can see that all the tables are created successfully.

Let’s see if they really exist in the database:

[8]:

1

# Display Tables

2

# check tables in database

3

def print_tables():

4

    try:

5

        # connect to server

6

        connection = connect_to_mysql()

7

        if connection is not None:

8

            # set the cursor

9

            cursor = connection.cursor()

10

            # set use db for cursor

11

            cursor.execute(f"USE heroes")

12

            # execute query to show dbs

13

            cursor.execute("SHOW TABLES")

14

            # print dbs

15

            for table in cursor:

16

                print(table)

17

    except mysql.connector.Error as e:

18

        print(f"An Error occurred in Displaying Tables:\n'{e}'")

19

    else:

20

        if connection is not None:

21

            # close the cursor and the connection

22

            cursor.close()

23

            connection.close()

24

25

# call the function to see tables

26

print_tables()

[8]:

Successfully connected to MySQL Server.

('heromovie',)

('movie',)

('superhero',)

Inserting Records

In this section we will insert some records to our database tables. First, let’s see the data which we will insert. Below are the tables displaying the records:

Superhero

hero_name

real_name

birth_date

city

Wonder Woman

Diana Prince

22.03.1976

Themyscira

Batman

Bruce Wayne

17.04.1915

Gotham

Superman

Clark Kent

18.04.1977

Metropolis

Spiderman

Peter Parker

1.08.1962

New York City

Table 13-2: Superhero data

Movie

movie_name

release_year

Spider-Man: No Way Home

2021

The Dark Knight

2008

Wonder Woman

2017

Man of Steel

2013

Table 13-3: Movie data

We will create two lists of dictionaries for these records. Then we will insert them to MySQL server. We also have two CSV files in the project; superhero.csv and movie.csv. These files are under the “data” folder and they contain the data in Tables 13-2 and 13-3. Let’s first start by reading these files:

[9]:

1

# read csv files

2

import csv

3

4

superheroes = []

5

movies = []

6

7

def get_data_as_dict(path):

8

    file = open(path)

9

    reader = csv.DictReader(file)

10

    list_of_data = []

11

    for row in reader:

12

        list_of_data.append(row)

13

    file.close()

14

    return list_of_data

15

16

# fill the lists

17

superheroes = get_data_as_dict('data/superhero.csv')

18

movies = get_data_as_dict('data/movie.csv')

19

20

for hero in superheroes:

21

    print(hero)

22

23

for movie in movies:

24

    print(movie)

[9]:

{'hero_name': 'Wonder Woman', 'real_name': 'Diana Prince', 'birth_date': '22.03.1976', 'city': 'Themyscira'}

{'hero_name': 'Batman', 'real_name': 'Bruce Wayne', 'birth_date': '17.04.1915', 'city': 'Gotham'}

{'hero_name': 'Superman', 'real_name': 'Clark Kent', 'birth_date': '18.04.1977', 'city': 'Metropolis'}

{'hero_name': 'Spiderman', 'real_name': 'Peter Parker', 'birth_date': '1.08.1962', 'city': 'New York City'}

{'movie_name': 'Spider-Man: No Way Home', 'release_year': '2021'}

{'movie_name': 'The Dark Knight', 'release_year': '2008'}

{'movie_name': 'Wonder Woman', 'release_year': ‘2017’}

{'movie_name': 'Man of Steel', 'release_year': '2013'}

In cell 9, we read the data in files, 'data/superhero.csv' and 'data/movie.csv'. And we fill a list for each file which are superheroes and movies. Now we will insert these lists to the related tables in our database.

[10]:

1

# import mysql classes

2

from mysql.connector import Error

3

4

# import custom connection function

5

from MySQLConnector import connect_to_mysql

6

7

# set db name

8

db_name = 'heroes'

9

10

# connect and set cursor

11

def start_record_insert():

12

    try:

13

        # connect to server

14

        connection = connect_to_mysql()

15

        if connection is not None:

16

            # set the cursor

17

            cursor = connection.cursor()

18

            # set use db for cursor

19

            cursor.execute(f"USE {db_name}")

20

            # insert superheroes

21

            insert_superheroes(cursor)

22

            # insert movies

23

            insert_movies(cursor)

24

    except Error as e:

25

        print(f"An Error occurred in Record Creation:\n'{e}'")

26

    else:

27

        # commit to the database

28

        connection.commit()

29

        cursor.close()

30

        connection.close()

In cell 10, we have the definition of the start_record_insert() function. It starts a connection to MySQL server and sets a cursor on our heroes database. Then in line 21, it calls insert_superheroes() function and in line 23, it calls the insert_movies() by passing the cursor as the argument. These functions will be responsible for formatting the columns and inserting records.

In line 28, inside the else block, we commit our changes to the database as: connection.commit(). When you made a modification in the database, you need to call the commit() method on the connection. This code will make sure that your data is committed to db. Otherwise, even if you don’t get any errors, the data will not be saved to the database.

[11]:

1

# function for inserting superheroes

2

def insert_superheroes(cursor):

3

    # convert str to date for birth_date

4

    str_to_date("birth_date")

5

6

    superhero_command = ("INSERT INTO superhero "

7

                   "(hero_name, real_name, birth_date, city) "

8

                   "VALUES (%(hero_name)s, %(real_name)s, %(birth_date)s, %(city)s)")

9

    for superhero in superheroes:

10

        cursor.execute(superhero_command, superhero)

11

    print("Superheroes inserted.")

12

13

# function for converting str to date

14

def str_to_date(column_name):

15

    from datetime import datetime

16

    for hero in superheroes:

17

        str_date = hero[column_name]

18

        date_format = "%d.%m.%Y"

19

        hero[column_name] = datetime.strptime(str_date, date_format).date()

The type for the birth_date column is date. So, we need to convert it for the records in the superheroes list. That’s why we have the str_to_date() function in cell 11, line 14.

Inside the insert_superheroes() function, we first define the SQL command for inserting. And we name it as superhero_command. In line 9, we set a for loop to execute each line one by one with the cursor.

[12]:

1

# function for inserting movies

2

def insert_movies(cursor):

3

    movie_command = ("INSERT INTO movie "

4

                   "(movie_name, release_year) "

5

                   "VALUES (%(movie_name)s, %(release_year)s)")

6

    for movie in movies:

7

        cursor.execute(movie_command, movie)

8

    print("Movies inserted.")

In cell 12, you see the definition of the insert_movies() function. It first creates an SQL command for insert and then loops over the items of the movies list. For each movie in the list, the cursor executes this insert command.

Let’s run the start_record_insert() function and do the actual insert operation:

[13]:

1

# call the start function

2

start_record_insert()

[13]:

Successfully connected to MySQL Server.

Superheroes inserted.

Movies inserted.

As you see in the output of cell 13, records are inserted to the tables in our database.

Querying Records

In this section, we will read the records which we inserted in the previous section. We will start by creating an SQL query statement. Let’s see how we set queries:

[14]:

1

# Querying Records

2

3

# import mysql classes

4

from mysql.connector import Error

5

6

# import custom connection function

7

from MySQLConnector import connect_to_mysql

8

9

# select query to get superheroes

10

query = ("SELECT * FROM superhero "

11

         "WHERE hero_name LIKE %s")

12

13

name_start_with = 'B%'

In cell 14, you see a simple SQL query statement with SELECT and WHERE clauses. The query will filter the results in which the hero_name starts with letter ‘B’. In SQL syntax, 'B%' means “start with b and the rest does not matter”. In the query variable, we have %s placeholder for this parameter.

[15]:

1

# connect and set cursor

2

def query_records():

3

    try:

4

        # connect to server

5

        connection = connect_to_mysql()

6

        if connection is not None:

7

            # set the cursor

8

            cursor = connection.cursor()

9

            # execute query with parameters

10

            cursor.execute(query, (name_start_with,))

11

            # get the result

12

            result = cursor.fetchall()

13

            # print the results

14

            for row in result:

15

                print(row)

16

    except Error as e:

17

        print(f"An Error occurred in Querying Records:\n'{e}'")

18

    else:

19

        cursor.close()

20

        connection.close()

21

22

# call the method to query

23

query_records()

[15]:

Successfully connected to MySQL Server.

(2, 'Batman', 'Bruce Wayne', datetime.date(1915, 4, 17), 'Gotham')

In cell 15, we define the query_records() function. In line 10, we execute our query with parameters. The second argument to the cursor.execute() function is a tuple containing the parameters.

In line 12, we get the query results by calling the cursor.fetchall() method. Then we print the results. As you see in the output, there is only one record which starts with the letter ‘B’.

We update the connect_to_mysql() function, to connect to our heroes database at the beginning. Here is the updated form of it:

[16]:

1

# define a connection function

2

def connect_to_mysql(host=mysql_host,

3

                     user=mysql_user,

4

                     password=mysql_password,

5

                     database='heroes'):

6

    connection = None

7

    try:

8

        # start the connection

9

        connection = mysql.connector.connect(

10

          host=host,

11

          user=user,

12

          password=password,

13

          database=database

14

        )

15

        print("Successfully connected to MySQL Server.")

16

    except mysql.connector.Error as e:

17

        print(f"An Error occurred: '{e}'")

18

    # else:

19

    #     connection.close()

20

21

    return connection

In cell 16, you see the final form of the connect_to_mysql() function. Since we are working on the heroes database, it is better to set it at the beginning of the connection. So, we pass the database parameter to the connect() function which is set to ‘heroes’ by default.

Updating Records

In this section we will update some records in our database. Let’s say we want to change the title and release year of one of the movies. To be able to update a specific record in our database, we need to know the unique id of that record. And to get the id’s we will query all the movies on our movie table.

[17]:

1

# Updating Records

2

3

# import mysql classes

4

from mysql.connector import Error

5

6

# import custom connection function

7

from MySQLConnector import connect_to_mysql

8

9

#--- GET ALL MOVIES ---#

10

# select query to get all movies

11

movie_select_query = ("SELECT * FROM movie")

12

13

# connect and set cursor

14

def get_movies():

15

    try:

16

        # connect to server

17

        connection = connect_to_mysql()

18

        if connection is not None:

19

            # set the cursor

20

            cursor = connection.cursor()

21

            # execute query

22

            cursor.execute(movie_select_query)

23

            # get the result

24

            result = cursor.fetchall()

25

            # print the results

26

            for (movie_id, movie_name, release_year) in result:

27

                print(f"{movie_id}\t{movie_name}\t{release_year}")

28

    except Error as e:

29

        print(f"An Error occurred in Querying Records:\n'{e}'")

30

    else:

31

        cursor.close()

32

        connection.close()

33

34

# call the method to query

35

get_movies()

[17]:

Successfully connected to MySQL Server.

1 Spider-Man: No Way Home 2021

2 The Dark Knight 2008

3 Wonder Woman 2017

4 Man of Steel 2013

In cell 17, we get all of the movies in the movie table. In line 26, we can directly address the names of the fields in the result set in a tuple as: for (movie_id, movie_name, release_year) in result.

In the output we see that the id of the Wonder Woman movie is 3. We want to update this record in the database. We will change its name to ‘Wonder Woman 1984’ and release year to 2000. Here is the code for update:

[18]:

1

#--- UPDATE A RECORD ---#

2

# set the update query

3

movie_update_query = ("UPDATE movie SET movie_name=%s, "

4

                      "release_year=2000 WHERE movie_id=%s")

5

6

# connect and update

7

def update_movie(id, name):

8

    try:

9

        # connect to server

10

        connection = connect_to_mysql()

11

        if connection is not None:

12

            # set the cursor

13

            cursor = connection.cursor()

14

            # execute query

15

            cursor.execute(movie_update_query, (name, id))

16

            # commit changes

17

            connection.commit()

18

    except Error as e:

19

        print(f"An Error occurred in Updating Records:\n'{e}'")

20

    else:

21

        print(f"{name} updated successfully.")

22

        cursor.close()

23

        connection.close()

24

25

# update the movie with id=3

26

update_movie(3, 'Wonder Woman 1984')

[18]:

Successfully connected to MySQL Server.

Wonder Woman 1984 updated successfully.

In cell 18, we update the movie with id = 3. We first define the update query in line 3. Then in the update_movie() function we execute this update query with parameters. Finally, in line 17, we commit our changes to the database by calling the connection.commit() method.

Now if you get all the movies one more time by calling the get_movies() function again, you will see that Wonder Woman movie is updated now:

[19]:

1

# get all movies again

2

get_movies()

[19]:

Successfully connected to MySQL Server.

1 Spider-Man: No Way Home 2021

2 The Dark Knight 2008

3 Wonder Woman 1984 2000

4 Man of Steel 2013

Deleting Records

Deleting records in a table in MySQL server is very easy. All you need is to define the unique id of the row which you want to delete. Let’s say we want to delete the movie with id = 4, which is “Man of Steel”.

[20]:

1

# Deleting Records

2

# import mysql classes

3

from mysql.connector import Error

4

# import custom connection function

5

from MySQLConnector import connect_to_mysql

6

# import UpdatingRecords.py file

7

import UpdatingRecords

8

9

# set the delete query

10

movie_update_query = ("DELETE FROM movie WHERE movie_id=%s")

11

12

# connect and update

13

def delete_movie(id):

14

    try:

15

        # connect to server

16

        connection = connect_to_mysql()

17

        if connection is not None:

18

            # set the cursor

19

            cursor = connection.cursor()

20

            # execute query

21

            cursor.execute(movie_update_query, (id,))

22

            # commit changes

23

            connection.commit()

24

    except Error as e:

25

        print(f"An Error occurred in Updating Records:\n'{e}'")

26

    else:

27

        print(f"Movie with id={id} deleted successfully.")

28

        cursor.close()

29

        connection.close()

30

31

# delete the movie with id=4

32

delete_movie(4)

[20]:

Successfully connected to MySQL Server.

Movie with id=4 deleted successfully.

In cell 20, we delete the movie record with id = 4. As you see the process is very similar to update operation. If we call get_movies() function from the UpdatingRecords.py file (the file for the previous section) again, you will see that “Man of Steel” no longer exists.

[21]:

1

# get all movies again

2

UpdatingRecords.get_movies()

[21]:

Successfully connected to MySQL Server.

1 Spider-Man: No Way Home 2021

2 The Dark Knight 2008

3 Wonder Woman 1984 2000

QUIZ – Database Operations

Now it’s time to solve the quiz for this chapter. You can download the quiz file, QUIZ_DatabaseOperations.zip, from the GitHub Repository of this book. You should put the quiz file in the DatabaseOperations project we build in this chapter.

Here are the questions for this chapter:

Q1:

Define a function to connect to a MySQL Server and print the connection result.

The function name will be mysql_connect.

It can be the local MySQL Server if you install it on your machine.

If you get an exception, print the error description.

The function will return an active connection object.

[1]:

1

# Q 1:

2

3

# import mysql

4

# ---- your solution here ---

5

6

# define the function

7

# ---- your solution here ---

8

9

# call the function

10

# mysql_connect()

[1]:

Successfully connected to MySQL Server.

Q2:

Create a new database named quiz_db.

You should call the connection function which you defined in Q1.

Print the result of the operation, either successful or an exception.

Hints:

* close connection

* close cursor

[2]:

1

# Q 2:

2

3

# import mysql

4

# ---- your solution here ---

5

6

# function for create the db

7

# ---- your solution here ---

8

9

# call the function and create the db

10

create_quiz_db("quiz_db")

[2]:

Successfully connected to MySQL Server.

Database created successfully.

Q3:

Define a function named create_student_table.

It will create a table in the quiz_db database.

The table name will be student.

You should call the connection function which you defined in Q1.

Here are the fields for the student table:

* id: int(11), not null, auto increment and primary key

* first_name: varchar(20)

* last_name: varchar(20)

* age: int(4)

Hints:

* set database as quiz_db on the cursor

* close connection

* close cursor

[3]:

1

# Q 3:

2

3

# import mysql

4

# ---- your solution here ---

5

6

# define Student table

7

# ---- your solution here ---

8

9

# connect and create table

10

# ---- your solution here ---

11

12

# call the function

13

create_student_table()

[3]:

Successfully connected to MySQL Server.

Student table created successfully.

Q4:

Define a function name insert_student_records.

The function will take a list of students and insert them to the student table in db.

We created the student table in quiz_db database in Q3.

You should call the connection function which you defined in Q1.

Here is the list:

students_to_insert = [

    ("Peter", "Parker", 24),

    ("Mary", "Jane", 23),

    ("Clark", "Kent", 32),

    ("Bruce", "Wayne", 29)]

Hints:

* commit()

* cursor.executemany() for executing sequences all at once

[4]:

1

# Q 4:

2

3

# import mysql

4

# ---- your solution here ---

5

6

# define insert query

7

# ---- your solution here ---

8

9

# define student list

10

# ---- your solution here ---

11

12

# connect and insert records

13

# ---- your solution here ---

14

15

# call the function

16

# ---- your solution here ---

[4]:

Successfully connected to MySQL Server.

Student records inserted.

Q5:

Define a function named query_students.

It will fetch and print all the student records in the quiz_db database.

You should call the connection function which you defined in Q1.

[5]:

1

# Q 5:

2

3

# import mysql

4

# ---- your solution here ---

5

6

# query to get all students

7

# ---- your solution here ---

8

9

# connect and set cursor

10

# ---- your solution here ---

11

12

# call the function

13

query_students()

[5]:

Successfully connected to MySQL Server.

(1, 'Peter', 'Parker', 24)

(2, 'Mary', 'Jane', 23)

(3, 'Clark', 'Kent', 32)

(4, 'Bruce', 'Wayne', 29)

SOLUTIONS – Database Operations

Here are the solutions for the quiz for this chapter.

S1:

[1]:

1

# S 1:

2

3

# import mysql

4

import mysql.connector

5

6

# define the function

7

def mysql_connect():

8

    connection = None

9

    try:

10

        # start the connection

11

        connection = mysql.connector.connect(

12

            host="localhost",

13

            user="root",

14

            password="<your_root_password>"

15

        )

16

        print("Successfully connected to MySQL Server.")

17

    except mysql.connector.Error as e:

18

        print(f"An Error occurred: '{e}'")

19

    return connection

20

21

# call the function

22

mysql_connect()

[1]:

Successfully connected to MySQL Server.

S2:

[2]:

1

# S 2:

2

3

# import mysql

4

import mysql.connector

5

6

# function for create the db

7

def create_quiz_db(db_name):

8

    try:

9

        # connect to server

10

        connection = mysql_connect()

11

        if connection is not None:

12

            # set the cursor

13

            cursor = connection.cursor()

14

            # create the database

15

            cursor.execute(f"CREATE DATABASE {db_name}")

16

    except mysql.connector.Error as e:

17

        print(f"An Error occurred in DB Creation:\n'{e}'")

18

    else:

19

        if connection is not None:

20

            # close the cursor and the connection

21

            cursor.close()

22

            connection.close()

23

            print("Database created successfully.")

24

25

# call the function and create the db

26

create_quiz_db("quiz_db")

[2]:

Successfully connected to MySQL Server.

Database created successfully.

S3:

[3]:

1

# S 3:

2

3

# import mysql

4

import mysql.connector

5

6

# define Student table

7

student_table = (

8

    "CREATE TABLE `student` ("

9

    "  `id` int(11) NOT NULL AUTO_INCREMENT,"

10

    "  `first_name` varchar(20) NOT NULL,"

11

    "  `last_name` varchar(20) NOT NULL,"

12

    "  `age` int(4) NOT NULL,"

13

    "  PRIMARY KEY (`id`)"

14

    ") ENGINE=InnoDB")

15

16

# connect and create table

17

def create_student_table():

18

    try:

19

        # connect to server

20

        connection = mysql_connect()

21

        if connection is not None:

22

            # set the cursor

23

            cursor = connection.cursor()

24

            # set use db for cursor

25

            cursor.execute("USE quiz_db")

26

            # create table

27

            cursor.execute(student_table)

28

    except mysql.connector.Error as e:

29

        print(f"An Error occurred in Table Creation:\n'{e}'")

30

    else:

31

        print("Student table created successfully.")

32

        cursor.close()

33

        connection.close()

34

35

# call the function

36

create_student_table()

[3]:

Successfully connected to MySQL Server.

Student table created successfully.

S4:

[4]:

1

# S 4:

2

3

# import mysql

4

import mysql.connector

5

6

# define insert query

7

insert_query = """

8

INSERT INTO student

9

(first_name, last_name, age)

10

VALUES (%s, %s, %s)

11

"""

12

13

# define student list

14

students_to_insert = [

15

    ("Peter", "Parker", 24),

16

    ("Mary", "Jane", 23),

17

    ("Clark", "Kent", 32),

18

    ("Bruce", "Wayne", 29)]

19

20

# connect and insert records

21

def insert_student_records(students_list):

22

    try:

23

        # connect to server

24

        connection = mysql_connect()

25

        if connection is not None:

26

            # set the cursor

27

            cursor = connection.cursor()

28

            # set use db for cursor

29

            cursor.execute("USE quiz_db")

30

            # create table

31

            cursor.executemany(insert_query, students_list)

32

            connection.commit()

33

    except mysql.connector.Error as e:

34

        print(f"An Error in insert operation:\n'{e}'")

35

    else:

36

        print("Student records inserted.")

37

        cursor.close()

38

        connection.close()

39

40

# call the function

41

insert_student_records(students_to_insert)

[4]:

Successfully connected to MySQL Server.

Student records inserted.

S5:

[5]:

1

# S 5:

2

3

# import mysql

4

import mysql.connector

5

6

# query to get all students

7

query = ("SELECT * FROM student")

8

9

# connect and set cursor

10

def query_students():

11

    try:

12

        # connect to server

13

        connection = mysql_connect()

14

        if connection is not None:

15

            # set the cursor

16

            cursor = connection.cursor()

17

            # set use db for cursor

18

            cursor.execute("USE quiz_db")

19

            # execute query

20

            cursor.execute(query)

21

            # get the result

22

            result = cursor.fetchall()

23

            # print the results

24

            for row in result:

25

                print(row)

26

    except mysql.connector.Error as e:

27

        print(f"An Error occurred in Querying Records:\n'{e}'")

28

    else:

29

        cursor.close()

30

        connection.close()

31

32

# call the function

33

query_students()

[5]:

Successfully connected to MySQL Server.

(1, 'Peter', 'Parker', 24)

(2, 'Mary', 'Jane', 23)

(3, 'Clark', 'Kent', 32)

(4, 'Bruce', 'Wayne', 29)

14.        Concurrency

In general, Concurrency is the fact of two or more events or circumstances happening or existing at the same time. In Computer Science, it is the ability to execute more than one program or task simultaneously. Concurrent programming may drastically improve the performance of your applications if it’s used properly.

In this chapter, we will learn how Python provides support for concurrent execution of code. You can find the PyCharm project for this chapter in the GitHub Repository of this book.

Chapter Outline:

CPU Bound vs. IO Bound

Multithreading vs. Multiprocessing

Threading

ThreadPoolExecutor – submit()

ThreadPoolExecutor – map()

ProcessPoolExecutor – submit()

ProcessPoolExecutor – map()

QUIZ – Concurrency

SOLUTIONS – Concurrency

CPU Bound vs. IO Bound

The tasks that computers perform can mainly be divided into two main categories which are CPU Bound tasks and IO Bound tasks.

CPU Bound means the rate of the execution of a task is limited by the speed of the CPU. Tasks as image resizing, large dataset operations like matrix multiplications, sorting of huge arrays, rendering operations may be examples of CPU Bound tasks.

IO Bound means the rate of the execution of a task is limited by the speed of the IO (input/output) subsystem. Tasks like reading from and writing to the disk, network or communication tasks can be examples of IO Bound.

In real-life projects, the difference between CPU Bound and IO Bound tasks is not always obvious. You should be aware that, most of the time it is not possible to strictly classify a task as either IO Bound or CPU Bound. You have to consider the tools for both cases and do some trials to decide based on your environment.

Multithreading vs. Multiprocessing

In concurrent programming, there are two concepts that you might encounter often. These are Multithreading and Multiprocessing. Both terms refer to operations which increase the computing power of a system. But there are some important differences between them.

Multithreading is a technique which enables a single process to be split into multiple code segments (threads). A thread, is a program execution context, within a process. A process can contain multiple threads. Threads run concurrently (in parallel) within the context of their parent process. Multithreaded applications are the ones that have two or more threads which run concurrently.

Multiprocessing refers to a system which has more than two CPUs (central processing units). In systems which have Multiprocessing, every CPU added helps to increase the computing power of the entire system. Each CPU has a main memory and can function independently.

Here are some key differences between Multithreading and Multiprocessing:

To increase computing power:

In Multithreading, multiple threads are created

In Multiprocessing, new CPUs are added to the system

Multiprocessing executes multiple processes on different CPUs, while Multithreading executes multiple threads in a single process.

Ease of implementation:

Multiprocessing needs considerable amount of investment (either time or resources)

Multithreading is easy to implement and requires less resources compared to Multiprocessing.

Python provides support for both Multithreading and Multiprocessing. In general, IO Bound tasks are handled with Multithreading and CPU Bound tasks are handling with Multiprocessing techniques. However, this might not be the case in every situation. It is up to the developer to decide which one to use when needed.

Threading

In this section we will learn how we create and run threads in Python. We will download some high-quality images from internet for this task. These are the images of IMDB Top 20 movies. You can find the URLs of the movies in the ‘data/imdb_top_20.csv’ file in the PyCharm Project of this chapter. We will download the images and save them to a folder called ‘images’. See the image below:

Figure 14-1:  imdb_top_20.csv file in the project

To be able to read the data in ‘data/imdb_top_20.csv’ file, we create a Python file named Utils.py. Here is the content in this file:

[1]:

1

# Utils.py file

2

3

import csv

4

5

# define a movies list

6

movies = []

7

8

# function to read movies

9

def read_csv():

10

    """reads the csv file and returns

11

        the list of movies"""

12

    movie_path = 'data/imdb_top_20.csv'

13

    with open(movie_path, 'r') as file:

14

        movie_dict = csv.DictReader(file, delimiter=';')

15

        for movie in movie_dict:

16

            movies.append(movie)

17

    # return the list

18

    return movies

In cell 1, we have Utils.py file and the read_csv() function in it. This function reads the movie data in ‘data/imdb_top_20.csv’ file and returns a list of dictionaries called movies.

Now let’s create another Python file named WithoutThreading.py. In this file we will download the movie images using the URLs provided in the movies list.

[2]:

1

# WithoutThreading.py file

2

3

import requests

4

import shutil

5

import time

6

import Utils

7

8

# call the function and fill movies

9

movies = Utils.read_csv()

10

11

# function to download a single image

12

def download_image(movie):

13

    title = movie['Title']

14

    file_name = 'images/' + str(title) + '.jpg'

15

    response = requests.get(movie['ImageURL'], stream=True)

16

    if response.status_code == 200:

17

        with open(file_name, 'wb') as file:

18

            shutil.copyfileobj(response.raw, file)

19

        print(f"{title} downloaded.")

20

    else:

21

        print(f"Error in downloading {title}")

22

23

# function to download images

24

def downlaod_all_images():

25

    # set start time

26

    start = time.perf_counter()

27

    # call download_image in a loop

28

    for movie in movies:

29

        download_image(movie)

30

    # set end time

31

    end = time.perf_counter()

32

    # print elapsed time

33

    print(f"\nDownload took {end-start:.2f} seconds without Threading.")

In cell 2, you see the content in the WithoutThreading.py file. We first get the movies list by calling the Utils.read_csv() function in line 8.

In line12, we define a function called download_image(). It downloads the image of a single movie, which is the parameter to it. It requests the image stream and saves it under the ‘images’ folder with the movie title.

In line 24, we have another function called downlaod_all_images(). This function loops over the movies list and call the download_image() function for each movie. We also keep track of the elapsed time in this function. And finally, we print the total time elapsed for downloading all of the movies. Here is the result when we call this function:

[3]:

1

# download all images

2

downlaod_all_images()

[3]:

The Shawshank Redemption downloaded.

The Godfather downloaded.

The Godfather: Part II downloaded.

The Dark Knight downloaded.

12 Angry Men downloaded.

Schindler's List downloaded.

The Lord of the Rings: The Return of the King downloaded.

Pulp Fiction downloaded.

Fight Club downloaded.

The Lord of the Rings: The Fellowship of the Ring downloaded.

Forrest Gump downloaded.

Star Wars: Episode V - The Empire Strikes Back downloaded.

Inception downloaded.

The Lord of the Rings: The Two Towers downloaded.

One Flew Over the Cuckoo's Nest downloaded.

Goodfellas downloaded.

The Matrix downloaded.

Se7en downloaded.

It's a Wonderful Life downloaded.

The Silence of the Lambs downloaded.

Download took 29.63 seconds without Threading.

As you see in the output of cell 3, it took about 29.63 seconds to download all movie images when we do it with a for loop. This is the result on my computer and my internet connection. Your result will be different. However, as you see, it took a bit long, because it tries to download each image one by one, one after another.

This example is a typical case, where IO Bound is the limit. We should expect a better result if we do the same operation using threads. Let’s test it and see if it’s true.

We will create a new Python file named Threading.py. And here is the content of this file:

[4]:

1

# Threading.py file

2

3

import requests

4

import shutil

5

import time

6

from threading import Thread

7

import Utils

8

9

# call the function and fill movies

10

movies = Utils.read_csv()

11

12

# create a threads list

13

threads = []

14

15

# function to download a single image

16

def download_image(movie):

17

    title = movie['Title']

18

    file_name = 'images/' + str(title) + '.jpg'

19

    response = requests.get(movie['ImageURL'], stream=True)

20

    if response.status_code == 200:

21

        with open(file_name, 'wb') as file:

22

            shutil.copyfileobj(response.raw, file)

23

        print(f"{title} downloaded.")

24

    else:

25

        print(f"Error in downloading {title}")

26

27

# function to download images

28

def downlaod_all_images():

29

    # set start time

30

    start = time.perf_counter()

31

    # fill the threads list

32

    for movie in movies:

33

        thread = Thread(target=download_image, args=(movie,))

34

        thread.start()

35

        threads.append(thread)

36

    # force for wait for finish

37

    for thread in threads:

38

        thread.join()

39

    # set end time

40

    end = time.perf_counter()

41

    # print elapsed time

42

    print(f"\nDownload took {end-start:.2f} seconds with Threading.")

In cell 4, line 13, we create an empty list for the threads which we will create in a minute. The list name is threads.

The definition of the download_image() function is exactly the same with the one in the previous section. It simply takes in a movie as the parameter, downloads its image and saves it to the ‘images’ folder.

We change the definition of the downlaod_all_images() function. Now instead of calling the download_image() function one after another, we create Thread objects for this. Here is the code for it:

thread = Thread(target=download_image, args=(movie,))

The target here, is the function which we want this thread to execute. And args is the arguments it will pass to the given function. We simply give the movie object. The Thread() constructor returns a Thread object which is ready to be started.

In line 34, we start this thread by calling start() method, which starts the thread’s activity.

In line 35, we append the current thread object to our threads list.

In line 37, we set a for loop on the threads list. Then inside the loop body, we call the join() method on each thread object. The join() method waits until the thread terminates. We need this to force the code execution to wait the thread to finish. Otherwise, the code would move on and we would not be able to track the end time.

And now, let’s see how long it takes to finish downloading all images with threads. Let’s call the function:

[5]:

1

# download all images

2

downlaod_all_images()

[5]:

12 Angry Men downloaded.

Forrest Gump downloaded.

The Lord of the Rings: The Return of the King downloaded.

The Lord of the Rings: The Two Towers downloaded.

The Godfather downloaded.

Pulp Fiction downloaded.

It's a Wonderful Life downloaded.

One Flew Over the Cuckoo's Nest downloaded.

Goodfellas downloaded.

Inception downloaded.

The Lord of the Rings: The Fellowship of the Ring downloaded.

The Silence of the Lambs downloaded.

The Shawshank Redemption downloaded.

Star Wars: Episode V - The Empire Strikes Back downloaded.

The Godfather: Part II downloaded.

Se7en downloaded.

Schindler's List downloaded.

The Matrix downloaded.

The Dark Knight downloaded.

Fight Club downloaded.

Download took 8.92 seconds with Threading.

As you see in the output of cell 5, the code execution takes 8.92 seconds with threading. It was 29.63 seconds without using threads. So, we have a considerable improvement here. That’s because downloading images via network is a good candidate for IO bound tasks. And if the images are not related to each other, like we have here, we can easily assign separate threads to them. None of them has to wait another one to complete.

ThreadPoolExecutor – submit()

In the previous section, we created each Thread object and start them one by one. Then in another loop, we call the join() method on each thread. Python has a better way to create and run threads. The module for this is called concurrent.futures and there are very handy classes in this module. ThreadPoolExecutor is one of these classes and it provides methods to execute thread calls easily. It is the class we use for Multithreading operations.

ThreadPoolExecutor has two important methods as submit() and map() for creating and managing threads.

submit(function, /, *args, **kwargs):

Schedules the callable, function, to be executed as function(*args, **kwargs) and returns a Future object representing the execution of the callable. Future instances are created by Executor.submit() and encapsulates the asynchronous execution of a callable.

We will create a new Python file named ThreadPoolExecutor_Submit.py. We will do the same image download operation but with concurrent.futures module, ThreadPoolExecutor class and the submit() method this time.

[6]:

1

# ThreadPoolExecutor_Submit.py file

2

3

import requests

4

import shutil

5

import time

6

from concurrent.futures import ThreadPoolExecutor, as_completed

7

import Utils

8

9

# call the function and fill movies

10

movies = Utils.read_csv()

11

12

# create a threads list

13

threads = []

14

15

# function to download a single image

16

def download_image(movie):

17

    title = movie['Title']

18

    file_name = 'images/' + str(title) + '.jpg'

19

    response = requests.get(movie['ImageURL'], stream=True)

20

    if response.status_code == 200:

21

        with open(file_name, 'wb') as file:

22

            shutil.copyfileobj(response.raw, file)

23

        print(f"{title} downloaded.")

24

    else:

25

        print(f"Error in downloading {title}")

26

27

# function to download images

28

def downlaod_all_images():

29

    # set start time

30

    start = time.perf_counter()

31

    # set ThreadPoolExecutor

32

    with ThreadPoolExecutor(max_workers=4) as executor:

33

        # submit the threads and get a list of futures

34

        future_list = [executor.submit(download_image, movie)

35

                       for movie in movies]

36

        for future in as_completed(future_list):

37

            future.result()

38

    # set end time

39

    end = time.perf_counter()

40

    # print elapsed time

41

    print(f"\nDownload took {end-start:.2f} sec, ThreadPoolExecutor.")

In cell 6, you see the content of the ThreadPoolExecutor_Submit.py file. We import the ThreadPoolExecutor class and the as_completed() function from the concurrent.futures module.

In the downlaod_all_images() function, we call the ThreadPoolExecutor(max_workers=4) constructor and it gives us an executor object.

In line 34, we set a list comprehension to call the submit() method with each movie. We pass movie objects one by one to the submit() method as: executor.submit(download_image, movie). And since the submit() method returns a Future object, list comprehension gives us a list of Future objects which we name it as future_list.

In line 36, we pass this future_list to the as_completed() function. This function returns an iterator over the Future instances as their related threads get completed. In other words, as_completed() function returns completed threads in form of an iterator. And we loop over this iterator with a for loop.

Inside the for loop, in line 37, we call the result() method for each Future object. Future.result(timeout=None) method returns the value returned by the call (the return value from the target function of the thread). If the call hasn’t yet completed then this method will wait up to timeout seconds. In our example, the target function is download_image() and it has no return value. It simply prints the download result. You can omit calling the result() method here, since it’s not needed. I added it to show you how you can get the return values from the functions you pass to the threads.

[7]:

1

# download all images

2

downlaod_all_images()

[7]:

The Godfather downloaded.

The Godfather: Part II downloaded.

12 Angry Men downloaded.

The Shawshank Redemption downloaded.

The Lord of the Rings: The Return of the King downloaded.

Pulp Fiction downloaded.

Schindler's List downloaded.

Forrest Gump downloaded.

The Lord of the Rings: The Fellowship of the Ring downloaded.

The Dark Knight downloaded.

The Lord of the Rings: The Two Towers downloaded.

Inception downloaded.

Fight Club downloaded.

Goodfellas downloaded.

One Flew Over the Cuckoo's Nest downloaded.

Star Wars: Episode V - The Empire Strikes Back downloaded.

It's a Wonderful Life downloaded.

The Matrix downloaded.

Se7en downloaded.

The Silence of the Lambs downloaded.

Download took 9.89 seconds with ThreadPoolExecutor.

As you see in cell 7, we got a similar result by using ThreadPoolExecutor.

ThreadPoolExecutor – map()

In the previous section we use the submit() method on the ThreadPoolExecutor objects to create and run threads. In this section we will learn the map() method to do the same operation.

map(func, *iterables, timeout=None, chunksize=1):

It is very similar to built-in map function in Python which is map(func, *iterables). The map() function on the ThreadPoolExecutor object returns an iterator of Future objects. Let’s see it in our download example:

[8]:

1

# ThreadPoolExecutor_Map.py file

2

3

import requests

4

import shutil

5

import time

6

from concurrent.futures import ThreadPoolExecutor, as_completed

7

import Utils

8

9

# call the function and fill movies

10

movies = Utils.read_csv()

11

12

# create a threads list

13

threads = []

14

15

# function to download a single image

16

def download_image(movie):

17

    title = movie['Title']

18

    file_name = 'images/' + str(title) + '.jpg'

19

    response = requests.get(movie['ImageURL'], stream=True)

20

    if response.status_code == 200:

21

        with open(file_name, 'wb') as file:

22

            shutil.copyfileobj(response.raw, file)

23

        print(f"{title} downloaded.")

24

    else:

25

        print(f"Error in downloading {title}")

26

27

# function to download images

28

def downlaod_all_images():

29

    # set start time

30

    start = time.perf_counter()

31

    # set ThreadPoolExecutor

32

    with ThreadPoolExecutor(max_workers=4) as executor:

33

        # map the threads

34

        executor.map(download_image, movies)

35

    # set end time

36

    end = time.perf_counter()

37

    # print elapsed time

38

    print(f"\nDownload took {end-start:.2f} sec, ThreadPoolExecutor.")

In cell 8, line 34, inside the downlaod_all_images() function we use the map() function on the executor object. The parameters are the target function (download_image) and the iterable (movies). The map() function instantiates and starts the threads all at once. And you will see a similar result when you run this code:

[9]:

1

# download all images

2

downlaod_all_images()

[9]:

The Godfather downloaded.

The Godfather: Part II downloaded.

12 Angry Men downloaded.

The Shawshank Redemption downloaded.

The Lord of the Rings: The Return of the King downloaded.

Schindler's List downloaded.

Pulp Fiction downloaded.

Forrest Gump downloaded.

The Lord of the Rings: The Fellowship of the Ring downloaded.

The Dark Knight downloaded.

The Lord of the Rings: The Two Towers downloaded.

Inception downloaded.

Star Wars: Episode V - The Empire Strikes Back downloaded.

Fight Club downloaded.

Goodfellas downloaded.

One Flew Over the Cuckoo's Nest downloaded.

Se7en downloaded.

It's a Wonderful Life downloaded.

The Silence of the Lambs downloaded.

The Matrix downloaded.

Download took 11.29 sec, ThreadPoolExecutor.

ProcessPoolExecutor – submit()

In the previous sections we learned ThreadPoolExecutor, which is the class used for Multithreading operations. In this section we will learn ProcessPoolExecutor, which is used for Multiprocessing tasks. The implementation of ProcessPoolExecutor is almost the same as ThreadPoolExecutor class. So, it is very easy to switch between them to be able to check if a task needs Multithreading or Multiprocessing.

For ProcessPoolExecutor, we will do a more computation heavy task. The task will be to decide if a given number is a Prime Number or not.

We will create a new Python file as ThreadPoolExecutor_Submit.py. Let’s start with a single processing example:

[10]:

1

# ProcessPoolExecutor_Submit.py file

2

3

from concurrent.futures import ProcessPoolExecutor

4

import time

5

6

# some numbers

7

primes = [7774777, 7778777, 111181111, 111191111, 777767777]

8

9

# function to check for being prime

10

def is_prime(num):

11

    if num > 1:

12

        for n in range(2, num):

13

            if (num % n) == 0:

14

                return False

15

        return True

16

    else:

17

        return False

18

19

def single_process_check():

20

    # set start time

21

    start = time.perf_counter()

22

    # check the primes list one by one

23

    for num in primes:

24

        print(f"Is Prime {num} - {is_prime(num)}")

25

    # set end time

26

    end = time.perf_counter()

27

    # print elapsed time

28

    print(f"\nTook {end - start:.2f} sec, Single Process.")

29

30

# call the single process check

31

single_process_check()

[10]:

Is Prime 7774777 - True

Is Prime 7778777 - True

Is Prime 111181111 - True

Is Prime 111191111 - True

Is Prime 777767777 - True

Took 29.52 sec, Single Process.

In cell 10, we have a function which checks if the given number is a prime number. Function name is is_prime() and as you may easily guess it is not suitable for high-performance operations.

We have another function named single_process_check(). This function loops over the primes list and checks each element one by one by calling the is_prime() function. And as you see in the cell output, it takes about 29.52 seconds to complete this task. Which is obviously an overkill for such a simple task.

Now let’s implement multiprocessing for the same task. We will use ProcessPoolExecutor and the submit() method. Here it is:

[11]:

1

### MULTIPROCESSING ###

2

def multi_proces_check():

3

    # set start time

4

    start = time.perf_counter()

5

    # set ProcessPoolExecutor

6

    with ProcessPoolExecutor() as executor:

7

        # submit the threads and get a list of futures

8

        future_list = [executor.submit(is_prime, n)

9

                       for n in primes]

10

        for num, future in zip(primes, as_completed(future_list)):

11

            print(f"Is Prime {num} - {future.result()}")

12

    # set end time

13

    end = time.perf_counter()

14

    # print elapsed time

15

    print(f"\nTook {end - start:.2f} sec, Multi Process.")

In cell 11, we define a new function named multi_proces_check(). In the function body, we set a with context manager to use ProcessPoolExecutor. In line 8, call the submit() method on the executor in a list comprehension. As we already know, this comprehension will return an iterator of Future objects. Then in line 10, we call as_completed() function on each Future object, then we print the result.

To be able to run this function you need to call it in a main.py file, especially on a macOS computer. Otherwise, you might get an error regarding subprocesses. That’s why we will create a main.py file in our project and call this function as follows:

[12]:

1

# main.py file

2

3

import ProcessPoolExecutor_Submit

4

5

if __name__ == '__main__':

6

    # call multi process check

7

    ProcessPoolExecutor_Submit.multi_proces_check()

[12]:

Is Prime 7774777 - True

Is Prime 7778777 - True

Is Prime 111181111 - True

Is Prime 111191111 - True

Is Prime 777767777 - True

Took 22.26 sec, Multi Process.

As you see in cell 12, we get an improvement in the same operation when we use Multiprocessing.

ProcessPoolExecutor – map()

In the previous section we used the submit() method for Multiprocessing. Now, let’s repeat the same operation using map() function this time.

We will create a new Python file named ProcessPoolExecutor_Map.py. And here is the multi_proces_check() function it this file:

[13]:

1

### MULTIPROCESSING ###

2

def multi_proces_check():

3

    # set start time

4

    start = time.perf_counter()

5

    # set ProcessPoolExecutor

6

    with ProcessPoolExecutor() as executor:

7

        # map the threads

8

        future_list = executor.map(is_prime, primes)

9

        for num, future in zip(primes, future_list):

10

            print(f"Is Prime {num} - {future}")

11

    # set end time

12

    end = time.perf_counter()

13

    # print elapsed time

14

    print(f"\nTook {end - start:.2f} sec, Multi Process.")

Let’s call this function from the main.py file:

[14]:

1

# main.py file

2

3

import ProcessPoolExecutor_Submit

4

import ProcessPoolExecutor_Map

5

6

if __name__ == '__main__':

7

    # call multi process check with submit()

8

    # ProcessPoolExecutor_Submit.multi_proces_check()

9

10

    # call multi process check with map()

11

    ProcessPoolExecutor_Map.multi_proces_check()

[14]:

Is Prime 7774777 - True

Is Prime 7778777 - True

Is Prime 111181111 - True

Is Prime 111191111 - True

Is Prime 777767777 - True

Took 22.40 sec, Multi Process.

Again, we see some improvement in the total execution time of the operation. This is the end of this chapter. Now it’s your turn to solve the quiz questions on Concurrency.

QUIZ – Concurrency

Now it’s time to solve the quiz for this chapter. You can download the quiz file, QUIZ_Concurrency.zip, from the GitHub Repository of this book. You should put the quiz file in the Concurrency project we build in this chapter.

Here are the questions for this chapter:

Q1:

We want to resize the images in the "images" folder in this project.

The initial size of this folder is around 27,5 MB with 20 images in it.

We want to create a "thumbnails" folder which includes the thumbnails of these images.

Thumbnail dimensions (width * height) of each image will 1/10 of the original image.

For example, the dimensions "Fight Club.jpg" file in the “images” directory is 2021*3000 pixels, but its thumbnail will be around 202*300 pixels.

And in terms of disk space, we expect the "thumbnails" folder to be around 200 KB.

You should keep track of elapsed time (start & end).

You need to define four functions:

* get_images

* create_directory

* resize_all_images

* resize_image

Packages you need:

* os

* pathlib

* time

* Pillow (for PIL)

[1]:

1

# Q 1:

2

3

# import packages

4

import os

5

from pathlib import Path

6

import time

7

# install Pillow package for PIL

8

from PIL import Image

9

10

# directory names

11

images_directory = "images"

12

thumbnails_directory = "thumbnails"

13

14

# get images fn

15

def get_images():

16

    """Returns a list of .jpg files

17

    in the images folder"""

18

    # ---- your solution here ---

19

20

# create directory fn

21

def create_directory(name):

22

    """Creates the directory with name.

23

    No errors if the directory exists."""

24

    # ---- your solution here ---

25

26

# resize all images fn

27

def resize_all_images():

28

    """Creates the directory.

29

    Gets images and call resize fn one by one.

30

    Keeps track of elapsed time."""

31

    try:

32

        # set start time

33

        # ---- your solution here ---

34

        # create directory

35

        # ---- your solution here ---

36

        # get images

37

        # ---- your solution here ---

38

        # resize each image one by one

39

        # ---- your solution here ---

40

        # set end time

41

        # ---- your solution here ---

42

    except:

43

        print("An Error occurred in resizing.")

44

    else:

45

        # print elapsed time

46

        # ---- your solution here ---

47

48

# resize a single image fn

49

def resize_image(image):

50

    # open the image

51

    # ---- your solution here ---

52

    # resize the image

53

    # ---- your solution here ---

54

    # Save the resized image

55

    # ---- your solution here ---

56

    # print result

57

    # ---- your solution here ---

58

59

# call resize all function

60

resize_all_images()

[1]:

…

The Godfather: Part II.jpg resized.

The Dark Knight.jpg resized.

Inception.jpg resized.

Images resized in 1.08 sec.

Q2:

"""

Redefine the resize_all_images() function you defined in Q1.

This time it will use Threading for resize operation.

Create and start a thread for each resize_image() function call.

Append these threads to a list called "threads".

Then join() the threads to force the code to wait them to finish.

Keep track of time again and print the final result.

Hints:

* create each thread using the Thread class

* start() each thread you create

* join() them in a separate for loop

[2]:

1

# Q 2:

2

3

from threading import Thread

4

5

# create a threads list

6

threads = []

7

8

# redefine resize all images fn

9

def resize_all_images():

10

    """Creates the directory.

11

        Gets images and call resize fn one by one.

12

        Keeps track of elapsed time."""

13

    try:

14

        # set start time

15

        # ---- your solution here ---

16

        # create directory

17

        # ---- your solution here ---

18

        # get images

19

        # ---- your solution here ---

20

        # fill the threads list

21

        # ---- your solution here ---

22

        # force for wait for finish

23

        # ---- your solution here ---

24

        # set end time

25

        # ---- your solution here ---

26

    except:

27

        print("An Error occurred in resizing.")

28

    else:

29

        # print elapsed time

30

        # ---- your solution here ---

31

32

# call resize all function

33

resize_all_images()

[2]:

…

The Dark Knight.jpg resized.

The Matrix.jpg resized.

Fight Club.jpg resized.

Images resized in 0.29 sec - Threading.

Q3:

Redefine the resize_all_images() function you defined in Q1.

Use ThreadPoolExecutor in this question.

Set a with context manager using ThreadPoolExecutor and use list comprehension to submit() threads.

Use a for loop to be able to get the result when the threads are completed.

Keep track of time again and print the final result.

Hints:

* ThreadPoolExecutor()

* [submit() in list comprehension]

* as_comleted()

* Future.result()

[3]:

1

# Q 3:

2

3

from concurrent.futures import ThreadPoolExecutor, as_completed

4

5

# create a threads list

6

threads = []

7

8

# redefine resize all images fn

9

def resize_all_images():

10

    """Creates the directory.

11

        Gets images and call resize fn one by one.

12

        Keeps track of elapsed time."""

13

    try:

14

        # set start time

15

        # ---- your solution here ---

16

        # create directory

17

        # ---- your solution here ---

18

        # get images

19

        # ---- your solution here ---

20

        # set ThreadPoolExecutor

21

        with # ---- your solution here ---:

22

            # submit the threads and get a list of futures

23

            # ---- your solution here ---

24

            # call as_complete() method for result

25

            # ---- your solution here ---

26

        # set end time

27

        # ---- your solution here ---

28

    except:

29

        print("An Error occurred in resizing.")

30

    else:

31

        # print elapsed time

32

        # ---- your solution here ---

33

34

# call resize all function

35

resize_all_images()

[3]:

…

The Dark Knight.jpg resized.

The Matrix.jpg resized.

Fight Club.jpg resized.

Images resized in 0.33 sec - ThreadPoolExecutor.

Q4:

Redefine the resize_all_images() function you defined in Q1.

Use ProcessPoolExecutor in this question.

Set a with context manager using ProcessPoolExecutor and use list comprehension to submit() threads.

Use a for loop to be able to get the result when the threads are completed.

Keep track of time again and print the final result.

Hints:

* ProcessPoolExecutor()

* [submit() in list comprehension]

* as_comleted()

* Future.result()

[4]:

1

# Q 4:

2

3

from concurrent.futures import ProcessPoolExecutor, as_completed

4

5

# create a threads list

6

threads = []

7

8

# redefine resize all images fn

9

def resize_all_images():

10

    """Creates the directory.

11

        Gets images and call resize fn one by one.

12

        Keeps track of elapsed time."""

13

    try:

14

        # set start time

15

        # ---- your solution here ---

16

        # create directory

17

        # ---- your solution here ---

18

        # get images

19

        # ---- your solution here ---

20

        # set ThreadPoolExecutor

21

        with # ---- your solution here ---:

22

            # submit the threads and get a list of futures

23

            # ---- your solution here ---

24

            # call as_complete() method for result

25

            # ---- your solution here ---

26

        # set end time

27

        # ---- your solution here ---

28

    except:

29

        print("An Error occurred in resizing.")

30

    else:

31

        # print elapsed time

32

        # ---- your solution here ---

33

34

# call resize all function

35

resize_all_images()

[4]:

…

The Matrix.jpg resized.

The Dark Knight.jpg resized.

Fight Club.jpg resized.

Images resized in 0.63 sec - ProcessPoolExecutor.

Q5:

Redefine the resize_all_images() function you defined in Q1.

Use ProcessPoolExecutor with map() function in this question.

Set a with context manager using ProcessPoolExecutor and map() the threads.

Keep track of time again and print the final result.

Hints:

* ProcessPoolExecutor()

* map(function, iterable)

[5]:

1

# Q 5:

2

3

from concurrent.futures import ProcessPoolExecutor

4

5

# create a threads list

6

threads = []

7

8

# redefine resize all images fn

9

def resize_all_images():

10

    """Creates the directory.

11

        Gets images and call resize fn one by one.

12

        Keeps track of elapsed time."""

13

    try:

14

        # set start time

15

        # ---- your solution here ---

16

        # create directory

17

        # ---- your solution here ---

18

        # get images

19

        # ---- your solution here ---

20

        # set ThreadPoolExecutor

21

        with # ---- your solution here ---:

22

            # map the threads

23

            # ---- your solution here ---

24

        # set end time

25

        # ---- your solution here ---

26

    except:

27

        print("An Error occurred in resizing.")

28

    else:

29

        # print elapsed time

30

        # ---- your solution here ---

31

32

# call resize all function

33

resize_all_images()

[5]:

…

Goodfellas.jpg resized.

The Dark Knight.jpg resized.

Fight Club.jpg resized.

Images resized in 0.61 sec-ProcessPoolExecutor-Map.

SOLUTIONS – Concurrency

Here are the solutions for the quiz for this chapter.

S1:

[1]:

1

# S 1:

2

3

# import packages

4

import os

5

from pathlib import Path

6

import time

7

# install Pillow package for PIL

8

from PIL import Image

9

10

# directory names

11

images_directory = "images"

12

thumbnails_directory = "thumbnails"

13

14

# get images fn

15

def get_images():

16

    """Returns a list of .jpg files

17

    in the images folder"""

18

    images = []

19

    # loop in the images folder

20

    for file in os.listdir(images_directory):

21

        # check if it is a .jpg file

22

        if file.endswith(".jpg"):

23

            images.append(file)

24

    return images

25

26

# create directory fn

27

def create_directory(name):

28

    """Creates the directory with name.

29

    No errors if the directory exists."""

30

    Path(name).mkdir(exist_ok=True)

31

32

# resize all images fn

33

def resize_all_images():

34

    """Creates the directory.

35

        Gets images and call resize fn one by one.

36

        Keeps track of elapsed time."""

37

    try:

38

        # set start time

39

        start = time.perf_counter()

40

        # create directory

41

        create_directory(thumbnails_directory)

42

        # get images

43

        images = get_images()

44

        # resize each image one by one

45

        for image in images:

46

            resize_image(image)

47

        # set end time

48

        end = time.perf_counter()

49

    except:

50

        print("An Error occurred in resizing.")

51

    else:

52

        # print elapsed time

53

        print(f"Images resized in {end - start:.2f} sec.")

54

55

# resize a single image fn

56

def resize_image(image):

57

    # open the image

58

    im = Image.open(images_directory + "/" + image)

59

    # resize the image

60

    resized_im = im.resize((round(im.size[0] * 0.1), round(im.size[1] * 0.1)))

61

    # Save the resized image

62

    resized_im.save(thumbnails_directory + "/" + image)

63

    # print result

64

    print(f"{image} resized.")

65

66

# call resize all function

67

resize_all_images()

[1]:

…

The Godfather: Part II.jpg resized.

The Dark Knight.jpg resized.

Inception.jpg resized.

Images resized in 1.08 sec.

S2:

[2]:

1

# S 2:

2

3

from threading import Thread

4

5

# create a threads list

6

threads = []

7

8

# redefine resize all images fn

9

def resize_all_images():

10

    """Creates the directory.

11

        Gets images and call resize fn one by one.

12

        Keeps track of elapsed time."""

13

    try:

14

        # set start time

15

        start = time.perf_counter()

16

        # create directory

17

        create_directory(thumbnails_directory)

18

        # get images

19

        images = get_images()

20

        # fill the threads list

21

        for image in images:

22

            thread = Thread(target=resize_image, args=(image,))

23

            thread.start()

24

            threads.append(thread)

25

        # force for wait for finish

26

        for thread in threads:

27

            thread.join()

28

        # set end time

29

        end = time.perf_counter()

30

    except:

31

        print("An Error occurred in resizing.")

32

    else:

33

        # print elapsed time

34

        print(f"Images resized in {end - start:.2f} sec - Threading.")

35

36

# call resize all function

37

resize_all_images()

[2]:

…

The Dark Knight.jpg resized.

The Matrix.jpg resized.

Fight Club.jpg resized.

Images resized in 0.29 sec - Threading.

S3:

[3]:

1

# S 3:

2

3

from concurrent.futures import ThreadPoolExecutor, as_completed

4

5

# create a threads list

6

threads = []

7

8

# redefine resize all images fn

9

def resize_all_images():

10

    """Creates the directory.

11

        Gets images and call resize fn one by one.

12

        Keeps track of elapsed time."""

13

    try:

14

        # set start time

15

        start = time.perf_counter()

16

        # create directory

17

        create_directory(thumbnails_directory)

18

        # get images

19

        images = get_images()

20

        # set ThreadPoolExecutor

21

        with ThreadPoolExecutor() as executor:

22

            # submit the threads and get a list of futures

23

            future_list = [executor.submit(resize_image, image)

24

                           for image in images]

25

            # call as_complete() method for result

26

            for future in as_completed(future_list):

27

                future.result()

28

        # set end time

29

        end = time.perf_counter()

30

    except:

31

        print("An Error occurred in resizing.")

32

    else:

33

        # print elapsed time

34

        print(f"Images resized in {end - start:.2f} sec - ThreadPoolExecutor.")

35

36

# call resize all function

37

resize_all_images()

[3]:

…

The Dark Knight.jpg resized.

The Matrix.jpg resized.

Fight Club.jpg resized.

Images resized in 0.33 sec - ThreadPoolExecutor.

S4:

[4]:

1

# S 4:

2

3

from concurrent.futures import ProcessPoolExecutor, as_completed

4

5

# create a threads list

6

threads = []

7

8

# redefine resize all images fn

9

def resize_all_images():

10

    """Creates the directory.

11

        Gets images and call resize fn one by one.

12

        Keeps track of elapsed time."""

13

    try:

14

        # set start time

15

        start = time.perf_counter()

16

        # create directory

17

        create_directory(thumbnails_directory)

18

        # get images

19

        images = get_images()

20

        # set ThreadPoolExecutor

21

        with ProcessPoolExecutor() as executor:

22

            # submit the threads and get a list of futures

23

            future_list = [executor.submit(resize_image, image)

24

                           for image in images]

25

            # call as_complete() method for result

26

            for future in as_completed(future_list):

27

                future.result()

28

        # set end time

29

        end = time.perf_counter()

30

    except:

31

        print("An Error occurred in resizing.")

32

    else:

33

        # print elapsed time

34

        print(f"Images resized in {end - start:.2f} sec - ProcessPoolExecutor.")

35

36

# call resize all function

37

resize_all_images()

[4]:

…

The Matrix.jpg resized.

The Dark Knight.jpg resized.

Fight Club.jpg resized.

Images resized in 0.63 sec - ProcessPoolExecutor.

S5:

[5]:

1

# S 5:

2

3

from concurrent.futures import ProcessPoolExecutor

4

5

# create a threads list

6

threads = []

7

8

# redefine resize all images fn

9

def resize_all_images():

10

    """Creates the directory.

11

        Gets images and call resize fn one by one.

12

        Keeps track of elapsed time."""

13

    try:

14

        # set start time

15

        start = time.perf_counter()

16

        # create directory

17

        create_directory(thumbnails_directory)

18

        # get images

19

        images = get_images()

20

        # set ThreadPoolExecutor

21

        with ProcessPoolExecutor() as executor:

22

            # map the threads

23

            executor.map(resize_image, images)

24

        # set end time

25

        end = time.perf_counter()

26

    except:

27

        print("An Error occurred in resizing.")

28

    else:

29

        # print elapsed time

30

        print(f"Images resized in {end-start:.2f} sec-ProcessPoolExecutor-Map.")

31

32

# call resize all function

33

resize_all_images()

[5]:

…

Goodfellas.jpg resized.

The Dark Knight.jpg resized.

Fight Club.jpg resized.

Images resized in 0.61 sec-ProcessPoolExecutor-Map.

15.        Project 2 – Web Scraping with Python

Project Setup

In this project, we will learn how to scrap data out of web pages using Python. You can find the PyCharm project for this chapter in the GitHub Repository of this book.

Chapter Outline:

Install Scrapy

Create a Scrapy Project

Define Our First Spider

Run Our Spider

Extracting Data

Extracting Data in Our Spider

Storing the Scraped Data

Following Links

Shortcut for Creating Requests

Install Scrapy

In this project we will use Scrapy as our primary tool for web crawling. Scrapy is an open source and collaborative framework built with Python, for extracting the data you need from websites. It is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like data mining, information processing or historical archival. Scrapy is one of the most widely used tool for large web crawling projects.

To install Scrapy we will simply use our PyCharm IDE. Here you can find the complete installation document. Let’s start by creating a new PyCharm project named WebScraping. Make sure you create your project with a new virtual environment. Inside PyCharm, navigate to Preferences, and under Project sub-section on the left, click Python Interpreter. Here we will install Scrapy package. See the image below:

Figure 15-1: Installing Scrapy in PyCharm IDE

Scrapy has some dependencies and PyCharm will install them too. So, it might take a bit long to install Scrapy and all the dependencies.

Scrapy is written in pure Python and depends on a few key Python packages (among others):

lxml, an efficient XML and HTML parser

parsel, an HTML/XML data extraction library written on top of lxml,

w3lib, a multi-purpose helper for dealing with URLs and web page encodings

twisted, an asynchronous networking framework

cryptography and pyOpenSSL, to deal with various network-level security needs

Create a Scrapy Project

Before we start using Scrapy we need to set up a new Scrapy project. In our WebScraping PyCharm project we will use Python terminal to create and run Scrapy project. You can have one or multiple Scrapy projects in a single PyCharm project. Let’s create the first one:

In PyCharm, right click on the project name and select Open In -> Terminal:

Figure 15-2: Open a Terminal Window in PyCharm

This will open a new Terminal Window in PyCharm. Here is the command to create a new Scrapy project:

scrapy startproject firstscraping

You will see a new directory in your PyCharm project named firstscraping when you run this code.

Figure 15-3: Scrapy project inside PyCharm project

Here are the files in our Scrapy project:

firstscraping/

    scrapy.cfg            # deploy configuration file

    firstscraping/        # project's Python module, you'll import your code from here

        __init__.py

        items.py          # project items definition file

        middlewares.py    # project middlewares file

        pipelines.py      # project pipelines file

        settings.py       # project settings file

        spiders/          # a directory where you'll later put your spiders

            __init__.py

Now that we successfully create a Scrapy project, let’s start by defining our first Spider class.

Define Our First Spider

Spiders are classes which we define and which Scrapy uses to scrape information from a website (or a group of websites). They must inherit from scrapy.Spider class and define the initial requests to make, optionally how to follow links in the pages, and how to parse the downloaded page content to extract data.

To learn the basic concepts of Scrapy we will be working on a dedicated web site which is https://quotes.toscrape.com. It is a simple web site, which has a list of famous quotes on different topics.

Create a new Python file named quotes_spider.py. under the firstscraping/spiders directory. And here is the content of this file:

[1]:

1

# quotes_spider.py file

2

3

from scrapy import Spider, Request

4

5

class QuotesSpider(Spider):

6

    # unique identifier of this spider

7

    name = "quotes"

8

9

    # begin crawl and return an iterable

10

    def start_requests(self):

11

        urls = [

12

            'https://quotes.toscrape.com/page/1/',

13

            'https://quotes.toscrape.com/page/2/',

14

        ]

15

        for url in urls:

16

            yield Request(url=url, callback=self.parse)

17

18

    # callback method to handle responses

19

    def parse(self, response):

20

        # get the page number from response

21

        page = response.url.split("/")[-2]

22

        # create a filename as quotes-1.html

23

        filename = f'quotes-{page}.html'

24

        # write into this file

25

        with open(filename, 'wb') as f:

26

            f.write(response.body)

27

        # log the activity

28

        self.log(f'Saved file {filename}')

In cell 1, we define our first Spider subclass. It inherits from scrapy.Spider and defines some attributes and methods:

name: identifies the Spider. It must be unique within a project, that is, you cannot set the same name for different Spiders.

start_requests(): must return an iterable of Requests (you can return a list of requests or write a generator function) which the Spider will begin to crawl from. Subsequent requests will be generated successively from these initial requests.

parse(): a method that will be called to handle the response downloaded for each of the requests made. The response parameter is an instance of TextResponse that holds the page content and has further helpful methods to handle it.

The parse() method usually parses the response, extracting the scraped data as dicts and also finding new URLs to follow and creating new requests (Request) from them.

Now that we have a Spider defined, let’s run it and do our first web scraping.

Run Our Spider

To run a spider in Scrapy, we need to go to Scrapy project’s top-level directory (firstscraping) and run the following command in the terminal:

scrapy crawl quotes

This will run the spider and create two .html files in the Scrapy project directory. See the image below:

Figure 15-4: Running the Spider in firstscraping folder

You should see two html files as: quotes-1.html and quotes-2.html. These are the page contents of the respective URLs. Our parse method gets and saves them.

The start_requests() method in line 10, loops over each url in the urls list. And it schedules an object of type scrapy.Request for each one. When it receives the response for each one, it instantiates a Response object and calls the callback method, which is the parse() method, associated with the request.

The parse() method simply gets the response, splits it and save its body into local html files.

A shortcut to the start_requests() method:

The main idea of the start_requests() method is to take a list of URLs list and loop through each item to initiate a Request object as: Request(url=url, callback=self.parse). However, we do not need to implement this method. There is a shortcut version, that can achieve the same result. There is a built-in attribute called start_urls which is the list of the URLs we want to crawl. If you define this list, Scrapy will create the respective Request objects for each item in it. Just like we did with implementing the start_requests() method.

Let’s redefine the QuotesSpider class by using start_urls this time:

[2]:

1

# redefine the same spider with start_urls

2

class QuotesSpider(Spider):

3

    # unique identifier of this spider

4

    name = "quotes"

5

    # start_urls for url list to crawl automatically

6

    start_urls = [

7

        'https://quotes.toscrape.com/page/1/',

8

        'https://quotes.toscrape.com/page/2/',

9

    ]

10

11

    # callback method to handle responses

12

    def parse(self, response):

13

        # get the page number from response

14

        page = response.url.split("/")[-2]

15

        # create a filename as quotes-1.html

16

        filename = f'quotes-{page}.html'

17

        # write into this file

18

        with open(filename, 'wb') as f:

19

            f.write(response.body)

The code in cell 2 gives exactly the same output with the cell 1. But with less lines of code. The parse() method will be called to handle each of the requests for those URLs in the start_urls list, even though we haven’t explicitly told Scrapy to do so. This happens because parse() is the default callback method, which is called for requests without an explicitly assigned callback.

Scrapy Shell

Before we start to extract data using Scrapy, there is an important tool which you should learn, which is the Scrapy shell. The Scrapy shell is an interactive shell where you can try and debug your scraping code very quickly, without having to run the spider. It’s meant to be used for testing data extraction code, but you can actually use it for testing any kind of code as it is also a regular Python shell.

The shell is used for testing XPath or CSS expressions and see how they work and what data they extract from the web pages you’re trying to scrape. It allows you to interactively test your expressions while you’re writing your spider, without having to run the spider to test every change.

Once you get familiarized with the Scrapy shell, you’ll see that it’s an invaluable tool for developing and debugging your spiders.

To launch the shell simply type the command below in your terminal which you run Scrapy:

scrapy shell <url>

So, to be able to inspect one of the URLs we used in the previous section, we need to run the command below in the firstscraping project:

scrapy shell 'https://quotes.toscrape.com/page/1/'

Please keep in mind that, on Windows, use double quotes instead of single ones. When we run this command successfully, we will have an interactive Scrapy shell.

Here is what we can do with it:

shelp() - print a help with the list of available objects and shortcuts

fetch(url[, redirect=True]) - fetch a new response from the given URL and update all related objects accordingly. You can optionally ask for HTTP 3xx redirections to not be followed by passing redirect=False

fetch(request) - fetch a new response from the given request and update all related objects accordingly.

view(response) - open the given response in your local web browser, for inspection. This will add a <base> tag to the response body in order for external links (such as images and style sheets) to display properly. Note, however, that this will create a temporary file in your computer, which won’t be removed automatically.

Let’s type view(response) command in the Scrapy shell. When you type this command and press Enter, Scrapy shell will open the response html file in your default web browser. This is a temporary file.

The shell creates objects from the downloaded page, like the Response object and the Selector objects (for both HTML and XML content).

Those objects are:

crawler - the current Crawler object.

spider - the Spider which is known to handle the URL, or a Spider object if there is no spider found for the current URL

request - a Request object of the last fetched page. You can modify this request using replace() or fetch a new request (without leaving the shell) using the fetch shortcut.

response - a Response object containing the last fetched page

settings - the current Scrapy settings

Figure 15-5: Scrapy shell (and the objects) in the terminal

Now that we have an active Scrapy shell, we can start playing with objects. We will use XPath expressions here just to get an idea of selecting objects in Scrapy:

Get the page title: '//title/text()'

>>> response.xpath('//title/text()').get()

'Quotes to Scrape'

Fetch another web site

>>> fetch("https://www.amazon.com/Best-Sellers/zgbs")

2022-04-22 22:13:36 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.com/robots.txt> (referer: None)

2022-04-22 22:13:37 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.com/Best-Sellers/zgbs> (referer: None)

Get the page title of the last request (to Amazon best sellers)

>>> response.xpath('//title/text()').get()

'Amazon.com Best Sellers: The most popular items on Amazon'

Print response headers

>>> from pprint import pprint

>>> pprint(response.headers)

{b'Accept-Ch': [b'ect,rtt,downlink,device-memory,sec-ch-device-memory,viewport'

                b'-width,sec-ch-viewport-width,dpr,sec-ch-dpr'],

b'Accept-Ch-Lifetime': [b'86400'],

b'Cache-Control': [b'no-cache'],

b'Content-Language': [b'en-US'],

b'Content-Type': [b'text/html;charset=UTF-8'],

b'Date': [b'Fri, 22 Apr 2022 19:13:37 GMT'],

b'Expires': [b'-1'],

b'Permissions-Policy': [b'interest-cohort=()'],

b'Pragma': [b'no-cache'],

b'Server': [b'Server'],

b'Strict-Transport-Security': [b'max-age=47474747; includeSubDomains; preload'],

b'Vary': [b'Content-Type,Accept-Encoding,X-Amzn-CDN-Cache,X-Amzn-AX-Treatmen'

           b't,User-Agent'],

b'Via': [b'1.1 9987fa8ab620895e83d1d8f10c40f6d2.cloudfront.net (CloudFront)'],

b'X-Amz-Cf-Id': [b'bW1ADijyxFtP4Pa0iLGZ2_z_w_oURnvnrwnDcmDG_UGtfsFviKORXg=='],

b'X-Amz-Cf-Pop': [b'FRA56-P4'],

b'X-Amz-Rid': [b'Y0FBYQZ58BTDJWZEMPA3'],

b'X-Cache': [b'Miss from cloudfront'],

b'X-Content-Type-Options': [b'nosniff'],

b'X-Frame-Options': [b'SAMEORIGIN'],

b'X-Xss-Protection': [b'1;']}

Extracting Data

Now that we have a basic understanding of how Scrapy works, what a Spider is and how to use Scrapy shell, we can start data extraction. The ideal approach is to see what we will extract. We will run a shell on one of our URLs and get some data using CSS selectors.

Open a new terminal, navigate to your Scrapy project top-level folder and type the shell command as:

scrapy shell 'https://quotes.toscrape.com/page/1/'

Let’s use some CSS selectors to extract data:

>>> response.css("title")

[<Selector xpath='descendant-or-self::title' data='<title>Quotes to Scrape</title>'>]

We use response.css() method to use CSS selectors and get data. Here we simply ask for the title tag which is <title>. And as you see in the result, we get this tag. And the result is a list-like object called SelectorList. SelectorList represents a list of Selector objects that wrap around XML/HTML elements and allow us to run further queries to fine-grain data extraction. Let’s extract just the text from title tag above:

>>> response.css('title::text').getall()

['Quotes to Scrape']

We added “::text” to the CSS query above. This means that we want to select only the text elements directly inside the <title> element. And getall() method is used to extract all the elements satisfying this query. If we want to get only the first result, we can simply call the get() method instead. Here is an example:

>>> response.css('title::text')[0].get()

'Quotes to Scrape'

CSS Selectors:

Selectors are patterns that match against elements in a tree, and as such form one of several technologies that can be used to select nodes in a document. Selectors have been optimized for use with HTML and XML, and are designed to be usable in performance-critical code. They are a core component of CSS (Cascading Style Sheets), which uses Selectors to bind style properties to elements in the document.

XPath:

Scrapy supports XPath expressions in addition to CSS selectors. XPath expressions are the foundation of Scrapy Selectors. CSS selectors are converted to XPath under-the-hood. You can see that if you read closely the text representation of the selector objects in the shell.

Now we are ready to complete our first spider by adding some code to extract the quotes from our example web site.

In the https://quotes.toscrape.com each quote is represented by HTML elements which look like the following:

<div class="quote">

    <span class="text">“The world as we have created it is a process of our

    thinking. It cannot be changed without changing our thinking.”</span>

    <span>

        by <small class="author">Albert Einstein</small>

        <a href="/author/Albert-Einstein">(about)</a>

    </span>

    <div class="tags">

        Tags:

        <a class="tag" href="/tag/change/page/1/">change</a>

        <a class="tag" href="/tag/deep-thoughts/page/1/">deep-thoughts</a>

        <a class="tag" href="/tag/thinking/page/1/">thinking</a>

        <a class="tag" href="/tag/world/page/1/">world</a>

    </div>

</div>

Let’s start a Scrapy shell on this website and see how we can extract data out of it:

scrapy shell 'https://quotes.toscrape.com'

And let’s extract the div element which has the class name as “quote”. The CSS selector for this is: div.quote.

>>> quote = response.css("div.quote")[0]

>>> quote

<Selector xpath="descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype...'>

We get the first div element with class name as “quote” and assign to a local variable called quote. Then we simply print it to the shell.

Let’s get more data. Let’s extract text, author and the tags from that quote using the quote object we created:

>>> text = quote.css("span.text::text").get()

>>> text

'“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'

>>> author = quote.css("small.author::text").get()

>>> author

'Albert Einstein'

>>> tags = quote.css("div.tags a.tag::text").getall()

>>> tags

['change', 'deep-thoughts', 'thinking', 'world']

Now that, we know how a quote looks like in the html response, we can set a loop to print the data of each one:

>>> for quote in response.css("div.quote"):

...     text = quote.css("span.text::text").get()

...     author = quote.css("small.author::text").get()

...     tags = quote.css("div.tags a.tag::text").getall()

...     print(dict(text=text, author=author, tags=tags))

...

{'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', 'author': 'Albert Einstein', 'tags': ['change', 'deep-thoughts', 'thinking', 'world']}

{'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”', 'author': 'J.K. Rowling', 'tags': ['abilities', 'choices']}

{'text': '“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”', 'author': 'Albert Einstein', 'tags': ['inspirational', 'life', 'live', 'miracle', 'miracles']}

{'text': '“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”', 'author': 'Jane Austen', 'tags': ['aliteracy', 'books', 'classic', 'humor']}

{'text': "“Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.”", 'author': 'Marilyn Monroe', 'tags': ['be-yourself', 'inspirational']}

{'text': '“Try not to become a man of success. Rather become a man of value.”', 'author': 'Albert Einstein', 'tags': ['adulthood', 'success', 'value']}

{'text': '“It is better to be hated for what you are than to be loved for what you are not.”', 'author': 'André Gide', 'tags': ['life', 'love']}

{'text': "“I have not failed. I've just found 10,000 ways that won't work.”", 'author': 'Thomas A. Edison', 'tags': ['edison', 'failure', 'inspirational', 'paraphrased']}

{'text': "“A woman is like a tea bag; you never know how strong it is until it's in hot water.”", 'author': 'Eleanor Roosevelt', 'tags': ['misattributed-eleanor-roosevelt']}

{'text': '“A day without sunshine is like, you know, night.”', 'author': 'Steve Martin', 'tags': ['humor', 'obvious', 'simile']}

Extracting Data in Our Spider

So far, we learned how to extract data in Scrapy shell. Now, it’s time to do the same thing in our spider. A Scrapy spider can be used to generate dictionaries containing the data extracted from the page. To do that, we use the yield keyword in the callback. Let’s define a new Spider named QuoteSpiderExtractor inside a new Python file named quote_spider_extractor.py under the “spiders” folder.

[3]:

1

# quote_spider_extractor.py file

2

3

from scrapy import Spider, Request

4

5

class QuoteSpiderExtractor(Spider):

6

    # unique identifier of this spider

7

    name = "quotes_extractor"

8

    # start_urls for url list to crawl automatically

9

    start_urls = [

10

        'https://quotes.toscrape.com/page/1/',

11

        'https://quotes.toscrape.com/page/2/',

12

    ]

13

14

    # callback method to handle responses

15

    def parse(self, response):

16

        # get the div elements with class name quote

17

        for quote in response.css('div.quote'):

18

            # yield a quote dictionary

19

            yield {

20

                'text': quote.css('span.text::text').get(),

21

                'author': quote.css('small.author::text').get(),

22

                'tags': quote.css('div.tags a.tag::text').getall()

23

            }

Let’s run this spider and see the output. Data will be extracted in the logs in terminal. And you will see a dictionary for each quote.

scrapy crawl quotes_extractor

…

2022-04-23 10:43:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://quotes.toscrape.com/page/1/>

{'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”', 'author': 'J.K. Rowling', 'tags': ['abilities', 'choices']}

2022-04-23 10:43:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://quotes.toscrape.com/page/1/>

{'text': '“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”', 'author': 'Albert Einstein', 'tags': ['inspirational', 'life', 'live', 'miracle', 'miracles']}

2022-04-23 10:43:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://quotes.toscrape.com/page/1/>

{'text': '“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”', 'author': 'Jane Austen', 'tags': ['aliteracy', 'books', 'classic', 'humor']}

2022-04-23 10:43:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://quotes.toscrape.com/page/1/>

{'text': "“Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.”", 'author': 'Marilyn Monroe', 'tags': ['be-yourself', 'inspirational']}

…

Storing the Scraped Data

Now let’s assume we want to store the scraped data instead of just printing them to the log console. The simplest way to store the scraped data is by using Feed exports, with the following command:

scrapy crawl quotes_extractor -O quotes_extractor.json

This command will generate a quotes_extractor.json file containing all scraped items, serialized in JSON. The -O (capital letter ‘O’) command-line switch overwrites any existing file; use -o (lower case letter ‘o’) instead to append new content to any existing file.

You should see a new file in your Scrapy project. See the image below:

Figure 15-6: quotes_extractor.json file in Scrapy project

Following Links

In the previous sections we mainly scraped a single page. Now, let’s see how we can scrape all the pages from https://quotes.toscrape.com web site. To do this, we need to learn how to follow the links. In most cases, the links are just <a> tags.

To extract a link in an HTML structure, we need to examine it. Here is, how we can examine the underlying HTML of a page in Google Chrome. Right click on the element you want to examine and click “Inspect”. See the image below:

Figure 15-7: Inspecting elements in Google Chrome

When you inspect an element in Google Chrome, it will open the Chrome Developer Tools extension. Inside the extension, under the Elements tab, you can see the entire HTML structure of the element.

The link which we want to inspect is the “Next” button, which contains a link to the second page. We need this element because we want to get the next page in our Spider. Here is the HTML code of this element:

<ul class="pager">

    <li class="next">

        <a href="/page/2/">Next <span aria-hidden="true">→</span></a>

    </li>

</ul>

The link here is the href attribute of the <a> tag and let’s extract it in our Spider now. Create a new Spider named QuoteSpiderFollowLinks inside a new Python file named quote_spider_follow_links.py under the “spiders” folder.

[4]:

1

# quote_spider_follow_links.py file

2

3

from scrapy import Spider, Request

4

5

class QuoteSpiderFollowLinks(Spider):

6

    # unique identifier of this spider

7

    name = "quotes_follow_links"

8

    # start_urls for url list to crawl automatically

9

    start_urls = [

10

        'https://quotes.toscrape.com/page/1/',

11

    ]

12

13

    # callback method to handle responses

14

    def parse(self, response):

15

        # get the div elements with class name quote

16

        for quote in response.css('div.quote'):

17

            # yield a quote dictionary

18

            yield {

19

                'text': quote.css('span.text::text').get(),

20

                'author': quote.css('small.author::text').get(),

21

                'tags': quote.css('div.tags a.tag::text').getall(),

22

            }

23

        # get next page element via the href attribute

24

        next_page = response.css('li.next a::attr(href)').get()

25

        if next_page is not None:

26

            # create the next page url

27

            next_page = response.urljoin(next_page)

28

            # request the next page with Scrapy Request

29

            yield Request(next_page, callback=self.parse)

In cell 4, we define a Spider which follows the links in a web site. After extracting the data, the parse() method looks for the link to the next page, builds a full absolute URL using the urljoin() method (since the links can be relative) and yields a new request to the next page, registering itself as callback to handle the data extraction for the next page and to keep the crawling going through all the pages.

This shows us how Scrapy follows the links. When it yields a Request in the callback method, it will schedule that request to be sent and register a callback method to be executed when that request finishes.

You can think this structure as some sort of a loop, which follows all the links to the next page until it doesn’t find one. This is very useful especially with sites that have pagination. You can implement a save mechanism in the parse() method depending on the page you visit.

Shortcut for Creating Requests

In the previous section we used the Request() constructor method to create and yield a Request object in the parse() method. However, we have a shortcut for creating Request objects. The syntax is: response.follow(). Let’s redefine the QuoteSpiderFollowLinks class with this shortcut this time:

[5]:

1

####  response.follow()  ###

2

class QuoteSpiderFollowLinks(Spider):

3

    # unique identifier of this spider

4

    name = "quotes_follow_links"

5

    # start_urls for url list to crawl automatically

6

    start_urls = [

7

        'https://quotes.toscrape.com/page/1/',

8

    ]

9

10

    # callback method to handle responses

11

    def parse(self, response):

12

        # get the div elements with class name quote

13

        for quote in response.css('div.quote'):

14

            # yield a quote dictionary

15

            yield {

16

                'text': quote.css('span.text::text').get(),

17

                'author': quote.css('small.author::text').get(),

18

                'tags': quote.css('div.tags a.tag::text').getall(),

19

            }

20

        # get next page element via the href attribute

21

        next_page = response.css('li.next a::attr(href)').get()

22

        if next_page is not None:

23

            # yield a Request object with response.follow()

24

            yield response.follow(next_page, callback=self.parse)

While using scrapy.Request we need to call the urljoin() method to create relative URLs.  However, with response.follow() method this is not necessary because it supports relative URLs out of the box. In line 24, we call response.follow() which returns a Request instance. Then, we yield this object.

It is also possible to pass a selector to the response.follow() method, which makes your code shorter:

[6]:

1

for href in response.css('ul.pager a::attr(href)'):

2

    yield response.follow(href, callback=self.parse)

One of the best things about the response.follow() method is that, it has a built-in mechanism to follow the <a> elements. It uses their href attributes automatically for following. So, we do not need to explicitly get the href attribute. Here is how:

[7]:

1

for a in response.css('ul.pager a'):

2

    yield response.follow(a, callback=self.parse)

In cell 7, we pass the <a> tag to the response.follow() method and it uses the href attributes automatically to follow.

response.follow_all(urls, callback):

If you need to create multiple requests from an iterable, there is a built-in method which is response.follow_all(). It is a generator that produces Request instances to follow all links in urls parameter. Let’s see how to use it:

[8]:

1

anchors = response.css('ul.pager a')

2

yield from response.follow_all(anchors, callback=self.parse)

In cell 8, we first get all the anchor tags (<a>) in line 1. Then in the second line, we call the response.follow_all() method by passing these anchors. And since it is a generator, we need to yield it as “yield from”.

This finalizes our project on web scraping with Python. Scrapy has lots of features which will help you to build complex and effective web crawlers. However, for the sake of this project, we believe it is enough to give you a basic understating on how web scraping and Scrapy works in Python.

In the next chapter, you will have an assignment related to this project. Good luck with your assignment.

16.        Assignment 2 – Web Scraping with Python

Assignment Setup

We finished the second project in this book which is Project 2 – Web Scraping with Python. Now it’s your turn to recreate the same project. This chapter is the assignment on the Project 2.

In this assignment you will start by installing the Scrapy package. You will use Scrapy Shell to inspect HTML structure of web pages. You create Spider classes and use them to extract and save scraped data.

To complete the assignment, follow the instructions where you see a #TODO statement. You can find the solutions in the previous chapter.

We will create a new PyCharm project. The project name is WebScraping. You should download the project from the GitHub Repository of this book before starting to code.

Chapter Outline:

Install Scrapy

Create a Scrapy Project

Define Our First Spider

Run Our Spider

Extracting Data

Extracting Data in Our Spider

Storing the Scraped Data

Following Links

Shortcut for Creating Requests

Install Scrapy

In this project we will use Scrapy as our primary tool for web crawling. Scrapy is an open source and collaborative framework built with Python, for extracting the data you need from websites. It is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like data mining, information processing or historical archival. Scrapy is one of the most widely used tool for large web crawling projects.

To install Scrapy we will simply use our PyCharm IDE. Here you can find the complete installation document. Let’s start by creating a new PyCharm project named WebScraping. Make sure you create your project with a new virtual environment. Inside PyCharm, navigate to Preferences, and under Project sub-section on the left, click Python Interpreter. Here we will install Scrapy package. See the image below:

Figure 15-1: Installing Scrapy in PyCharm IDE

Scrapy has some dependencies and PyCharm will install them too. So, it might take a bit long to install Scrapy and all the dependencies.

Scrapy is written in pure Python and depends on a few key Python packages (among others):

lxml, an efficient XML and HTML parser

parsel, an HTML/XML data extraction library written on top of lxml,

w3lib, a multi-purpose helper for dealing with URLs and web page encodings

twisted, an asynchronous networking framework

cryptography and pyOpenSSL, to deal with various network-level security needs

Create a Scrapy Project

Before we start using Scrapy we need to set up a new Scrapy project. In our WebScraping PyCharm project we will use Python terminal to create and run Scrapy project. You can have one or multiple Scrapy projects in a single PyCharm project. Let’s create the first one:

In PyCharm, right click on the project name and select Open In -> Terminal:

Figure 15-2: Open a Terminal Window in PyCharm

This will open a new Terminal Window in PyCharm. Here is the command to create a new Scrapy project:

scrapy startproject firstscraping

You will see a new directory in your PyCharm project named firstscraping when you run this code.

Figure 15-3: Scrapy project inside PyCharm project

Here are the files in our Scrapy project:

firstscraping/

    scrapy.cfg            # deploy configuration file

    firstscraping/        # project's Python module, you'll import your code from here

        __init__.py

        items.py          # project items definition file

        middlewares.py    # project middlewares file

        pipelines.py      # project pipelines file

        settings.py       # project settings file

        spiders/          # a directory where you'll later put your spiders

            __init__.py

Now that we successfully create a Scrapy project, let’s start by defining our first Spider class.

Define Our First Spider

Spiders are classes which we define and which Scrapy uses to scrape information from a website (or a group of websites). They must inherit from scrapy.Spider class and define the initial requests to make, optionally how to follow links in the pages, and how to parse the downloaded page content to extract data.

To learn the basic concepts of Scrapy we will be working on a dedicated web site which is https://quotes.toscrape.com. It is a simple web site, which has a list of famous quotes on different topics.

Create a new Python file named quotes_spider.py. under the firstscraping/spiders directory. And here is the content of this file:

[1]:

1

# quotes_spider.py file

2

3

from scrapy import Spider, Request

4

5

class QuotesSpider(Spider):

6

    # unique identifier of this spider

7

    name = "quotes"

8

9

    # begin crawl and return an iterable

10

    def start_requests(self):

11

        urls = [

12

            'https://quotes.toscrape.com/page/1/',

13

            'https://quotes.toscrape.com/page/2/',

14

        ]

15

        for url in urls:

16

            # TODO - yield Request objects

17

18

    # callback method to handle responses

19

    def parse(self, response):

20

        # get the page number from response

21

        page = # TODO - get page number from response.url

22

        # create a filename as quotes-1.html

23

        filename = f'quotes-{page}.html'

24

        # write into this file

25

        with open(filename, 'wb') as f:

26

            # TODO - write the response body

27

        # log the activity

28

        self.log(f'Saved file {filename}')

In cell 1, we define our first Spider subclass. It inherits from scrapy.Spider and defines some attributes and methods:

name: identifies the Spider. It must be unique within a project, that is, you can’t set the same name for different Spiders.

start_requests(): must return an iterable of Requests (you can return a list of requests or write a generator function) which the Spider will begin to crawl from. Subsequent requests will be generated successively from these initial requests.

parse(): a method that will be called to handle the response downloaded for each of the requests made. The response parameter is an instance of TextResponse that holds the page content and has further helpful methods to handle it.

The parse() method usually parses the response, extracting the scraped data as dicts and also finding new URLs to follow and creating new requests (Request) from them.

Now that we have a Spider defined, let’s run it and do our first web scraping.

Run Our Spider

To run a spider in Scrapy, we need to go to Scrapy project’s top-level directory (firstscraping) and run the following command in the terminal:

scrapy crawl quotes

This will run the spider and create two .html files in the Scrapy project directory. See the image below:

Figure 15-4: Running the Spider in firstscraping folder

You should see two html files as: quotes-1.html and quotes-2.html. These are the page contents of the respective URLs. Our parse method gets and saves them.

The start_requests() method in line 10, loops over each url in the urls list. And it schedules an object of type scrapy.Request for each one. When it receives the response for each one, it instantiates a Response object and calls the callback method, which is the parse() method, associated with the request.

The parse() method simply gets the response, splits it and save its body into local html files.

A shortcut to the start_requests() method:

The main idea of the start_requests() method is to take a list of URLs list and loops through each item to initiate a Request object as: Request(url=url, callback=self.parse). However, we do not need to implement this method. There is a shortcut version, that can achieve the same result. There is a built-in attribute called start_urls which is the list of the URLs we want to crawl. If you define this list, Scrapy will create the respective Request objects for each item in it. Just like we did with implementing the start_requests() method.

Let’s redefine the QuotesSpider class by using start_urls this time:

[2]:

1

# redefine the same spider with start_urls

2

class QuotesSpider(Spider):

3

    # unique identifier of this spider

4

    name = # TODO - define a unique name for spider

5

    # start_urls for url list to crawl automatically

6

    start_urls = # TODO - define urls list

7

8

    # callback method to handle responses

9

    def parse(self, response):

10

        # get the page number from response

11

        page = # TODO - get page number from response.url

12

        # create a filename as quotes-1.html

13

        filename = # TODO - create a page name

14

        # write into this file

15

        with # TODO - open filename in write binary mode:

16

            # TODO - write the response body

The code in cell 2 gives exactly the same output with the cell 1. But with less lines of code. The parse() method will be called to handle each of the requests for those URLs in the start_urls list, even though we haven’t explicitly told Scrapy to do so. This happens because parse() is the default callback method, which is called for requests without an explicitly assigned callback.

Scrapy Shell

Before start to extract data using Scrapy, there is an important tool which you should learn, which is the Scrapy shell. The Scrapy shell is an interactive shell where you can try and debug your scraping code very quickly, without having to run the spider. It’s meant to be used for testing data extraction code, but you can actually use it for testing any kind of code as it is also a regular Python shell.

The shell is used for testing XPath or CSS expressions and see how they work and what data they extract from the web pages you’re trying to scrape. It allows you to interactively test your expressions while you’re writing your spider, without having to run the spider to test every change.

Once you get familiarized with the Scrapy shell, you’ll see that it’s an invaluable tool for developing and debugging your spiders.

To launch the shell simply type the command below in your terminal which you run Scrapy:

scrapy shell <url>

So, to be able to inspect one of the URLs we used in the previous section, we need to run the command below in the firstscraping project:

scrapy shell 'https://quotes.toscrape.com/page/1/'

Please keep in mind that, on Windows, use double quotes instead of single ones. When we run this command successfully, we will have an interactive Scrapy shell.

Here is what we can do with it:

shelp() - print a help with the list of available objects and shortcuts

fetch(url[, redirect=True]) - fetch a new response from the given URL and update all related objects accordingly. You can optionally ask for HTTP 3xx redirections to not be followed by passing redirect=False

fetch(request) - fetch a new response from the given request and update all related objects accordingly.

view(response) - open the given response in your local web browser, for inspection. This will add a <base> tag to the response body in order for external links (such as images and style sheets) to display properly. Note, however, that this will create a temporary file in your computer, which won’t be removed automatically.

Let’s type view(response) command in the Scrapy shell. When you type this command and press Enter, Scrapy shell will open the response html file in your default web browser. This is a temporary file.

The shell creates objects from the downloaded page, like the Response object and the Selector objects (for both HTML and XML content).

Those objects are:

crawler - the current Crawler object.

spider - the Spider which is known to handle the URL, or a Spider object if there is no spider found for the current URL

request - a Request object of the last fetched page. You can modify this request using replace() or fetch a new request (without leaving the shell) using the fetch shortcut.

response - a Response object containing the last fetched page

settings - the current Scrapy settings

Figure 15-5: Scrapy shell (and the objects) in the terminal

Now that we have an active Scrapy shell, we can start playing with objects. We will use XPath expressions here just to get an idea of selecting objects in Scrapy.

Get the page title: '//title/text()'

>>> response.xpath('//title/text()').get()

'Quotes to Scrape'

Fetch another web site

>>> fetch("https://www.amazon.com/Best-Sellers/zgbs")

2022-04-22 22:13:36 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.com/robots.txt> (referer: None)

2022-04-22 22:13:37 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.com/Best-Sellers/zgbs> (referer: None)

Get the page title of the last request (to Amazon best sellers)

>>> response.xpath('//title/text()').get()

'Amazon.com Best Sellers: The most popular items on Amazon'

Print response headers

>>> from pprint import pprint

>>> pprint(response.headers)

{b'Accept-Ch': [b'ect,rtt,downlink,device-memory,sec-ch-device-memory,viewport'

                b'-width,sec-ch-viewport-width,dpr,sec-ch-dpr'],

b'Accept-Ch-Lifetime': [b'86400'],

b'Cache-Control': [b'no-cache'],

b'Content-Language': [b'en-US'],

b'Content-Type': [b'text/html;charset=UTF-8'],

b'Date': [b'Fri, 22 Apr 2022 19:13:37 GMT'],

b'Expires': [b'-1'],

b'Permissions-Policy': [b'interest-cohort=()'],

b'Pragma': [b'no-cache'],

b'Server': [b'Server'],

b'Strict-Transport-Security': [b'max-age=47474747; includeSubDomains; preload'],

b'Vary': [b'Content-Type,Accept-Encoding,X-Amzn-CDN-Cache,X-Amzn-AX-Treatmen'

           b't,User-Agent'],

b'Via': [b'1.1 9987fa8ab620895e83d1d8f10c40f6d2.cloudfront.net (CloudFront)'],

b'X-Amz-Cf-Id': [b'bW1ADijyxFtP4Pa0iLGZ2_z_w_oURnvnrwnDcmDG_UGtfsFviKORXg=='],

b'X-Amz-Cf-Pop': [b'FRA56-P4'],

b'X-Amz-Rid': [b'Y0FBYQZ58BTDJWZEMPA3'],

b'X-Cache': [b'Miss from cloudfront'],

b'X-Content-Type-Options': [b'nosniff'],

b'X-Frame-Options': [b'SAMEORIGIN'],

b'X-Xss-Protection': [b'1;']}

Extracting Data

Now that we have a basic understanding of how Scrapy works, what a Spider is and how to use Scrapy shell, we can start data extraction. The ideal approach is to see what we will extract. We will run a shell on one of our URLs and get some data using CSS selectors.

Open a new terminal, navigate to your Scrapy project top-level folder and type the shell command as:

scrapy shell 'https://quotes.toscrape.com/page/1/'

Let’s use some CSS selectors to extract data:

>>> response.css("title")

[<Selector xpath='descendant-or-self::title' data='<title>Quotes to Scrape</title>'>]

We use response.css() method to use CSS selectors and get data. Here we simply ask for the title tag which is <title>. And as you see in the result, we get his tag. And the result is a list-like object called SelectorList. SelectorList represents a list of Selector objects that wrap around XML/HTML elements and allow us to run further queries to fine-grain data extraction. Let’s extract just the text from title tag above:

>>> response.css('title::text').getall()

['Quotes to Scrape']

We added ::text to the CSS query above. This means that we want to select only the text elements directly inside the <title> element. And getall() method is used to extract all the elements satisfying this query. If we want to get only the first result, we can simply call the get() method instead. Here is an example:

>>> response.css('title::text')[0].get()

'Quotes to Scrape'

CSS Selectors:

Selectors are patterns that match against elements in a tree, and as such form one of several technologies that can be used to select nodes in a document. Selectors have been optimized for use with HTML and XML, and are designed to be usable in performance-critical code. They are a core component of CSS (Cascading Style Sheets), which uses Selectors to bind style properties to elements in the document.

XPath:

Scrapy supports XPath expressions in addition to CSS selectors. XPath expressions are the foundation of Scrapy Selectors. CSS selectors are converted to XPath under-the-hood. You can see that if you read closely the text representation of the selector objects in the shell.

Now we are ready to complete our first spider by adding some code to extract the quotes from our example web site.

In the https://quotes.toscrape.com each quote is represented by HTML elements which look like the following:

<div class="quote">

    <span class="text">“The world as we have created it is a process of our

    thinking. It cannot be changed without changing our thinking.”</span>

    <span>

        by <small class="author">Albert Einstein</small>

        <a href="/author/Albert-Einstein">(about)</a>

    </span>

    <div class="tags">

        Tags:

        <a class="tag" href="/tag/change/page/1/">change</a>

        <a class="tag" href="/tag/deep-thoughts/page/1/">deep-thoughts</a>

        <a class="tag" href="/tag/thinking/page/1/">thinking</a>

        <a class="tag" href="/tag/world/page/1/">world</a>

    </div>

</div>

Let’s start a Scrapy shell on this website and see how we can extract data out of it:

scrapy shell 'https://quotes.toscrape.com'

And let’s extract the div element which has the class name as “quote”. The CSS selector for this is: div.quote.

>>> quote = response.css("div.quote")[0]

>>> quote

<Selector xpath="descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype...'>

We get the first div element with class name as “quote” and assign to a local variable called quote. Then we simply print it to the shell.

Let’s get more data. Let’s extract text, author and the tags from that quote using the quote object we created:

>>> text = quote.css("span.text::text").get()

>>> text

'“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'

>>> author = quote.css("small.author::text").get()

>>> author

'Albert Einstein'

>>> tags = quote.css("div.tags a.tag::text").getall()

>>> tags

['change', 'deep-thoughts', 'thinking', 'world']

Now that, we know how a quote looks like in the html response, we can set a loop to print the data of each one:

>>> for quote in response.css("div.quote"):

...     text = quote.css("span.text::text").get()

...     author = quote.css("small.author::text").get()

...     tags = quote.css("div.tags a.tag::text").getall()

...     print(dict(text=text, author=author, tags=tags))

...

{'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', 'author': 'Albert Einstein', 'tags': ['change', 'deep-thoughts', 'thinking', 'world']}

{'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”', 'author': 'J.K. Rowling', 'tags': ['abilities', 'choices']}

{'text': '“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”', 'author': 'Albert Einstein', 'tags': ['inspirational', 'life', 'live', 'miracle', 'miracles']}

{'text': '“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”', 'author': 'Jane Austen', 'tags': ['aliteracy', 'books', 'classic', 'humor']}

{'text': "“Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.”", 'author': 'Marilyn Monroe', 'tags': ['be-yourself', 'inspirational']}

{'text': '“Try not to become a man of success. Rather become a man of value.”', 'author': 'Albert Einstein', 'tags': ['adulthood', 'success', 'value']}

{'text': '“It is better to be hated for what you are than to be loved for what you are not.”', 'author': 'André Gide', 'tags': ['life', 'love']}

{'text': "“I have not failed. I've just found 10,000 ways that won't work.”", 'author': 'Thomas A. Edison', 'tags': ['edison', 'failure', 'inspirational', 'paraphrased']}

{'text': "“A woman is like a tea bag; you never know how strong it is until it's in hot water.”", 'author': 'Eleanor Roosevelt', 'tags': ['misattributed-eleanor-roosevelt']}

{'text': '“A day without sunshine is like, you know, night.”', 'author': 'Steve Martin', 'tags': ['humor', 'obvious', 'simile']}

Extracting Data in Our Spider

So far, we learned how to extract data in Scrapy shell. Now, it’s time to do the same thing in our spider. A Scrapy spider can be used to generate dictionaries containing the data extracted from the page. To do that, we use the yield keyword in the callback. Let’s define a new Spider named QuoteSpiderExtractor inside a new Python file named quote_spider_extractor.py under the “spiders” folder.

[3]:

1

# quote_spider_extractor.py file

2

3

# TODO - import Spider and Request classes

4

#  from scrapy package

5

6

class QuoteSpiderExtractor(Spider):

7

    # unique identifier of this spider

8

    name =  # TODO - define a unique name for spider

9

    # start_urls for url list to crawl automatically

10

    start_urls = # TODO - define urls list

11

12

    # callback method to handle responses

13

    def parse(self, response):

14

        # get the div elements with class name quote

15

        for quote in # TODO - get div elements with class 'quote' in response:

16

            # yield a quote dictionary

17

            yield {

18

                'text': # TODO - get the text attribute of

19

                        # span elements with class "text"

20

                'author': # TODO - get the text attribute of

21

                          # small elements with class "author"

22

                'tags': quote.css('div.tags a.tag::text').getall()

23

            }

Let’s run this spider and see the output. Data will be extracted in the logs in terminal. And you will see a dictionary for each quote.

scrapy crawl quotes_extractor

…

2022-04-23 10:43:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://quotes.toscrape.com/page/1/>

{'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”', 'author': 'J.K. Rowling', 'tags': ['abilities', 'choices']}

2022-04-23 10:43:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://quotes.toscrape.com/page/1/>

{'text': '“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”', 'author': 'Albert Einstein', 'tags': ['inspirational', 'life', 'live', 'miracle', 'miracles']}

2022-04-23 10:43:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://quotes.toscrape.com/page/1/>

{'text': '“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”', 'author': 'Jane Austen', 'tags': ['aliteracy', 'books', 'classic', 'humor']}

2022-04-23 10:43:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://quotes.toscrape.com/page/1/>

{'text': "“Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.”", 'author': 'Marilyn Monroe', 'tags': ['be-yourself', 'inspirational']}

…

Storing the Scraped Data

Now let’s assume we want to store the scraped data instead of just printing them to the log console. The simplest way to store the scraped data is by using Feed exports, with the following command:

scrapy crawl quotes_extractor -O quotes_extractor.json

This command will generate a quotes_extractor.json file containing all scraped items, serialized in JSON. The -O (capital letter ‘O’) command-line switch overwrites any existing file; use -o (lower case letter ‘o’) instead to append new content to any existing file.

You should see a new file in your Scrapy project. See the image below:

Figure 15-6: quotes_extractor.json file in Scrapy project

Following Links

In the previous sections we mainly scraped a single page. Now, let’s see how we can scrape all the pages from https://quotes.toscrape.com web site. To do this, we need to learn how to follow the links. In most cases, the links are just <a> tags.

To extract a link in an HTML structure, we need to examine it. Here is, how we can examine the underlying HTML of a page in Google Chrome. Right click on the element you want to examine and click “Inspect”. See the image below:

Figure 15-7: Inspecting elements in Google Chrome

When you inspect an element in Google Chrome, it will open the Chrome Developer Tools extension. Inside the extension, under the Elements tab, you can see the entire HTML structure of the element.

The link which we want to inspect is the “Next” button, which contains a link to the second page. We need this element because we want to get the next page in our Spider. Here is the HTML code of this element:

<ul class="pager">

    <li class="next">

        <a href="/page/2/">Next <span aria-hidden="true">→</span></a>

    </li>

</ul>

The link here is the href attribute of the <a> tag here and let’s extract it in our Spider now. Create a new Spider named QuoteSpiderFollowLinks inside a new Python file named quote_spider_follow_links.py under the “spiders” folder.

[4]:

1

# quote_spider_follow_links.py file

2

3

from scrapy import Spider, Request

4

5

class QuoteSpiderFollowLinks(Spider):

6

    # unique identifier of this spider

7

    name = # TODO - define a unique name for spider

8

    # start_urls for url list to crawl automatically

9

    start_urls = # TODO - define urls list

10

11

    # callback method to handle responses

12

    def parse(self, response):

13

        # get the div elements with class name quote

14

        for quote in  # TODO - get div elements with class 'quote' in response:

15

            # yield a quote dictionary

16

            yield {

17

                'text':  # TODO - get the text attribute of

18

                # span elements with class "text"

19

                    'author':  # TODO - get the text attribute of

20

            # small elements with class "author"

21

            'tags': quote.css('div.tags a.tag::text').getall()

22

            }

23

24

        # get next page element via the href attribute

25

        next_page = # TODO - get next page element via the href attribute

26

                    # find li with class "next" and the <a> tag in it

27

        if next_page is not None:

28

            # create the next page url

29

            next_page = response.urljoin(next_page)

30

            # request the next page with Scrapy Request

31

            # TODO - yield the Request object

In cell 4, we define a Spider which follows the links in a web site. After extracting the data, the parse() method looks for the link to the next page, builds a full absolute URL using the urljoin() method (since the links can be relative) and yields a new request to the next page, registering itself as callback to handle the data extraction for the next page and to keep the crawling going through all the pages.

This shows us how Scrapy follows the links. When it yields a Request in the callback method, it will schedule that request to be sent and register a callback method to be executed when that request finishes.

You can think this structure as some sort of loop, which follows all the links to the next page until it doesn’t find one. This is very useful especially with sites that have pagination. You can implement a save mechanism in the parse() method depending on the page you visit now.

Shortcut for Creating Requests

In the previous section we used the Request() constructor method to create and yield a Request object in the parse() method. However, we have a shortcut for creating Request objects. The syntax is: response.follow(). Let’s redefine the QuoteSpiderFollowLinks class with this shortcut this time:

[5]:

1

####  response.follow()  ###

2

class QuoteSpiderFollowLinks(Spider):

3

    # unique identifier of this spider

4

    name =  # TODO - define a unique name for spider

5

    # start_urls for url list to crawl automatically

6

    start_urls =  # TODO - define urls list

7

8

    # callback method to handle responses

9

    def parse(self, response):

10

        # get the div elements with class name quote

11

        for quote in response.css('div.quote'):

12

            # yield a quote dictionary

13

            yield {

14

                'text':  # TODO - get the text attribute of

15

                # span elements with class "text"

16

                    'author':  # TODO - get the text attribute of

17

            # small elements with class "author"

18

            'tags': quote.css('div.tags a.tag::text').getall()

19

            }

20

        # get next page element via the href attribute

21

        next_page = # TODO - get next page element via the href attribute

22

        if next_page is not None:

23

            # TODO - yield a Request object with response.follow()

While using scrapy.Request we need to call the urljoin() method to create relative URLs.  However, with response.follow() method this is not necessary because it supports relative URLs out of the box. In line 24, we call response.follow() which returns a Request instance. Then, we yield this object.

It is also possible to pass a selector to the response.follow() method, which makes your code shorter:

[6]:

1

for href in response.css('ul.pager a::attr(href)'):

2

    yield response.follow(href, callback=self.parse)

One of the best things about the response.follow() method is that, it has a built-in mechanism to follow the <a> elements. It uses their href attributes automatically for following. So, we do not need to explicitly get the href attribute. Here is how:

[7]:

1

for a in response.css('ul.pager a'):

2

    yield response.follow(a, callback=self.parse)

In cell 7, we pass the <a> tag to the response.follow() method and it uses the href attributes automatically to follow.

response.follow_all(urls, callback):

If you need to create multiple requests from an iterable, there is a built-in method which is response.follow_all(). It is a generator that produces Request instances to follow all links in urls parameter. Let’s see how to use it:

[8]:

1

anchors = response.css('ul.pager a')

2

yield from response.follow_all(anchors, callback=self.parse)

In cell 8, we first get all the anchor tags (<a>) in line 1. Then in the second line, we call the response.follow_all() method by passing these anchors. And since it is a generator, we need to yield it as “yield from”.

This finalizes our project on web scraping with Python. Scrapy has lots of features which will help you to build complex and effective web crawlers. However, for the sake of this project, we believe it is enough to give you a basic understating on how web scraping and Scrapy works in Python.

17.        Project 3 – API Development with Flask

In this project, we will create a simple movie review web application using Flask and Python. You can find the PyCharm project for this chapter in the GitHub Repository of this book.

Chapter Outline:

Project Layout

Introduction to Flask

The Application Factory

Create the Database

Movies Data and The Utils Module

Views and Blueprints

Register Blueprint and View Function

Templates

The Base Template

Register Template

Login Blueprint & Template

Home Blueprint & Template

Review Blueprint & Template

Static Files

Project Layout

Our project is a simple app which we will use for reviewing some movies in the IMDB Top 250 list. The movie data will be provided as a CSV file in the project. We will use the built-in SQLite database for this project. Here are the pages in our application:

Home Page:

In the home page, we will see the list of movies in a responsive grid. By clicking any of the movies, you can either create a review or update/delete it if it is already created. The movies with a review will be displayed with a green border around them.

The first tile in the grid is the “All Reviews” button, which will navigate you to a list of all reviews.

We have a menu bar on top of the page. On the menu bar on the left, we have the app title which is also an anchor tag for the Home Page. On the right, you see the logged in user’s full name and a Logout button.

Figure 17-1: Home Page

Register:

To be able to use the app, the user needs to register and login in to the system. We have two different pages for this. The first page is the Register page. In which, user will enter the full name, user name and a password. All of which are required to fill.

Figure 17-2: Register

Login:

After successful registration, user needs to login in. We save user data in the user table, so, the user can login any time with correct user name and password. Inside the app, we check if the credentials are true or not. If not, we notify the user to try again.

User can easily navigate to the Register page from the Login screen.

Figure 17-3: Login

Create a Review:

The logged in user can create a review for any movie on the Home Page. By clicking its tile in the Home Page, the user will be navigated to the Create page. In Figure 17-4, in the page url, you can see the Id of the movie, for which the user will create a review.

Figure 17-4: Create a Review

Update the Review:

The user can update a review, by clicking its tile in the Home Page. He/She will be navigated to the Update page. In Figure 17-5, in the page url, you can see the Id of the movie, which the user will update the existing review.

Before leaving the Home Page, we check if there is already a review for that movie. If there is, then the Update page will render. If not, then the user will see the Create page.

Figure 17-5: Update or Delete a Review

Reviews:

In the Home Page, we have a tile called “All Reviews”. When clicked, it will navigate the user to the Reviews Page. This is where we see all the reviews in the database. If the review is created by the current user, then you will see an “Edit” button on the right-hand side of that review. So, the user can view all of the reviews, but can edit only the ones which he/she created.

Figure 17-6: All Reviews

Introduction to Flask

We will use Flask for building our app. Flask was created by Armin Ronacher in 2004. It is a microframework written in Python. It is a lightweight WSGI (Web Server Gateway Interface) web application framework, which is designed to make getting started quick and easy, with the ability to scale up to complex applications. Flask supports extensions that can add application features like database abstraction layer, form validation, etc.

A Flask application is an instance of the Flask class. Everything about the application, such as configuration and URLs, will be registered with this class.

To understand Flask better, let’s start with a small example. We will create a simple web API, which simply renders “Hi from Flask App” in the web browser.

We will create a new PyCharm project named MovieReview with a new virtual environment. And create a Python file named hi.py.

We have to install the Flask package first. You can easily install it in PyCharm or you can use pip to install it. Here is the official documentation if you need a detailed installation guideline.

Figure 17-7: Install Flask package in PyCharm IDE

[1]:

1

# hi.py file

2

3

# import Flask class from flask package

4

from flask import Flask

5

6

# create a Flask app

7

app = Flask(__name__)

8

9

# define a root

10

@app.route('/hi')

11

def hi_flask():

12

    return "Hi from Flask App"

In cell 1, we have a very simple Flask app in the hi.py file. We import the Flask class first. Then in line 7, we create an instance of the Flask class and name it simply as app. This object will be our WSGI application. The first argument is the name of the application’s module or package. __name__ is a convenient shortcut for this that is appropriate for most cases. This is needed so that Flask knows where to look for resources such as templates and static files.

In line 10, we use the route() decorator to tell Flask what URL should trigger the function in the next line. Here, the URL will be ‘/hi’. Our hi_flask() function will return a text message which will be displayed in the user’s browser.

To run a Flask application, we need to use the ‘flask’ command or ‘python -m flask’ command. In this project, we prefer to use the ‘flask’ command. We need to make some settings in the terminal before running the ‘flask’ command. We need to tell the terminal the application to work with by exporting the FLASK_APP environment variable. Here are the commands for this, on different operating systems:

Bash (macOS and Linux):

------------------------

$ export FLASK_APP=hi

$ export FLASK_ENV=development

$ flask run

 * Running on http://127.0.0.1:5000/

CMD (Windows):

------------------------

> set FLASK_APP=hi

> set FLASK_ENV=development

> flask run

 * Running on http://127.0.0.1:5000/

Powershell (Windows):

------------------------

> $env:FLASK_APP = "hi"

> $env:FLASK_ENV = "development"

> flask run

 * Running on http://127.0.0.1:5000/

To be able to run these commands in your terminal (either standalone or in PyCharm) you need to be in the top-level directory of the project. The top-level directory for this project is the MovieReview project folder. In PyCharm to start a terminal, right click on the top-level folder and select Open In -> Terminal. See the image below:

Figure 17-8: Open In Terminal in PyCharm

Now we can run the terminal command to start our Flask app. See the image below on macOS machine:

Figure 17-9: Running a Flask App in PyCharm Terminal on macOS

When you run these commands, your Flask app will start on your local machine. By default, the IP address for the localhost will be http://127.0.0.1 and the port will be 5000. So, the complete URL will become http://127.0.0.1:5000. Open this URL in a browser window and add the ‘/hi’ route to the end as: http://127.0.0.1:5000/hi. When you call this URL in the browser, it will render a string as “Hi from Flask App”. See the image below:

Figure 17-10: Flask App running on the browser

If you can see a screen like the one in Figure 17-10, then congratulations, your first Flask app is up and running. You can move to the next section.

The Application Factory

In the previous section, we created a Flask app instance inside the hi.py file. It was just a simple example and in the real-world project, there is a better way of creating Flask apps. We have a special function known as The Application Factory. The Application Factory is responsible for any configuration, registration and other setup operations. And it returns the configured app object, which is an instance of the Flask class.

flaskapp directory:

In a Flask project the actual project has its own directory. Keep in mind that, the Flask project is not the same thing as the PyCharm project. Flask project is a sub-folder of the PyCharm project. So, inside our MovieReview PyCharm project we will have a Flask project as flaskapp. Here is the folder structure:

Figure 17-11: Flask App inside the PyCharm project

As you see in Figure 17-11, MovieReview is our PyCharm Project, and the flaskapp is the Flask project in it. flaskapp is the directory where we will instantiate a Flask instance. Actually, it will be a Python package because it will contain the __init__.py file.

Now let’s create the flaskapp directory (package) in our MovieReview project, if you haven’t already. Make sure you have __init__.py file in it. The __init__.py will contain the application factory for our Flask app, and it tells Python that the flaskapp directory should be treated as a package. Here is the content of this file:

[2]:

1

# __init__.py file

2

3

import os

4

from flask import Flask

5

6

# application factory

7

def create_app(test_config=None):

8

    # create a Flask app object

9

    app = Flask(__name__, instance_relative_config=True)

10

11

    # configure the app

12

    app.config.from_mapping(

13

        SECRET_KEY='dev',

14

        DATABASE=os.path.join(app.instance_path, 'flaskapp.sqlite'),

15

    )

16

17

    # create the instance folder if not exists

18

    try:

19

        os.makedirs(app.instance_path)

20

    except OSError:

21

        pass

22

23

    # a simple test to see if it works

24

    @app.route('/test')

25

    def test():

26

        return 'Flaskapp init file works fine :)'

27

28

    # return the configured Flask app object

29

    return app

In cell 2, you see __init__.py file content. This is the initial form of this file and we will modify it constantly. The create_app() function is the application factory itself. Make sure you give exactly the same name, because “create_app” is the name Flask uses for application factory by default.

In line 9 we create a Flask app instance as: app = Flask(__name__, instance_relative_config=True). The parameters are:

__name__: name of the current Python module. This tells the app where it’s located to set up relative paths.

instance_relative_config=True: This parameter tells the app that configuration files are relative to the instance folder. The instance folder is a special folder which is located outside the flaskapp package and holds local data like configuration secrets and database files. It will automatically be created when we run Flask in a minute.

In line 12, we configure our Flask app object. The app.config.from_mapping() method sets some default configuration that the app will use like SECRET_KEY and DATABASE. We will use SQLite database for this project.

SECRET_KEY is used by Flask to keep the app data safe. During development you can set it to 'dev', but you should override it with a random value before deploying the app.

DATABASE is the path where we save the SQLite database file. It’s under app.instance_path, which is the path that Flask has chosen for the instance folder. Don’t worry, it will make sense when we create our database.

In line 19, we create the instance folder as: os.makedirs(app.instance_path). This folder will hold our local database file when we initialize the db.

In line 24, we have a simple route for test purposes. We want to see if our Flask app works when we run this code. The url for this route is simply ‘/test’.

And finally, in line 29, we return the configured Flask app object.

Now let’s run this code and test it. To run it, start a new Terminal window in PyCharm and set a new environment variable for our flaskapp project. Make sure you are at the top-level folder in the PyCharm project which is MovieReview directory.

Bash (macOS and Linux):

------------------------

$ export FLASK_APP=flaskapp

$ export FLASK_ENV=development

$ flask run

CMD (Windows):

------------------------

> set FLASK_APP=flaskapp

> set FLASK_ENV=development

> flask run

Powershell (Windows):

------------------------

> $env:FLASK_APP = "flaskapp"

> $env:FLASK_ENV = "development"

> flask run

This time the environment variable name is flaskapp, which is the same name with our flask project. If you run this code in your PyCharm terminal, then it should start a new Flask app on the given local IP address and port. You should see an output similar to the one below when you run these commands:

(venv) MovieReview % export FLASK_APP=flaskapp  

(venv) MovieReview % export FLASK_ENV=development

(venv) MovieReview % flask run                  

* Serving Flask app 'flaskapp' (lazy loading)

* Environment: development

* Debug mode: on

* Running on http://127.0.0.1:5000 (Press CTRL+C to quit)

* Restarting with stat

* Debugger is active!

* Debugger PIN: 885-298-466

Now let’s check it on the web browser. When you type http://127.0.0.1:5000/test in the browser, you should see a text of “Flaskapp init file works fine :)”. See the images below:

Figure 17-12: Running flaskapp in PyCharm

Figure 17-13: The result in the browser for http://127.0.0.1:5000/test

If you see a similar result like the one in Figure 17-13, then your application factory is set up correctly. You can move on to the next section now. Which is about creating a local database.

Create The Database

As we mentioned earlier, we will use SQLite database to store our app data. SQLite is a light-weight database server which doesn’t require setting up a separate db server and is built-in to Python. It is convenient for small projects, however for large applications it may not serve well due to its limitations.

We will create a separate Python file in our flaskapp folder, called db.py. This file will setup and initialize the database. Here is the code in this file:

[3]:

1

# db.py file

2

3

import sqlite3

4

import click

5

from flask import current_app, g

6

from flask.cli import with_appcontext

7

8

# db connection

9

def get_db():

10

    if 'db' not in g:

11

        g.db = sqlite3.connect(

12

            current_app.config['DATABASE'],

13

            detect_types=sqlite3.PARSE_DECLTYPES

14

        )

15

        g.db.row_factory = sqlite3.Row

16

    return g.db

17

18

# close connection

19

def close_db(e=None):

20

    db = g.pop('db', None)

21

    # check if connection exists

22

    if db is not None:

23

        db.close()

24

25

# initialize db

26

def init_db():

27

    db = get_db()

28

    with current_app.open_resource('schema.sql') as f:

29

        db.executescript(f.read().decode('utf8'))

30

31

# command line command

32

@click.command('init-db')

33

@with_appcontext

34

def init_db_command():

35

    """Clear the existing data and create new tables."""

36

    init_db()

37

    click.echo('Successfully initialized the database.')

38

39

# register functions with app instance

40

def init_app(app):

41

    app.teardown_appcontext(close_db)

42

    app.cli.add_command(init_db_command)

In cell 3, you see the code in our db.py file. In line 5, we import current_app and g objects from the flask package:

g is a special object which is unique for each request. It stores the data which is accessible for multiple functions during the current request.

current_app is an object that points to the current Flask application request.

sqlite3.connect() establishes a connection to the file pointed at by the DATABASE configuration key.

sqlite3.Row tells the connection to return rows that behave like dicts, which allows accessing the columns by their names.

The functions are:

get_db(): creates a db connection and sets it to the g.db attribute

close_db(): checks and closes the db connection if it exists

init_db(): initializes a database by reading the 'schema.sql' file, which contains the SQL commands to create the database tables.

init_db_command(): it calls the init_db() function and prints the result. It is decorated with @click.command('init-db') and @with_appcontext.

click.command() defines a command line command called init-db which calls the init_db() function and shows a success message to the user.

with_appcontext decorator is in the flask.cli module and wraps a callback to guarantee it will be called with a script's application context.

init_app() is used to register the functions with the app instance, which is the incoming parameter. We will call this function from the application factory.

app.teardown_appcontext() tells Flask to call that function when cleaning up after returning the response.

app.cli.add_command() adds a new command that can be called with the flask command. This enables us to use “flask init-db” command in the terminal.

Now let’s create the 'schema.sql' file under the flaskapp directory. Here is the code in this file:

[4]:

1

-- schema.sql file

2

3

DROP TABLE IF EXISTS user;

4

DROP TABLE IF EXISTS review;

5

6

CREATE TABLE user (

7

  id INTEGER PRIMARY KEY AUTOINCREMENT,

8

  username TEXT UNIQUE NOT NULL,

9

  password TEXT NOT NULL,

10

  fullname TEXT NOT NULL

11

);

12

13

CREATE TABLE review (

14

  id INTEGER PRIMARY KEY,

15

  author_id INTEGER NOT NULL,

16

  created TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,

17

  title TEXT NOT NULL,

18

  body TEXT NOT NULL,

19

  FOREIGN KEY (author_id) REFERENCES user (id)

20

);

In cell 4, you see the SQL commands to create two database tables; user and review. Each table has fields in their definition commands. Before running these commands, we need to call the init_app() function from the application factory. Here is updated __init__.py file content:

[5]:

1

# __init__.py file

2

3

import os

4

from flask import Flask

5

from . import db

6

7

# application factory

8

def create_app(test_config=None):

9

    # create a Flask app object

10

    app = Flask(__name__, instance_relative_config=True)

11

12

    # configure the app

13

    app.config.from_mapping(

14

        SECRET_KEY='dev',

15

        DATABASE=os.path.join(app.instance_path, 'flaskapp.sqlite'),

16

    )

17

18

    # create the instance folder if not exists

19

    try:

20

        os.makedirs(app.instance_path)

21

    except OSError:

22

        pass

23

24

    # db registration

25

    db.init_app(app)

26

27

    # a simple test to see if it works

28

    @app.route('/test')

29

    def test():

30

        return 'Flaskapp init file works fine :)'

31

32

    # return the configured Flask app object

33

    return app

In cell 5, line 35, we register the database with our application by calling the db.init_app() function and passing the current app object as the argument.

Now we are ready to initialize the database file. All we need to do is to call the “flask init-db” command in the terminal. Make sure you stop the Flask server by pressing CTRL+C to quit. And set the FLASK_APP and the FLASK_ENV environment variables again before running the “flask init-db” command. Here is the output on a macOS computer:

(venv) MovieReview % export FLASK_APP=flaskapp  

(venv) MovieReview % export FLASK_ENV=development

(venv) MovieReview % flask init-db

Successfully initialized the database.

If you see the text of ‘Successfully initialized the database.’ in the output, then your database tables are created and the database is initialized in the application. If you check the “instance” folder in the project now, you should see the flaskapp.sqlite file in it. This is the file where our database will be stored. You can now run “flask run” command to start the server again.

Movies Data and The Utils Module

We will be using movies data which belongs to the top 20 movies in the IMDB Top 250 movies. The data will be in a CSV file inside a folder called ‘data’. File name is imdb_top_20.csv and you can find it in the project directory in GitHub repository of this book.

In our project, we will also have a module called utils.py. This module will contain some common functions, like reading CSV data and getting movies, getting the user and review data from database etc. Here is the content of this file:

[6]:

1

# utils.py file

2

3

from flask import (g, redirect, url_for)

4

from flaskapp.db import get_db

5

import csv

6

import functools

7

8

# function to read all movies

9

def get_movies():

10

    """reads the csv file and returns

11

        the list of movies"""

12

    movies = []

13

    movie_path = 'data/imdb_top_20.csv'

14

    with open(movie_path, 'r') as file:

15

        movie_dict = csv.DictReader(file, delimiter=';')

16

        for movie in movie_dict:

17

            movies.append(movie)

18

    # return the list

19

    return movies

20

21

# function to read a single movie

22

def get_movie(id):

23

    """returns a single movie"""

24

    movie = None

25

    movie_path = 'data/imdb_top_20.csv'

26

    with open(movie_path, 'r') as file:

27

        movie_dict = csv.DictReader(file, delimiter=';')

28

        for m in movie_dict:

29

            if m['id'] == str(id):

30

                movie = m

31

    return movie

32

33

# get movie image

34

def get_movie_img(id):

35

    return get_movie(id)['imageURL']

36

37

# get all reviews

38

def get_all_reviews():

39

    db = get_db()

40

    return db.execute(

41

        'SELECT r.id, title, body, created, author_id, username, fullname'

42

        ' FROM review r JOIN user u ON r.author_id = u.id'

43

        ' ORDER BY created DESC'

44

    ).fetchall()

45

46

# get review from db

47

def get_review_from_db(id):

48

    return get_db().execute(

49

        'SELECT r.id, title, body, created, author_id, username'

50

        ' FROM review r JOIN user u ON r.author_id = u.id'

51

        ' WHERE r.id = ?',

52

        (id,)

53

    ).fetchone()

54

55

# get user

56

def get_user(username):

57

    db = get_db()

58

    user = db.execute(

59

        'SELECT * FROM user WHERE username = ?', (username,)

60

    ).fetchone()

61

    return user

62

63

# Require Authentication

64

def login_required(view):

65

    @functools.wraps(view)

66

    def wrapped_view(**kwargs):

67

        if g.user is None:

68

            return redirect(url_for('login.login'))

69

        return view(**kwargs)

70

    return wrapped_view

In cell 6, we have the utils.py file content. Here are the functions in it:

get_movies(): This function reads the csv file and returns the list of movies.

get_movie(): This function returns a single movie based on the given id parameter.

get_movie_img(): This function returns the movie image URL based on the given id.

get_all_reviews(): It returns the list of all reviews in the database. It uses a SQL join statement to join the review and user tables. We call the fetchall() method to get a list of results.

get_review_from_db(): It returns a single review from the database. We use fetchone() for returning a single result.

get_user(): It returns the user data from the database based on the username provided.

login_required(): We will require the user to login to our application before creating, editing and deleting movie reviews. login_required is the decorator which will check if the user is logged in before any of these operations. It returns a new view function that wraps the original view it’s applied to. The new function simply checks if the user is logged in as: if g.user is None. Remember that the object g stores the data in the current request. If g.user is not None, this means that the user is logged in. You will see how we fill it later on. If it is None, then we redirect the user to the login page.

url_for(function, parameters):

The url_for() function in Flask, generates a URL to an endpoint using the function passed in as an argument.

For example, the test() function that was added to the application factory in the previous section, has the name 'test' and can be linked to with url_for('test'). You can think of it as requesting the URL of http://127.0.0.1:5000/test in your web browser.

If it took an argument, which you’ll see later, it would be linked to using url_for('test', param_1=value_1, param_2=value_2).

In line 68, we use this function as url_for('login.login'). The parameter is ‘login.login’. The first ‘login’ is the Blueprint name and the second one is the endpoint name. So, this means that, we redirect the user to an endpoint (a function here) called ‘login’ inside a Blueprint named ‘login’. Don’t worry it will make sense when you learn about the Blueprints.

Views and Blueprints

In a Flask application the UI (user interface) is mainly created by Templates (HTML templates) and the back-end is created by Views and Blueprints.

Views are the functions which respond to the requests. Views handle the incoming request based on the request URL patterns. And they return data that Flask turns into an outgoing response.

Blueprints are the way to organize a group of related views. They can contain any number of view functions. Views are registered with a Blueprint and the Blueprint is registered with the application factory.

In our project we will have four Blueprints which are: home, login, register, and review. Each one will have a separate module and in each of them we will have different View functions to handle incoming requests.

Register Blueprint and View Function

A Blueprint is a class in the flask module, which is used to register the Views (View functions). The name of our first Blueprint will be Register and it will have a View function called register(). Create a new Python file in the flaskapp directory called register.py. And here is the code in this file:

[7]:

1

# register.py file

2

from flask import (Blueprint, flash, redirect,

3

                   render_template, request, session, url_for)

4

from flaskapp.db import get_db

5

from werkzeug.security import generate_password_hash

6

from .utils import get_user

7

8

# initialize the Blueprint object

9

bp = Blueprint('register', __name__, url_prefix='/')

10

11

# Register View Code

12

@bp.route('/register', methods=('GET', 'POST'))

13

def register():

14

    if request.method == 'POST':

15

        # get input fields

16

        username, password, fullname = get_input_fields()

17

        # get db

18

        db = get_db()

19

        # check input fields

20

        error = check_input_fields(username, password, fullname)

21

        # check error

22

        if error is None:

23

            try:

24

                # insert new user

25

                insert_into_db(db, username, password, fullname)

26

            except db.IntegrityError:

27

                error = f"User {username} is already registered."

28

            else:

29

                session.clear()

30

                session['user_id'] = get_user(username)['id']

31

                return redirect(url_for('home.index'))

32

        flash(error)

33

    return render_template('register/register.html')

34

35

# fn for getting inputs

36

def get_input_fields():

37

    username = request.form['username']

38

    password = request.form['password']

39

    fullname = request.form['fullname']

40

    return username, password, fullname

41

42

# fn for checking input fields

43

def check_input_fields(username, password, fullname):

44

    if not username or not password or not fullname:

45

        return 'Full Name, Username or Password can not be empty.'

46

    else:

47

        return None

48

49

# fn for inserting a DB row

50

def insert_into_db(db, username, password, fullname):

51

    db.execute("INSERT INTO user (username, password, fullname) "

52

               "VALUES (?, ?, ?)",

53

        (username, generate_password_hash(password), fullname),)

54

    db.commit()

In cell 7, line 9, we create a Blueprint object as: bp = Blueprint('register', __name__, url_prefix='/'). The first parameter is the unique name of this Blueprint the second one is the name of the current application and the third parameter is the URL prefix.

In line 13 you see the definition of the register() view function. Here is what this function does:

@bp.route decorator associates the URL “/register” with the register() view function. When Flask receives a request to http://127.0.0.1:5000/register, it will call the register() view and use the return value as the response. This view function can handle both GET and POST request types.

In line 14, we check the request method being POST, which means the user submits the form in this view. If the method is POST, then we have some validations for the form fields like username, password and fullname. We don’t want any of them to be empty, that’s why we call the check_input_fields() function.

If all of the fields are filled by the user, then we can try to insert this into the user table in db. That’s what we do in line 25, by calling the insert_into_db() function.

If db insert operation is successful, then in the else block in line 29, we clear the old session and add the user id to it. Then we redirect the user to the home page by calling the url_for('home.index') function. We will define the home.index view function later on.

In line 32, we call a special function called flash() by passing the error as the argument. This function stores the error messages that can be retrieved when rendering the template. We will print the error messages later on in our HTML template.

Finally, in line 33, we call the render_template() function by passing the file location of the template. This will render the HTML template for the 'register/register.html' file in the web browser. We will create this file soon.

In line 50, we have the definition of the insert_into_db() function. It simply inserts a new row to the database. For security reasons, we call the generate_password_hash() function instead of saving the password as it is. This will store the hash value of the password.

To be able to use our Register Blueprint, we need to register it inside our application factory as follows:

[8]:

1

# __init__.py file

2

3

import os

4

from flask import Flask

5

from . import db, register

6

7

# application factory

8

def create_app(test_config=None):

9

    # create a Flask app object

10

    app = Flask(__name__, instance_relative_config=True)

11

12

    # configure the app

13

    app.config.from_mapping(

14

        SECRET_KEY='dev',

15

        DATABASE=os.path.join(app.instance_path, 'flaskapp.sqlite'),

16

    )

17

18

    # create the instance folder if not exists

19

    try:

20

        os.makedirs(app.instance_path)

21

    except OSError:

22

        pass

23

24

    # db registration

25

    db.init_app(app)

26

27

    # register Blueprints

28

    app.register_blueprint(register.bp)

29

30

    # a simple test to see if it works

31

    @app.route('/test')

32

    def test():

33

        return 'Flaskapp init file works fine :)'

34

35

    # return the configured Flask app object

36

    return app

Now that we have our Register Blueprint and a view function for it, let’s request its URL in the web browser. If you type http://127.0.0.1:5000/register in your browser and press on Enter, you will see an error page as TemplateNotFound. It will say: jinja2.exceptions.TemplateNotFound: register/register.html. This tells us that Flask couldn’t find any Templates for rendering. We will create HTML templates in the succeeding sections.

Templates

Templates are files that contain static data as well as placeholders for dynamic data. A template is rendered with specific data to produce a final document. All the template files in a Flask app need to be located under a special folder called “templates”. This folder should be directly under the flaskapp directory and it is the default location where Flask looks for templates at run time.

Flask uses the Jinja template library to render templates. In our application, we will use templates to render HTML which will display in the user’s browser.

A Jinja template is simply a text file. Jinja can generate any text-based format (HTML, XML, CSV, LaTeX, etc.). A Jinja template doesn’t need to have a specific extension: .html, .xml, or any other extension is just fine.

A template contains variables and/or expressions, which get replaced with values when the template is rendered; and tags, which control the logic of the template.

The template syntax in Jinja is heavily inspired by Python. Special delimiters are used to distinguish Jinja syntax from the static data in the template. Anything between {{ and }} is an expression that will be output to the final document. {% and %} denotes a control flow statement like if and for. Unlike Python, blocks are denoted by start and end tags rather than indentation since static text within a block could change indentation. Here are a few kinds of delimiters:

{% ... %} for Statements

{{ ... }} for Expressions to print to the template output

{# ... #} for Comments not included in the template output

The Base Template

In our application, each page will have the same basic layout around a different body. For example, they will all include a navigation bar at the top. Instead of writing the entire HTML structure in each template, each template will extend a base template and override specific sections. You can think of the base templates as the master pages.

In our flaskapp project directory, we will create a new folder called templates, and inside this folder, create a new HTML file named base.html. Here is the code in this file:

[9]:

1

<!doctype html>

2

<html lang="en" >

3

<head>

4

  <meta charset="UTF-8">

5

  <link rel='stylesheet' href='https://fonts.googleapis.com/css?family=Rubik:400,700'>

6

  <link rel="stylesheet" href="./style.css">

7

  <link rel="stylesheet" href="{{ url_for('static', filename='style.css') }}">

8

  <title>Flask App</title>

9

</head>

10

<body>

11

12

<div class="menu">

13

  <a class="title-anchor" href="{{ url_for('home.index') }}"><h1>Movie Review with Flask</h1></a>

14

  <div class="menu-right">

15

    {% if g.user %}

16

      <h2>{{ g.user['fullname'] }}</h2>

17

      <a href="{{ url_for('login.logout') }}">Log Out</a>

18

    {% else %}

19

      <a href="{{ url_for('register.register') }}">Register</a>

20

      <a href="{{ url_for('login.login') }}">Log In</a>

21

    {% endif %}

22

  </div>

23

</div>

24

<div class="content">

25

  <header>

26

    {% block header %}{% endblock %}

27

  </header>

28

  {% for message in get_flashed_messages() %}

29

    <div class="flash">{{ message }}</div>

30

  {% endfor %}

31

  {% block content %}{% endblock %}

32

</div>

33

</body>

34

</html>

In cell 9, we have the content of base.html file in the templates folder. It is a regular HTML file which includes Jinja commands in it.

In line 15, we check the user in the current request as: {% if g.user %}. And if g.user is not None, we are sure that the user is logged in and we render a logout link button. Else, we render Register and Log In buttons.

In line 16, inside the <h2> tag, we render the user fullname as: {{ g.user['fullname'] }}.

In line 26, we have a placeholder for the header block as: {% block header %}{% endblock %}.

There are two blocks defined in the base template which will be overridden in the child templates:

{% block header %} is where the unique header of each page will replace.

{% block content %} is where the content of each page goes, such as the register or login forms.

In line 28, we check if there are any error messages in the view functions. Remember that we called the flash(error) function in the view code for storing the error messages. And this is where we get these messages by calling the get_flashed_messages() function. We set a for loop to iterate over the messages and render a div element for each one. Here is the code: {% for message in get_flashed_messages() %}.

And finally, in line 31, we have the content block.

Register Template

Now that we have the base template, we can extend it and create a new template called the Register Template. It will contain the HTML code to create our register page. Inside the templates folder create a sub-folder called register. And inside this folder create an HTML document as register.html. Here is the code in this file:

[10]:

1

{% extends 'base.html' %}

2

3

{% block content %}

4

    <div class="login-form">

5

      <form method="post">

6

        <h1>Register</h1>

7

        <div class="content">

8

          <div class="input-field">

9

            <input class="input-text" id="fullname" name="fullname" placeholder="Full Name">

10

          </div>

11

          <div class="input-field">

12

            <input class="input-text" id="username" name="username" placeholder="User Name">

13

          </div>

14

          <div class="input-field">

15

            <input class="input-text" type="password" id="password" name="password" placeholder="Password">

16

          </div>

17

        </div>

18

        <div class="action">

19

          <button type="submit" class="actionbtn">Sign Up</button>

20

        </div>

21

      </form>

22

    </div>

23

{% endblock %}

In cell 10, we have the register template. In the first line you see that it extends the base template as: {% extends 'base.html' %}.

In line 3, we have the structure for the content block. This will replace the content block in the base template. The register Template is a basic HTML form element with some input fields in it.

Remember that in cell 7 line 33, inside the register() view function we had the return statement as: return render_template('register/register.html'). This is where we tell Flask to find and render the template when it matches the specified route, which was: @bp.route('/register', methods=('GET', 'POST')).

Now if request the URL http://127.0.0.1:5000/register in the browser and press on Enter, Flask will try to render the register template. You will still get a BuildError, because we do not have a template named home.index yet. Don’t worry we will create it in a minute.

Login Blueprint & Template

Before creating the Blueprint for Home Page, we will create the Login Page. As we did for the Register page, first we will create a Blueprint with some View functions in it. Then we will create a template as login.html.

Login Blueprint and View Functions:

For the Blueprint, create a Python file named login.py under the flaskapp directory. Here is the code for this file:

[12]:

1

# login.py file

2

3

from flask import (Blueprint, flash, g, redirect,

4

                   render_template, request, session, url_for)

5

from flaskapp.db import get_db

6

from werkzeug.security import check_password_hash

7

8

# initialize the Blueprint object

9

bp = Blueprint('login', __name__, url_prefix='/')

10

11

# Login View Code

12

@bp.route('/login', methods=['GET', 'POST'])

13

def login():

14

    if request.method == 'POST':

15

        username = request.form['username']

16

        password = request.form['password']

17

        db = get_db()

18

        error = None

19

        user = db.execute(

20

            'SELECT * FROM user WHERE username = ?', (username,)

21

        ).fetchone()

22

        # check user from db

23

        if user is None:

24

            error = "Incorrect username. Please try again."

25

        elif not check_password_hash(user['password'], password):

26

            error = "Incorrect password. Please try again."

27

        # check error

28

        if error is None:

29

            session.clear()

30

            session['user_id'] = user['id']

31

            return redirect(url_for('home.index'))

32

        # flash error

33

        flash(error)

34

    # render template

35

    return render_template('login/login.html')

36

37

# Run before all view functions

38

@bp.before_app_request

39

def load_logged_in_user():

40

    user_id = session.get('user_id')

41

    if user_id is None:

42

        g.user = None

43

    else:

44

        g.user = get_db().execute(

45

            'SELECT * FROM user WHERE id = ?', (user_id,)

46

        ).fetchone()

47

48

# logout route

49

@bp.route('/logout')

50

def logout():

51

    session.clear()

52

    return redirect(url_for('login.login'))

In cell 12, you see the code in the login.py file. It creates a Blueprint called ‘login’ and define routes for '/login' and '/logout'. Now we have to register this Blueprint with the application factory. Here it is:

[13]:

1

# __init__.py file

2

3

import os

4

from flask import Flask

5

from . import db, register, login

6

7

# application factory

8

def create_app(test_config=None):

9

    # create a Flask app object

10

    app = Flask(__name__, instance_relative_config=True)

11

12

    # configure the app

13

    app.config.from_mapping(

14

        SECRET_KEY='dev',

15

        DATABASE=os.path.join(app.instance_path, 'flaskapp.sqlite'),

16

    )

17

18

    # create the instance folder if not exists

19

    try:

20

        os.makedirs(app.instance_path)

21

    except OSError:

22

        pass

23

24

    # db registration

25

    db.init_app(app)

26

27

    # register Blueprints

28

    app.register_blueprint(register.bp)

29

    app.register_blueprint(login.bp)

30

31

    # a simple test to see if it works

32

    @app.route('/test')

33

    def test():

34

        return 'Flaskapp init file works fine :)'

35

36

    # return the configured Flask app object

37

    return app

Login Template:

Now that we have a Blueprint for the login page, we can define a Template for it. To do this, create a new folder called login under the templates directory. And in that folder, create a file called login.html. Here is the code in this file:

[14]:

1

{% extends 'base.html' %}

2

3

{% block content %}

4

    <div class="login-form">

5

      <form method="post">

6

        <h1>Login</h1>

7

        <div class="content">

8

          <div class="input-field">

9

            <input class="input-text" id="username" name="username" placeholder="User Name">

10

          </div>

11

          <div class="input-field">

12

            <input class="input-text" type="password" id="password" name="password" placeholder="Password">

13

          </div>

14

          <a href="#" class="link">Forgot Your Password?</a>

15

        </div>

16

        <div class="action">

17

          <button class="actionbtn" type="submit">Sign in</button>

18

          <a class="actionbtn" href="{{ url_for('register.register') }}">Register</a>

19

        </div>

20

      </form>

21

    </div>

22

{% endblock %}

Home Blueprint & Template

Now let’s create a Blueprint and a Template for the home page which we will call as index.html.

Home Blueprint and View Function:

For the Blueprint, create a Python file named home.py under the flaskapp directory. Here is the code for this file:

[15]:

1

# home.py file

2

from flask import (Blueprint, g, render_template)

3

from .utils import login_required, get_all_reviews, \

4

    get_movies, get_review_from_db

5

6

# initialize the Blueprint object

7

bp = Blueprint('home', __name__, url_prefix='/')

8

9

# Index View Code

10

@bp.route('/', methods=['GET'])

11

@login_required

12

def index():

13

    # get all movies

14

    movies = get_movies()

15

    # get all reviews

16

    all_reviews = get_all_reviews()

17

    # render index.html

18

    return render_template('home/index.html', movies=movies,

19

                           all_reviews=all_reviews,

20

                           get_if_reviewed=get_if_reviewed)

21

22

def get_if_reviewed(movie):

23

    review = get_review_from_db(movie['id'])

24

    if review is not None and g.user['id'] == review['author_id']:

25

        return True

26

    else:

27

        return False

In the Home Blueprint, we don’t have a specific route for the index() view function. The route is: @bp.route('/', methods=['GET']). Which means it will be the main page when the user opens the application by requesting http://127.0.0.1:5000/.

In line 15 and 16, it gets all the movies and the reviews by calling the respective functions in the utils file.

It renders a template in the 'home/index.html' location which we will create next. Here, we also pass some parameters while rendering the template. Here is the code:

    return render_template('home/index.html', movies=movies,

                           all_reviews=all_reviews,

                           get_if_reviewed=get_if_reviewed)

The first two parameters are list type objects and the third one is a function, which we will call from the HTML template. This is how we pass parameters to templates from the view functions.

Now we have to register this Blueprint with the application factory. Here it is:

[16]:

1

# __init__.py file

2

3

import os

4

from flask import Flask

5

from . import db, register, login, home

6

7

# application factory

8

def create_app(test_config=None):

9

    # create a Flask app object

10

    app = Flask(__name__, instance_relative_config=True)

11

12

    # configure the app

13

    app.config.from_mapping(

14

        SECRET_KEY='dev',

15

        DATABASE=os.path.join(app.instance_path, 'flaskapp.sqlite'),

16

    )

17

18

    # create the instance folder if not exists

19

    try:

20

        os.makedirs(app.instance_path)

21

    except OSError:

22

        pass

23

24

    # db registration

25

    db.init_app(app)

26

27

    # register Blueprints

28

    app.register_blueprint(register.bp)

29

    app.register_blueprint(login.bp)

30

    app.register_blueprint(home.bp)

31

32

    # a simple test to see if it works

33

    @app.route('/test')

34

    def test():

35

        return 'Flaskapp init file works fine :)'

36

37

    # return the configured Flask app object

38

    return app

Index Template:

Now that we have a Blueprint for the home page, we can define a Template for it. To do this, create a new folder called home under the templates directory. And in that folder, create a file called index.html. Here is the code in this file:

[17]:

1

{% extends 'base.html' %}

2

3

{% block content %}

4

    <div class="login-form">

5

      <form method="post">

6

        <h1>Login</h1>

7

        <div class="content">

8

          <div class="input-field">

9

            <input class="input-text" id="username" name="username" placeholder="User Name">

10

          </div>

11

          <div class="input-field">

12

            <input class="input-text" type="password" id="password" name="password" placeholder="Password">

13

          </div>

14

          <a href="#" class="link">Forgot Your Password?</a>

15

        </div>

16

        <div class="action">

17

          <button class="actionbtn" type="submit">Sign in</button>

18

          <a class="actionbtn" href="{{ url_for('register.register') }}">Register</a>

19

        </div>

20

      </form>

21

    </div>

22

{% endblock %}

At this point, if you run the app and type http://127.0.0.1:5000/ in your web browser, you should a screen like the one below:

Figure 17-14: Login page which is redirected from the Home page

For the time being, the app doesn’t look good because we have no formatting yet. Don’t worry we will add some CSS and it will look pretty.

Review Blueprint & Template

The last Blueprint which we will add is the Review Blueprint. It will be responsible for creating new reviews, updating existing ones, deleting reviews and displaying a list of all reviews.

Review Blueprint and View Functions:

For the Blueprint, create a Python file named review.py under the flaskapp directory. Here is the code for this file:

[18]:

1

# review.py file

2

from flask import (

3

    Blueprint, flash, g, redirect,

4

    render_template, request, url_for)

5

from werkzeug.exceptions import abort

6

from flaskapp.db import get_db

7

from .utils import login_required, get_all_reviews, \

8

    get_movie, get_review_from_db, get_movie_img

9

10

bp = Blueprint('review', __name__)

11

12

# reviews route

13

@bp.route('/reviews')

14

@login_required

15

def reviews():

16

    all_reviews = get_all_reviews()

17

    return render_template('review/reviews.html',

18

                           all_reviews=all_reviews,

19

                           get_movie_img=get_movie_img)

20

21

# create a new review

22

@bp.route('/<int:id>/create', methods=('GET', 'POST'))

23

@login_required

24

def create(id):

25

    # get movie

26

    movie = get_movie(id)

27

    if request.method == 'POST':

28

        title = request.form['title']

29

        body = request.form['body']

30

        error = None

31

        if not body:

32

            error = 'You need to add a review.'

33

        if error is not None:

34

            flash(error)

35

        else:

36

            db = get_db()

37

            db.execute(

38

                'INSERT INTO review (id, title, body, author_id)'

39

                ' VALUES (?, ?, ?, ?)',

40

                (id, title, body, g.user['id'])

41

            )

42

            db.commit()

43

            return redirect(url_for('review.reviews'))

44

    return render_template('review/create.html', movie=movie)

45

46

# get a single review

47

def get_review(id, check_author=True):

48

    review = get_review_from_db(id)

49

    if review is None:

50

        abort(404, f"Post id {id} doesn't exist.")

51

    if check_author and review['author_id'] != g.user['id']:

52

        abort(403)

53

    return review

54

55

# update review

56

@bp.route('/<int:id>/update', methods=('GET', 'POST'))

57

@login_required

58

def update(id):

59

    review = get_review(id)

60

    if request.method == 'POST':

61

        title = request.form['title']

62

        body = request.form['body']

63

        error = None

64

        if not title:

65

            error = 'Title is required.'

66

        if error is not None:

67

            flash(error)

68

        else:

69

            db = get_db()

70

            db.execute(

71

                'UPDATE review SET title = ?, body = ?'

72

                ' WHERE id = ?',

73

                (title, body, id)

74

            )

75

            db.commit()

76

            return redirect(url_for('review.reviews'))

77

    return render_template('review/update.html', review=review)

78

79

# delete review

80

@bp.route('/<int:id>/delete', methods=('POST',))

81

@login_required

82

def delete(id):

83

    get_review(id)

84

    db = get_db()

85

    db.execute('DELETE FROM review WHERE id = ?', (id,))

86

    db.commit()

87

    return redirect(url_for('review.reviews'))

Review Templates:

We will have three separate templates for review operations:

create.html

update.html

reviews.html

To define these files, create a new folder called review under the templates directory. Here are the file contents:

review/create.html

[19]:

1

{% extends 'base.html' %}

2

3

{% block content %}

4

  <div class="login-form">

5

      <form method="post">

6

        <h1>{{ movie['title'] }}</h1>

7

        <div class="content">

8

          <div class="input-field">

9

            <input class="input-text" id="title" name="title" value="Review for: {{ movie['title'] }}">

10

          </div>

11

          <div class="input-field">

12

            <textarea class="input-text" id="body"

13

                      name="body" placeholder="Review" rows="6"></textarea>

14

          </div>

15

        </div>

16

        <div class="action">

17

          <button class="actionbtn" type="submit">Create Review</button>

18

        </div>

19

      </form>

20

    </div>

21

{% endblock %}

review/update.html:

[20]:

1

{% extends 'base.html' %}

2

3

{% block content %}

4

  <div class="login-form">

5

      <form method="post">

6

        <h1>{{ review['title'] }}</h1>

7

        <div class="content">

8

          <div class="input-field">

9

            <input class="input-text" id="title" name="title"

10

                   value="{{ request.form['title'] or review['title'] }}">

11

          </div>

12

          <div class="input-field">

13

            <textarea class="input-text" id="body"

14

                      name="body" rows="6">

15

              {{ request.form['body'] or review['body'] }}

16

            </textarea>

17

          </div>

18

        </div>

19

        <div class="action">

20

          <button class="actionbtn" type="submit">Save</button>

21

        </div>

22

      </form>

23

24

      <form class="login-form"

25

            action="{{ url_for('review.delete', id=review['id']) }}"

26

            method="post">

27

        <div class="action">

28

          <input class="actionbtn danger" type="submit"

29

                 value="Delete" onclick="return confirm('Are you sure?');">

30

        </div>

31

      </form>

32

    </div>

33

{% endblock %}

review/reviews.html:

[21]:

1

{% extends 'base.html' %}

2

3

{% block header %}

4

  <div class="pageheader">

5

    <h1>{% block title %}All Reviews{% endblock %}</h1>

6

  </div>

7

{% endblock %}

8

9

{% block content %}

10

  {% for review in all_reviews %}

11

    <article class="review">

12

      <div class="review-left">

13

        <div class="article-img">

14

          <img src="{{ get_movie_img(review['id']) }}" style="width: 100px;" />

15

        </div>

16

        <div class="article-content">

17

          <header>

18

            <div>

19

              <h3>{{ review['title'] }}</h3>

20

              <h5 class="about">

21

                {{ review['fullname'] }}, {{ review['created'].strftime('%Y-%m-%d') }}

22

              </h5>

23

            </div>

24

          </header>

25

          <p class="article-body">{{ review['body'] }}</p>

26

        </div>

27

      </div>

28

      <div class="review-right">

29

        {% if g.user['id'] == review['author_id'] %}

30

            <a class="edit-button"

31

               href="{{ url_for('review.update', id=review['id']) }}">Edit</a>

32

          {% endif %}

33

      </div>

34

    </article>

35

    {% if not loop.last %}

36

      <hr>

37

    {% endif %}

38

  {% endfor %}

39

{% endblock %}

Finally, please do not forget to register the Review Blueprint with the application factory. Here is the final form of the __init__.py file:

[22]:

1

# __init__.py file

2

3

import os

4

from flask import Flask

5

from . import db, register, login, home, review

6

7

# application factory

8

def create_app(test_config=None):

9

    # create a Flask app object

10

    app = Flask(__name__, instance_relative_config=True)

11

12

    # configure the app

13

    app.config.from_mapping(

14

        SECRET_KEY='dev',

15

        DATABASE=os.path.join(app.instance_path, 'flaskapp.sqlite'),

16

    )

17

18

    # create the instance folder if not exists

19

    try:

20

        os.makedirs(app.instance_path)

21

    except OSError:

22

        pass

23

24

    # db registration

25

    db.init_app(app)

26

27

    # register Blueprints

28

    app.register_blueprint(register.bp)

29

    app.register_blueprint(login.bp)

30

    app.register_blueprint(home.bp)

31

    app.register_blueprint(review.bp)

32

33

    # a simple test to see if it works

34

    @app.route('/test')

35

    def test():

36

        return 'Flaskapp init file works fine :)'

37

38

    # return the configured Flask app object

39

    return app

Static Files

Static files are CSS files and other types of files with JavaScript functions, or a logo image. They should be placed under the flaskapp/static directory and referenced with url_for('static', filename='...').

We have already addressed the CSS file named style.css in the base.html file as follows:

<link rel="stylesheet" href="{{ url_for('static', filename='style.css') }}">

Now let’s create this file. You can find the file style.css file in the project folder in the GitHub repository of the book. Here is the final image of the PyCharm project:

Figure 17-15: PyCharm project structure finalized

After you add the CSS file, the app will look really nice. Here is the image of the home page:

Figure 17-16: Movie Review app completed

This finalizes our project of API Development with Flask. In the next chapter, you will have an assignment on this project. Good luck with your assignment.

18.        Assignment 3 – API Development with Flask

Assignment Setup

We finished the third project in this book which is Project 3 – API Development with Flask. Now it’s your turn to recreate the same project. This chapter is the assignment on the Project 3.

To complete the assignment, follow the instructions where you see a #TODO statement. You can find the solutions in the previous chapter.

We will create a new PyCharm project. The project name is MovieReview. You should download the project from the GitHub Repository of this book before starting to code.

Chapter Outline:

Project Layout

Introduction to Flask

The Application Factory

Create the Database

Movies Data and The Utils Module

Views and Blueprints

Register Blueprint and View Function

Templates

The Base Template

Register Template

Login Blueprint & Template

Home Blueprint & Template

Review Blueprint & Template

Static Files

Project Layout

Our project is a simple app which we will use for reviewing some movies in the IMDB Top 250 list. The movie data will be provided as a CSV file in the project. We will use the built-in SQLite database for this project. Here are the pages in our application:

Home Page:

In the home page, we will see the list of movies in a responsive grid. By clicking any of the movies, you can either create a review or update/delete it if it is already created. The movies with a review will be displayed with a green border around them.

The first tile in the grid is the “All Reviews” button, which will navigate you to a list of all reviews.

We have a menu bar on top of the page. On the menu bar on the left, we have the app title which is also an anchor tag for the Home Page. On the right, you see the logged in user’s full name and a Logout button.

Figure 17-1: Home Page

Register:

To be able to use the app, the user needs to register and login in to the system. We have two different pages for this. The first page is the Register page. In which, user will enter the full name, user name and a password. All of which are required to fill.

Figure 17-2: Register

Login:

After successful registration, user needs to login in. We save user data in the user table, so, the user can login any time with correct user name and password. Inside the app, we check if the credentials are true or not. If not, we notify the user to try again.

User can easily navigate to the Register page from the Login screen.

Figure 17-3: Login

Create a Review:

The logged in user can create a review for any movie on the Home Page. By clicking its tile in the Home Page, the user will be navigated to the Create page. In Figure 17-4, in the page url, you can see the Id of the movie, for which the user will create a review.

Figure 17-4: Create a Review

Update the Review:

The user can update a review, by clicking its tile in the Home Page. He/She will be navigated to the Update page. In Figure 17-5, in the page url, you can see the Id of the movie, which the user will update the existing review.

Before leaving the Home Page, we check if there is already a review for that movie. If there is, then the Update page will render. If not, then the user will see the Create page.

Figure 17-5: Update or Delete a Review

Reviews:

In the Home Page, we have a tile called “All Reviews”. When clicked, it will navigate the user to the Reviews Page. This is where we see all the reviews in the database. If the review is created by the current user, then you will see an “Edit” button on the right-hand side of that review. So, the user can view all of the reviews, but can edit only the ones which he/she created.

Figure 17-6: All Reviews

Introduction to Flask

We will use Flask for building our app. Flask was created by Armin Ronacher in 2004. It is a microframework written in Python. It is a lightweight WSGI (Web Server Gateway Interface) web application framework, which is designed to make getting started quick and easy, with the ability to scale up to complex applications. Flask supports extensions that can add application features like database abstraction layer, form validation, etc.

A Flask application is an instance of the Flask class. Everything about the application, such as configuration and URLs, will be registered with this class.

To understand Flask better, let’s start with a small example. We will create a simple web API, which simply renders “Hi from Flask App” in the web browser.

We will create a new PyCharm project named MovieReview with a new virtual environment. And create a Python file named hi.py.

We have to install the Flask package first. You can easily install it in PyCharm or you can use pip to install it. Here is the official documentation if you need a detailed installation guideline.

Figure 17-7: Install Flask package in PyCharm IDE

[1]:

1

# hi.py file

2

3

# TODO - import Flask class from flask package

4

5

# TODO - create a Flask app

6

7

# TODO - define a root for '/hi'

8

def hi_flask():

9

    return "Hi from Flask App"

In cell 1, we have a very simple Flask app in the hi.py file. We import the Flask class first. Then in line 7, we create an instance of the Flask class and name it simply as app. This object will be our WSGI application. The first argument is the name of the application’s module or package. __name__ is a convenient shortcut for this that is appropriate for most cases. This is needed so that Flask knows where to look for resources such as templates and static files.

In line 10, we use the route() decorator to tell Flask what URL should trigger the function in the next line. Here, the URL will be ‘/hi’. Our hi_flask() function will return a text message which will be displayed in the user’s browser.

To run a Flask application, we need to use the ‘flask’ command or ‘python -m flask’ command. In this project, we prefer to use the ‘flask’ command. We need to make some settings in the terminal before running the ‘flask’ command. We need to tell the terminal the application to work with by exporting the FLASK_APP environment variable. Here are the commands for this, on different operating systems:

Bash (macOS and Linux):

------------------------

$ export FLASK_APP=hi

$ export FLASK_ENV=development

$ flask run

 * Running on http://127.0.0.1:5000/

CMD (Windows):

------------------------

> set FLASK_APP=hi

> set FLASK_ENV=development

> flask run

 * Running on http://127.0.0.1:5000/

Powershell (Windows):

------------------------

> $env:FLASK_APP = "hi"

> $env:FLASK_ENV = "development"

> flask run

 * Running on http://127.0.0.1:5000/

To be able to run these commands in your terminal (either standalone or in PyCharm) you need to be in the top-level directory of the project. The top-level directory for this project is the MovieReview project folder. In PyCharm to start a terminal, right click on the top-level folder and select Open In -> Terminal. See the image below:

Figure 17-8: Open In Terminal in PyCharm

Now we can run the terminal command to start our Flask app. See the image below on macOS machine:

Figure 17-9: Running a Flask App in PyCharm Terminal on macOS

When you run these commands, your Flask app will start on your local machine. By default, the IP address for the localhost will be http://127.0.0.1 and the port will be 5000. So, the complete URL will become http://127.0.0.1:5000. Open this URL in a browser window and add the ‘/hi’ route to the end as: http://127.0.0.1:5000/hi. When you call this URL in the browser, it will render a string as “Hi from Flask App”. See the image below:

Figure 17-10: Flask App running on the browser

If you can see a screen like the one in Figure 17-10, then congratulations, your first Flask app is up and running. You can move to the next section.

The Application Factory

In the previous section, we created a Flask app instance inside the hi.py file. It was just a simple example and in the real-world project, there is a better way of creating Flask apps. We have a special function known as The Application Factory. The Application Factory is responsible for any configuration, registration and other setup operations. And it returns the configured app object, which is an instance of the Flask class.

flaskapp directory:

In a Flask project the actual project has its own directory. Keep in mind that, the Flask project is not the same thing as the PyCharm project. Flask project is a sub-folder of the PyCharm project. So, inside our MovieReview PyCharm project we will have a Flask project as flaskapp. Here is the folder structure:

Figure 17-11: Flask App inside the PyCharm project

As you see in Figure 17-11, MovieReview is our PyCharm Project, and the flaskapp is the Flask project in it. flaskapp is the directory where we will instantiate a Flask instance. Actually, it will be a Python package because it will contain the __init__.py file.

Now let’s create the flaskapp directory (package) in our MovieReview project, if you haven’t already. Make sure you have __init__.py file in it. The __init__.py will contain the application factory for our Flask app, and it tells Python that the flaskapp directory should be treated as a package. Here is the content of this file:

[2]:

1

# __init__.py file

2

3

import os

4

from flask import Flask

5

from . import db, register, login, home, review

6

7

# application factory

8

def create_app(test_config=None):

9

    # TODO - create a Flask app object

10

    # with __name__ and instance_relative_config=True

11

12

    # configure the app

13

    app.config.from_mapping(

14

        SECRET_KEY='dev',

15

        DATABASE=os.path.join(app.instance_path, 'flaskapp.sqlite'),

16

    )

17

18

    # create the instance folder if not exists

19

    try:

20

        os.makedirs(app.instance_path)

21

    except OSError:

22

        pass

23

24

    # db registration

25

    # TODO - register the database

26

27

    # register Blueprints

28

    # register

29

    # login

30

    # home

31

    # review

32

33

    # TODO - create a simple test to see if it works

34

    # '/test'

35

36

    # TODO - return the configured Flask app object

In cell 2, you see __init__.py file content. This is the initial form of this file and we will modify it constantly. The create_app() function is the application factory itself. Make sure you give exactly the same name, because “create_app” is the name Flask uses for application factory by default.

In line 9 we create a Flask app instance as: app = Flask(__name__, instance_relative_config=True). The parameters are:

__name__: name of the current Python module. This tells the app where it’s located to set up relative paths.

instance_relative_config=True: This parameter tells the app that configuration files are relative to the instance folder. The instance folder is a special folder which is located outside the flaskapp package and holds local data like configuration secrets and database files. It will automatically be created when we run Flask in a minute.

In line 12, we configure our Flask app object. The app.config.from_mapping() method sets some default configuration that the app will use like SECRET_KEY and DATABASE. We will use SQLite database for this project.

SECRET_KEY is used by Flask to keep the app data safe. During development you can set it to 'dev', but you should override it with a random value before deploying the app.

DATABASE is the path where we save the SQLite database file. It’s under app.instance_path, which is the path that Flask has chosen for the instance folder. Don’t worry, it will make sense when we create our database.

In line 19, we create the instance folder as: os.makedirs(app.instance_path). This folder will hold our local database file when we initialize the db.

In line 24, we have a simple route for test purposes. We want to see if our Flask app works when we run this code. The url for this route is simply ‘/test’.

And finally, in line 29, we return the configured Flask app object.

Now let’s run this code and test it. To run it, start a new Terminal window in PyCharm and set a new environment variable for our flaskapp project. Make sure you are at the top-level folder in the PyCharm project which is MovieReview directory.

Bash (macOS and Linux):

------------------------

$ export FLASK_APP=flaskapp

$ export FLASK_ENV=development

$ flask run

CMD (Windows):

------------------------

> set FLASK_APP=flaskapp

> set FLASK_ENV=development

> flask run

Powershell (Windows):

------------------------

> $env:FLASK_APP = "flaskapp"

> $env:FLASK_ENV = "development"

> flask run

This time the environment variable name is flaskapp, which is the same name with our flask project. If you run this code in your PyCharm terminal, then it should start a new Flask app on the given local IP address and port. You should see an output similar to the one below when you run these commands:

(venv) MovieReview % export FLASK_APP=flaskapp  

(venv) MovieReview % export FLASK_ENV=development

(venv) MovieReview % flask run                  

* Serving Flask app 'flaskapp' (lazy loading)

* Environment: development

* Debug mode: on

* Running on http://127.0.0.1:5000 (Press CTRL+C to quit)

* Restarting with stat

* Debugger is active!

* Debugger PIN: 885-298-466

Now let’s check it on the web browser. When you type http://127.0.0.1:5000/test in the browser, you should see a text of “Flaskapp init file works fine :)”. See the images below:

Figure 17-12: Running flaskapp in PyCharm

Figure 17-13: The result in the browser for http://127.0.0.1:5000/test

If you see a similar result like the one in Figure 17-13, then your application factory is set up correctly. You can move on to the next section now. Which is about creating a local database.

Create The Database

As we mentioned earlier, we will use SQLite database to store our app data. SQLite is a light-weight database server which doesn’t require setting up a separate db server and is built-in to Python. It is convenient for small projects, however for large applications it may not serve well due to its limitations.

We will create a separate Python file in our flaskapp folder, called db.py. This file will setup and initialize the database. Here is the code in this file:

[3]:

1

# db.py file

2

3

import sqlite3

4

import click

5

from flask import current_app, g

6

from flask.cli import with_appcontext

7

8

# db connection

9

def get_db():

10

    if 'db' not in g:

11

        g.db = # TODO - create sqlite3 connection

12

        g.db.row_factory = sqlite3.Row

13

    return g.db

14

15

# close connection

16

def close_db(e=None):

17

    db = g.pop('db', None)

18

    # TODO - check if connection exists and close

19

20

# initialize db

21

def init_db():

22

    db = # TODO - get db

23

    with current_app.open_resource('schema.sql') as f:

24

        # TODO - execute script on db

25

26

# command line command

27

# TODO - define a click command as 'init-db'

28

# TODO - call with_appcontext decorator

29

def init_db_command():

30

    """Clear the existing data and create new tables."""

31

    # TODO - initialize the db and print the result

32

33

# register functions with app instance

34

def init_app(app):

35

    # TODO - call teardown_appcontext fn to close db

36

    # TODO - add cli command to the app

In cell 3, you see the code in our db.py file. In line 5, we import current_app and g objects from the flask package:

g is a special object which is unique for each request. It stores the data which is accessible for multiple functions during the current request.

current_app is an object that points to the current Flask application request.

sqlite3.connect() establishes a connection to the file pointed at by the DATABASE configuration key.

sqlite3.Row tells the connection to return rows that behave like dicts, which allows accessing the columns by their names.

The functions are:

get_db(): creates a db connection and sets it to the g.db attribute

close_db(): checks and closes the db connection if it exists

init_db(): initializes a database by reading the 'schema.sql' file, which contains the SQL commands to create the database tables.

init_db_command(): it calls the init_db() function and prints the result. It is decorated with @click.command('init-db') and @with_appcontext.

click.command() defines a command line command called init-db which calls the init_db() function and shows a success message to the user.

with_appcontext decorator is in the flask.cli module and wraps a callback to guarantee it will be called with a script's application context.

init_app() is used to register the functions with the app instance, which is the incoming parameter. We will call this function from the application factory.

app.teardown_appcontext() tells Flask to call that function when cleaning up after returning the response.

app.cli.add_command() adds a new command that can be called with the flask command. This enables us to use “flask init-db” command in the terminal.

Now let’s create the 'schema.sql' file under the flaskapp directory. Here is the code in this file:

[4]:

1

-- schema.sql file

2

3

DROP TABLE IF EXISTS user;

4

DROP TABLE IF EXISTS review;

5

6

CREATE TABLE user (

7

  id INTEGER PRIMARY KEY AUTOINCREMENT,

8

  username TEXT UNIQUE NOT NULL,

9

  password TEXT NOT NULL,

10

  fullname TEXT NOT NULL

11

);

12

13

CREATE TABLE review (

14

  id INTEGER PRIMARY KEY,

15

  author_id INTEGER NOT NULL,

16

  created TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,

17

  title TEXT NOT NULL,

18

  body TEXT NOT NULL,

19

  FOREIGN KEY (author_id) REFERENCES user (id)

20

);

In cell 4, you see the SQL commands to create two database tables; user and review. Each table has fields in their definition commands. Before running these commands, we need to call the init_app() function from the application factory. Here is updated __init__.py file content:

[5]:

1

# __init__.py file

2

3

import os

4

from flask import Flask

5

from . import db, register, login, home, review

6

7

# application factory

8

def create_app(test_config=None):

9

    # TODO - create a Flask app object

10

    # with __name__ and instance_relative_config=True

11

12

    # configure the app

13

    app.config.from_mapping(

14

        SECRET_KEY='dev',

15

        DATABASE=os.path.join(app.instance_path, 'flaskapp.sqlite'),

16

    )

17

18

    # create the instance folder if not exists

19

    try:

20

        os.makedirs(app.instance_path)

21

    except OSError:

22

        pass

23

24

    # db registration

25

    # TODO - register the database

26

27

    # register Blueprints

28

    # register

29

    # login

30

    # home

31

    # review

32

33

    # TODO - create a simple test to see if it works

34

    # '/test'

35

36

    # TODO - return the configured Flask app object

In cell 5, line 35, we register the database with our application by calling the db.init_app() function and passing the current app object as the argument.

Now we are ready to initialize the database file. All we need to do is to call the “flask init-db” command in the terminal. Make sure you stop the Flask server by pressing CTRL+C to quit. And set the FLASK_APP and the FLASK_ENV environment variables again before running the “flask init-db” command. Here is the output on a macOS computer:

(venv) MovieReview % export FLASK_APP=flaskapp  

(venv) MovieReview % export FLASK_ENV=development

(venv) MovieReview % flask init-db

Successfully initialized the database.

If you see the text of ‘Successfully initialized the database.’ in the output, then your database tables are created and the database is initialized in the application. If you check the “instance” folder in the project now, you should see the flaskapp.sqlite file in it. This is the file where our database will be stored. You can now run “flask run” command to start the server again.

Movies Data and The Utils Module

We will be using movies data which belongs to the top 20 movies in the IMDB Top 250 movies. The data will be in a CSV file inside a folder called ‘data’. File name is imdb_top_20.csv and you can find it in the project directory in GitHub repository of this book.

In our project, we will also have a module called utils.py. This module will contain some common functions, like reading CSV data and getting movies, getting the user and review data from database etc. Here is the content of this file:

[6]:

1

# utils.py file

2

3

from flask import (g, redirect, url_for)

4

from flaskapp.db import get_db

5

import csv

6

import functools

7

8

# function to read all movies

9

def get_movies():

10

    """reads the csv file and returns

11

        the list of movies"""

12

    movies = []

13

    movie_path = 'data/imdb_top_20.csv'

14

    # TODO - read movies at movie_path

15

    # and fill the movies list

16

17

    # return the list

18

    return movies

19

20

# function to read a single movie

21

def get_movie(id):

22

    """returns a single movie"""

23

    movie = None

24

    movie_path = 'data/imdb_top_20.csv'

25

    # TODO - get a single movie with id

26

27

    return movie

28

29

# get movie image

30

def get_movie_img(id):

31

    # TODO - return 'imageURL' of the movie with id

32

33

# get all reviews

34

def get_all_reviews():

35

    db = # TODO - get db

36

    # TODO - get all reviews from DB

37

    # 'SELECT r.id, title, body, created, author_id, username, fullname'

38

    #         ' FROM review r JOIN user u ON r.author_id = u.id'

39

    #         ' ORDER BY created DESC'

40

41

# get review from db

42

def get_review_from_db(id):

43

    # TODO - get a single movie from db with id

44

    # 'SELECT r.id, title, body, created, author_id, username'

45

    #         ' FROM review r JOIN user u ON r.author_id = u.id'

46

    #         ' WHERE r.id = ?'

47

48

# get user

49

def get_user(username):

50

    db = # TODO - get db

51

    # TODO - get as single user from db with username

52

    # 'SELECT * FROM user WHERE username = ?'

53

    return user

54

55

# Require Authentication

56

def login_required(view):

57

    @functools.wraps(view)

58

    def wrapped_view(**kwargs):

59

        if g.user is None:

60

            return redirect(url_for('login.login'))

61

        return view(**kwargs)

62

    return wrapped_view

In cell 6, we have the utils.py file content. Here are the functions in it:

get_movies(): This function reads the csv file and returns the list of movies.

get_movie(): This function returns a single movie based on the given id parameter.

get_movie_img(): This function returns the movie image URL based on the given id.

get_all_reviews(): It returns the list of all reviews in the database. It uses a SQL join statement to join the review and user tables. We call the fetchall() method to get a list of results.

get_review_from_db(): It returns a single review from the database. We use fetchone() for returning a single result.

get_user(): It returns the user data from the database based on the username provided.

login_required(): We will require the user to login to our application before creating, editing and deleting movie reviews. login_required is the decorator which will check if the user is logged in before any of these operations. It returns a new view function that wraps the original view it’s applied to. The new function simply checks if the user is logged in as: if g.user is None. Remember that the object g stores the data in the current request. If g.user is not None, this means that the user is logged in. You will see how we fill it later on. If it is None, then we redirect the user to the login page.

url_for(function, parameters):

The url_for() function in Flask, generates a URL to an endpoint using the function passed in as an argument.

For example, the test() function that was added to the application factory in the previous section, has the name 'test' and can be linked to with url_for('test'). You can think of it as requesting the URL of http://127.0.0.1:5000/test in your web browser.

If it took an argument, which you’ll see later, it would be linked to using url_for('test', param_1=value_1, param_2=value_2).

In line 68, we use this function as url_for('login.login'). The parameter is ‘login.login’. The first ‘login’ is the Blueprint name and the second one is the endpoint name. So, this means that, we redirect the user to an endpoint (a function here) called ‘login’ inside a Blueprint named ‘login’. Don’t worry it will make sense when you learn about the Blueprints.

Views and Blueprints

In a Flask application the UI (user interface) is mainly created by Templates (HTML templates) and the back-end is created by Views and Blueprints.

Views are the functions which respond to the requests. Views handle the incoming request based on the request URL patterns. And they return data that Flask turns into an outgoing response.

Blueprints are the way to organize a group of related views. They can contain any number of view functions. Views are registered with a Blueprint and the Blueprint is registered with the application factory.

In our project we will have four Blueprints which are: home, login, register, and review. Each one will have a separate module and in each of them we will have different View functions to handle incoming requests.

Register Blueprint and View Function

A Blueprint is a class in the flask module, which is used to register the Views (View functions). The name of our first Blueprint will be Register and it will have a View function called register(). Create a new Python file in the flaskapp directory called register.py. And here is the code in this file:

[7]:

1

# register.py file

2

from flask import (Blueprint, flash, redirect,

3

                   render_template, request, session, url_for)

4

from flaskapp.db import get_db

5

from werkzeug.security import generate_password_hash

6

from .utils import get_user

7

8

# initialize the Blueprint object

9

bp = Blueprint('register', __name__, url_prefix='/')

10

11

# Register View Code

12

@bp.route('/register', methods=('GET', 'POST'))

13

def register():

14

    if request.method == 'POST':

15

        # get input fields

16

        username, password, fullname = get_input_fields()

17

        # get db

18

        db = # TODO - get db

19

        # check input fields

20

        error = check_input_fields(username, password, fullname)

21

        # check error

22

        if error is None:

23

            try:

24

                # TODO - insert new user into db

25

            except db.IntegrityError:

26

                error = f"User {username} is already registered."

27

            else:

28

                # TODO - clear session

29

                session.clear()

30

                session['user_id'] = get_user(username)['id']

31

                return # TODO - redirect to url for 'home.index'

32

        flash(error)

33

    return # TODO - render template at 'register/register.html'

34

35

# fn for getting inputs

36

def get_input_fields():

37

    username = request.form['username']

38

    # TODO - get password from request

39

    # TODO - get fullname from request

40

    return username, password, fullname

41

42

# fn for checking input fields

43

def check_input_fields(username, password, fullname):

44

    if not username or not password or not fullname:

45

        return 'Full Name, Username or Password can not be empty.'

46

    else:

47

        return None

48

49

# fn for inserting a DB row

50

def insert_into_db(db, username, password, fullname):

51

    # TODO - insert new row to db

52

    # "INSERT INTO user (username, password, fullname) "

53

    #                "VALUES (?, ?, ?)"

54

55

    # TODO - commit to db

In cell 7, line 9, we create a Blueprint object as: bp = Blueprint('register', __name__, url_prefix='/'). The first parameter is the unique name of this Blueprint the second one is the name of the current application and the third parameter is the URL prefix.

In line 13 you see the definition of the register() view function. Here is what this function does:

@bp.route decorator associates the URL “/register” with the register() view function. When Flask receives a request to http://127.0.0.1:5000/register, it will call the register() view and use the return value as the response. This view function can handle both GET and POST request types.

In line 14, we check the request method being POST, which means the user submits the form in this view. If the method is POST, then we have some validations for the form fields like username, password and fullname. We don’t want any of them to be empty, that’s why we call the check_input_fields() function.

If all of the fields are filled by the user, then we can try to insert this into the user table in db. That’s what we do in line 25, by calling the insert_into_db() function.

If db insert operation is successful, then in the else block in line 29, we clear the old session and add the user id to it. Then we redirect the user to the home page by calling the url_for('home.index') function. We will define the home.index view function later on.

In line 32, we call a special function called flash() by passing the error as the argument. This function stores the error messages that can be retrieved when rendering the template. We will print the error messages later on in our HTML template.

Finally, in line 33, we call the render_template() function by passing the file location of the template. This will render the HTML template for the 'register/register.html' file in the web browser. We will create this file soon.

In line 50, we have the definition of the insert_into_db() function. It simply inserts a new row to the database. For security reasons, we call the generate_password_hash() function instead of saving the password as it is. This will store the hash value of the password.

To be able to use our Register Blueprint, we need to register it inside our application factory as follows:

[8]:

1

# __init__.py file

2

3

import os

4

from flask import Flask

5

from . import db, register, login, home, review

6

7

# application factory

8

def create_app(test_config=None):

9

    # TODO - create a Flask app object

10

    # with __name__ and instance_relative_config=True

11

12

    # configure the app

13

    app.config.from_mapping(

14

        SECRET_KEY='dev',

15

        DATABASE=os.path.join(app.instance_path, 'flaskapp.sqlite'),

16

    )

17

18

    # create the instance folder if not exists

19

    try:

20

        os.makedirs(app.instance_path)

21

    except OSError:

22

        pass

23

24

    # db registration

25

    # TODO - register the database

26

27

    # register Blueprints

28

    # register

29

    # login

30

    # home

31

    # review

32

33

    # TODO - create a simple test to see if it works

34

    # '/test'

35

36

    # TODO - return the configured Flask app object

Now that we have our Register Blueprint and a view function for it, let’s request its URL in the web browser. If you type http://127.0.0.1:5000/register in your browser and press on Enter, you will see an error page as TemplateNotFound. It will say: jinja2.exceptions.TemplateNotFound: register/register.html. This tells us that Flask couldn’t find any Templates for rendering. We will create HTML templates in the succeeding sections.

Templates

Templates are files that contain static data as well as placeholders for dynamic data. A template is rendered with specific data to produce a final document. All the template files in a Flask app need to be located under a special folder called “templates”. This folder should be directly under the flaskapp directory and it is the default location where Flask looks for templates at run time.

Flask uses the Jinja template library to render templates. In our application, we will use templates to render HTML which will display in the user’s browser.

A Jinja template is simply a text file. Jinja can generate any text-based format (HTML, XML, CSV, LaTeX, etc.). A Jinja template doesn’t need to have a specific extension: .html, .xml, or any other extension is just fine.

A template contains variables and/or expressions, which get replaced with values when the template is rendered; and tags, which control the logic of the template.

The template syntax in Jinja is heavily inspired by Python. Special delimiters are used to distinguish Jinja syntax from the static data in the template. Anything between {{ and }} is an expression that will be output to the final document. {% and %} denotes a control flow statement like if and for. Unlike Python, blocks are denoted by start and end tags rather than indentation since static text within a block could change indentation. Here are a few kinds of delimiters:

{% ... %} for Statements

{{ ... }} for Expressions to print to the template output

{# ... #} for Comments not included in the template output

The Base Template

In our application, each page will have the same basic layout around a different body. For example, they will all include a navigation bar at the top. Instead of writing the entire HTML structure in each template, each template will extend a base template and override specific sections. You can think of the base templates as the master pages.

In our flaskapp project directory, we will create a new folder called templates, and inside this folder, create a new HTML file named base.html. Here is the code in this file:

[9]:

1

<!doctype html>

2

<html lang="en" >

3

<head>

4

  <meta charset="UTF-8">

5

  <link rel='stylesheet' href='https://fonts.googleapis.com/css?family=Rubik:400,700'>

6

  <link rel="stylesheet" href="./style.css">

7

  <link rel="stylesheet" href="{{ url_for('static', filename='style.css') }}">

8

  <title>Flask App</title>

9

</head>

10

<body>

11

12

<div class="menu">

13

  <a class="title-anchor" href="{{ url_for('home.index') }}"><h1>Movie Review with Flask</h1></a>

14

  <div class="menu-right">

15

    {% if g.user %}

16

      <h2>{{ g.user['fullname'] }}</h2>

17

      <a href="{{ url_for('login.logout') }}">Log Out</a>

18

    {% else %}

19

      <a href="{{ url_for('register.register') }}">Register</a>

20

      <a href="{{ url_for('login.login') }}">Log In</a>

21

    {% endif %}

22

  </div>

23

</div>

24

<div class="content">

25

  <header>

26

    {% block header %}{% endblock %}

27

  </header>

28

  {% for message in get_flashed_messages() %}

29

    <div class="flash">{{ message }}</div>

30

  {% endfor %}

31

  {% block content %}{% endblock %}

32

</div>

33

</body>

34

</html>

In cell 9, we have the content of base.html file in the templates folder. It is a regular HTML file which includes Jinja commands in it.

In line 15, we check the user in the current request as: {% if g.user %}. And if g.user is not None, we are sure that the user is logged in and we render a logout link button. Else, we render Register and Log In buttons.

In line 16, inside the <h2> tag, we render the user fullname as: {{ g.user['fullname'] }}.

In line 26, we have a placeholder for the header block as: {% block header %}{% endblock %}.

There are two blocks defined in the base template which will be overridden in the child templates:

{% block header %} is where the unique header of each page will replace.

{% block content %} is where the content of each page goes, such as the register or login forms.

In line 28, we check if there are any error messages in the view functions. Remember that we called the flash(error) function in the view code for storing the error messages. And this is where we get these messages by calling the get_flashed_messages() function. We set a for loop to iterate over the messages and render a div element for each one. Here is the code: {% for message in get_flashed_messages() %}.

And finally, in line 31, we have the content block.

Register Template

Now that we have the base template, we can extend it and create a new template called the Register Template. It will contain the HTML code to create our register page. Inside the templates folder create a sub-folder called register. And inside this folder create an HTML document as register.html. Here is the code in this file:

[10]:

1

{% extends 'base.html' %}

2

3

{% block content %}

4

    <div class="login-form">

5

      <form method="post">

6

        <h1>Register</h1>

7

        <div class="content">

8

          <div class="input-field">

9

            <input class="input-text" id="fullname" name="fullname" placeholder="Full Name">

10

          </div>

11

          <div class="input-field">

12

            <input class="input-text" id="username" name="username" placeholder="User Name">

13

          </div>

14

          <div class="input-field">

15

            <input class="input-text" type="password" id="password" name="password" placeholder="Password">

16

          </div>

17

        </div>

18

        <div class="action">

19

          <button type="submit" class="actionbtn">Sign Up</button>

20

        </div>

21

      </form>

22

    </div>

23

{% endblock %}

In cell 10, we have the register template. In the first line you see that it extends the base template as: {% extends 'base.html' %}.

In line 3, we have the structure for the content block. This will replace the content block in the base template. The register Template is a basic HTML form element with some input fields in it.

Remember that in cell 7 line 33, inside the register() view function we had the return statement as: return render_template('register/register.html'). This is where we tell Flask to find and render the template when it matches the specified route, which was: @bp.route('/register', methods=('GET', 'POST')).

Now if request the URL http://127.0.0.1:5000/register in the browser and press on Enter, Flask will try to render the register template. You will still get a BuildError, because we do not have a template named home.index yet. Don’t worry we will create it in a minute.

Login Blueprint & Template

Before creating the Blueprint for Home Page, we will create the Login Page. As we did for the Register page, first we will create a Blueprint with some View functions in it. Then we will create a template as login.html.

Login Blueprint and View Functions:

For the Blueprint, create a Python file named login.py under the flaskapp directory. Here is the code for this file:

[12]:

1

# login.py file

2

3

from flask import (Blueprint, flash, g, redirect,

4

                   render_template, request, session, url_for)

5

from flaskapp.db import get_db

6

from werkzeug.security import check_password_hash

7

8

# initialize the Blueprint object

9

bp = Blueprint('login', __name__, url_prefix='/')

10

11

# Login View Code

12

@bp.route('/login', methods=['GET', 'POST'])

13

def login():

14

    if request.method == 'POST':

15

        username = request.form['username']

16

        password = request.form['password']

17

        db = # TODO - get db

18

        error = None

19

        # TODO - get the user from eb with username

20

21

        # check user from db

22

        if user is None:

23

            error = "Incorrect username. Please try again."

24

        elif not check_password_hash(user['password'], password):

25

            error = "Incorrect password. Please try again."

26

        # check error

27

        if error is None:

28

            # TODO - clear session

29

            session['user_id'] = # TODO - set user id

30

            # TODO - return redirect url for 'home.index'

31

32

        # TODO - flash the error

33

34

    # render template

35

    return # TODO - render template at 'login/login.html'

36

37

# Run before all view functions

38

@bp.before_app_request

39

def load_logged_in_user():

40

    user_id = # TODO - get user id from session

41

    if user_id is None:

42

        g.user = None

43

    else:

44

        g.user = # TODO - get user from db with user_id

45

46

# logout route

47

@bp.route('/logout')

48

def logout():

49

    # TODO - clear session

50

    # TODO - return redirect url for 'login.login'

In cell 12, you see the code in the login.py file. It creates a Blueprint called ‘login’ and define routes for '/login' and '/logout'. Now we have to register this Blueprint with the application factory. Here it is:

[13]:

1

# __init__.py file

2

3

import os

4

from flask import Flask

5

from . import db, register, login, home, review

6

7

# application factory

8

def create_app(test_config=None):

9

    # TODO - create a Flask app object

10

    # with __name__ and instance_relative_config=True

11

12

    # configure the app

13

    app.config.from_mapping(

14

        SECRET_KEY='dev',

15

        DATABASE=os.path.join(app.instance_path, 'flaskapp.sqlite'),

16

    )

17

18

    # create the instance folder if not exists

19

    try:

20

        os.makedirs(app.instance_path)

21

    except OSError:

22

        pass

23

24

    # db registration

25

    # TODO - register the database

26

27

    # register Blueprints

28

    # register

29

    # login

30

    # home

31

    # review

32

33

    # TODO - create a simple test to see if it works

34

    # '/test'

35

36

    # TODO - return the configured Flask app object

Login Template:

Now that we have a Blueprint for the login page, we can define a Template for it. To do this, create a new folder called login under the templates directory. And in that folder, create a file called login.html. Here is the code in this file:

[14]:

1

{% extends 'base.html' %}

2

3

{% block content %}

4

    <div class="login-form">

5

      <form method="post">

6

        <h1>Login</h1>

7

        <div class="content">

8

          <div class="input-field">

9

            <input class="input-text" id="username" name="username" placeholder="User Name">

10

          </div>

11

          <div class="input-field">

12

            <input class="input-text" type="password" id="password" name="password" placeholder="Password">

13

          </div>

14

          <a href="#" class="link">Forgot Your Password?</a>

15

        </div>

16

        <div class="action">

17

          <button class="actionbtn" type="submit">Sign in</button>

18

          <a class="actionbtn" href="{{ url_for('register.register') }}">Register</a>

19

        </div>

20

      </form>

21

    </div>

22

{% endblock %}

Home Blueprint & Template

Now let’s create a Blueprint and a Template for the home page which we will call as index.html.

Home Blueprint and View Function:

For the Blueprint, create a Python file named home.py under the flaskapp directory. Here is the code for this file:

[15]:

1

# home.py file

2

from flask import (Blueprint, g, render_template)

3

from .utils import login_required, get_all_reviews, \

4

    get_movies, get_review_from_db

5

6

# initialize the Blueprint object

7

bp = Blueprint('home', __name__, url_prefix='/')

8

9

# Index View Code

10

@bp.route('/', methods=['GET'])

11

@login_required

12

def index():

13

    # get all movies

14

    movies = # TODO - get movies

15

    # get all reviews

16

    all_reviews = # TODO - get all reviews

17

    # render index.html

18

    return # TODO - render template at 'home/index.html'

19

    # with parameters as: movies, all_reviews, get_if_reviewed

20

21

def get_if_reviewed(movie):

22

    review = # TODO - get review from db for the movie with 'id'

23

    if review is not None and g.user['id'] == review['author_id']:

24

        return True

25

    else:

26

        return False

In the Home Blueprint, we don’t have a specific route for the index() view function. The route is: @bp.route('/', methods=['GET']). Which means it will be the main page when the user opens the application by requesting http://127.0.0.1:5000/.

In line 15 and 16, it gets all the movies and the reviews by calling the respective functions in the utils file.

It renders a template in the 'home/index.html' location which we will create next. Here, we also pass some parameters while rendering the template. Here is the code:

    return render_template('home/index.html', movies=movies,

                           all_reviews=all_reviews,

                           get_if_reviewed=get_if_reviewed)

The first two parameters are list type objects and the third one is a function, which we will call from the HTML template. This is how we pass parameters to templates from the view functions.

Now we have to register this Blueprint with the application factory. Here it is:

[16]:

1

# __init__.py file

2

3

import os

4

from flask import Flask

5

from . import db, register, login, home, review

6

7

# application factory

8

def create_app(test_config=None):

9

    # TODO - create a Flask app object

10

    # with __name__ and instance_relative_config=True

11

12

    # configure the app

13

    app.config.from_mapping(

14

        SECRET_KEY='dev',

15

        DATABASE=os.path.join(app.instance_path, 'flaskapp.sqlite'),

16

    )

17

18

    # create the instance folder if not exists

19

    try:

20

        os.makedirs(app.instance_path)

21

    except OSError:

22

        pass

23

24

    # db registration

25

    # TODO - register the database

26

27

    # register Blueprints

28

    # register

29

    # login

30

    # home

31

    # review

32

33

    # TODO - create a simple test to see if it works

34

    # '/test'

35

36

    # TODO - return the configured Flask app object

Index Template:

Now that we have a Blueprint for the home page, we can define a Template for it. To do this, create a new folder called home under the templates directory. And in that folder, create a file called index.html. Here is the code in this file:

[17]:

1

{% extends 'base.html' %}

2

3

{% block content %}

4

    <div class="login-form">

5

      <form method="post">

6

        <h1>Login</h1>

7

        <div class="content">

8

          <div class="input-field">

9

            <input class="input-text" id="username" name="username" placeholder="User Name">

10

          </div>

11

          <div class="input-field">

12

            <input class="input-text" type="password" id="password" name="password" placeholder="Password">

13

          </div>

14

          <a href="#" class="link">Forgot Your Password?</a>

15

        </div>

16

        <div class="action">

17

          <button class="actionbtn" type="submit">Sign in</button>

18

          <a class="actionbtn" href="{{ url_for('register.register') }}">Register</a>

19

        </div>

20

      </form>

21

    </div>

22

{% endblock %}

At this point, if you run the app and type http://127.0.0.1:5000/ in your web browser, you should a screen like the one below:

Figure 17-14: Login page which is redirected from the Home page

For the time being, the app doesn’t look good because we have no formatting yet. Don’t worry we will add some CSS and it will look pretty.

Review Blueprint & Template

The last Blueprint which we will add is the Review Blueprint. It will be responsible for creating new reviews, updating existing ones, deleting reviews and displaying a list of all reviews.

Review Blueprint and View Functions:

For the Blueprint, create a Python file named review.py under the flaskapp directory. Here is the code for this file:

[18]:

1

# review.py file

2

from flask import (

3

    Blueprint, flash, g, redirect,

4

    render_template, request, url_for)

5

from werkzeug.exceptions import abort

6

from flaskapp.db import get_db

7

from .utils import login_required, get_all_reviews, \

8

    get_movie, get_review_from_db, get_movie_img

9

10

bp = Blueprint('review', __name__)

11

12

# reviews route

13

@bp.route('/reviews')

14

@login_required

15

def reviews():

16

    TODO - get all reviews

17

    # TODO - render template at 'review/reviews.html'

18

    # with parameters: all_reviews, get_movie_img

19

20

# create a new review

21

@bp.route('/<int:id>/create', methods=('GET', 'POST'))

22

@login_required

23

def create(id):

24

    # TODO - get movie with id

25

26

    if # TODO - check if method is POST

27

        # TODO - get title and body from the form

28

        error = None

29

        if not body:

30

            error = 'You need to add a review.'

31

        if error is not None:

32

            flash(error)

33

        else:

34

            # TODO - get db

35

            # TODO - insert a new review into db

36

            # 'INSERT INTO review (id, title, body, author_id)'

37

            #                 ' VALUES (?, ?, ?, ?)',

38

39

            # TODO - commit to db

40

41

            # TODO - redirect to 'review.reviews'

42

43

    # TODO - render template at 'review/create.html'

44

    # with parameter: movie

45

46

# get a single review

47

def get_review(id, check_author=True):

48

    # TODO - get review from db with id

49

    if review is None:

50

        abort(404, f"Post id {id} doesn't exist.")

51

    if check_author and review['author_id'] != g.user['id']:

52

        abort(403)

53

    return review

54

55

# update review

56

@bp.route('/<int:id>/update', methods=('GET', 'POST'))

57

@login_required

58

def update(id):

59

    # TODO - get review from db with id

60

    if request.method == 'POST':

61

        # TODO - get title and body from the form

62

        error = None

63

        if not title:

64

            error = 'Title is required.'

65

        if error is not None:

66

            flash(error)

67

        else:

68

            # TODO - get db

69

            # TODO - update the review in db

70

            # 'UPDATE review SET title = ?, body = ?'

71

            #                 ' WHERE id = ?'

72

73

            # TODO - commit to db

74

75

            # TODO - redirect to 'review.reviews'

76

77

    # TODO - render template at 'review/update.html'

78

    # with parameter: review

79

80

# delete review

81

@bp.route('/<int:id>/delete', methods=('POST',))

82

@login_required

83

def delete(id):

84

    # TODO - get review with id

85

86

    # TODO - get db

87

    # TODO - delete the review with id

88

    # 'DELETE FROM review WHERE id = ?'

89

90

    # TODO - commit to db

91

92

    # TODO - redirect to 'review.reviews'

Review Templates:

We will have three separate templates for review operations:

create.html

update.html

reviews.html

To define these files, create a new folder called review under the templates directory. Here are the file contents:

review/create.html

[19]:

1

{% extends 'base.html' %}

2

3

{% block content %}

4

  <div class="login-form">

5

      <form method="post">

6

        <h1>{{ movie['title'] }}</h1>

7

        <div class="content">

8

          <div class="input-field">

9

            <input class="input-text" id="title" name="title" value="Review for: {{ movie['title'] }}">

10

          </div>

11

          <div class="input-field">

12

            <textarea class="input-text" id="body"

13

                      name="body" placeholder="Review" rows="6"></textarea>

14

          </div>

15

        </div>

16

        <div class="action">

17

          <button class="actionbtn" type="submit">Create Review</button>

18

        </div>

19

      </form>

20

    </div>

21

{% endblock %}

review/update.html:

[20]:

1

{% extends 'base.html' %}

2

3

{% block content %}

4

  <div class="login-form">

5

      <form method="post">

6

        <h1>{{ review['title'] }}</h1>

7

        <div class="content">

8

          <div class="input-field">

9

            <input class="input-text" id="title" name="title"

10

                   value="{{ request.form['title'] or review['title'] }}">

11

          </div>

12

          <div class="input-field">

13

            <textarea class="input-text" id="body"

14

                      name="body" rows="6">

15

              {{ request.form['body'] or review['body'] }}

16

            </textarea>

17

          </div>

18

        </div>

19

        <div class="action">

20

          <button class="actionbtn" type="submit">Save</button>

21

        </div>

22

      </form>

23

24

      <form class="login-form"

25

            action="{{ url_for('review.delete', id=review['id']) }}"

26

            method="post">

27

        <div class="action">

28

          <input class="actionbtn danger" type="submit"

29

                 value="Delete" onclick="return confirm('Are you sure?');">

30

        </div>

31

      </form>

32

    </div>

33

{% endblock %}

review/reviews.html:

[21]:

1

{% extends 'base.html' %}

2

3

{% block header %}

4

  <div class="pageheader">

5

    <h1>{% block title %}All Reviews{% endblock %}</h1>

6

  </div>

7

{% endblock %}

8

9

{% block content %}

10

  {% for review in all_reviews %}

11

    <article class="review">

12

      <div class="review-left">

13

        <div class="article-img">

14

          <img src="{{ get_movie_img(review['id']) }}" style="width: 100px;" />

15

        </div>

16

        <div class="article-content">

17

          <header>

18

            <div>

19

              <h3>{{ review['title'] }}</h3>

20

              <h5 class="about">

21

                {{ review['fullname'] }}, {{ review['created'].strftime('%Y-%m-%d') }}

22

              </h5>

23

            </div>

24

          </header>

25

          <p class="article-body">{{ review['body'] }}</p>

26

        </div>

27

      </div>

28

      <div class="review-right">

29

        {% if g.user['id'] == review['author_id'] %}

30

            <a class="edit-button"

31

               href="{{ url_for('review.update', id=review['id']) }}">Edit</a>

32

          {% endif %}

33

      </div>

34

    </article>

35

    {% if not loop.last %}

36

      <hr>

37

    {% endif %}

38

  {% endfor %}

39

{% endblock %}

Finally, please do not forget to register the Review Blueprint with the application factory. Here is the final form of the __init__.py file:

[22]:

1

# __init__.py file

2

3

import os

4

from flask import Flask

5

from . import db, register, login, home, review

6

7

# application factory

8

def create_app(test_config=None):

9

    # create a Flask app object

10

    app = Flask(__name__, instance_relative_config=True)

11

12

    # configure the app

13

    app.config.from_mapping(

14

        SECRET_KEY='dev',

15

        DATABASE=os.path.join(app.instance_path, 'flaskapp.sqlite'),

16

    )

17

18

    # create the instance folder if not exists

19

    try:

20

        os.makedirs(app.instance_path)

21

    except OSError:

22

        pass

23

24

    # db registration

25

    db.init_app(app)

26

27

    # register Blueprints

28

    app.register_blueprint(register.bp)

29

    app.register_blueprint(login.bp)

30

    app.register_blueprint(home.bp)

31

    app.register_blueprint(review.bp)

32

33

    # a simple test to see if it works

34

    @app.route('/test')

35

    def test():

36

        return 'Flaskapp init file works fine :)'

37

38

    # return the configured Flask app object

39

    return app

Static Files

Static files are CSS files and other types of files with JavaScript functions, or a logo image. They should be placed under the flaskapp/static directory and referenced with url_for('static', filename='...').

We have already addressed the CSS file named style.css in the base.html file as follows:

<link rel="stylesheet" href="{{ url_for('static', filename='style.css') }}">

Now let’s create this file. You can find the file style.css file in the project folder in the GitHub repository of the book. Here is the final image of the PyCharm project:

Figure 17-15: PyCharm project structure finalized

After you add the CSS file, the app will look really nice. Here is the image of the home page:

Figure 17-16: Movie Review app completed

This finalizes our project of API Development with Flask.

19.        Final Exam

The Final Exam includes the topics we have covered in this book.

It is a multiple-choice exam with 20 questions.

Each question is 5 points, which makes 100 points in total.

You should get at least 70 points to be successful. The duration is 120 minutes.

Here is the number of questions in the respective chapters:

Topic

Questions

Collections

2

Iterators

2

Generators

2

Date and Time

2

Decorators

2

Context Managers

2

Functional Programming

2

Regular Expressions

2

Database Operations

2

Concurrency

2

Total

20

If you want to solve the final exam in Jupyter environment, you can download the notebook file for the final exam, Final_Exam.ipynb, from the GitHub Repository of this book.

Here are the questions for the Final Exam:

QUESTIONS:

Q1:

We have some dummy text as: "lorem ipsum dolor sit amet consectetur adipiscing elit".

We want to count occurrences of each letter in this text.

We will print the most common four letters in this text.

We don't want to count space characters.

Which function below can succeed this task?

A

from collections import Counter

text = "lorem ipsum dolor sit amet consectetur adipiscing elit"

def count_and_print_most_common(text):

    text_counter = Counter(text)

    return text_counter.most_common(4)

most_5 = count_and_print_most_common(text)

print(most_5)

B

from collections import Counter

text = "lorem ipsum dolor sit amet consectetur adipiscing elit"

def count_and_print_most_common(text):

    text = ''.join(text.split())

    text_counter = Counter(text)

    return text_counter.most_common(4)

most_5 = count_and_print_most_common(text)

print(most_5)

C

from collections import Counter

text = "lorem ipsum dolor sit amet consectetur adipiscing elit"

def count_and_print_most_common(text):

    text = ''.join(text.split())

    text_counter = Counter(text)

    return text_counter

most_5 = count_and_print_most_common(text)

print(most_5)

D

from collections import Counter

text = "lorem ipsum dolor sit amet consectetur adipiscing elit"

def count_and_print_most_common(text):

    text = ''.join(text.split())

    return text.most_common(4)

most_5 = count_and_print_most_common(text)

print(most_5)

Q2:

We want to define a namedtuple as SuperHero.

SuperHero has three attributes:

hero

name

age

Create two SuperHero instance as follows:

Batman, Bruce Wayne, 28

Superman, Clark Kent, 26

We will print the names of SuperHeroes using keys as follows:

Bruce Wayne is Batman

Clark Kent is Superman

Which option can print this output?

A

from collections import namedtuple

SuperHero = namedtuple(['hero', 'name', 'age'])

S_1 = SuperHero('Batman', 'Bruce Wayne', '28')

S_2 = SuperHero('Superman', 'Clark Kent', '26')

print(f"{S_1.name} is {S_1.hero}")

print(f"{S_2.name} is {S_2.hero}")

B

from collections import namedtuple

SuperHero = namedtuple('SuperHero', 'hero', 'name', 'age')

S_1 = SuperHero('Batman', 'Bruce Wayne', '28')

S_2 = SuperHero('Superman', 'Clark Kent', '26')

print(f"{S_1.name} is {S_1.hero}")

print(f"{S_2.name} is {S_2.hero}")

C

from collections import namedtuple

SuperHero = namedtuple('SuperHero', ['hero', 'name', 'age'])

S_1 = SuperHero('Batman', 'Bruce Wayne', '28')

S_2 = SuperHero('Superman', 'Clark Kent', '26')

print(f"{S_1.name} is {S_1.hero}")

print(f"{S_2.name} is {S_2.hero}")

D

from collections import namedtuple

SuperHero = namedtuple('SuperHero', ['hero', 'name', 'age'])

S_1 = SuperHero('Batman', 'Bruce Wayne', '28')

S_2 = SuperHero('Superman', 'Clark Kent', '26')

print(f"{S_1.hero} is {S_1.name}")

print(f"{S_2.hero} is {S_2.name}")

Q3:

In Python, Iterator objects are required to support two special methods, which together form the Iterator Protocol.

What are these methods?

A

* __init__()

* __self__()

B

* __init__()

* __iter__()

C

* __next__()

* __call__()

D

* __iter__()

* __next__()

Q4:

Define an iterator class which will generate a sequence of odd numbers like 1, 3, 5, 7, 9, … etc.

Class name will be OddsIterator.

Instantiate an object of this class with name odd_numbers.

The odd_numbers object will be an iterator that provides odd numbers from 1 to 19.

Print the odd numbers using this object inside a for loop.

Make sure it raises StopIteration exception if you want to print a number bigger than 19.

Which one below is NOT a correct definition for this iterator?

A

class OddsIterator:

    def __init__(self, limit):

        self.current = 1

        self.limit = limit

    def __iter__(self):

        return self

    def __next__(self):

        if self.current <= self.limit:

            current_value = self.current

            self.current += 2

            return current_value

odd_numbers = OddsIterator(19)

for i in range(20):

    print(odd_numbers.__next__())

B

class OddsIterator:

    def __init__(self, limit):

        self.current = 1

        self.limit = limit

    def __iter__(self):

        return self

    def __next__(self):

        if self.current <= self.limit:

            current_value = self.current

            self.current += 2

            return current_value

        raise StopIteration

odd_numbers = OddsIterator(19)

for i in range(20):

    print(next(odd_numbers))

C

class OddsIterator:

    def __init__(self, limit):

        self.current = 1

        self.limit = limit

    def __iter__(self):

        return self

    def __next__(self):

        if self.current <= self.limit:

            current_value = self.current

            self.current += 2

            return current_value

        else:

            raise StopIteration

odd_numbers = OddsIterator(19)

for i in range(20):

    print(odd_numbers.__next__())

D

class OddsIterator:

    counter = 0

    def __init__(self, limit):

        global counter

        self.current = 1

        counter = limit

    def __iter__(self):

        return self

    def __next__(self):

        if self.current <= counter:

            current_value = self.current

            self.current += 2

            return current_value

        raise StopIteration

odd_numbers = OddsIterator(19)

for i in range(20):

    print(next(odd_numbers))

Q5:

We want to define a generator function that gives the list of first n positive integers which are multiples of m.

Function name will be positive_multiples and it will take two parameters:

n: number of positive numbers

m: number which we will use to get multiples

For example, if n=7 and m=5 then we will get the following list:

[5, 10, 15, 20, 25, 30, 35]

We will call this function and print the first 7 positive multiples of 5.

Which one below is appropriate for this task?

A

def positive_multiples(n, m):

    for i in range(n):

        yield i * m

multiples = positive_multiples(7, 5)

print(list(multiples))

B

def positive_multiples(n, m):

    for i in range(n):

        return (i+1) * m

multiples = positive_multiples(7, 5)

print(list(multiples))

C

def positive_multiples(n, m):

    for i in range(n):

        yield (i+1) * m

multiples = positive_multiples(7, 5)

print(list(multiples))

D

def positive_multiples(n, m):

    counter = 1

    while counter <= n:

        return counter * m

        counter += 1

multiples = positive_multiples(7, 5)

print(list(multiples))

Q6:

We want to define a generator expression.

This is going to be a one-line generator.

It will return an iterator which contains the cubes of even elements in the numbers list.

Here is the numbers list:

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

Convert the resulting iterator to a list.

Then print the items in this list.

Here is the expected output:

[8, 64, 216, 512, 1000]

Which cell below completes this operation successfully?

A

numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

even_cubes_gen = (i**3 for i in numbers if i % 2 == 1)

even_cubes_list = list(even_cubes_gen)

print(even_cubes_list)

B

numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

even_cubes_gen = (i**2 for i in numbers if i % 2 == 0)

even_cubes_list = list(even_cubes_gen)

print(even_cubes_list)

C

numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

even_cubes_gen = (i*3 for i in numbers if i % 2 == 1)

even_cubes_list = list(even_cubes_gen)

print(even_cubes_list)

D

numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

even_cubes_gen = (i**3 for i in numbers if i % 2 == 0)

even_cubes_list = list(even_cubes_gen)

print(even_cubes_list)

Q7:

Create a datetime object corresponding to date string of "%d/%m/%y %H:%M".

Here are the date values:

year: 2022

month: 4

day: 19

hour: 16

minutes: 36

Then print the following text by formatting this datetime object:

"Tuesday, 19. April 2022 04:36PM"

Which cell below completes this operation successfully?

A

from datetime import datetime

dt = datetime.strptime("22/4/19 16:36", "%d/%m/%y %H:%M")

formatted_dt = dt.strftime("%A, %d. %B %Y %I:%M%p")

print(formatted_dt)

B

from datetime import datetime

dt = datetime.strptime("19/4/22 16:36", "%d/%m/%y %H:%M")

formatted_dt = dt.strftime("%A, %d. %B %Y %I:%M%p")

print(formatted_dt)

C

from datetime import datetime

dt = datetime.strptime("19/4/22 16:36", "%d/%m/%y")

formatted_dt = dt.strftime("%A, %d. %B %Y %I:%M%p")

print(formatted_dt)

D

from datetime import datetime

dt = datetime.strptime("19/4/22 16:36", "%d/%m/%Y %H:%M")

formatted_dt = dt.strftime("%A, %d. %B %Y %I:%M%p")

print(formatted_dt)

Q8:

Create a date object from ISO format.

The date will be: 2021-10-25.

Then print the date in following format:

"Monday 25. October 2021"

Which cell below completes this operation successfully?

A

from datetime import date

d = date.fromisoformat('2021-10-25')

print(d.isoformat())

B

from datetime import date

d = date.fromisoformat('2021-10-25')

print(d.strftime("%d/%m/%y"))

C

from datetime import date

d = date.fromisoformat('2021-10-25')

print(d.strftime("%A %d. %B %Y"))

D

from datetime import date

d = date.fromisoformat('2021-10-25')

print(d.strftime("%d. %m %y"))

Q9:

We have a basic function which takes one argument and simply returns it.

Here is this function:

def text_to_list(text):

    return text

We want this function to always return the characters of the text in a list.

For example, when we call it as:

text_to_list('python advanced')

We want it to return:

['p', 'y', 't', 'h', 'o', 'n', ' ', 'a', 'd', 'v', 'a', 'n', 'c', 'e', 'd', '.']

But we do not want to modify the function body.

Function definition will not be changed.

What we need is a decorator to do the list conversion for our function.

So, you need to define this decorator.

Which cell below completes this operation successfully?

A

def list_decorator(func):

    def wrapper(text):

        func_result = func(text)

        return list(func_result)

    return wrapper

@list_decorator

def text_to_list(text):

    return text

list_from_text = text_to_list('python advanced.')

print(list_from_text)

B

def list_decorator(func):

    def wrapper(text):

        func_result = func(text)

        yield list(func_result)

    return wrapper

@list_decorator

def text_to_list(text):

    return text

list_from_text = text_to_list('python advanced.')

print(list_from_text)

C

def list_decorator(func):

    def wrapper(text):

        func_result = func(text)

        return func_result

    return wrapper

@list_decorator

def text_to_list(text):

    return text

list_from_text = text_to_list('python advanced.')

print(list_from_text)

D

def list_decorator(func):

    def wrapper(text):

        return func(text)

    return wrapper

@list_decorator

def text_to_list(text):

    return text

list_from_text = text_to_list('python advanced.')

print(list_from_text)

Q10:

Define a class decorator named FunctionDetails.

This decorator will print the function arguments as follows:

Parameters: (a tuple of parameters)

Here is the function which we want to decorate:

def some_function(first, last, age, a_list):

    return (first, last, age, a_list)

Call the function after you decorate it with FunctionDetails.

When you call it, it should print the parameters as following:

some_function('jane', 'doe', 28, [1,2,3])

-----

Parameters: ('jane', 'doe', 28, [1, 2, 3])

Which one below can achieve this?

A

class FunctionDetails:

    def __call__(self, *args):

        print(f"Parameters: {args}")

@FunctionDetails

def some_function(first, last, age, a_list):

    return (first, last, age, a_list)

some_function('jane', 'doe', 28, [1,2,3])

B

class FunctionDetails:

    def __init__(self, func):

        self.func = func

    def __iter__(self, *args):

        print(f"Parameters: {args}")

@FunctionDetails

def some_function(first, last, age, a_list):

    return (first, last, age, a_list)

some_function('jane', 'doe', 28, [1,2,3])

C

class FunctionDetails:

    def __init__(self, func):

        self.func = func

    def __call__(self):

        print(f"Parameters: {args}")

@FunctionDetails

def some_function(first, last, age, a_list):

    return (first, last, age, a_list)

some_function('jane', 'doe', 28, [1,2,3])

D

class FunctionDetails:

    def __init__(self, func):

        self.func = func

    def __call__(self, *args):

        print(f"Parameters: {args}")

@FunctionDetails

def some_function(first, last, age, a_list):

    return (first, last, age, a_list)

some_function('jane', 'doe', 28, [1,2,3])

Q11:

Define a custom context manager class as CustomContextManager.

Define the necessary methods in it.

Your context manager should open and return the file object at the specified path.

Also, it should close the file on exit.

The parameters to the class constructor should be:

path: file path for the file to read

mode: operation mode (read, write, append etc.)

Which one is the correct definition of this context manager class?

A

class CustomContexManager:

    def __init__(self, path, mode):

        self.path = path

        self.mode = mode

        self.file = None

    def __iter__(self):

        self.file = open(self.path, self.mode)

        return self.file

    def __exit__(self, exc_type, exc_value, exc_traceback):

        self.file.close()

B

class CustomContexManager:

    def __init__(self, path, mode):

        self.path = path

        self.mode = mode

        self.file = None

    def __enter__(self):

        self.file = open(self.path, self.mode)

        return self.file

    def __exit__(self, exc_type, exc_value, exc_traceback):

        self.file.close()

C

class CustomContexManager:

    def __init__(self, path, mode):

        self.path = path

        self.mode = mode

        self.file = None

    def __enter__(self):

        self.file = open(self.path, self.mode)

        return self.file

D

class CustomContexManager:

    def __init__(self, path, mode):

        self.path = path

        self.mode = mode

        self.file = None

    def __enter__(self):

        self.file = open(self.path, self.mode)

    def __exit__(self, exc_type, exc_value, exc_traceback):

        self.file.close()

Q12:

Define a function-based context manager which creates a file and writes some text in it.

The name will be file_writer and it will take two parameters:

path: file path for the file to read

mode: operation mode (read, write, append etc.)

Use this context manager in a with statement to create a file and add the text below in this file.

File name should be: "text_file_for_writing.txt".

And the text is:

"The file is created in the final exam.

It is a function-based context manager,

decorated with contextlib.contextmanager."

Which one below can achieve this?

A

from contextlib import contextmanager

@contextmanager

def file_writer(path, mode='w'):

    file_object = None

    try:

        file_object = open(path, mode=mode)

        return file_object

    finally:

        if file_object:

            file_object.close()

with file_writer("text_file_for_writing.txt") as file:

    file.write("The file is created in the final exam.\n"

               "It is a function-based context manager,\n"

               "decorated with contextlib.contextmanager.")

B

from contextlib import contextmanager

@contextmanager

def file_writer(path, mode='r'):

    file_object = None

    try:

        file_object = open(path, mode=mode)

        yield file_object

    finally:

        if file_object:

            file_object.close()

with file_writer("text_file_for_writing.txt") as file:

    file.write("The file is created in the final exam.\n"

               "It is a function-based context manager,\n"

               "decorated with contextlib.contextmanager.")

C

from contextlib import contextmanager

@contextmanager

def file_writer(path, mode='w'):

    file_object = None

    try:

        file_object = open(path, mode=mode)

        yield file_object

    finally:

        if file_object:

            file_object.close()

with file_writer("text_file_for_writing.txt") as file:

    file.write("The file is created in the final exam.\n"

               "It is a function-based context manager,\n"

               "decorated with contextlib.contextmanager.")

D

from contextlib import contextmanager

def file_writer(path, mode='w'):

    file_object = None

    try:

        file_object = open(path, mode=mode)

        yield file_object

    finally:

        if file_object:

            file_object.close()

with file_writer("text_file_for_writing.txt") as file:

    file.write("The file is created in the final exam.\n"

               "It is a function-based context manager,\n"

               "decorated with contextlib.contextmanager.")

Q13:

We have a list of numbers which we want to filter only the positive ones.

Here is the list:

[5, -16, 4, -3, 0, -12, 9, 1, 2, -7, 8]

Define a filter() function which gives us the positive numbers in this list.

It should take a lambda function and this list as the parameters.

The filter function will return a filter object.

You should convert this filter object to a list.

Finally, you should sort the list in ascending order in-place.

The resulting list should be:

[1, 2, 4, 5, 8, 9]

Which one below can achieve this?

A

all_numbers = [5, -16, 4, -3, 0, -12, 9, 1, 2, -7, 8]

numbers_filtered = filter(lambda x: x < 0, all_numbers)

positives = list(numbers_filtered)

positives.sort()

print(positives)

B

all_numbers = [5, -16, 4, -3, 0, -12, 9, 1, 2, -7, 8]

numbers_filtered = filter(lambda x: x > 0, all_numbers)

positives = list(numbers_filtered)

positives.sort()

print(positives)

C

all_numbers = [5, -16, 4, -3, 0, -12, 9, 1, 2, -7, 8]

numbers_filtered = filter(lambda x: x, all_numbers)

positives = list(numbers_filtered)

positives.sort()

print(positives)

D

all_numbers = [5, -16, 4, -3, 0, -12, 9, 1, 2, -7, 8]

numbers_filtered = filter(all_numbers, lambda x: x > 0)

positives = list(numbers_filtered)

positives.sort()

print(positives)

Q14:

We want define a function which multiplies all the items in the given list.

The initial value will be 100, which means it will start by multiplying the first item with 100.

Here is the list to multiply:

[4, 5, 6, 7, 8]

The expected result is:

672000

Which one below can achieve this task?

A

from functools import reduce

list_mult = [4, 5, 6, 7, 8]

multiplication = reduce(lambda x, y: x * y, list_mult, 100)

print(multiplication)

B

from functools import reduce

list_mult = [4, 5, 6, 7, 8]

multiplication = map(lambda x, y: x * y, list_mult, 100)

print(multiplication)

C

from functools import reduce

list_mult = [4, 5, 6, 7, 8]

multiplication = map(lambda x, y: x * y, list_mult)

print(multiplication)

D

from functools import reduce

list_mult = [4, 5, 6, 7, 8]

multiplication = reduce(lambda x, y: x * y, list_mult)

print(multiplication)

Q15:

Define a regular expression pattern in the following way. It should:

start with 2 digits

a dot (.) after 2 digits

3 alphabetical upper case letters (A to Z)

a dash (-)

zero or more word characters after the dash

Here are some examples:

"47.UPS-333"

"16.ARC-c"

"04.XYZ-"

What is the correct RE pattern for this?

A

import re

pattern = re.compile(r"\d{2}.[A-Z]{3}-\w+")

m = pattern.search("47.UPS-333")

m_2 = pattern.search("16.ARC-c")

m_3 = pattern.search("04.XYZ-")

print(m.group())

print(m_2.group())

print(m_3.group())

B

import re

pattern = re.compile(r"\d{2}.[A-Z]{3}-\w")

m = pattern.search("47.UPS-333")

m_2 = pattern.search("16.ARC-c")

m_3 = pattern.search("04.XYZ-")

print(m.group())

print(m_2.group())

print(m_3.group())

C

import re

pattern = re.compile(r"\d{2}.[A-Z]{3}-\w*")

m = pattern.search("47.UPS-333")

m_2 = pattern.search("16.ARC-c")

m_3 = pattern.search("04.XYZ-")

print(m.group())

print(m_2.group())

print(m_3.group())

D

import re

pattern = re.compile(r"\d{2}.[A-Z]-\w*")

m = pattern.search("47.UPS-333")

m_2 = pattern.search("16.ARC-c")

m_3 = pattern.search("04.XYZ-")

print(m.group())

print(m_2.group())

print(m_3.group())

Q16:

Define a function named find_all_with_re.

The function will take a regular expression and a text as parameters.

Ant will return the list of all possible matches of this regex in the text.

The regex will be: all words starting with 'c' or 'e' (a word must include at least two characters).

The text is:

"Lorem ipsum dolor sit amet, a consectetur adipiscing elit. Sed id cid tempor risus. Quisque imperdiet, neque c pulvinar sollicitudin, augue nisl varius nibh, e suscipit cerat non lectus."

Expected Result:

['consectetur', 'elit', 'cid', 'cerat']

Which one below can achieve this task?

A

import re

def find_all_with_re(regex, text):

    return re.find(regex, text)

text = """Lorem ipsum dolor sit amet,

a consectetur adipiscing elit.

Sed id cid tempor risus.

Quisque imperdiet, neque c pulvinar sollicitudin,

augue nisl varius nibh, e suscipit cerat non lectus."""

regex = r"\b[ce]\w+"

match_list = find_all_with_re(regex, text)

print(match_list)

B

import re

def find_all_with_re(regex, text):

    return re.findall(regex, text)

text = """Lorem ipsum dolor sit amet,

a consectetur adipiscing elit.

Sed id cid tempor risus.

Quisque imperdiet, neque c pulvinar sollicitudin,

augue nisl varius nibh, e suscipit cerat non lectus."""

regex = r"\b(ce)\w+"

match_list = find_all_with_re(regex, text)

print(match_list)

C

import re

def find_all_with_re(regex, text):

    return re.findall(regex, text)

text = """Lorem ipsum dolor sit amet,

a consectetur adipiscing elit.

Sed id cid tempor risus.

Quisque imperdiet, neque c pulvinar sollicitudin,

augue nisl varius nibh, e suscipit cerat non lectus."""

regex = r"\b[ce]\w"

match_list = find_all_with_re(regex, text)

print(match_list)

D

import re

def find_all_with_re(regex, text):

    return re.findall(regex, text)

text = """Lorem ipsum dolor sit amet,

a consectetur adipiscing elit.

Sed id cid tempor risus.

Quisque imperdiet, neque c pulvinar sollicitudin,

augue nisl varius nibh, e suscipit cerat non lectus."""

regex = r"\b[ce]\w+"

match_list = find_all_with_re(regex, text)

print(match_list)

Q17:

We will define a function to connect to a MySQL Server and print the connection result.

The function name will be mysql_connector.

The server will be the local MySQL Server on your machine (localhost).

The function has to complete these three tasks:

1- If it gets an exception, it should print the error description.

2- If it successfully connects to the server it should print it as:

"Successfully connected to MySQL Server."

3- It should return an active mysql.connector.connection.MySQLConnection object.

Which one below can achieve this task?

A

import mysql.connector

def mysql_connector():

    connection = None

    try:

        connection = connect(

            host="localhost",

            user="root",

            password="<your_root_password>"

        )

        print("Successfully connected to MySQL Server.")

    except mysql.connector.Error as e:

        print(f"An Error occurred: '{e}'")

    return connection

mysql_connector()

B

import mysql.connector

def mysql_connector():

    connection = None

    try:

        connection = mysql.connector.connect(

            host="localhost",

            user="root",

            password="<your_root_password>"

        )

        print("Successfully connected to MySQL Server.")

    except mysql.connector.Error as e:

        print(f"An Error occurred: '{e}'")

    return connection

mysql_connector()

C

import mysql.connector

# define the function

def mysql_connector():

    try:

        yield mysql.connector.connect(

            host="localhost",

            user="root",

            password="<your_root_password>"

        )

        print("Successfully connected to MySQL Server.")

    except mysql.connector.Error as e:

        print(f"An Error occurred: '{e}'")

mysql_connector()

D

import mysql.connector

# define the function

def mysql_connector():

    try:

        connection = mysql.connector.connect(

            host="localhost",

            user="root",

            password="<your_root_password>"

        )

        print("Successfully connected to MySQL Server.")

        yield connection

    finally:

        pass

mysql_connector()

Q18:

We want to define a function named create_a_database.

It will create a new database named my_local_db on the local MySQL server.

To be able to create a database, you need to connect to the server first.

The function should print the result of the operation, either successful or an exception.

The function should also drop the database if it exists with the same name.

What is the correct definition for this function?

A

from mysql.connector.connection import MySQLConnection

def create_a_database(db_name):

    try:

        connection = MySQLConnection()

        if connection is not None:

            cursor = connection.cursor()

            cursor.execute(f"DROP DATABASE {db_name}")

            cursor.execute(f"CREATE DATABASE {db_name}")

    except mysql.connector.Error as e:

        print(f"An Error occurred in DB Creation:\n'{e}'")

    else:

        if connection is not None:

            cursor.close()

            connection.close()

            print("Database created successfully.")

create_a_database("my_local_db")

B

import mysql.connector

def create_a_database(db_name):

    try:

        connection = mysql.connector.connect(

            user="root",

            password="<your_root_password>"

        )

        if connection is not None:

            cursor = connection.cursor()

            cursor.execute(f"DROP DATABASE {db_name}")

    except mysql.connector.Error as e:

        print(f"An Error occurred in DB Creation:\n'{e}'")

    else:

        if connection is not None:

            cursor.close()

            connection.close()

            print("Database created successfully.")

create_a_database("my_local_db")

C

import mysql.connector

def create_a_database(db_name):

    try:

        connection = mysql.connector.connect(

            user="root",

            password="<your_root_password>"

        )

        if connection is not None:

            cursor = connection.cursor()

            cursor.execute(f"CREATE DATABASE {db_name}")

    except mysql.connector.Error as e:

        print(f"An Error occurred in DB Creation:\n'{e}'")

    else:

        if connection is not None:

            cursor.close()

            connection.close()

            print("Database created successfully.")

create_a_database("my_local_db")

D

import mysql.connector

def create_a_database(db_name):

    try:

        connection = mysql.connector.connect(

            host="localhost",

            user="root",

            password="<your_root_password>"

        )

        if connection is not None:

            cursor = connection.cursor()

            cursor.execute(f"DROP DATABASE {db_name}")

            cursor.execute(f"CREATE DATABASE {db_name}")

    except mysql.connector.Error as e:

        print(f"An Error occurred in DB Creation:\n'{e}'")

    else:

        if connection is not None:

            cursor.close()

            connection.close()

            print("Database created successfully.")

create_a_database("my_local_db")

Q19:

Let's assume we have a folder named "images" which contains some image files, and an empty folder called "thumbnails".

Let's also assume, the initial size of the "images" folder is around 27,5 MB with 20 images in it.

The size is quite large, so we want to resize the images in this folder.

We want to save the thumbnails of these images in the "thumbnails" folder.

Thumbnail dimensions (width * height) of each image will 1/10 of the original image.

And in terms of disk space, we expect the "thumbnails" folder to be around 200 KB.

We also want to keep track of elapsed time (start & end).

We want to use Threading for resize operation.

We have a function named get_images() which returns a list of images in the "images" folder.

There is also a function named resize_image() which resizes the given image based on width * height values.

Which function below is defined correctly for creating and running threads for resize operation?

import os

from PIL import Image

# directory names

images_directory = "images"

thumbnails_directory = "thumbnails"

# get images fn

def get_images():

    """Returns a list of .jpg files

    in the images folder"""

    images = []

    for file in os.listdir(images_directory):

        if file.endswith(".jpg"):

            images.append(file)

    return images

# resize a single image fn

def resize_image(image):

    """Resizes the image and saves to thumbnails_directory"""

    im = Image.open(images_directory + "/" + image)

    resized_im = im.resize((round(im.size[0] * 0.1), round(im.size[1] * 0.1)))

    resized_im.save(thumbnails_directory + "/" + image)

    print(f"{image} resized.")

A

from threading import Thread

threads = []

def resize_all_images():

    try:

        start = time.perf_counter()

        images = get_images()

        for image in images:

            thread = Thread(target=resize_image, args=(image,))

            thread.start()

            threads.append(thread)

        for thread in threads:

            thread.join()

        end = time.perf_counter()

    except:

        print("An Error occurred in resizing.")

    else:

        print(f"Images resized in {end - start:.2f} sec - Threading.")

B

from threading import Thread

threads = []

def resize_all_images():

    try:

        start = time.perf_counter()

        images = get_images()

        for image in images:

            thread = Thread(target=resize_image, args=(image,))

            thread.start()

        end = time.perf_counter()

    except:

        print("An Error occurred in resizing.")

    else:

        print(f"Images resized in {end - start:.2f} sec - Threading.")

C

from threading import Thread

threads = []

def resize_all_images():

    try:

        start = time.perf_counter()

        images = get_images()

        for image in images:

            thread = Thread(target=resize_image, args=(image,))

            threads.append(thread)

        for thread in threads:

            thread.join()

        end = time.perf_counter()

    except:

        print("An Error occurred in resizing.")

    else:

        print(f"Images resized in {end - start:.2f} sec - Threading.")

D

from threading import Thread

threads = []

def resize_all_images():

    try:

        start = time.perf_counter()

        images = get_images()

        for image in images:

            thread = Thread(target=resize_image, args=(image,))

            threads.append(thread)

        end = time.perf_counter()

    except:

        print("An Error occurred in resizing.")

    else:

        print(f"Images resized in {end - start:.2f} sec - Threading.")

Q20:

Let's assume we have a folder named "images" which contains some image files, and an empty folder called "thumbnails".

Let's also assume, the initial size of the "images" folder is around 27,5 MB with 20 images in it.

The size is quite large, so we want to resize the images in this folder.

We want to save the thumbnails of these images in the "thumbnails" folder.

Thumbnail dimensions (width * height) of each image will 1/10 of the original image.

And in terms of disk space, we expect the "thumbnails" folder to be around 200 KB.

We also want to keep track of elapsed time (start & end).

We want to use ThreadPoolExecutor for resize operation.

We have a function named get_images() which returns a list of images in the "images" folder.

There is also a function named resize_image() which resizes the given image based on width * height values.

Which function below is defined correctly for using ThreadPoolExecutor and a list comprehension to submit() threads.?

import os

from PIL import Image

# directory names

images_directory = "images"

thumbnails_directory = "thumbnails"

# get images fn

def get_images():

    """Returns a list of .jpg files

    in the images folder"""

    images = []

    for file in os.listdir(images_directory):

        if file.endswith(".jpg"):

            images.append(file)

    return images

# resize a single image fn

def resize_image(image):

    """Resizes the image and saves to thumbnails_directory"""

    im = Image.open(images_directory + "/" + image)

    resized_im = im.resize((round(im.size[0] * 0.1), round(im.size[1] * 0.1)))

    resized_im.save(thumbnails_directory + "/" + image)

    print(f"{image} resized.")

A

from concurrent.futures import ThreadPoolExecutor, as_completed

threads = []

def resize_all_images():

    try:

        start = time.perf_counter()

        images = get_images()

        with ThreadPoolExecutor() as executor:

            executor.submit(resize_image, image)

            for future in as_completed(executor):

                future.result()

        end = time.perf_counter()

    except:

        print("An Error occurred in resizing.")

    else:

        print(f"Images resized in {end - start:.2f} sec - ThreadPoolExecutor.")

B

from concurrent.futures import ThreadPoolExecutor, as_completed

threads = []

def resize_all_images():

    try:

        start = time.perf_counter()

        images = get_images()

        with ThreadPoolExecutor() as executor:

            future_list = [submit(resize_image, image)

                           for image in images]

        end = time.perf_counter()

    except:

        print("An Error occurred in resizing.")

    else:

        print(f"Images resized in {end - start:.2f} sec - ThreadPoolExecutor.")

C

from concurrent.futures import ThreadPoolExecutor, as_completed

threads = []

def resize_all_images():

    try:

        start = time.perf_counter()

        images = get_images()

        with ThreadPoolExecutor() as executor:

            future_list = [executor.submit(resize_image, image)

                           for image in images]

            for future in as_completed(future_list):

                future.result()

        end = time.perf_counter()

    except:

        print("An Error occurred in resizing.")

    else:

        print(f"Images resized in {end - start:.2f} sec - ThreadPoolExecutor.")

D

from concurrent.futures import ThreadPoolExecutor, as_completed

threads = []

def resize_all_images():

    try:

        start = time.perf_counter()

        images = get_images()

        with ThreadPoolExecutor() as executor:

            executor.submit(resize_image, image)

        end = time.perf_counter()

    except:

        print("An Error occurred in resizing.")

    else:

        print(f"Images resized in {end - start:.2f} sec - ThreadPoolExecutor.")

ANSWERS OF THE FINAL EXAM QUESTIONS:

Question

Answer

Q1

B

Q2

C

Q3

D

Q4

A

Q5

C

Q6

D

Q7

B

Q8

C

Q9

A

Q10

D

Q11

B

Q12

C

Q13

B

Q14

A

Q15

C

Q16

D

Q17

B

Q18

D

Q19

A

Q20

C

20.        Conclusion

This is the last chapter in this book. We have covered almost all of the advanced level concepts in Python and we covered them in great detail. We started with setting our development environment, the PyCharm IDE.

Then we learned Collections, Iterators, Generators, Date and Time, Decorators, Context Managers and Functional Programming in Python.

 After these topics, we had our first project which was Sending Emails with Python. We learned how to use smtplib module to send messages on both a test server and an actual Gmail account.

Then we learned, Regular Expressions, Database Operations and Concurrency in Python.

We had our second project after these topics, which was about Web Scraping. You learned how to scrap data out of web pages using Python and Scrapy.

Finally, we built our third project, which was a Movie Review web application using Python and Flask. We built the project step by step from scratch and you learned all of the fundamental concepts of building API’s using Flask.

You also had assignments after each project. And finally, you got the Final Exam to test yourself.

The approach we followed in this book is quite unique and intense. It aims to teach you Python in a solid and unforgettable way. That’s why we had more than 52 coding exercises, quizzes and assignments. The idea is to make sure we used all the possible ways to help you learn Python programming. And I hope, I achieved this.

This book is a part of the Hands-On Python Series. There are three books in this series: Beginner, Intermediate and Advanced. This book is the third one, the Advanced level. By completing this series, I believe you can feel comfortable with almost all of the fundamentals of Python. The coding exercises, projects, assignments and exam questions will give you a great resource for your programming life and job interviews on Python.

Dear reader!

I want to thank you with all my heart, for your interest in my book, your patience and your desire to learn Python. You did a great job finishing this intense book. And I am very happy to be a part of this. I hope we see each other again in another programming book. Till then, I hope, you have wonderful life and reach your dreams.

Good bye.

Musa Arda

[1] Hands-On Python Series: https://www.amazon.com/gp/product/B09JM26C3Z

[2] Python official website: https://www.python.org/

[3] Python Virtual Environments: https://docs.python.org/3/tutorial/venv.html

[4] PyCharm’s official website: https://www.jetbrains.com/pycharm/

[5] PyCharm official documentation: https://www.jetbrains.com/help/pycharm/quick-start-guide.html