Learn Python like a Professional Start from the basics and go all the way to creating your own applications.

Chapter 1

Introduction

1.1 Introduction

I have heard many people over the years say that Python is an easy language to lean and that Python is also a simple language.

To some extent both of these statements are true; but only to some extent.

While the core of the Python language is easy to lean and relatively simple (in part thanks to its consistency); the sheer richness of the language constructs and flexibility available can be overwhelming. In addition the Python environment, its eco system, the range of libraries available, the often competing options available etc., can make moving to the next level daunting.

Once you have learned the core elements of the language such as how classes and inheritance work, how functions work, what are protocols and Abstract Base Classes etc. Where do you go next?

The aim of this book is to delve into those next steps. The book is organized into eight different topics:

Computer Graphics. The book covers Computer Graphics and Computer Generated Art in Python as well as Graphical User Interfaces and Graphing/ Charting via MatPlotLib.
Games Programming. This topic is covered using the pygame library.
Testing and Mocking. Testing is an important aspect of any software devel- opment; this book introduces testing in general and the PyTest module in detail. It also considers mocking within testing including what and when to mock.
File Input/Output. The book covers text ﬁle reading and writing as well as reading and writing CSV and Excel ﬁles. Although not strictly related to ﬁle input, regulator expressions are included in this section as they can be used to process textual data held in ﬁles.
Database Access. The book introduces databases and relational database in particular. It then presents the Python DB-API database access standard and one implementation of this standard, the PyMySQL module used to access a MySQL database.
Logging. An often missed topic is that of logging. The book therefore intro- duces logging the need for logging, what to log and what not to log as well as the Python logging module.
Concurrency and Parallelism. The book provides extensive coverage of concurrency topics including Threads, Processes and inter thread or process synchronisation. It also presents Futures and AsyncIO.
Reactive Programming. This section of the book introduces Reactive Programming using the PyRx reactive programming library.
Network Programming. The book concludes by introducing socket and web service communications in Python.

Each section is introduced by a chapter providing the background and key concepts of that topic. Subsequent chapters then cover various aspects of the topic.

For example, the ﬁrst topic covered is on Computer Graphics. This section has an introductory chapter on Computer Graphics in general. It then introduces the Turtle Graphics Python library which can be used to generate a graphical display. The following chapter considers the subject of Computer-Generated Art and uses the Turtle Graphics library to illustrate these ideas. Thus, several examples are presented that might be considered art. The chapter concludes by presenting the

well known Koch Snowflake and the Mandelbrot Fractal set.

This is followed by a chapter presenting the MatPlotLib library used for gen- erating 2D and 3D charts and graphs (such as a line chart, bar chart or scatter graph).

The section concludes with a chapter on Graphical User Interfaces (or GUIs) using the wxpython library. This chapter explores what we mean by a GUI and some of the alternatives available in Python for creating a GUI.

Subsequent topics follow a similar pattern.

Each programming or library-oriented chapter also includes numerous sample programs that can be downloaded from the google repository and executed. These chapters also include one or more end of chapter exercises (with sample solutions also in the google repository).

The topics within the book can be read mostly independently of each other. This allows the reader to dip into subject areas as and when required. For example, the File Input/Output section and the Database Access section can be read independently of each other (although in this case assessing both technologies may be useful in selecting an appropriate approach to adopt for the long-term persistent storage of data in a particular system).

Chapter 2

Introduction to Computer Graphics

2.1 Introduction

Computer Graphics are everywhere; they are on your TV, in cinema adverts, the core of many ﬁlms, on your tablet or mobile phone and certainly on your PC or Mac as well as on the dashboard of your car, on your smart watch and in childrens electronic toys.

However, what do we mean by the term Computer Graphics? The term goes back to a time when many (most) computers were purely textual in terms of their input and output and very few computers could generate graphical displays let alone handle input via such a display. However, in terms of this book we take the term Computer Graphics to include the creation of Graphical User Interfaces (or GUIs), graphs and charts such as bar charts or line plots of data, graphics in computer games (such as Space Invaders or Flight Simulator) as well as the generation of 2D and 3D scenes or images. We also use the term to include Computer Generated Art.

The availability of Computer Graphics is very important for the huge acceptance of computer systems by non computer scientists over the last 40 years. It is in part thanks to the accessibility of computer systems via computer graphic interfaces that almost everybody now uses some form of computer system (whether that is a PC, a tablet, a mobile phone or a smart TV).

A Graphical User Interface (GUI) can capture the essence of an idea or a situation, often avoiding the need for a long passage of text or textual commands. It is also because a picture can paint a thousand words; as long as it is the right picture.

In many situations where the relationships between large amounts of information must be conveyed, it is much easier for the user to assimilate this graphically than textually. Similarly, it is often easier to convey some meaning by manipulating some system entities on screen, than by combinations of text commands.

For example, a well chosen graph can make clear information that is hard to determine from a table of the same data. In turn an adventure style game can become engaging and immersive with computer graphics which is in marked contrast to the textual versions of the 1980s. This highlights the advantages of a visual presentation compared to a purely textual one.

2.2 Background

Every interactive software system has a Human Computer Interface, whether it be a single text line system or an advanced graphic display. It is the vehicle used by developers for obtaining information from their user(s), and in turn, every user has to face some form of computer interface in order to perform any desired computer operation.

Historically computer systems did not have a Graphical User Interface and rarely generated a graphical view. These systems from the 60s, 70s and 80s typically focussed on numerical or data processing tasks. They were accessed via green or grey screens on a text oriented terminal. There was little or no opportunity for graphical output.

However, during this period various researchers at laboratories such as Stanford, MIT, Bell Telephone Labs and Xerox were looking at the possibilities that graphic systems might offer to computers. Indeed even as far back as 1963 Ivan Sutherland showed that interactive computer graphics were feasible with his Ph.D. thesis on the Sketchpad system.

2.3 The Graphical Computer Era

Graphical computer displays and interactive graphical interfaces became a common means of human–computer interaction during the 1980s. Such interfaces can save a user from the need to learn complex commands. They are less likely to intimidate computer naives and can provide a large amount of information quickly in a form which can be easily assimilated by the user.

The widespread use of high quality graphical interfaces (such as those provided by the Apple Macintosh and the early Windows interface) led many computer users to expect such interfaces to any software they use. Indeed these systems paved the way for the type of interface that is now omnipresent on PCs, Macs, Linux boxes, tablets and smart phones etc. This graphical user interface is based on the WIMP paradigm (Windows, Icons, Menus and Pointers) which is now the prevalent type of graphical user interface in use today.

The main advantage of any window-based system, and particularly of a WIMP environment, is that it requires only a small amount of user training. There is no need to learn complex commands, as most operations are available either as icons, operations on icons, user actions (such as swiping) or from menu options, and are easy to use. (An icon is a small graphic object that is usually symbolic of an

The Graphical Computer Era 7

operation or of a larger entity such as an application program or a ﬁle). In general, WIMP based systems are simple to learn, intuitive to use, easy to retain and straightforward to work with.

These WIMP systems are exempliﬁed by the Apple Macintosh interface (see Goldberg and Robson as well as Tesler), which was influenced by the pioneering work done at the Palo Alto Research Center on the Xerox Star Machine. It was, however, the Macintosh which brought such interfaces to the mass market, and ﬁrst gained acceptance for them as tools for business, home and industry. This interface transformed the way in which humans expected to interact with their computers, becoming a de facto standard, which forced other manufacturers to provide similar interfaces on their own machines, for example Microsoft Windows for the PC.

This type of interface can be augmented by providing direct manipulation graphics. These are graphics which can be grabbed and manipulated by the user, using a mouse, to perform some operation or action. Icons are a simple version of this, the “opening” of an icon causes either the associated application to execute or the associated window to be displayed.

2.4 Interactive and Non Interactive Graphics

Computer graphics can be broadly subdivided into two categories:

Non Interactive Computer Graphics
Interactive Computer Graphics.

In Non Interactive Computer Graphics (aka Passive Computer Graphics) an image is generated by a computer typically on a computer screen; this image can be viewed by the user (however they cannot interact with the image). Examples of non-interactive graphics presented later in this book include Computer Generated Art in which an image is generated using the Python Turtle Graphics library. Such an image can viewed by the user but not modiﬁed. Another example might be a basic bar chart generated using MatPlotLib which presents some set of data.

Interactive Computer Graphics by contrast, involve the user interacting with the image displayed in the screen in some way, this might be to modify the data being displayed or to change they way in which the image is being rendered etc. It is typiﬁed by interactive Graphical User Interfaces (GUIs) in which a user interacts with menus, buttons, input ﬁeld, sliders, scrollbars etc. However, other visual displays can also be interactive. For example, a slider could be used with a MatplotLib chart. This display could present the number of sales made on a particular date; as the slider is moved so the data changes and the chart is modiﬁed to show different data sets.

Another example is represented by all computer games which are inherently interactive and most, if not all, update their visual display in response to some user inputs. For example in the classic flight simulator game, as the user moves the joystick or mouse, the simulated plane moves accordingly and the display presented to the user updates.

2.5 Pixels

A key concept for all computer graphics systems is the pixel. Pixel was originally a word formed from combining and shortening the words picture (or pix) and ele- ment. A pixel is a cell on the computer screen. Each cell represents a dot on the screen. The size of this dot or cell and the number of cells available will vary depending upon the type, size and resolution of the screen. For example, it was common for early Windows PCs to have a 640 by 480 resolution display (using a VGA graphics card). This relates to the number of pixels in terms of the width and height. This meant that there were 640 pixels across the screen with 480 rows of pixels down the screen. By contrast todays 4K TV displays have 4096 by 2160 pixels.

The size and number of pixels available affects the quality of the image as presented to a user. With lower resolution displays (with fewer individual pixels) the image may appear blocky or poorly deﬁned; where as with a higher resolution it may appear sharp and clear.

Each pixel can be referenced by its location in the display grid. By ﬁlling a pixel on the screen with different colors various images/displays can be created. For example, in the following picture a single pixel has been ﬁlled at position 4 by 4:

A sequence of pixels can form a line, a circle or any number of different shapes. However, since the grid of pixels is based on individual points, a diagonal line or a circle may need to utilise multiple pixels which when zoomed may have jagged edges. For example, the following picture shows part of a circle on which we have zoomed in:

Each pixel can have a colour and a transparency associated with it. The range of colors available depends on the display system being used. For example, mono chrome displays only allow black and white, whereas a grey scale display only allows various shades of grey to be displayed. On modern systems it is usually possible to represent a wide range of colours using the tradition RGB colour codes (where R represents Red, G represents Green, and B represents Blue). In this encoding solid Red is represented by a code such as [255, 0, 0] whereas solid Green is represented by [0, 255, 0] and solid Blue by [0, 0, 255]. Based on this idea various shades can be represented by combination of these codes such as Orange which might be represented by [255, 150, 50]. This is illustrated below for a set of RGB colours using different red, green and blue values:

In addition it is possible to apply a transparency to a pixel. This is used to indicate how solid the ﬁll colour should be. The above grid illustrates the effect of applying a 75%, 50% and 25% transparency to colours displayed using the Python wxPython GUI library. In this library the transparency is referred to as the alpha opaque value. It can have values in the range 0–255 where 0 is completely trans- parent and 255 is completely solid.

2.1 Bit Map Versus Vector Graphics

There are two ways of generating an image/display across the pixels on the screen. One approach is known as bit mapped (or raster) graphics and the other is known as vector graphics. In the bit mapped approach each pixel is mapped to the values to be displayed to create the image. In the vector graphics approach geometric shapes are described (such as lines and points) and these are then rendered onto a display. Raster graphics are simpler but vector graphics provide much more flexibility and scalability.

2.2 Buffering

One issue for interactive graphical displays is the ability to change the display as smoothly and cleanly as possible. If a display is jerky or seems to jump from one image to another, then users will ﬁnd it uncomfortable. It is therefore common to drawn the next display on some in memory structure; often referred to as a buffer. This buffer can then be rendered on the display once the whole image has been created. For example Turtle Graphics allows the user to deﬁne how many changes should be made to the display before it is rendered (or drawn) on to the screen. This can signiﬁcantly speed up the performance of a graphic application.

In some cases systems will use two buffers; often referred to as double buffering. In this approach one buffer is being rendered or drawn onto the screen while the other buffer is being updated. This can signiﬁcantly improve the overall perfor- mance of the system as modern computers can perform calculations and generate data much faster than it can typically be drawn onto a screen.

2.3 Python and Computer Graphics

In the remainder of this section of the book we will look at generating computer graphics using the Python Turtle Graphics library. We will also discuss using this library to create Computer Generated Art. Following this we will explore the MatPlotLib library used to generate charts and data plots such as bar charts, scatter graphs, line plots and heat maps etc. We will then explore the use of Python libraries to create GUIs using menus, ﬁelds, tables etc.

Chapter 3

Python Turtle Graphics

3.1 Introduction

Python is very well supported in terms of graphics libraries. One of the most widely used graphics libraries is the Turtle Graphics library introduced in this chapter. This is partly because it is straight forward to use and partly because it is provided by default with the Python environment (and this you do not need to install any additional libraries to use it).

The chapter concludes by briefly considering a number of other graphic libraries including PyOpen GL. The PyOpenGL library can be used to create sophisticated 3D scenes.

3.2 The Turtle Graphics Library

3.2.1 The Turtle Module

This provides a library of features that allow what are known as vector graphics to be created. Vector graphics refers to the lines (or vectors) that can be drawn on the screen. The drawing area is often referred to as a drawing plane or drawing board and has the idea of x, y coordinates.

The Turtle Graphics library is intended just as a basic drawing tool; other libraries can be used for drawing two and three dimensional graphs (such as MatPlotLib) but those tend to focus on speciﬁc types of graphical displays.

The idea behind the Turtle module (and its name) derives from the Logo pro- gramming language from the 60s and 70s that was designed to introduce program- ming to children. It had an on screen turtle that could be controlled by commands such as forward (which would move the turtle forward), right (which would turn the turtle by a certain number of degrees), left (which turns the turtle left by a certain number of degrees) etc.

This idea has continued into the current Python Turtle Graphics library where commands such as turtle.forward(10) moves the turtle (or cursor as it is now) forward 10 pixels etc. By combining together these apparently simple commands, it is possible to create intricate and quiet complex shapes.

3.2.2 Basic Turtle Graphics

Although the turtle module is built into Python 3 it is necessary to import the module before you use it:

import turtle

There are in fact two ways of working with the turtle module; one is to use the classes available with the library and the other is to use a simpler set of functions that hide the classes and objects. In this chapter we will focus on the set of functions you can use to create drawings with the Turtle Graphics library.

The ﬁrst thing we will do is to set up the window we will use for our drawings; the TurtleScreen class is the parent of all screen implementations used for whatever operating system you are running on.

If you are using the functions provided by the turtle module, then the screen object is initialised as appropriate for your operating system. This means that you can just focus on the following functions to conﬁgure the layout/display such as this screen can have a title, a size, a starting location etc.

The key functions are:

setup(width, height, startx, starty) Sets the size and position of the main window/screen. The parameters are:
- width—if an integer, a size in pixels, if a float, a fraction of the screen; default is 50% of screen.
- height—if an integer, the height in pixels, if a float, a fraction of the screen; default is 75% of screen.
- startx—if positive, starting position in pixels from the left edge of the screen, if negative from the right edge, if None, center window horizontally.
- starty—if positive, starting position in pixels from the top edge of the screen, if negative from the bottom edge, if None, center window vertically.
title(titlestring) sets the title of the screen/window.
exitonclick() shuts down the turtle graphics screen/window when the use clicks on the screen.
bye() shuts down the turtle graphics screen/window.
done() starts the main event loop; this must be the last statement in a turtle graphics program.

speed(speed) the drawing speed to use, the default is 3. The higher the value the faster the drawing takes place, values in the range 0–10 are accepted.
turtle.tracer(n = None) This can be used to batch updates to the turtle graphics screen. It is very useful when a drawing become large and complex. By setting the number (n) to a large number (say 600) then 600 elements will be drawn in memory before the actual screen is updated in one go; this can sig- niﬁcantly speed up the generation of for example, a fractal picture. When called without arguments, returns the currently stored value of n.
turtle.update() Perform an update of the turtle screen; this should be called at the end of a program when tracer() has been used as it will ensure that all elements have been drawn even if the tracer threshold has not yet been reached.
pencolor(color) used to set the colour used to draw lines on the screen; the color can be speciﬁed in numerous ways including using named colours set as ‘red’, ‘blue’, ‘green’ or using the RGB colour codes or by specifying the color using hexadecimal numbers. For more information on the named colours and RGB colour codes to use see https://www.tcl.tk/man/tcl/TkCmd/colors.htm. Note all colour methods use American spellings for example this method is pencolor (not pencolour).
ﬁllcolor(color) used to set the colour to use to ﬁll in closed areas within drawn lines. Again note the spelling of colour!

The following code snippet illustrates some of these functions:

import turtle

# set a title for your canvas window turtle.title(‘My Turtle Animation‘)

# set up the screen size (in pixels)

# set the starting point of the turtle (0, 0) turtle.setup(width=200, height=200, startx=0, starty=0)

# sets the pen color to red turtle.pencolor(‘red‘)

# …

# Add this so that the window will close when clicked on turtle.exitonclick()

We can now look at how to actually draw a shape onto the screen.

The cursor on the screen has several properties; these include the current drawing colour of the pen that the cursor moves, but also its current position (in the x, y coordinates of the screen) and the direction it is currently facing. We have already seen that you can control one of these properties using the pencolor() method, other methods are used to control the cursor (or turtle) and are presented below.

The direction in which the cursor is pointing can be altered using several functions including:

right(angle) Turn cursor right by angle units.
left(angle) Turn the cursor left by angle units.
setheading(to_angle) Set the orientation of the cursor to to_angle. Where 0 is east, 90 is north, 180 is west and 270 is south.

You can move the cursor (and if the pen is down this will draw a line) using:

forward(distance) move the cursor forward by the speciﬁed distance in the direction that the cursor is currently pointing. If the pen is down then draw a line.
backward(distance) move the cursor backward by distance in the opposite direction that in which the cursor is pointing.

And you can also explicitly position the cursor:

goto(x, y) move the cursor to the x, y location on the screen speciﬁed; if the pen is down draw a line. You can also use steps and set position to do the same thing.
setx(x) sets the cursor’s x coordinate, leaves the y coordinate unchanged.
sety(y) sets the cursor’s y coordinate, leaves the x coordinate unchanged.

It is also possible to move the cursor without drawing by modifying whether the pen is up or down:

penup() move the pen up—moving the cursor will no longer draw a line.
pendown() move the pen down—moving the cursor will now draw a line in the current pen colour.

The size of the pen can also be controlled:

pensize(width) set the line thickness to width. The method width() is an alias for this method.

It is also possible to draw a circle or a dot:

circle(radius, extent, steps) draws a circle using the given radius. The extent determines how much of the circle is drawn; if the extent is not given then the whole circle is drawn. Steps indicates the number of steps to be used to drawn the circle (it can be used to draw regular polygons).
dot(size, color) draws a ﬁlled circle with the diameter of size using the speciﬁed color.

You can now use some of the above methods to draw a shape on the screen. For this ﬁrst example, we will keep it very simple, we will draw a simple square:

# Draw a square
turtle.forward(50)
turtle.right(90)
turtle.forward(50)
turtle.right(90)
turtle.forward(50)
turtle.right(90)
turtle.forward(50)
turtle.right(90)

The above moves the cursor forward 50 pixels then turns 90° before repeating these steps three times. The end result is that a square of 50 x 50 pixels is drawn on the screen:

Note: that the cursor is displayed during drawing (this can be turned off with turtle.hideturtle() as the cursor was originally referred to as the turtle).

3.1.1 Drawing Shapes

Of course you do not need to just use ﬁxed values for the shapes you draw, you can use variables or calculate positions based on expressions etc.

For example, the following program creates a sequence of squares rotated around a central location to create an engaging image:

import turtle

def setup():
“”” Provide the config for the screen “””
turtle.title(‘Multiple Squares Animation’)
turtle.setup(100, 100, 0, 0)
turtle.hideturtle()

def draw_square(size):
“”” Draw a square in the current direction “””
turtle.forward(size)
turtle.right(90)
turtle.forward(size)
turtle.right(90)
turtle.forward(size)
turtle.right(90)
turtle.forward(size)

setup()

for _ in range(0, 12): draw_square(50)
# Rotate the starting direction
turtle.right(120)

# Add this so that the window will close when clicked on
turtle.exitonclick()

In this program two functions have been deﬁned, one to setup the screen or window with a title and a size and to turn off the cursor display. The second function takes a size parameter and uses that to draw a square. The main part of the program then sets up the window and uses a for loop to draw 12 squares of 50 pixels each by continuously rotating 120° between each square. Note that as we do not need to reference the loop variable we are using the ‘_’ format which is considered an anonymous loop variable in Python.

The image generated by this program is shown below:

3.1.1 Filling Shapes

It is also possible to ﬁll in the area within a drawn shape. For example, you might wish to ﬁll in one of the squares we have drawn as shown below:

To do this we can use the begin_ﬁll() and end_ﬁll() functions:

begin_ﬁll() indicates that shapes should be ﬁlled with the current ﬁll col- our, this function should be called just before drawing the shape to be ﬁlled.
end_ﬁll() called after the shape to be ﬁlled has been ﬁnished. This will cause the shape drawn since the last call to begin_ﬁll() to be ﬁlled using the current ﬁll colour.
ﬁlling() Return the current ﬁll state (True if ﬁlling, False if not).

The following program uses this (and the earlier draw_square() function) to draw the above ﬁlled square:

turtle.title(‘Filled Square Example’)
turtle.setup(100, 100, 0, 0)
turtle.hideturtle()

turtle.pencolor(‘red’)
turtle.fillcolor(‘yellow’)
turtle.begin_fill()

draw_square(60)

turtle.end_fill() turtle.done()

3.1 Other Graphics Libraries

Of course Turtle Graphics is not the only graphics option available for Python; however other graphics libraries do not come pre-packed with Python and must be downloaded using a tool such as Anaconda, PIP or PyCharm.

PyQtGraph. The PyQtGraph library is pure Python library oriented towards mathematics, scientiﬁc and engineering graphic applications as well as GUI applications.
Pillow. Pillow is a Python imaging library (based on PIL the Python Imaging library) that provides image processing capabilities for use in Python.
Pyglet. pyglet is another windowing and multimedia library for Python.

3.2 3D Graphics

Although it is certainly possible for a developer to create convincing 3D images using Turtle Graphics; it is not the primary aim of the library. This means that there is no direct support for creating 3D images other than the basic cursor moving facilities and the programers skill.

However, there are 3D graphics libraries available for Python. One such library is Panda3D (https://www.panda3d.org) while another is VPython (https://vpython.org) while a third is pi3d (https://pypi.org/project/pi3d). However we will briefly look at the PyOpenGL library as this builds on the very widely used OpenGL library.

3.2.1 PyOpenGL

PyOpenGL is an open-source project that provides a set of bindings (or wrappings around) the OpenGL library. OpenGL is the Open Graphics Library which is a cross language, cross platform API for rendering 2D and 3D vector graphics. OpenGL is used in a wide range of applications from games to virtual reality, through data and information visualization systems to Computer Aided Design (CAD) systems. PyOpenGL provides a set of Python functions that call out from Python to the underlying OpenGL libraries. This makes it very easy to create 3D vector-based images in Python using the industry standard OpenGL library. A very simple examples of an image created using PyOpenGL is given below:

Chapter 4

Computer Generated Art

4.1 Creating Computer Art

Computer Art is deﬁned as any art that uses a computer. However, in the context of this book we mean it to be art that is generated by a computer or more speciﬁcally a computer program. The following example, illustrates how in a very few lines of Python code, using the Turtle graphics library, you can create images that might be considered to be computer art.

The following image is generated by a recursive function that draws a circle at a given x, y location of a speciﬁed size. This function recursively calls itself by modifying the parameters so that smaller and smaller circles are drawn at different locations until the size of the circles goes below 20 pixels.

The program used to generate this picture is given below for reference:

import turtle

WIDTH = 640

HEIGHT = 360

def setup_window():

There are a few points to note about this program. It uses recursion to draw the circles with smaller and smaller circles being drawn until the radius of the circles falls below a certain threshold (the termination point).

It also uses the turtle.tracer() function to speed up drawing the picture as 2000 changes will be buffered before the screen is updated.

Finally, the colours used for the circles are changed at each level of recession; a very simple approach is used so that the Red, Green and Blue codes are changed resulting in different colour circles. Also a line width is used to reduce the size of the circle outline to add more interest to the image.

4.1 A Computer Art Generator

As another example of how you can use Turtle graphics to create computer art, the following program randomly generates RGB colours to use for the lines being drawn which gives the pictures more interest. It also allows the user to input an angle to use when changing the direction in which the line is drawn. As the drawing happens within a loop even this simple change to the angle used to draw the lines can generate very different pictures.

Some sample images generated from this program are given below. The left most picture is generated by inputting an angle of 38 degrees, the picture on the right uses an angle of 68 degrees and the bottom picture an angle of 98 degrees.

The following pictures below use angles of 118, 138 and 168 degrees respectively.

What is interesting about these images is how different each is; even though they use exactly the same program. This illustrates how algorithmic or computer gen- erated art can be as subtle and flexible as any other art form. It also illustrates that even with such a process it is still up to the human to determine which image (if any) is the most aesthetically pleasing.

4.1 Fractals in Python

Within the arena of Computer Art fractals are a very well known art form. Factrals are recurring patterns that are calculated either using an iterative approach (such as a for loop) or a recursive approach (when a function calls itself but with modiﬁed parameters). One of the really interesting features of fractals is that they exhibit the same pattern (or nearly the same pattern) at successive levels of granularity. That is, if you magniﬁed a fractal image you would ﬁnd that the same pattern is being repeated at successively smaller and smaller magniﬁcations. This is known as ex- panding symmetry or unfolding symmetry; if this replication is exactly the same at every scale, then it is called afﬁne self-similar.

Fractals have their roots in the world of mathematics starting in the 17th century, with the term fractal being coined in the 20th century by mathematical Benoit Mandelbrot in 1975. One often cited description that Mandelbrot published to describe geometric fractals is a rough or fragmented geometric shape that can be split into parts, each of which is (at least approximately) a reduced-size copy of the whole.

For more information see Mandelbrot, Benoît B. (1983). The fractal geometry of nature. Macmillan. ISBN (978-0-7167-1186-5).

Since the later part of the 20th century fractals have been a commonly used way of creating computer art.

One example of a fractal often used in computer art is the Koch snowflake, while another is the Mandelbrot set. Both of these are used in this chapter as examples to illustrate how Python and the Turtle graphics library can be used to create fractal based art.

4.1.1 The Koch Snowflake

The Koch snowflake is a fractal that begins with equilateral triangle and then replaces the middle third of every line segment with a pair of line segments that form an equilateral bump. This replacement can be performed to any depth generating ﬁner and ﬁner grained (smaller and smaller) triangles until the overall shape resembles a snow flake.
The following program can be used to generate a Koch snowflake with different levels of recursion. The larger the number of levels of recursion the more times each line segment is dissected.

Several different runs of the program are shown below with the depth set at 0, 1, 3 and 7.

Running the simple draw_koch() function with different depths makes it easy to see the way in which each side of a triangle can be dissected into a further triangle like shape. This can be repeated to multiple depths giving a more detailed structured in which the same shape is repeated again and again.

4.1.1 Mandelbrot Set

Probably one of the most famous fractal images is based on the Mandelbrot set. The Mandelbrot set is the set of complex numbers c for which the function z* z+c does not diverge when iterated from z = 0 for which the sequence of functions (func(0), func(func(0)) etc.) remains bounded by an absolute value. The deﬁnition of the Mandelbrot set and its name is down to the French mathematician Adrien Douady, who named it as a tribute to the mathematician Benoit Mandelbrot.

Mandelbrot set images may be created by sampling the complex numbers and testing, for each sample point c, whether the sequence func(0), func(func(0)) etc. ranges to inﬁnity (in practice this means that a test is made to see if it leaves some predetermined bounded neighborhood of 0 after a predetermined number of iterations). Treating the real and imaginary parts of c as image coordinates on the complex plane, pixels may then be colored according to how soon the sequence crosses an arbitrarily chosen threshold, with a special color (usually black) used for the values of c for which the sequence has not crossed the threshold after the predetermined number of iterations (this is necessary to clearly distinguish the Mandelbrot set image from the image of its complement).

The following image was generated for the Mandelbrot set using Python and Turtle graphics.

The program used to generate this image is given below:

Chapter 5

Introduction to Matplotlib

5.1 Introduction

Matplotlib is a Python graphing and plotting library that can generate a variety of different types of graph or chart in a variety of different formats. It can be used to generate line charts, scatter graphs, heat maps, bar charts, pie charts and 3D plots. It can even support animations and interactive displays.

An example of a graph generated using Matplotlib is given below. This shows a line chart used to plot a simple sign wave:

Matplotlib is a very flexible and powerful graphing library. It can support a variety of different Python graphics platforms and operating system windowing environments. It can also generate output graphics in a variety of different formats including PNG, JPEG, SVG and PDF etc.

Matplotlib can be used on its own or in conjunction with other libraries to provide a wide variety of facilities. One library that is often used in conjunction with Matplotlib is NumPy which is a library often used in Data Science applications that provides a variety of functions and data structures (such as n-dimensional arrays) that can be very useful when processing data for display within a chart.

However, Matplotlib does not come prebuilt into the Python environment; it is an optional module which must be added to your environment or IDE.

In this chapter we will introduce the Matplotlib library, its architecture, the components that comprise a chart and the pyplot API. The pyplot API is the simplest and most common way in which a programmer interacts with Matplotlib. We will then explore a variety of different types of chart and how they can be created using Matplotlib, from simple line charts, through scatter charts, to bar charts and pie charts. We will ﬁnish by looking at a simple 3D chart.

5.2 Matplotlib

Matplotlib is a graph plotting library for Python. For simple graphs Matplotlib is very easy to use, for example to create a simple line graph for a set of x and y coordinates you can use the matplotlib.pyplot.plot function:

import matplotlib.pyplot as pyplot

# Plot a sequence of values

pyplot.plot([1, 0.25, 0.5, 2, 3, 3.75, 3.5])

# Display the chart in a window

pyplot.show()

This very simple program generates the following graph:

In this example, the plot() function takes a sequence of values which will be treated as the y axis values; the x axis values are implied by the position of the y values within the list. Thus as the list has six elements in it the x axis has the range 0–6. In turn as the maximum value contained in the list is 3.75, then the y axis ranges from 0 to 4.

5.3 Plot Components

Although they may seem simple, there are numerous elements that comprise a Matplotlib graph or plot. These elements can all be manipulated and modiﬁed independently. It is therefore useful to be familiar with the Matplotlib terminology associated with these elements, such as ticks, legends, labels etc.

The elements that make up a plot are illustrated below:

The diagram illustrates the following elements:

Axes An Axes is deﬁned by the matplotlib.axes.Axes class. It is used to maintain most of the elements of a ﬁgure namely the X and Y Axis, the Ticks, the Line plots, any text and any polygon shapes.
Title This is the title of the whole ﬁgure.
Ticks (Major and Minor) The Ticks are represented by the class matplotlib.axis.Tick. A Tick is the mark on the Axis indicating a new

value. There can be Major ticks which are larger and may be labeled. There are also minor ticks which can be smaller (and may also be labelled).

Tick Labels (Major and Minor) This is a label on a Tick.
Axis The maplotlib.axis.Axis class deﬁnes an Axis object (such as an X or Y axis) within a parent Axes instance. It can have formatters used to format the labels used for the major and minor ticks. It is also possible to set the locations of the major and minor ticks.
Axis Labels (X, Y and in some cases Z) These are labels used to describe the Axis.
Plot types such as line and scatter plots. Various types of plots and graphs are supported by Matplotlib including line plots, scatter graphs, bar charts and pie charts.
Grid This is an optional grid displayed behind a plot, graph or chart. The grid can be displayed with a variety of different line styles (such as solid or dashed lines), colours and line widths.

5.4 Matplotlib Architecture

The Matplotlib library has a layered architecture that hides much of the complexity associated with different windowing systems and graphic outputs. This architecture has three main layers, the Scripting Layer, the Artist Layer and the Backend Layer. Each layer has speciﬁc responsibilities and components. For example, the Backend is responsible for reading and interacting with the graph or plot being generated. In turn the Artist Layer is responsible for creating the graph objects that will be rendered by the Backend Layer. Finally, the Scripting Layer is used by the developer to create the graphs.

This architecture is illustrated below:

5.1.1 Backend Layer

The Matplotlib backend layer handles the generation of output to different target formats. Matplotlib itself can be used in many different ways to generate many different outputs.

Matplotlib can be used interactively, it can be embedded in an application (or graphical user interface), it may be used as part of a batch application with plots being stored as PNG, SVG, PDF or other images etc.

To support all of these use cases, Matplotlib can target different outputs, and each of these capabilities is called a backend; the “frontend” is the developer facing code. The Backend Layer maintains all the different backends and the programmer can either use the default backend or select a different backend as required.

The backend to be used can be set via the matplotlib.use() function. For example, to set the backend to render Postscript use: matplotlib.use(‘PS’) this is illustrated below:

import matplotlib

if ‘matplotlib.backends‘ not in sys.modules: matplotlib.use(‘PS’)

import matplotlib.pyplot as pyplot

It should be noted that if you use the matplotlib.use() function, this must be done before importing matplotlib.pyplot. Calling matplotlib.use () after matplotlib.pyplot has been imported will have no effect. Note that the argument passed to the matplotlib.use() function is case sensitive.

The default renderer is the ‘Agg’ which uses the Anti-Grain Geometry C++ library to make a raster (pixel) image of the ﬁgure. This produces high quality raster graphics based images of the data plots.

The ‘Agg’ backend was chosen as the default backend as it works on a broad selection of Linux systems as its supporting requirements are quite small; other backends may run on one particular system, but may not work on another system. This occurs if a particular system does not have all the dependencies loaded that the speciﬁed Matplotlib backend relies on.

The Backend Layer can be divided into two categories:

User interface backends (interactive) that support various Python windowing systems such as wxWidgets (discussed in the next chapter), Qt, TK etc.
Hardcopy Backends (non interactive) that support raster and vector graphic outputs.

The User Interface and Hardcopy backends are built upon common abstractions referred to as the Backend base classes.

5.1.1 The Artist Layer

The Artist layer provides the majority of the functionality that you might consider to be what Matplotlib actually does; that is the generation of the plots and graphs that are rendered/ displayed to the user (or output in a particular format).

The artist layer is concerned with things such as the lines, shapes, axis, and axes, text etc. that comprise a plot.

The classes used by the Artist Layer can be classiﬁed into one of the following three groups: primitives, containers and collections:

Primitives are classes used to represent graphical objects that will be drawn on to a ﬁgures canvas.
Containers are objects that hold primitives. For example, typically a ﬁgure would be instantiated and used to create one or more Axes etc.
Collections are used to efﬁciently handle large numbers of similar types of objects.

Although it is useful to be aware of these classes; in many cases you will not need to work with them directly as the pyplot API hides much of the detail. However, it is possible to work at the level of ﬁgures, axes, ticks etc. if required.

5.1.2 The Scripting Layer

The scripting layer is the developer facing interface that simpliﬁes the task of working with the other layers.

Note that from the programmers point of view, the Scripting Layer is represented by the pyplot module. Under the covers pyplot uses module-level objects to track the state of the data, handle drawing the graphs etc.

When imported pyplot selects either the default backend for the system or the one that has been conﬁgured; for example via the matplotlib.use() function.

It then calls a setup() function that:

creates a ﬁgure manager factory function, which when called will create a new

ﬁgure manager appropriate for the selected backend,

prepares the drawing function that should be used with the selected backend,
identiﬁes the callable function that integrates with the backend mainloop function,
provides the module for the selected backend.

The pyplot interface simpliﬁes interactions with the internal wrappers by providing methods such as plot(), pie(), bar(), title(), saveﬁg(), draw() and ﬁgure() etc.

Most of the examples presented in the next chapter will use the functions pro- vided by the pyplot module to create the required charts; thereby hiding the lower-level details.

Chapter 6

Graphing with Matplotlib pyplot

6.1 Introduction

In this chapter we will explore the Matplotlib pyplot API. This is the most common way in which developers generate different types of graphs or plots using Matplotlib.

6.2 The pyplot API

The purpose of the pyplot module and the API it presents is to simplify the generation and manipulation of Matplotlib plots and charts. As a whole the Matplotlib library tries to make simple things easy and complex things possible. The primary way in which it achieves the ﬁrst of these aims is through the pyplot API as this API has high level functions such as bar(), plot(), scatter() and pie() that make it easy to create bar charts, line plots, scatter graphs and pie charts. One point to note about the functions provided by the pyplot API is that they can often take very many parameters; however, most of these parameters will have default values that in many situations will give you a reasonable default behaviour/ default visual representation. You can therefore ignore most of the parameters available until such time as you actually need to do something different; at which point you should refer to the Matplotlib documentation as this has extensive material as well as numerous examples.

It is of course necessary to import the pyplot module; as it is a module within the Matplotlib (e.g. matplotlib.pyplot) library. It is often given an alias within a program to make it easier to reference. Common alias for this module are pyplot or plt.

A typical import for the pyplot module is given below:

import matplotlib.pyplot as pyplot

The plyplot API can be used to

construct the plot,
conﬁgure labels and axis,
manage color and line styles,
handles events/allows plots to be interactive,
display (show) the plot.

We will see examples of using the pyplot API in the following sections.

6.1 Line Graphs

A Line Graph or Line Plot is a graph with the points on the graph (often referred to as markers) connected by lines to show how something changes in value as some set of values (typically the x axis) changes; for example, over a series to time intervals (also known as a time series). Time Series line charts are typically drawn in chronological order; such charts are known as run charts.

The following chart is an example of a run chart; it charts time across the bottom (x axis) against speed (represented by the y axis).

The ﬁrst thing that this program does is to import the matplotlib.pyplot module and give it an alias of pyplot (as this is a shorter name it makes the code easier to read).

Two lists of values are then created for the x and y coordinates of each marker or plot point.

The graph itself is then conﬁgured with labels being provided for the x and y axis (using the pyplot functions xlabel() and ylabel()). The title of the graph is then set (again using a pyplot function).

After this the x and y values are then plotted as a line chart on the graph. This is done using the pyplot.plot() function. This function can take a wide range of parameters, the only compulsory parameters being the data used to deﬁne the plot points. In the above example a third parameter is provided; this is a string ‘bo-’. This is a coded format string in that each element of the string is meaningful to the pyplot.plot() function. The elements of the string are:

b—this indicates the colour to use when drawing the line; in this case the letter ‘b’ indicates the colour blue (in the same way ‘r’ would indicate red and ‘g’ would indicate green).
o—this indicates that each marker (each point being plotted) should be repre- sented by a cirlce. The lines between the markers then create the line plot.
‘–’—This indicates the line style to use. A single dash (‘-’) indicates a solid line, where as a double dash (‘–’) indicates a dashed line.

Finally the program then uses the show() function to render the ﬁgure on the screen; alternatively saveﬁg() could have been used to save the ﬁgure to a ﬁle.

6.1.1 Coded Format Strings

There are numerous options that can be provided via the format string, the fol- lowing tables summarises some of these:

The following colour abbreviations are supported by the format string:

Different ways of representing the markers (points on the graph) connected by the lines are also supported including:

Finally, the format string supports different line styles:

Some examples of formatting strings:

‘r’ red line with default markers and line style.
‘g-’ green solid line.
‘–’ dashed line with the default colour and default markers.
‘yo:’ yellow dotted line with circle markers.

6.2 Scatter Graph

A Scatter Graph or Scatter Plot is type of plot where individual values are indicated using cartesian (or x and y) coordinates to display values. Each value is indicated via a mark (such as a circle or triangle) on the graph. They can be used to represent values obtained for two different variables; one plotted on the x axis and the other plotted on the y axis.

An example of a scatter chart with three sets of scatter values is given below

In this graph each dot represents the amount of time people of different ages spend on three different activities.

The program that was used to generate the above graph is shown below:

import matplotlib.pyplot as pyplot

# Create data

riding = ((17, 18, 21, 22, 19, 21, 25, 22, 25, 24),
(3, 6, 3.5, 4, 5, 6.3, 4.5, 5, 4.5, 4))

swimming = ((17, 18, 20, 19, 22, 21, 23, 19, 21, 24),
(8, 9, 7, 10, 7.5, 9, 8, 7, 8.5, 9))

sailing = ((31, 28, 29, 36, 27, 32, 34, 35, 33, 39),
(4, 6.3, 6, 3, 5, 7.5, 2, 5, 7, 4))

# Plot the data
pyplot.scatter(x=riding[0], y=riding[1], c=’red’, marker=’o’, label=’riding’)
pyplot.scatter(x=swimming[0], y=swimming[1], c=’green’, marker=’^’, label=’swimming’)
pyplot.scatter(x=sailing[0], y=sailing[1], c=’blue’, marker=’*’, label=’sailing’)

# Configure graph
pyplot.xlabel(‘Age’)
pyplot.ylabel(‘Hours’)
pyplot.title(‘Activities Scatter Graph’)
pyplot.legend()

# Display the chart
pyplot.show()

In the above example the plot.scatter() function is used to generate the scatter graph for the data deﬁned by the riding, swimming and sailing tuples.

The colours of the markers have been speciﬁed using the named parameter c. This parameter can take a string representing the name of a colour or a two dimensional array with a single row in which each value in the row represents an RGB color code. The marker Indicates the marker style such as ‘o’ for a circle, a ‘^’ for a triangle and ‘*’ for a star shape. The label is used in the chart legend for the marker.

Other options available on the pyplot.scatter() function include:

alpha : indicates the alpha blending value, between 0 (transparent) and 1 (opaque).

linewidths : which is used to indicate the line width of the marker edges.
edgecolors : indicates the color to use for the marker edges if different from the ﬁll colour used for the marker (indicates by the parameter ‘c’).

6.4.1 When to Use Scatter Graphs

A useful question to consider is when should a scatter plot be used? In general scatter plats are used when it is necessary to show the relationship between two variables. Scatter plots are sometimes called correlation plots because they show how two variables are correlated.

In many cases a trend can be discerned between the points plotted on a scatter chart (although there may be outlying values). To help visualise the trend it can be useful to draw a trend line along with the scatter graph. The trend line helps to make the relationship of the scatter plots to the general trend clearer.

The following chart represents a set of values as a scatter graph and draws the trend line of this scatter graph. As can be seen some values are closer to the trendline than others.

The trend line has been created in this case using the numpy function polyﬁt().

The polyﬁt() function performs a least squares polynomial ﬁt for the data it is given. A poly1d class is then created based on the array returned by polyﬁt(). This class is a one-dimensional polynomial class. It is a convenience class, used to encapsulate “natural” operations on polynomials. The poly1d object is then used to generate a set of values for use with the set of x values for the function py- plot.plot().

6.4 Pie Charts

A Pie Chart is a type of graph in which a circle is divided into sectors (or wedges) that each represent a proportion of the whole. A wedge of the circle represents a category’s contribution to the overall total. As such the graph resembles a pie that has been cut into different sized slices.

Typically, the different sectors of the pie chart are presented in different colours and are arranged clockwise around the chart in order of magnitude. However, if there is a slice that does not contain a unique category of data but summarizes several, for example “other types” or “other answers”, then even if it is not the smallest category, it is usual to display it last in order that it does not detract from the named categories of interest.

The following chart illustrates a pie chart used to represent programming language usage within a particular organization.

The pyplot.pie() function takes several parameters, most of which are optional. The only required parameter is the ﬁrst one that provides the values to be used for the wedge or segment sizes. The following optional parameters are used in the above example:

The labels parameter is an optional parameter that can take a sequence of strings that are used to provide labels for each wedge.
The autopct parameter takes a string (or function) to be used to format the numeric values used with each wedge.
The counterclockwise parameter. By default wedges are plotted counter clockwise in pyplot and so to ensure that the layout is more like the traditional clockwise approach the counterclock parameter is set to False.
The startangle parameter. The starting angle has also been moved 90° using the startangle parameter so that the ﬁrst segment starts at the top of the chart.

6.4.1 Expanding Segments

It can be useful to emphasis a particular segment of the pie chart by exploding it; that is separating it out from the rest of the pie chart. This can be done using the explode parameter of the pie() function that takes a sequence of values indi- cating how much a segment should be exploded by.

The visual impact of the pie chart can also be enhanced in this case by adding a shadow to the segments using the named shadow Boolean parameter. The effect of these are shown below:

The program that generated this modiﬁed chart is given below for reference:

import matplotlib.pyplot as pyplot

labels = (‘Python’, ‘Java’, ‘Scala’, ‘C#’) sizes = [45, 30, 15, 10]

# only “explode” the 1st slice (i.e. ‘Python’) explode = (0.1, 0, 0, 0)

6.4.1 When to Use Pie Charts

It is useful to consider what data can be/should be presented using a pie chart. In general pie charts are useful for displaying data that can be classiﬁed into nominal or ordinal categories. Nominal data is categorised according to descriptive or qualitative information such as program languages, type of car, country of birth etc. Ordinal data is similar but the categories can also be ranked, for example in a survey people may be asked to say whether they classed something as very poor, poor, fair, good, very good.

Pie charts can also be used to show percentage or proportional data and usually the percentage represented by each category is provided next to the corresponding slice of pie.

Pie charts are also typically limited to presenting data for six or less categories. When there are more categories it is difﬁcult for the eye to distinguish between the relative sizes of the different sectors and so the chart becomes difﬁcult to interpret.

6.5 Bar Charts

A Bar Chart is a type of chart or graph that is used to present different discrete categories of data. The data is usually presented vertically although in some cases horizontal bar charts may be used. Each category is represented by a bar whose height (or length) represents the data for that category.

Because it is easy to interpret bar charts, and how each category relates to another, they are one of the most commonly used types of chart. There are also several different common variations such as grouped bar charts and stacked bar charts.

The following is an example of a typical bar chart. Five categories of programming languages are presented along the x axis while the y axis indicates percentage usage. Each bar then represents the usage percentage associated with each programming language.

The program used to generate the above ﬁgure is given below:

import matplotlib.pyplot as pyplot

# Set up the data
labels = (‘Python’, ‘Scala’, ‘C#’, ‘Java’, ‘PHP’)
index = (1, 2, 3, 4, 5)
# provides locations on x axis
sizes = [45, 10, 15, 30, 22]

# Set up the bar chart
pyplot.bar(index, sizes, tick_label=labels)

# Configure the layout pyplot.ylabel(‘Usage’)
pyplot.xlabel(‘Programming Languages’)

# Display the chart pyplot.show()

The chart is constructed such that the lengths of the different bars are propor- tional to the size of the category they represent. The x-axis represents the different categories and so has no scale. In order to emphasise the fact that the categories are discrete, a gap is left between the bars on the x-axis. The y-axis does have a scale and this indicates the units of measurement.

6.4.1 Horizontal Bar Charts

Bar charts are normally drawn so that the bars are vertical which means that the taller the bar, the larger the category. However, it is also possible to draw bar charts so that the bars are horizontal which means that the longer the bar, the larger the category. This is a particularly effective way of presenting a large number of different categories when there is insufﬁcient space to ﬁt all the columns required for a vertical bar chart across the page.

In Matplotlib the pyplot.barh() function can be used to generate a hori- zontal bar chart:

In this case the only line of code to change from the previous example is:

pyplot.barh(x_values, sizes, tick_label = labels)

6.4.1 Coloured Bars

It is also common to colour different bars in the chart in different colours or using different shades. This can help to distinguish one bar from another. An example is given below:

The colour to be used for each category can be provided via the color parameter to the bar() (and barh()) function. This is a sequence of the colours to apply. For example, the above coloured bar chart can be generated using:

pyplot.bar(x_values, sizes, tick_label=labels, color=(‘red’,‘green’, ‘blue’, ‘yellow’, ‘orange’))

6.4.1 Stacked Bar Charts

Bar Charts can also be stacked. This can be a way of showing total values (and what contributes to those total values) across several categories. That is, it is a way of viewing overall totals, for several different categories based on how different ele- ments contribute to those totals.

Different colours are used for the different sub-groups that contribute to the overall bar. In such cases, a legend or key is usually provided to indicate what sub-group each of the shadings/colours represent. The legend can be placed in the plot area or may be located below the chart.

For example, in the following chart the total usage of a particular programming language is composed of its use in games and web development as well as data science analytics.

From this ﬁgure we can see how much each use of a programming language contributes to the overall usage of that language. The program that generated this chart is given below:

import matplotlib.pyplot as pyplot

# Set up the data
labels = (‘Python’, ‘Scala’, ‘C#’, ‘Java’, ‘PHP’) index = (1, 2, 3, 4, 5)
web_usage = [20, 2, 5, 10, 14]
data_science_usage = [15, 8, 5, 15, 2]
games_usage = [10, 1, 5, 5, 4]

# Set up the bar chart
pyplot.bar(index, web_usage, tick_label=labels, label=’web’)
pyplot.bar(index, data_science_usage, tick_label=labels, label=’data science’, bottom=web_usage)

web_and_games_usage = [web_usage[i] + data_science_usage[i]
for i in range(0, len(web_usage))]
pyplot.bar(index, games_usage, tick_label=labels, label=’games’, bottom=web_and_games_usage)

# Configure the layout pyplot.ylabel(‘Usage’) pyplot.xlabel(‘Programming Languages’) pyplot.legend()

# Display the chart
pyplot.show()

One thing to note from this example is that after the ﬁrst set of values are added using the pyplot.bar() function, it is necessary to specify the bottom locations for the next set of bars using the bottom parameter. We can do this just using the values already used for web_usage for the second bar chart; however for the third bar chart we must add the values used for web_usage and data_- science_usage together (in this case using a for list comprehension).

6.4.1 Grouped Bar Charts

Finally, Grouped Bar Charts are a way of showing information about different sub-groups of the main categories. In such cases, a legend or key is usually pro- vided to indicate what sub-group each of the shadings/colours represent. The legend can be placed in the plot area or may be located below the chart.

For a particular category separate bar charts are drawn for each of the subgroups. For example, in the following chart the results obtained for two sets of teams across a series of lab exercises are displayed. Thus each team has a bar for lab1, lab2, lab3 etc. A space is left between each category to make it easier to compare the subcategories.

The following program generates the grouped bar chart for the lab exercises example:

import matplotlib.pyplot as pyplot

BAR_WIDTH = 0.35
# set up grouped bar charts
teama_results = (60, 75, 56, 62, 58)
teamb_results = (55, 68, 80, 73, 55)
# Set up the index for each bar
index_teama = (1, 2, 3, 4, 5)
index_teamb = [i + BAR_WIDTH for i in index_teama]

# Determine the mid point for the ticks
ticks = [i + BAR_WIDTH / 2 for i in index_teama]
tick_labels = (‘Lab 1’, ‘Lab 2’, ‘Lab 3’, ‘Lab 4’, ‘Lab 5’)

# Plot the bar charts
pyplot.bar(index_teama, teama_results, BAR_WIDTH, color=’b’,
label=’Team A’)
pyplot.bar(index_teamb, teamb_results, BAR_WIDTH, color=’g’,
label=’Team B’)

# Set up the graph
pyplot.xlabel(‘Labs’)
pyplot.ylabel(‘Scores’)
pyplot.title(‘Scores by Lab’)
pyplot.xticks(ticks, tick_labels)
pyplot.legend()

# Display the graph
pyplot.show()

Notice in the above program that it has been necessary to calculate the index for the second team as we want the bars presented next to each other. Thus the index for the teams includes the width of the bar for each index point, thus the ﬁrst bar is at index position 1.35, the second at index position 2.35 etc. Finally the tick positions must therefore be between the two bars and thus is calculated by taking into account the bar widths.

This program generates the following grouped bar chart:

6.4 Figures and Subplots

A Matplotlib ﬁgure is the object that contains all the graphical elements displayed on a plot. That is the axes, the legend, the title as well as the line plot or bar chart itself. It thus represents the overall window or page and is the top, out graphical component.

In many cases the ﬁgure is implicit as the developer interacts with the pyplot

API; however the ﬁgure can be accessed directly if required.

The matplotlib.pyplot.ﬁgure() function generates a ﬁgure object. This function returns a matplotlib.ﬁgure.Figure object. It is then possible to interact directly with the ﬁgure object. For example it is possible to add axes to the ﬁgure, to add sub plots to a graph etc.

Working directly with the ﬁgure is necessary if you want to add multiple sub- plots to a ﬁgure. This can be useful if what is required is to be able to compare different views of the same data side by side. Each subplot has its own axes which can coexist within the ﬁgure.

One or more subplots can be added to a ﬁgure using the ﬁgure.add_ subplot() method. This method adds an Axes to the ﬁgure as one of a set of one or more subplots. A subplot can be added using a 3-digit integer (or three separate integers) describing the position of the subplot. The digits represent the number of rows, columns and the index of the sub plot within the resulting matrix.

Thus 2, 2, 1 (and 221) all indicate that the subplot will take the 1st index within a two by two grid of plots. In turn 2, 2, 3 (223) indicates that the sub plot will be at index 3 which will be row 2 and column 1 within the 2 by 2 grid of plots. Where as 2, 2, 4 (or 224) indicates that the plot should be added as at index 4 or the fourth subplot within the grid (so position 2 by 2) etc.

For example, the following ﬁgure illustrates four subplots presented within a single ﬁgure. Each subplot is added via the ﬁgure.add_subplot() method.

This ﬁgure is generated by the following program:
import matplotlib.pyplot as pyplot

t = range(0, 20)
s = range(30, 10, -1)
# Set up the grid of subplots to be 2 by 2
grid_size=’22’

# Initialize a Figure
figure = pyplot.figure()

# Add first subplot
position = grid_size + ‘1’
print(‘Adding first subplot to position’, position)
axis1 = figure.add_subplot(position)
axis1.set(title=’subplot(2,2,1)’)
axis1.plot(t, s)

# Add second subplot
position = grid_size + ‘2’
print(‘Adding second subplot to position’, position)
axis2 = figure.add_subplot(position)
axis2.set(title=’subplot(2,2,2)’)
axis2.plot(t, s, ‘r-‘)

# Add third subplot
position = grid_size + ‘3’
print(‘Adding third subplot to position’, position)
axis3 = figure.add_subplot(position)
axis3.set(title=’subplot(2,2,3)’)
axis3.plot(t, s, ‘g-‘)

# Add fourth subplot
position = grid_size + ‘4’
print(‘Adding fourth subplot to position’, position)
axis4 = figure.add_subplot(position)
axis4.set(title=’subplot(2,2,4)’)
axis4.plot(t, s, ‘y-‘)

# Display the chart
pyplot.show()

The console output from this program is given below:
Adding first subplot to position 221
Adding second subplot to position 222
Adding third subplot to position 223
Adding fourth subplot to position 224

6.4 3D Graphs

A three dimensional graph is used to plot the relationships between three sets of values (instead of the two used in the examples presented so far in this chapter). In a three dimensional graph as well as the x and y axis there is also a z axis.

The following program creates a simple 3D graph using two sets of values generated using the numpy range function. These are then converted into a coordinate matrix using the numpy meshgrid() function. The z axis values are created using the numpy sin() function. The 3D graph surface is plotted using the plot_surface() function of the futures axes object. This takes the x, y and z coordinates. The function is also given a colour map to use when rendering the surface (in this case the Matplotlib cool to warm colour map is used).

import matplotlib.pyplot as pyplot
# Import matplotlib colour map
from matplotlib import cm as colourmap
# Required for £D Projections
from mpl_toolkits.mplot3d import Axes3D

# Provide access to numpy functions import numpy as np
# Make the data to be displayed
x_values = np.arange(-6, 6, 0.3)
y_values = np.arange(-6, 6, 0.3)

# Generate coordinate matrices from coordinate vectors
x_values, y_values = np.meshgrid(x_values, y_values)

# Generate Z values as sin of x plus y values
z_values = np.sin(x_values + y_values)

# Obtain the figure object
figure = pyplot.figure()

# Get the axes object for the 3D graph
axes = figure.gca(projection=’3d’)

# Plot the surface.
surf = axes.plot_surface(x_values,
y_values, z_values,
cmap=colourmap.coolwarm)

# Add a color bar which maps values to colors.
figure.colorbar(surf)

# Add labels to the graph
pyplot.title(“3D Graph”)
axes.set_ylabel(‘y values’, fontsize=8)
axes.set_xlabel(‘x values’, fontsize=8)
axes.set_zlabel(‘z values’, fontsize=8)

# Display the graph
pyplot.show()

This program generates the following 3D graph:

One point to note about three dimensional graphs is that they are not universally accepted as being a good way to present data. One of the maxims of data visual- isation is keep it simple/keep it clean. Many consider that a three dimensional chart does not do this and that it can be difﬁcult to see what is really being shown or that it can be hard to interpret the data appropriately. For example, in the above chart what are the values associated with any of the peaks? This is difﬁcult to determine as it is hard to see where the peaks are relative to the X, Y and Z axis. Many consider such 3D charts to be eye candy; pretty to look at but not providing much information. As such the use of a 3D chart should be minimised and only used when actually necessary.

6.4 Exercises

The following table provides information on cities in the UK and their populations (note that London has been omitted as its population is so much larger than that of any other city and this would distort the graph).

City	Population
Bristol	617,280
Cardiff	447,287
Bath	94,782
Liverpool	864,122
Glasgow	591,620
Edinburgh	464,990
Leeds	455,123
Reading	318,014
Swansea	300,352
Manchester	395,515

Using this data create:

A scatter plot for the city to population data.
A bar chart for the city to population data.

Chapter 7

Graphical User Interfaces

7.1 Introduction

A Graphical User Interface can capture the essence of an idea or a situation, often avoiding the need for a long passage of text. Such interfaces can save a user from the need to learn complex commands. They are less likely to intimidate computer users and can provide a large amount of information quickly in a form which can be easily assimilated by the user.

The widespread use of high quality graphical interfaces has led many computer users to expect such interfaces to any software they use. Most programming lan- guages either incorporate a Graphical User Interface (GUI) library or have third party libraries available.

Python is of course a cross platform programming language and this brings in additional complexities as the underlying operating system may provide different windowing facilities depending upon whether the program is running on Unix, Linux, Mac OS or Windows operating systems.

In this chapter we will ﬁrst introduce what we mean by a GUI and by WIMP based UIs in particular. We will then consider the range of libraries available for Python before selecting one to use. This chapter will then describe how to create rich client graphical displays (desktop application) using one of these GUI libraries. Thus in this chapter we consider how windows, buttons, text ﬁelds and labels etc. are created, added to windows, positioned and organised.

7.1 GUIs and WIMPS

GUIs (Graphical User Interfaces) and WIMP (Windows, Icons, Mice and Pop-up Menus) style interfaces have been available within computer systems for many years but they are still one of the most signiﬁcant developments to have occurred. These interfaces were originally developed out of a desire to address many of the perceived weaknesses of purely textual interfaces.

The textual interface to an operating system was typiﬁed by a peremptory prompt. In Unix/Linux systems for example, the prompt is often merely a single character such as %, > or $, which can be intimidating. This is true even for experienced computer users if they are not familiar with the Unix/Linux family of operating systems.

For example, a user wishing to copy a ﬁle from one directory to another might have to type something like:

> cp file.pdf ~otheruser/projdir/srcdir/newfile.pdf

This long sequence needs to be entered with no mistakes in order to be accepted. Any error in this command will cause the system to generate an error message which might or might not be enlightening. Even where systems attempt to be more “user friendly’’ through features like command histories, much typing of arrow keys and ﬁlenames is typically needed.

The main issue on both input and output is one of bandwidth. For example, in situations where the relationships between large amounts of information must be described, it is much easier to assimilate this if output is displayed graphically than if it is displayed as a tables of ﬁgures. On input, combinations of mouse actions can be given a meaning that could otherwise only be conveyed by several lines of text. WIMP stands for Windows (or Window Managers), Icons, Mice and Pop-up menus. WIMP interfaces allow the user to overcome at least some of the weak- nesses of their textual counterparts—it is possible to provide a pictorial image of the operating system which can be based on a concept the user can relate to, menus can be used instead of textual commands and information in general can be displayed graphically.

The fundamental concepts presented via a WIMP interface were originally developed at XEROX’s Palo Alto Research Center and used on the Xerox Star machine, but gained much wider acceptance through ﬁrst the Apple Macintosh and then IBM PC implementations of WIMP interfaces.

Most WIMP style environments use a desktop analogy (although this is less true of mobile devices such as phones and tablets):

the whole screen represents a working surface (a desktop),
graphic windows that can overlap represent sheets of paper on that desktop,
graphic objects are used for speciﬁc concepts, for example ﬁling cabinets for disks or a waste bin for ﬁle disposal (these could be regarded as desk accessories),
various application programs are displayed on the screen, these stand for tools that you might use on your desktop.

In order to interact with this display, the WIMP user is provided with a mouse (or a light pen or a touch sensitive screen), which can be used to select icons and menus or to manipulate windows.

The software basis of any WIMP style environment is the window manager. It controls the multiple, possibly overlapping windows and icons displayed on the screen. It also handles the transfer of information about events which occur in those windows to the appropriate application and generates the various menus and prompts used.

A window is an area of the graphic screen in which a page or piece of a page of information may be displayed; it may display text, graphics or a combination of both. These windows may be overlapping, and associated with the same process, or they may be associated with separate processes. Windows can generally be created, opened, closed, moved and resized.

An icon is a small graphic object that is usually symbolic of an operation or of a larger entity such as an application program or a ﬁle. The opening of an icon causes either the associated application to execute or the associated window to be displayed.

At the heart of the users ability to interact with such WIMP based programs is the event loop. This loop listens for events such as the user clicking a button or selecting a menu item or entering a text ﬁeld. When such an event occurs it triggers the associated behaviour (such as running a function linked with a button).

7.2 Windowing Frameworks for Python

Python is a cross platform programming language. As such Python programs can be written on one platform (such as a Linux box) and then run on that platform or another operating system platform (such as Windows or Mac OS). This can however generate issues for libraries that need to be available across multiple operating system platforms. The area of GUIs is particularly an issue as a library written to exploit features available in the Microsoft Windows system may not be available (or may look different) on Mac OS or Linux systems.

Each operating system that Python runs on may have one or more windowing systems written for it and these systems may or may not be available on other operating systems. This makes the job of providing a GUI library for Python that much more difﬁcult.

Developers of Python GUIs have taken one of two approaches to handle this:

One approach is to write a wrapper that abstracts the underlying GUI facilities so that the developer works at a level above a speciﬁc windowing system’s facilities. The Python library then maps (as best it can) the facilities to the underlying system that is currently being used.
The other approach is to provide a closer wrapping to a particular set of facilities on the underlying GUI system and to only target systems that support those facilities.

Some of the libraries available for Python are listed below and have been cat- egorised into platform-independent libraries and platform-speciﬁc libraries:

7.2.1 Platform-Independent GUI Libraries

Tkinter. This is the standard built-in Python GUI library. It is built on top of the Tcl/Tk widget set that has been around for very many years for many different operating systems. Tcl stands for Tool Command Language while Tk is the graphical user interface toolkit for Tcl.
wxPython. wxWidgets is a free, highly portable GUI library. Its is written in C++ and it can provide a native look and feel on operating systems such as Windows, Mac OS, Linux etc. wxPython is a set of Python bindings for wxWidgets. This is the library that we will be using in this chapter.
PyQT or PySide both of these libraries wrap the Qt toolkit facilities. Qt is a cross platform software development system for the implementation of GUIs and applications.

7.2.2 Platform-Speciﬁc GUI Libraries

PyObjc is a Mac OS speciﬁc library that provides an Objective-C bridge to the Apple Mac Cocoa GUI libraries.
PythonWin provides a set of wrappings around the Microsoft Windows Foundation classes and can be used to create Windows based GUIs.

Chapter 8

The wxPython GUI Library

8.1 The wxPython Library

The wxPython library is a cross platform GUI library (or toolkit) for Python. It allows programmers to develop highly graphical user interfaces for their programs using common concepts such as menu bars, menus, buttons, ﬁelds, panels and frames.

In wxPython all the elements of a GUI are contained within top level windows such as a wx.Frame or a wx.Dialog. These windows contain graphical com- ponents known as widgets or controls. These widgets/controls may be grouped together into Panels (which may or may not have a visible representation).

Thus in wxPython we might construct a GUI from:

Frames which provide the basic structure for a window: borders, a label and some basic functionality (e.g. resizing).
Dialogs which are like Frames but provide fewer border controls.
Widgets/Controls that are graphical objects displayed in a frame. Some other languages refer to them as UI components. Examples of widgets are buttons, checkboxes, selection lists, labels and text ﬁelds.
Containers are component that are made up of one or more other components (or containers). All the components within a container (such as a panel) can be treated as a single entity.

Thus, a GUI is constructed hierarchically from a set of widgets, containers and one or more Frames (or in the case of a pop-up dialog then Dialogs). This is illustrated below for a window containing several panels and widgets:

Windows such as Frames and Dialogs have a component hierarchy that is used (amongst other things) to determine how and when elements of the window are drawn and redrawn. The component hierarchy is rooted with the frame, within which components and containers can be added.

The above ﬁgure illustrates a component hierarchy for a frame, with two con- tainer Panels and a few basic widgets/ui components held within the Panels. Note that a panel can contain another sub panel with different widgets in.

8.1.1 wxPython Modules

The wxPython library is comprised of many different modules. These modules provide different features from the core wx module to the html oriented wx.html and wx.html2 modules. These modules include:

wx which holds the core widgets and classes in the wx library.
wx.adv that provides less commonly used or more advanced widgets and classes.
wx.grid contains widgets and classes supporting the display and editing of tabular data.
wx.richtext consists of widgets and classes used for displaying multiple text styles and images.
wx.html comprises widgets and supporting classes for a generic html renderer.
wx.html2 provides further widget and supporting classes for a native html renderer, with CSS and javascript support.

8.1.2 Windows as Objects

In wxPython, Frames and Dialogs as well as their contents are instances of appropriate classes (such as Frame, Dialog, Panel, Button or StaticText). Thus when you create a window, you create an object that knows how to display itself on the computer screen. You must tell it what to display and then tell it to show its contents to the user.

You should bear the following points in mind during your reading of this chapter; they will help you understand what you are required to do:

You create a window by instantiating a Frame or Dialog object.
You deﬁne what the window displays by creating a widget that has an appropriate parent component. This adds the widget to a container, such as a type of panel or a frame.
You can send messages to the window to change its state, perform an operation, and display a graphic object. The window, or components within the window, can send messages to other objects in response to user (or program) actions. Everything displayed by a window is an instance of a class and is potentially subject to all of the above.
wx.App handles the main event loop of the GUI application.

8.1.3 A Simple Example

An example of creating a very simple window using wxPython is given below. The result of running this short program is shown here for both a Mac and a Windows PC:

This program creates a top level window (the wx.Frame) and gives it a title. It also creates a label (a wx.StaticText object) to be displayed within the frame.

To use the wxPython library it is necessary to import the wx module.

import wx

# Create the Application Object
app = wx.App()
# Now create a Frame (representing the window)
frame = wx.Frame(parent=None, title=’ Simple Hello World’)
# And add a text label to it
text = wx.StaticText(parent=frame, label= ‘Hello Python’)

# Display the window (frame)
frame.Show()

# Start the event loop
app.MainLoop()

The program also creates a new instance of the Application Object called wx.

App().

Every wxPython GUI program must have one Application Object. It is the equivalent of the main() function in many non-GUI applications as it will run the GUI application for you. It also provides default facilities for deﬁning startup and shutdown operations and can be subclassed to create custom behaviour.

The wx.StaticText class is used to create a single (or multiple) line label. In this case the label shows the string ‘Hello Python’. The StaticText object is constructed with reference to its parent container. This is the container within which the text will be displayed. In this case the StaticText is being displayed directly within the Frame and thus the frame object is its containing parent object. In contrast the Frame which is a top level window, does not have a parent container.

Also notice that the frame must be shown (displayed) for the user to see it. This is because there might be multiple different windows that need to be shown (or hidden) in different situations for an application.

Finally the program starts the applications’ main event loop; within this loop the program listens for any user input (such as requesting that the window is closed).

8.2 The wx.App Class

The wx.App class represents the application and is used to:

start up the wxPython system and initialise the underlying GUI toolkit,
set and get application-wide properties,
implement the native windowing system main message or event loop, and to dispatch events to window instances.

Every wxPython application must have a single wx.App instance. The creation of all of the UI objects should be delayed until after the wx.App object has been created in order to ensure that the GUI platform and wxWidgets have been fully initialised.

It is common to subclass the wx.App class and override methods such as OnPreInit and OnExit to provide custom behaviour. This ensures that the required behaviour is run at appropriate times. The methods that can be overridden for this purpose are:

OnPreInit, This method can be overridden to deﬁne behaviour that should be run once the application object is created, but before the OnInit method has been called.
OnInit This is expected to create the applications main window, display that window etc.
OnRun, This is the method used to start the execution of the main program.
OnExit, This can be overridden to provide any behaviour that should be called just before the application exits.

As an example, if we wish to set up a GUI application such that the main frame is initialized and shown after the wx.App has been instantiated then the safest way is to override the OnInit() method of the wx.App class in a suitable subclass. The method should return True of False; where True is used to indicate that processing of the application should continue and False indicates that the application should terminate immediately (usually as the result of some unexpected issue).

An example wx.App subclass is shown below:

8.3 Window Classes

The window or widget container classes that are commonly used within a wxPython application are:

wx.Dialog A Dialog is a top level window used for popups where the user has limited ability to interact with the window. In many cases the user can only input some data and/or accept or decline an option.
wx.Frame A Frame is a top level window whose size and position can be set and can (usually) be controlled by the user.
wx.Panel Is a container (non top level window) on which controls/widgets can be placed. This is often used in conjunction with a Dialog or a Frame to manage the positioning of widgets within the GUI.

The inheritance hierarchy for these classes is given below for reference:

As an example of using a Frame and a Panel, the following application creates two Panels and displays them within a top level Frame. The background colour of the Frame is the default grey; while the background colour for the ﬁrst Panel is blue and for the second Panel it is red. The resulting display is shown below:

The program that generated this GUI is given below:

import wx

class SampleFrame(wx.Frame):

def init (self):

super(). init (parent=None, title=‘Sample App’, size=(300, 300))

# Set up the first Panel to be at position 1, 1 # (The default) and of size 300 by 100

# with a blue background self.panel1 = wx.Panel(self) self.panel1.SetSize(300, 100)

self.panel1.SetBackgroundColour(wx.Colour(0, 0, 255))

# Set up the second Panel to be at position 1, 110 # and of size 300 by 100 with a red background self.panel2 = wx.Panel(self) self.panel2.SetSize(1, 110, 300, 100)

self.panel2.SetBackgroundColour(wx.Colour(255, 0, 0))

class MainApp(wx.App):

The SampleFrame is a subclass of the wx.Frame class; it thus inherits all of the functionality of a Top Level Frame (window). Within the init () method of the SampleFrame the super classes init () method is called. This is used to set the size of the Frame and to give the Frame a title. Note that the Frame also indicates that it does not have a parent window.

When the Panel is created it is necessary to specify the window (or in this case Frame) within which it will be displayed. This is a common pattern within wxPython.

Also note that the SetSize method of the Panel class also allows the position to be speciﬁed and that the Colour class is the wxPython Colour class.

8.4 Widget/Control Classes

Although there are very many widgets/controls available to the developer, the most commonly used include:

wx.Button/wx.ToggleButton/wx.RadioButton These are widgets that provide button like behaviour within a GUI.
wx.TextCtrl This widget allows text to be displayed and edited. I can be a single line or multiple line widget depending upon conﬁguration.
wx.StaticText Used to display one or more lines of read-only text. In many libraries this widgets is known as a label.
wx.StaticLine A line used in dialogs to separate groups of widgets. The line may be vertical or horizontal.
wx.ListBox This widget is used to allow a user to select one option from a list of options.
wx.MenuBar/wx.Menu/wx.MenuItem. The components that can be used to construct a set of menus for a User Interface.
wx.ToolBar This widget is used to display a bar of buttons and/or other widgets usually placed below the menubar in a wx.Frame.

The inheritance hierarchy of these widgets is given below. Note that they all inherit from the class Control (hence why they are often referred to as Controls as well as Widgets or GUI components).

Whenever a widget is created it is necessary to provide the container window class that will hold it, such as a Frame or a Panel, for example:

enter_button = wx.Button(panel, label=‘Enter’)

In this code snippet a wx.Button is being created that will have a label ‘Enter’

and will be displayed within the given Panel.

8.5 Dialogs

The generic wx.Dialog class can be used to build any custom dialog you require. It can be used to create modal and modeless dialogs:

A modal dialog blocks program flow and user input on other windows until it is dismissed.
A modeless dialog behaves more like a frame in that program flow continues, and input in other windows is still possible.
The wx.Dialog class provides two versions of the show method to support modal and modeless dialogs. The ShowModal() method is used to display a modal dialog, while the Show() is used to show a modeless dialog.

As well as the generic wx.Dialog class, the wxPython library provides numerous prebuilt dialogs for common situations. These pre built dialogs include:

wx.ColourDialog This class is used to generate a colour chooser dialog.
wx.DirDialog This class provides a directory chooser dialog.
wx.FileDialog This class provides a ﬁle chooser dialog.
wx.FontDialog This class provides a font chooser dialog.
wx.MessageDialog This class can be used to generate a single or multi-line message or information dialog. It can support Yes, No and Cancel options. It can be used for generic messages or for error messages.
wx.MultiChoiceDialog This dialog can be used to display a lit of strings and allows the user to select one or more values for the list.
wx.PasswordEntryDialog This class represents a dialog that allows a user to enter a one-line password string from the user.
wx.ProgressDialog If supported by the GUI platform, then this class will provide the platforms native progress dialog, otherwise it will use the pure Python wx.GenericProgressDialog. The wx.GenericProgressDialog shows a short message and a progress bar.
wx.TextEntryDialog This class provides a dialog that requests a one-line text string from the user.

Most of the dialogs that return a value follow the same pattern. This pattern returns a value from the ShowModel() method that indicates if the user selected OK or CANCEL (using the return value wx.ID_OK or wx.ID_CANCEL). The selected/entered value can then be obtained from a suitable get method such as GetColourData() for the ColourDialog or GetPath() for the DirDialog.

8.6 Arranging Widgets Within a Container

Widgets can be located within a window using speciﬁc coordinates (such as 10 pixels down and 5 pixels across). However, this can be a problem if you are considering cross platform applications, this is because how a button is rendered (drawn) on a Mac is different to Windows and different again from the windowing systems on Linux/Unix etc.

This means that different amount of spacing must be given on different plat- forms. In addition the fonts used with text boxes and labels differ between different platforms also requiring differences in the layout of widgets.

To overcome this wxPython provides Sizers. Sizers work with a container such as a Frame or a Panel to determine how the contained widgets should be laid out. Widgets are added to a sizer which is then set onto a container such as a Panel.

A Sizer is thus an object which works with a container and the host windowing platform to determine the best way to display the objects in the window. The developer does not need to worry about what happens if a user resizes a window or if the program is executed on a different windowing platform.

Sizers therefore help to produce portable, presentable user interfaces. In fact one Sizer can be placed within another Sizer to create complex component layouts.

There are several sizers available including:

wx.BoxSizer This sizer can be used to place several widgets into a row or column organisation depending upon the orientation. When the BoxSizer is created the orientation can be speciﬁed using wx.VERTICAL or wx, HORIZONTAL.
wx.GridSizer This sizer lays widgets out in a two dimensional grid. Each cell within the grid has the same size. When the GridSizer object is created it is possible to specify the number of rows and columns the grid has. It is also possible to specify the spacing between the cells both horizontally and vertically.
wx.FlexGridSizer This sizer is a slightly more flexible version of the GridSizer. In this version not all columns and rows need to be the same size (although all cells in the same column are the same width and all cells in the same row are the same height).

wx.GridBagSizer is the most flexible sizer. It allows widgets to be posi- tioned relative to the grid and also allows widgets to span multiple rows and/or columns.

To use a Sizer it must ﬁrst be instantiated. When widgets are created they should be added to the sizer and then the sizer should be set on the container.

For example, the following code uses a GridSizer used with a Panel to layout out four widgets comprised of two buttons, a StaticText label and a TextCtrl input ﬁeld:

# Create the panel panel = wx.Panel(self)
# Create the sizer to use with 4 rows and 1 column
# And 5 spacing around each cell
grid = wx.GridSizer(4, 1, 5, 5)

# Create the widgets
text = wx.TextCtrl(panel, size=(150, -1))
enter_button = wx.Button(panel, label=’Enter’)
label = wx.StaticText(panel,label=’Welcome’)
message_button = wx.Button(panel, label=’Show Message’)

# Add the widgets to the grid sizer
grid.AddMany([text, enter_button, label, message_button])
# Set the sizer on the panel
panel.SetSizer(grid)

The resulting display is shown below:

8.7 Drawing Graphics

In earlier chapters we looked at the Turtle graphics API for generating vector and raster graphics in Python.

The wxPython library provides its own facilities for generating cross platform graphic displays using lines, squares, circles, text etc. This is provided via the Device Context.

A Device Context (often shortened to just DC) is an object on which graphics and text can be drawn.

It is intended to allow different output devices to all have a common graphics API (also known as the GDI or Graphics Device Interface). Speciﬁc device contexts can be instantiate depending on whether the program is to use a window on a computer screen or some other output medium (such as a printer).

There are several Device Context types available such as wx.WindowDC, wx.

PaintDC and wx.ClientDC:

The wx.WindowDC is used if we want to paint on the whole window (Windows only). This includes window decorations.
The wx.ClientDC is used to draw on the client area of a window. The client area is the area of a window without its decorations (title and border).
The wx.PaintDC is used to draw on the client area as well but is intended to support the window refresh paint event handling mechanism.

Note: that the wx.PaintDC should be used only from a wx.PaintEvent handler while the wx.ClientDC should never be used from a wx.PaintEvent handler.

Whichever Device Context is used, they all support a similar set of methods that are used to generate graphics, such as:

DrawCircle (x, y, radius) Draws a circle with the given centre and radius.
DrawEllipse (x, y, width, height) Draws an ellipse contained in the rectangle speciﬁed either with the given top left corner and the given size or directly.
DrawPoint (x, y) Draws a point using the color of the current pen.
DrawRectangle (x, y, width, height) Draws a rectangle with the given corner coordinate and size.
DrawText (text, x, y) Draws a text string at the speciﬁed point, using the current text font, and the current text foreground and background colours.
DrawLine (pt1, pt2)/DrawLine (x1, y1, x2, y2) This method draws a line from the ﬁrst point to the second.

It is also important to understand when the device context is refreshed/redrawn. For example, if you resize a window, maximise it, minimise it, move it, or modify its contents the window is redrawn. This generates an event, a PaintEvent.

You can bind a method to the PaintEvent (using wx.EVT_PAINT) that can be called each time the window is refreshed.

This method can be used to draw whatever the contents of the window should be. If you do not redraw the contents of the device context in such a method than whatever you previously drew will display when the window is refreshed.

The following simple program illustrates the use of some of the Draw methods listed above and how a method can be bound to the paint event so that the display is refreshed appropriately when using a device context:

When this program is run the following display is generated:

Chapter 9

Events in wxPython User Interfaces

9.1 Event Handling

Events are an integral part of any GUI; they represent user interactions with the interface such as clicking on a button, entering text into a ﬁeld, selecting a menu option etc.

The main event loop listens for an event; when one occurs it processes that event (which usually results in a function or method being called) and then waits for the next event to happen. This loop is initiated in wxPython via the call to the MainLoop() method on the wx.App object.

This raises the question ‘what is an Event?’. An event object is a piece of information representing some interaction that occurred typically with the GUI (although an event can be generated by anything). An event is processed by an Event Handler. This is a method or function that is called when the event occurs. The event is passed to the handler as a parameter. An Event Binder is used to bind an event to an event handler.

9.2 Event Deﬁnitions

It is useful to summarise the deﬁnitions around events as the terminology used can be confusing and is very similar:

Event represents information from the underlying GUI framework that describes something that has happened and any associated data. The speciﬁc data available will differ depending on what has occurred. For example, if a window has been moved then the associated data will relate to the window’s new location. Where as a CommandEvent generated by a selection action from a ListBox provides the item index for the selection.
Event Loop the main processing loop of the GUI that waits for an event to occur. When an event occurs the associated event handler is called.
Event Handlers these are methods (or functions) that are called when an event occurs.
Event Binders associate a type of event with an event handler. There are different event binders for different types of event. For example, the event binder associated with the wx.MoveEvent is named wx.EVT_MOVE.

The relationship between the Event, the Event Handler via the Event Binder is illustrated below:

The top three boxes illustrate the concepts while the lower 3 boxes provide a concrete example of binding a Move_Event to an on_move() method via the EVT_MOVE binder.

9.3 Types of Events

There are numerous different types of event including:

wx.CloseEvent used to indicate that a Frame or Dialog has been closed. The event binder for this event is named wx.EVT_CLOSE.
wx.CommandEvent used with widgets such as buttons, list boxes, menu items, radio buttons, scrollbars, sliders etc. Depending upon the type of widget that generated the event different information may be provided. For example, for a Button a CommandEvent indicates that a button has been clicked where as for a ListBox it indicates that an option has been selected, etc. Different event binders are used for different event situations. For example, to bind a command event to a event handler for a button then the wx.EVT_BUTTON binder is used; while for a ListBox a wx.EVT_LISTBOX binder can be used.
wx.FocusEvent This event is sent when a window’s focus changes (loses or gains focus). You can pick up a window gaining focus using the wx. EVT_SET_FOCUS event binder. The wx.EVT_KILL_FOCUS is used to bind an event handler that will be called when a window loses focus.
wx.KeyEvent This event contains information relating to a key press or release.
wx.MaximizeEvent This event is generated when a top level window is maximised.
wx.MenuEvent This event is used for menu oriented actions such as the menu being opened or closed; however it should be noted that this event is not used when a menu item is selected (MenuItems generate CommandEvents).
wx.MouseEvent This event class contains information about the events generated by the mouse: this includes information on which mouse button was pressed (and released) and whether the mouse was double clicked etc.
wx.WindowCreateEvent This event is sent just after the actual window is created.
wx.WindowDestoryedEvent This event is sent as early as possible during the window destruction process.

9.4 Binding an Event to an Event Handler

An event is bound to an Event Handler using the Bind() method of an event generating object (such as a button, ﬁeld, menu item etc.) via a named Event Binder.

For example:

button.Bind(wx.EVT_BUTTON, self.event_handler_method)

9.5 Implementing Event Handling

There are four steps involved in implementing event handling for a widget or window, these are:

Identify the event of interest. Many widgets will generate different events in different situations; it may therefore be necessary to determine which event you are interested in.
Find the correct Event Binder name, e.g. wx.EVT_CLOSE, wx.EVT_MOVE or wx.EVT_BUTTON etc. Again you may ﬁnd that the widget you are inter- ested in supports numerous different event binders which may be used in dif- ferent situations (even for the same event).
Implement an event handler (i.e. a suitable method or function) that will be called when the event occurs. The event handler will be supplied with the event object.
Bind the Event to the Event Handler via the Binder Name using the Bind()

method of the widget or window.

To illustrate this we will use a simple example.

We will write a very simple event handling application. This application will have a Frame containing a Panel. The Panel will contain a label using the wx. StaticText class.

We will deﬁne an event handler called on_mouse_click() that will move the StaticText label to the current mouse location when the left mouse button is pressed. This means that we can move the label around the screen.

To do this we ﬁrst need to determine the widget that will be used to generate the event. In this case it is the panel that contains the text label. Having done this we can look at the Panel class to see what events and Event Bindings it supports. It turns out that the Panel class only directly deﬁnes support for NavigationKeyEvents. This is not actually what we want; however the Panel class extends the Window class.

The Window class supports numerous event bindings, from those associated with setting the focus (wx.EVT_SET_FOCUS and wx.EVT_KILL_FOCUS) to key presses (wx.EVT_KEY_DOWN and wx.EVT_KEY_UP) as well as mouse events. There are however numerous different mouse event bindings. These allow left, middle and right mouse button clicks to be picked up, down clicks to be identiﬁed, situations such as the mouse entering or leaving the window etc. However, the binding we are interested in for a MouseEvent is the wx. EVT_LEFT_DOWN binding; this picks up on the MoueEvent when the left mouse button is pressed (there is also the wx.EVT_LEFT_UP binding which can be used to pick up an event that occurs when the left mouse button is released).

We now know that we need to bind the on_mouse_click() event handler to the MouseEvent via the wx.EVT_LEFT_DOWN event binder, for example:

self.panel.Bind(wx.EVT_LEFT_DOWN, self.on_mouse_click)

All event handler methods takes two parameters, self and the mouse event.

Thus the signature of the on_mouse_click() method is:

def on_mouse_click(self, mouse_event):

The mouse event object has numerous methods deﬁned that allow information about the mouse to be obtained such as the number of mouse clicks involved (GetClickCount()), which button was pressed (GetButton()) and the current mouse position within the containing widget or window (GetPosition ()). We can therefore use this last method to obtain the current mouse location and then use the SetPosition(x, y) method on the StaticText object to set its position.

The end result is the program shown below:

When this program is run; the window is displayed with the ‘Hello’ StaticText label in the top left hand corner of the Frame (actually it is added to the Panel, however the Panel ﬁlls the Frame in this example). If the user then clicks the left mouse button anywhere within the Frame then the ‘Hello’ label jumps to that location.

This is shown below for the initial setup and then for two locations within the window.

9.6 An Interactive wxPython GUI

An example of a slightly larger GUI application, that brings together many of the ideas presented in this chapter, is given below.

In this application we have a text input ﬁeld (a wx.TextCtrl) that allows a user to enter their name. When they click on the Enter button (wx.Button) the welcome label (a wx.StaticText) is updated with their name. The ‘Show Message’ button is used to display a wx.MessageDialog which will also contain their name.

The initial display is shown below for both a Mac and a Windows PC, note that the default background colour for a Frame is different on a Windows PC than on a Mac and thus although the GUI runs on both platforms, the look differs between the two:

The code used to implement this GUI application is given below:

If the user enters their name in the top TextCtrl ﬁeld, for example ‘Phoebe’, then when they click on the ‘Enter’ button the welcome label changes to ‘Welcome Phoebe’:

If they now click on the ‘Show Message’ button then the wx. MessageDialog (a speciﬁc type of wx.Dialog) will display a welcome message to Phoebe:

9.7 Exercises

9.7.1 Simple GUI Application

This exercise builds on the GUI you created in the last chapter.

The application should allow a user to enter their name and age. You will need to check that the value entered into the age ﬁeld is a numeric value (for example using isnumeric()). If the value is not a number, then an error message dialog should be displayed.

A button should be provided labelled ‘Birthday’; when clicked it should increment the age by one and display a Happy Birthday message. The age should be updated within the GUI.

An example of the user interface you created in the last chapter is given below:

As an example, the user might enter their name and age as shown below:

When the user clicks on the ‘birthday’ button then the Happy Birthday message dialog is displayed:

9.7.2 GUI Interface to a Tic Tac Toe Game

The aim of this exercise is to implement a simple Tic Tac Toe game. The game should allow two users to play interactive using the same mouse. The ﬁrst user will have play as the ‘X’ player and the second user as the ‘0’ player.

When each user selects a button you can set the label for the button to their symbol.

You will need two check after each move to see if someone has won (or if the game is a draw).

You will still need an internal representation of the grid so that you can deter- mine who, if anyone, has won.

An example of how the GUI for the TicTacToe game might look is given below:

You can also add dialogs to obtain the players names and to notify them who won or whether there was a draw.

Chapter 10

PyDraw wxPython Example Application

10.1 Introduction

This chapter builds on the GUI library presented in the last two chapters to illustrate how a larger application can be built. It presents a case study of a drawing tool akin to a tool such as Visio etc.

10.2 The PyDraw Application

The PyDraw application allows a user to draw diagrams using squares, circles, lines and text. At present there is no select, resize, reposition or delete option available (although these could be added if required). PyDraw is implemented using the wxPython set of components as deﬁned in version 4.0.6.

When a user starts the PyDraw application, they see the interface shown above (for both the Microsoft Windows and Apple Mac operating systems). Depending on

the operating system it has a menu bar across the top (on a Mac this menu bar is at the Top of the Mac display), a tool bar below the menu bar and a scrollable drawing area below that.

The ﬁrst button on the tool bar clears the drawing area. The second and third buttons are only implemented so that they print out a message into the Python console, but are intended to allow a user to load and save drawings.

The tool bar buttons are duplicated on the menus deﬁned for the application, along with a drawing tool selection menu, as shown below:

10.3 The Structure of the Application

The user interface created for the PyDraw application is made up of a number of elements (see below): the PyDrawMenuBar, the PyDrawToolbar containing a sequence of buttons across the top of the window, the drawing panel, and the window frame (implemented by the PyDrawFrame class).

The following diagram shows the same information as that presented above, but as a containment hierarchy, this means that the diagram illustrates how one object is contained within another. The lower level objects are contained within the higher level objects.

It is important to visualize this as the majority of wxPython interfaces are built up in this way, using containers and sizers.

The inheritance structure between the classes used in the PyDraw application is illustrated below. This class hierarchy is typical of an application which incorpo- rates user interface features with graphical elements.

10.3.1 Model, View and Controller Architecture

The application adopts the well established Model-View-Controller (or MVC) design pattern for separating out the responsibilities between the view element (e.g. the Frame or Panel), the control element (for handling user input) and the model element (which holds the data to be displayed).

This separation of concerns is not a new idea and allows the construction of GUI applications that mirror the Model-View-Controller architecture. The intention of the MVC architecture is the separation of the user display, from the control of user input, from the underlying information model as illustrated below.

There are a number of reasons why this separation is useful:

reusability of application and/or user interface components,
ability to develop the application and user interface separately,
ability to inherit from different parts of the class hierarchy.
ability to deﬁne control style classes which provide common features separately from how these features may be displayed.

This means that different interfaces can be used with the same application, without the application knowing about it. It also means that any part of the system can be changed without affecting the operation of the other. For example, the way that the graphical interface (the look) displays the information could be changed without modifying the actual application or how input is handled (the feel). Indeed the application need not know what type of interface is currently connected to it at all.

10.3.2 PyDraw MVC Architecture

The MVC structure of the PyDraw application has a top level controller class PyDrawController and a top level view class the PyDrawFrame (there is no model as the top level MVC triad does not hold any explicit data itself). This is shown below:

At the next level down there is another MVC structure; this time for the drawing element of the application. There is a DrawingController, with a DrawingModel and a DrawingPanel (the view) as illustrated below:

The DrawingModel, DrawingPanel and DrawingController classes exhibit the classic MVC structure. The view and the controller classes (DrawingPanel and DrawingController) know about each other and the drawing model, whereas the DrawingModel knows nothing about the view or the controller. The view is notiﬁed of changes in the drawing through the paint event.

10.3.3 Additional Classes

There are also four types of drawing object (of Figure): Circle, Line, Square and Text ﬁgures. The only difference between these classes is what is drawn on the graphic device context within the on_paint() method. The Figure class, from which they all inherit, deﬁnes the common attributes used by all objects within a Drawing (e.g. point representing an x and y location and size).

The PyDrawFrame class also uses a PyDrawMenuBar and a PyDrawToolBar class. The ﬁrst of these extends the wx.MenuBar with menu items for use within the PyDraw application. In turn the PyDrawToolBar extends the wx.ToolBar and provides icons for use in PyDraw.

The ﬁnal class is the PyDrawApp class that extends the wx.App class.

10.3.4 Object Relationships

However, the inheritance hierarchy is only part of the story for any object oriented application. The following ﬁgure illustrates how the objects relate to one another within the working application.

The PyDrawFrame is responsible for setting up the controller and the

DrawingPanel.

The PyDrawController is responsible for handling menu and tool bar user interactions.

This separates graphical elements from the behaviour triggered by the user.

The DrawingPanel is responsible le for displaying any ﬁgures held by the DrawingModel. The DrawingController manages all user interactions with the DrawingPanel including adding ﬁgures and clearing all ﬁgures from the model. The DrawingModel holds list of ﬁgures to be displayed.

10.4 The Interactions Between Objects

We have now examined the physical structure of the application but not how the objects within that application interact.

In many situations this can be extracted from the source code of the application (with varying degrees of difﬁculty). However, in the case of an application such as PyDraw, which is made up of a number of different interacting components, it is useful to describe the system interactions explicitly.

The diagrams illustrating the interactions between the objects use the following conventions:

a solid arrow indicates a message send,
a square box indicates a class,
a name in brackets indicates the type of instance,
numbers indicate the sequence of message sends.

These diagrams are based on the collaboration diagrams found in the UML (Uniﬁed Modelling Language) notation.

10.4.1 The PyDrawApp

When the PyDrawApp is instantiated the PyDrawFrame in created and displayed using the OnInit() method. The MainLoop() method is then invoked. This is shown below:

10.4.2 The PyDrawFrame Constructor

The PyDrawFrame constructor method sets up the main display of the UI application and also initialises the controllers and drawing elements. This is shown below using a collaboration diagram:

The PyDrawFrame constructor sets up the environment for the application. It creates the top level PyDrawController. It creates the DrawingPanel and initialises the display layout. It initialises the menu bar and tool bar. It binds the controllers menu handler to the menus and centers itself.

10.4.3 Changing the Application Mode

One interesting thing to note is what happens when the user selects an option from the Drawing menu. This allows the mode to be changed to a square, circle, line or text. The interactions involved are shown below for the situation where a user selects the ‘Circle’ menu item on the Drawing menu (using a collaboration diagram):

When the user selects one of the menu items the command_menu_handler () method of the PyDrawController is invoked. This method determines which menu item has been selected; it then calls an appropriate setter method (such as set_circle_mode() or set_line_mode() etc.). These methods set the mode attribute of the controller to an appropriate value.

10.4.4 Adding a Graphic Object

A user adds a graphic object to the drawing displayed by the DrawingPanel by pressing the mouse button.

When the user clicks on the drawing panel, the DrawingController

responds as shown below:

The above illustrates what happens when the user presses and releases a mouse button over the drawing panel, to create a new ﬁgure.

When the user presses the mouse button, a mouse clicked message is sent to the DrawingController, which decides what action to perform in response (see above). In PyDraw, it obtains the cursor point at which the event was generated by calling the GetPosition() method on the mouse_event.

The controller then calls its own add() method passing in the current mode and the current mouse location. The controller obtains the current mode (from the

PyDrawController using the method callback provided when the DrawingController is instantiated) and adds the appropriate type of ﬁgure to the DrawingModel.

The add() method then adds a new ﬁgure to the drawing model based on the speciﬁed mode.

10.5 The Classes

This section presents the classes in the PyDraw application. As these classes build on concepts already presented in the last few chapters, they shall be presented in their entirety with comments highlighting speciﬁc points of their implementations. Note that the code imports the wx module from the wxPython library, e.g.

import wx

10.5.1 The PyDrawConstants Class

The purpose of this class is to provide a set of constants that can be referenced in the remainder of the application. It is used to provide constants for the IDs used with menu items and toolbar tools. It also provides constants used to represent the current mode (to indicate whether a line, square, circle or test should be added to the display).

10.5.2 The PyDrawFrame Class

The PyDrawFrame class provides the main window for the application. Note that due to the separation of concerns introduced via the MVC architecture, the view class is only concerned with the layout of the components:

10.5.3 The PyDrawMenuBar Class

The PyDrawMenuBar class is a subclass of the wx.MenuBar class which deﬁnes the contents of the menu bar for the PyDraw application. It does this by creating two wx.Menu objects and adding them to the menu bar. Each wx.Menu implements a drop down menu from the menu bar. To add individual menu items the wx. MenuItem class is used. These menu items are appended to the menu. The menus are themselves appended to the menu bar. Note that each menu item has an id that can be used to identify the source of a command event in an event handler. This allows a single event handler to deal with events generated by multiple menu items.

class PyDrawMenuBar(wx.MenuBar):

10.5.4 The PyDrawToolBar Class

The DrawToolBar class is a subclass of wx.ToolBar. The classes constructor initialises three tools that are displayed within the toolbar. The Realize() method is used to ensure that the tools are rendered appropriately. Note that appropriate ids have been used to allow an event handler to identify which tools generated a particular command event. By reusing the same ids for related menu items and command tools, a single handler can be used to manage events from both types of sources.

10.5.5 The PyDrawController Class

This class provides the control element of the top level view. It maintains the current mode and implements a handler that can handle events from menu items and from the tool bar tools. An id is used to identify each individual menu or tool which allows a single handler to be registered with the frame.

10.5.6 The DrawingModel Class

The DrawingModel class has a contents attribute that is used to hold all the ﬁgures in the drawing. It also provides some convenience methods to reset the contents and to add a ﬁgure to the contents.

The DrawingModel is a relatively simple model which merely records a set of graphical ﬁgures in a List. These can be any type of object and can be displayed in any way as long as they implement the on_paint() method. It is the objects themselves which determine what they look like when drawn.

10.5.7 The DrawingPanel Class

The DrawingPanel class is a subclass of the wx.Panel class. It provides the view for the drawing data model. This uses the classical MVC architecture and has a model (DrawingModel), a view (the DrawingPanel) and a controller (the DrawingController).

The DrawingPanel instantiates its own DrawingController to handle mouse events.

It also registers for paint events so that it knows when to refresh the display.

10.5.8 The DrawingController Class

The DrawingController class provides the control class for the top level MVC architecture used with the DrawingModel (model) and DrawingPanel (view) classes. In particular it handles the mouse events in the DrawingPanel via the on_mouse_click() method.

It also deﬁnes an add method that is used to add a ﬁgure to the DrawingModel (the actual ﬁgure depends on the current mode of the PyDrawController). A ﬁnal method, the clear() method, removes all ﬁgures from the drawing model and refreshes the display.

10.5.9 The Figure Class

The Figure class (an abstract superclass of the Figure class hierarchy) captures the elements which are common to graphic objects displayed within a drawing. The point deﬁnes the position of the ﬁgure, while the size attribute deﬁnes the size of the ﬁgure. Note that the Figure is a subclass of a wx.Panel and thus the display is constructed from inner panels onto which the various ﬁgure shapes are drawn.

The Figure class deﬁnes a single abstract method on_paint(dc) that must be implemented by all concrete subclasses. This method should deﬁne how the shape is drawn on the drawing panel.

10.5.10 The Square Class

This is a subclass of Figure that speciﬁes how to draw a square shape in a drawing. It implements the on_paint() method inherited from Figure.

10.5.11 The Circle Class

This is another subclass of Figure. It implements the on_paint() method by drawing a circle. Note that the shape will be drawn within the panel size deﬁned via the Figure class (using the call to super). It is therefore necessary to see the circle to ﬁt within these bounds. This means that the size attribute must be used to generate an appropriate radius. Also note that the DrawCircle() method of the device context takes a point that is the centre of the circle so this must also be calculated.

10.5.12 The Line Class

This is another subclass of Figure. In this very simple example, a default end point for the line is generated. Alternatively the program could look for a mouse released event and pick up the mouse at this location and use this as the end point of the line.

10.5.13 The Text Class

This is also a subclass of Figure. A default value is used for the text to display; however a dialog could be presented to the user allowing them to input the text they wish to display:

Chapter 11

Introduction to Games Programming

11.1 Introduction

Games programming is performed by developers/coders who implement the logic that drives a game.

Historically games developers did everything; they wrote the code, designed the sprites and icons, handled the game play, dealt with sounds and music, generated any animations required etc. However, as the game industry has matured games companies have developed speciﬁc roles including Computer Graphics (CG) animators, artists, games developers and games engine and physics engine developers etc.

Those involved with code development may develop a physics engine, a games engine, the games themselves, etc. Such developers focus on different aspects of a game. For examples a game engine developer focusses on creating the framework within which the game will run. In turn a physics engine developer will focus on implementing the mathematics behind the physics of the simulated games world (such as the effect of gravity on characters and components within that world). In many cases there will also be developers working on the AI engine for a game. These developers will focus on providing facilities that allow the game or characters in the game to operate intelligently.

Those developing the actual game play will use these engines and frameworks to create the overall end result. It is they who give life to the game and make it an enjoyable (and playable) experience.

11.2 Games Frameworks and Libraries

There are many frameworks and libraries available that allow you to create anything from simple games to large complex role playing games with inﬁnite worlds.

One example is the Unity framework that can be used with the C# programming language. Another such framework is the Unreal engine used with the C++ pro- gramming language.

Python has also been used for games development with several well known games titles depending on it in one way or another. For example, Battleﬁeld 2 by Digital Illusions CE is a military simulator ﬁrst-person shooter game. Battleﬁeld Heroes handles portions of the game logic involving game modes and scoring using Python.

Other games that use Python include Civilisation IV (for many of the tasks), Pirates of the Caribbean Online and Overwatch (which makes its choices with Python). Python is also embedded as a scripting engine within tools such as Autodesk’s Maya which is a computer animation toolkit that is often used with games.

11.1 Python Games Development

For those wanting to learn more about game development; Python has much to offer. There are many examples available online as well as several game oriented frameworks.

The frameworks/libraries available for games development in Python including:

Arcade. This is a Python library for creating 2D style video games.
pyglet is a windowing and multimedia library for Python that can also be used for games development.
Cocos2d is a framework for building 2D games that is built on top of pyglet.
pygame is probably the most widely used library for creating games within the Python world. There are also many extensions available for pygame that help to create a wide range of different types of games.

We will focus on pygame in the next two chapters in this book. Other libraries of interest to Python games developers include:

PyODE. This is an open-source Python binding for the Open Dynamics Engine which is an open-source physics engine.
pymunk Pymunk is a easy-to-use 2D physics library that can be used whenever you need 2d rigid body physics with Python. It is very good when you need 2D physics in your game, demo or other application. It is built on top of the 2D physics library Chipmunk.
pyBox2D pybox2d is a 2D physics library for your games and simple simu- lations. It’s based on the Box2D library written in C++. It supports several shape types (circle, polygon, thin line segments) as well as a number of joint types (revolute, prismatic, wheel, etc.).
Blender. This is a open-source 3D computer graphics software toolset used for creating animated ﬁlms, visual effects, art, 3D printed models, interactive 3D applications and video games. Blender’s features include 3D modeling, tex- turing, raster graphics editing, rigging and skinning, etc. Python can be used as a scripting tool for creation, prototyping, game logic and more.
Quake Army Knife which is an environment for developing 3D maps for games based on the Quake engine. It is written in Delphi and Python.

11.2 Using Pygame

In the next two chapters we will explore the core pygame library and how it can be used to develop interactive computer games. The next chapter explores pygame itself and the facilities it provides. The following chapter developers a simple interactive game in which the user moves a starship around avoiding meteors which scroll vertically down the screen.

Chapter 12

Building Games with pygame

12.1 Introduction

pygame is a cross-platform, free and Open Source Python library designed to make building multimedia applications such as games easy. Development of pygame started back in October 2000 with pygame version 1.0 being released six months later. The version of pygame discussed in this chapter is version 1.9.6. If you have a later version check to see what changes have been made to see if they have any impact on the examples presented here.

pygame is built on top of the SDL library. SDL (or Simple Directmedia Layer) is a cross platform development library designed to provide access to audio, key- boards, mouse, joystick and graphics hardware via OpenGL and Direct3D. To promote portability, pygame also supports a variety of additional backends including WinDIB, X11, Linux Frame Buffer etc.

SDL ofﬁcially supports Windows, Mac OS X, Linux, iOS and Android (although other platforms are unofﬁcially supported). SDL itself is written in C and pygame provides a wrapper around SDL. However, pygame adds functionality not found in SDL to make the creation of graphical or video games easier. These functions include vector maths, collision detection, 2D sprite scene graph management, MIDI support, camera, pixel array manipulation, transformations, ﬁltering, advanced freetype font support and drawing.

The remainder of this chapter introduces pygame, the key concepts; the key modules, classes and functions and a very simple ﬁrst pygame application. The next chapter steps through the development of a simple arcade style video game which illustrates how a game can be created using pygame.

12.2 The Display Surface

The Display Surface (aka the display) is the most important part of a pygame game. It is the main window display of your game and can be of any size, however you can only have one Display Surface.

In many ways the Display Surface is like a blank piece of paper on which you can draw. The surface itself is made up of pixels which are numbered from 0,0 in the top left hand corner with the pixel locations being indexed in the x axis and the y axis. This is shown below:

The above diagram illustrates how pixels within a Surface are indexed. Indeed a Surface can be used to draw lines, shapes (such as rectangles, squares, circles and elipses), display images, manipulate individual pixels etc. Lines are drawn from one pixel location to another (for example from location 0,0 to location 9,0 which would draw a line across the top of the above display surface). Images can be displayed within the display surface given a starting point such as 1, 1.

The Display Surface is created by the pygame.display.set_mode() function. This function takes a tuple that can be used to specify the size of the Display Surface to be returned. For example:

display_surface = pygame.display.set_mode((400, 300))

This will create a Display Surface (window) of 400 by 300 pixels.

Once you have the Display Surface you can ﬁll it with an appropriate back- ground colour (the default is black) however if you want a different background colour or want to clear everything that has previously been drawn on the surface, then you can use the surface’s ﬁll() method:

WHITE = (255, 255, 255)

display_surface.fill(WHITE)

The ﬁll method takes a tuple that is used to deﬁne a colour in terms of Red, Green and Blue (or RGB) colours. Although the above examples uses a meaningful name for the tuple representing the RGB values used for white; there is of course no requirement to do this (although it is considered good practice).

To aid in performance any changes you make to the Display Surface actually happen in the background and will not be rendered onto the actual display that the user sees until you call the update() or ﬂip() methods on the surface. For example:

pygame.display.update()
pygame.display.flip()

The update() method will redraw the display with all changes made to the display in the background. It has an optional parameter that allows you to specify just a region of the display to update (this is deﬁned using a Rect which represents a rectangular area on the screen). The ﬂip() method always refreshes the whole of the display (and as such does exactly the same as the update() method with no parameters).

Another method, which is not speciﬁcally a Display Surface method, but which is often used when the display surface is created, provides a caption or title for the top level window. This is the pygame.display.set_caption() function. For example:

pygame.display.set_caption(‘Hello World’)

This will give the top level window the caption (or title) ‘Hello World’.

12.3 Events

Just as the Graphical User Interface systems described in earlier chapters have an event loop that allows the programmer to work out what the user is doing (in those cases this is typically selecting a menu item, clicking a button or entering data etc.); pygame has an event loop that allows the game to work out what the player is doing. For example, the user may press the left or right arrow key. This is repre- sented by an event.

12.3.1 Event Types

Each event that occurs has associated information such as the type of that event. For example:

Pressing a key will result in a KEYDOWN type of event, while releasing a key will result in a KEYUP event type.
Selecting the window close button will generate a QUIT event type etc.
Using the mouse can generate MOUSEMOTION events as well as

MOUSEBUTTONDOWN and MOUSEBUTTONUP event types.

Using a Joystick can generate several different types of event including JOYAXISMOTION, JOYBALLMOTION, JOYBUTTONDOWN and JOYBU TTONUP.

These event types tell you what occurred to generate the event. This means that you can choose which types of events you want to deal with and ignore other events.

12.3.2 Event Information

Each type of event object provides information associated with that event. For example a Key oriented event object will provide the actual key pressed while a mouse oriented event object will provide information on the position of the mouse, which button was pressed etc. If you try an access an attribute on an event that does not support that attribute, then an error will be generated.

The following lists some of the attributes available for different event types:

KEYDOWN and KEYUP, the event has a key attribute and a mod attribute (indicating if any other modifying keys such as Shift are also being pressed).
MOUSEBUTTONUP and MOUSEBUTTONDOWN has an attribute pos that holds a tuple indicating the mouse location in terms of x and y coordinates on the underlying surface. It also has a button attribute indicating which mouse was pressed.
MOUSEMOTION has pos, rel and buttons attributes. The pos is a tuple indi- cating the x and y location of mouse cursor. The real attribute indicates the amount of mouse movement and buttons indicates the state of the mouse buttons.

As an example if we want to check for a keyboard event type and then check that the key pressed was the space bar, then we can write:

if event.type == pygame.KEYDOWN:

# Check to see which key is pressed

if event.key == pygame.K_SPACE: print(‘space’)

This indicates that if it is a key pressed event and that the actual key was the space bar; then print the string ‘space’.

There are many keyboard constants that are used to represent the keys on the keyboard and pygame.K_SPACE constant used above is just one of them.

All the keyboard constants are preﬁxed with ‘K_’ followed by the key or the

name of the key, for example:

Events 129

K_TAB, K_SPACE, K_PLUS, K_0, K_1, K_AT, K_a, K_b, K_z, K_DELTE, K_DOWN, K_LEFT, K_RIGHT, K_LEFT etc.

Further keyboard constants are provided for modiﬁer states that can be combined with the above such as KMOD_SHIFT, KMOD_CAPS, KMOD_CTRL and KMOD_ALT.

12.3.3 The Event Queue

Events are supplied to a pygame application via the Event Queue.

The Event Queue is used to collect together events as they happen. For example, let us assume that a user clicks on the mouse twice and a key twice before a program has a chance to process them; then there will be four events in the Event Queue as shown below:

The application can then obtain an iterable from the event queue and process through the events in turn. While the program is processing these events further events may occur and will be added to the Event Queue. When the program has ﬁnished processing the initial collection of events it can obtain the next set of events to process.

One signiﬁcant advantage of this approach is that no events are ever lost; that is if the user clicks the mouse twice while the program is processing a previous set of events; they will be recorded and added to the event queue. Another advantage is that the events will be presented to the program in the order that they occurred.

The pygame.event.get() function will read all the events currently on the Event Queue (removing them from the event queue). The method returns an EventList which is an iterable list of the events read. Each event can then be processed in turn. For example:

for event in pygame.event.get():

if event.type == pygame.QUIT: print(‘Received Quit Event:’)

elif event.type == pygame.MOUSEBUTTONDOWN: print(‘Received Mouse Event’)

elif event.type == pygame.KEYDOWN: print(‘Received KeyDown Event’)

In the above code snippet an EventList is obtained from the Event Queue containing the current set of events. The for loop then processes each event in turn checking the type and printing an appropriate message.

You can use this approach to trigger appropriate behaviour such as moving an image around the screen or calculating the players score etc. However, be aware that if this behaviour takes too long it can make the game difﬁcult to play (although the examples in this chapter and the next are simple enough that this is not a problem).

12.4 A First pygame Application

We are now at the point where we can put together what we have looked at so far and create a simple pygame application.

It is common to create a hello world style program when using a new pro- gramming language or using a new application framework etc. The intention is that the core elements of the language or framework are explored in order to generate the most basic form of an application using the language or framework. We will therefore implement the most basic application possible using pygame.

The application we will create will display a pygame window, with a ‘Hello World’ title. We will then be able to quit the game. Although technically speaking this isn’t a game, it does possess the basic architecture of a pygame application. The simple HelloWorld game will initialise pygame and the graphical dis- play. It will then have a main game playing loop that will continue until the user selects to quit the application. It will then shut down pygame. The display created by the program is shown below for both Mac and Windows operating systems:

To quit the program click on the exit button for the windowing system you are using.

The simple HelloWorld game is given below:

import pygame

def main():

print(‘Starting Game’)

print(‘Initialising pygame’)

pygame.init() # Required by every pygame application

print(‘Initialising HelloWorldGame’) pygame.display.set_mode((200, 100)) pygame.display.set_caption(‘Hello World’)

print(‘Update display’) pygame.display.update()

print(‘Starting main Game Playing Loop’) running = True

while running:

for event in pygame.event.get():

if event.type == pygame.QUIT: print(‘Received Quit Event:’, event) running = False

print(‘Game Over’) pygame.quit()if name == ‘ main ‘:

There are several key steps highlighted by this example, these steps are:

1. Import pygame. pygame is of course not one of the default modules available within Python. You must ﬁrst import pygame into you code. The import pygame statement imports the pygame module into your code and makes the functions and classes in pygame available to you (note the capitalisation – pygame is not the same module name as PyGame). It is also common to ﬁnd that programs import
• from pygame.locals import *
• This adds several constants and functions into the namespace of your pro- gram. In this very simple example we have not needed to do this.

2.Initialise pygame. Almost every pygame module needs to be initialised in some way and the simplest way to do this is to call pygame.init(). This will do what is required to set the pygame environment up for use. If you forget to call this function you will typically get an error message such as pygame.error: video system not initialised (or something similar). If you get such a method check to see that you have called pygame.init(). Note that you can initialise individual pygame modules (for example the pygame.font module can be initialised using pygame.font.init()) if required. However pygame.init() is the most commonly used approach to setting up pygame.

3.Set up the display. Once you have initialised the pygame framework you can setup the display. In the above code example, the display is set up using the pygame.display.set_mode() function. This function takes a tuple specifying the size of the window to be created (in this case 200 pixels wide by 100 pixels high). Note that if you try and invoke this function by passing in two parameters instead of a tuple, then you will get an error. This function returns the drawing surface or screen/window that can be used to display items within the game such as icons, messages, shapes etc. As our example is so simple we do not bother saving it into a variable. However, anything more complex than this will need to do so. We also set the window/frame’s caption (or title). This is displayed in the title bar of the window.
4.Render the display. We now call the pygame.display.update() func- tion. This function causes the current details of the display to be drawn. At the moment this is a blank window. However, it is common in games to perform a series of updates to the display in the background and then when the program is ready to update the display to call this function. This batches a series of updates and the causes the display to be refreshed. In a complex display it is possible to indicate which parts of the display need to be redrawn rather than redrawing the whole window. This is done by passing a parameter into the update() function to indicate the rectangle to be redrawn. However, our example is so simple we are ok with redrawing the whole window and therefore we do not need to pass any parameters to the function.

5.Main game playing loop. It is common to have a main game playing loop that drives the processing of user inputs, modiﬁes the state of the game and updates the display. This is represented above by the while running: loop. The local variable running is initialised to True. This means that the while loop ensures that the game continues until the user selects to quit the game at which point the running variable is set to False which causes the loop to exit. In many cases this loop will call update() to refresh the display. The above example does not do this as nothing is changed in the display. However the example developed later in this chapter will illustrate this idea.

6. Monitor for events that drive the game. As mentioned earlier the event queue is used to allow user inputs to be queued and then processed by the game. In the simple example shown above this is represented by a for loop that receives events using pygame.event.get() and then checking to see if the event is a pygame.QUIT event. If it is, then it sets the running flag to False. Which will cause the main while loop of the game to terminate.

7. Quit pygame once ﬁnished. In pygame any module that has an init() function also has an equivalent quit() function that can be used to perform any cleanup operations. As we called init() on the pygame module at the

start of our program we will therefore need to call pygame.quit() at the end of the program to ensure everything is tidied up appropriately.

12.5 Further Concepts

There are very many facilities in pygame that go beyond what we can cover in this book, however a few of the more common are discussed below.

Surfaces are a hierarchy. The top level Display Surface may contain other surfaces that may be used to draw images or text. In turn containers such as Panels may render surfaces to display images or text etc.

Other types of surface. The primary Display Surface is not the only surface in pygame. For example, when an image, such as a PNG or JPEG image is loaded into a game then it is rendered onto a surface. This surface can then be displayed within another surface such as the Display Surface. This means that anything you can do to the Display Surface you can do with any other surface such as draw on it, put text on it, colour it, add another icon onto the surface etc.

Fonts. The pygame.font.Font object is used to create a Font that can be used to render text onto a surface. The render method returns a surface with the text rendered on it that can be displayed within another surface such as the Display Surface. Note that you cannot write text onto an existing surface you must always obtain a new surface (using render) and then add that to an existing surface. The text can only be displayed in a single line and the surface holding the text will be of the dimensions required to render the text. For example:

text_font = pygame.font.Font(‘freesansbold.ttf’, 18) text_surface = text_font.render(‘Hello World’, antialias=True, color=BLUE)

This creates a new Font object using the speciﬁed font with the speciﬁed font size (in this case 18). It will then render the string ‘Hello World’ on to a new surface using the speciﬁed font and font size in Blue. Specifying that antialias is True indicates that we would like to smooth the edges of the text on the screen.

Rectangles (or Rects). The pygame.Rect class is an object used to represent rectangular coordinates. A Rect can be created from a combination of the top left corner coordinates plus a width and height. For flexibility many functions that expect a Rect object can also be given a Rectlike list; this is a list that contains the data necessary to create a Rect object. Rects are very useful in a pygame Game as they can be used to deﬁne the borders of a game object. This means that they can be used within games to detect if two objects have collided. This is made particularly easy because the Rect class provides several collision detection methods:

pygame.Rect.contains() test if one rectangle is inside another
pygame.Rect.collidepoint() test if a point is inside a rectangle
pygame.Rect.colliderect() test if two rectangles overlap
pygame.Rect.collidelist() test if one rectangle in a list intersects
pygame.Rect.collidelistall() test if all rectangles in a list intersect
pygame.Rect.collidedict() test if one rectangle in a dictionary intersects
pygame.Rect.collidedictall() test if all rectangles in a dictionary intersect

The class also provides several other utility methods such as move() which moves the rectangle and inﬂate() which can grow or shrink the rectangles size.

Drawing shapes. The pygame.draw module has numerous functions that can be used to draw lines and shapes onto a surface, for example:

pygame.draw.rect(display_surface, BLUE, [x, y, WIDTH, HEIGHT])

This will draw a ﬁlled blue rectangle (the default) onto the display surface. The rectangle will be located at the location indicated by x and y (on the surface). This indicates the top left hand corner of the rectangle. The width and height of the rectangle indicate its size. Note that these dimensions are deﬁned within a list which is a structure referred to as being rect like (see below). If you do not want a ﬁlled rectangle (i.e. You just want the outline) then you can use the optional width parameter to indicate the thickness of the outer edge. Other methods available include:

pygame.draw.polygon() draw a shape with any number of sides
pygame.draw.circle() draw a circle around a point
pygame.draw.ellipse() draw a round shape inside a rectangle
pygame.draw.arc() draw a partial section of an ellipse
pygame.draw.line() draw a straight line segment
pygame.draw.lines() draw multiple contiguous line segments
pygame.draw.aaline() draw ﬁne antialiased lines
pygame.draw.aalines() draw a connected sequence of antialiased lines

Images. The pygame.image module contains functions for loading, saving and transforming images. When an image is loaded into pygame, it is represented by a Surface object. This means that it is possible to draw, manipulate and process an image in exactly the same way as any other surface which provides a great deal of flexibility.

At a minimum the module only supports loading uncompressed BMP images but usually also supports JPEG, PNG, GIF (non-animated), BMP, TIFF as well as other formats. However, it only supports a limited set of formats when saving images; these are BMP, TGA, PNG and JPEG.

An image can be loaded from a ﬁle using:

image_surface = pygame.image.load(filename).convert()

This will load the image from the speciﬁed ﬁle onto a surface.

One thing you might wonder at is the use of the convert() method on the object returned from the pygame.image.load() function. This function returns a Surface that is used to display the image contained in the ﬁle. We call the method convert() on this Surface, not to convert the image from a particular ﬁle format (such as PNG, or JPEG) instead this method is used to convert the pixel format used by the Surface. If the pixel format used by the Surface is not the same as the display format, then it will need to be converted on the fly each time the image is displayed on the screen; this can be a fairly time consuming (and unnecessary) process. We therefore do this once when the image is loaded which means that it should not hinder runtime performance and may improve performance signiﬁcantly on some systems.

Once you have a surface containing an image it can be rendered onto another surface, such as the display surface using the Surface.blit() method. For example:

display_surface.blit(image_surface, (x, y))

Note that the position argument is a tuple specifying the x and y coordinates to the image on the display surface.

Strictly speaking the blit() method draws one surface (the source surface) onto another surface at the destination coordinates. Thus the target surface does not beed to be the top level display surface.

Clock. A Clock object is an object that can be used to track time. In particular it can be used to deﬁne the frame rate for the game. That is the number of frames rendered per second. This is done using the Clock.tick() method. This method should be called once (and only once) per frame. If you pass the optional framerate argument to the tick() the function, then pygame will ensure that

the games refresh rate is slower then the the given ticks per second. This can be used to help limit the runtime speed of a game. By calling clock.tick

(30) once per frame, the program will never run at more than 30 frames per second.

12.6 A More Interactive pygame Application

The ﬁrst pygame application we looked at earlier just displayed a window with the caption ‘Hello World’. We can now extend this a little by playing with some of the features we have looked at above.

The new application will add some mouse event handling. This will allow us to pick up the location of the mouse when the user clicked on the window and draw a small blue box at that point.

If the user clicks the mouse multiple times we will get multiple blue boxes being drawn. This is shown below.

This is still not much of a game but does make the pygame application more interactive.

The program used to generate this application is presented below:

import pygame

FRAME_REFRESH_RATE = 30

BLUE = (0, 0, 255)

BACKGROUND = (255, 255, 255) # White

WIDTH = 10

HEIGHT = 10

def main():

print(‘Initialising PyGame’)

pygame.init() # Required by every PyGame application

print(‘Initialising Box Game’)

display_surface = pygame.display.set_mode((400, 300)) pygame.display.set_caption(‘Box Game’)

print(‘Update display’) pygame.display.update() print(‘Setup the Clock’) clock = pygame.time.Clock()

# Clear the screen of current contents

display_surface.fill(BACKGROUND)

print(‘Starting main Game Playing Loop’) running = True

while running:

for event in pygame.event.get():

if event.type == pygame.QUIT: print(‘Received Quit Event:’, event) running = False

elif event.type == pygame.MOUSEBUTTONDOWN: print(‘Received Mouse Event’, event) x, y = event.pos

pygame.draw.rect(display_surface, BLUE, [x, y,

WIDTH, HEIGHT])

second

# Update the display

pygame.display.update()

# Defines the frame rate – the number of frames per

# Should be called once per frame (but only once)

clock.tick(FRAME_REFRESH_RATE)

print(‘Game Over’)

# Now tidy up and quit Python

pygame.quit()

if name == ‘ main ‘: main()

Note that we now need to record the display surface in a local variable so that we can use it to draw the blue rectangles. We also need to call the pygame.dis- play.update() function each time round the main while loop so that the new rectangles we have drawn as part of the event processing for loop are displayed to the user.

We also set the frame rate each time round the main while loop. This should happen once per frame (but only once) and uses the clock object initialised at the start of the program.

12.7 Alternative Approach to Processing Input Devices

There are actually two ways in which inputs from a device such as a mouse, joystick or the keyboard can be processed. One approach is the Event based model described earlier. The other approach is the State based approach.
Although the Event based approach has many advantages is has two disadvantages:
• Each event represents a single action and continuous actions are not explicitly represented. Thus if the user presses both the X key and the Z key then this will generate two events and it will be up to the program to determine that they have been pressed at the same time.
• It is also up to the program to determine that the user is still pressing a key (by noting that no KEYUP event has occurred).
• Both of these are possible but can be error prone.
An alternative approach is to use the State based approach. In the state based approach the program can directly check the state of a input device (such as a key or mouse or keyboard). For example, you can use pygame.key.get_pressed() which returns the state of all the keys. This can be used to determine if a speciﬁc key is being pressed at this moment in time. For example, pygame.key. get_pressed()[pygame.K_SPACE] can be used to check to see if the space bar is being pressed.
This can be used to determine what action to take. If you keep checking that the key is pressed then you can keep performing the associated action. This can be very useful for continues actions in a game such as moving an object etc.
However, if the user presses a key and then releases it before the program checks the state of the keyboard then that input will be missed.

12.8 pygame Modules

There are numerous modules provided as part of pygame as well as associated libraries. Some of the core modules are listed below:
• pygame.display This module is used to control the display window or screen. It provides facilities to initialise and shutdown the display module. It can be used to initialise a window or screen. It can also be used to cause a window or screen to refresh etc.

• pygame.event This module manages events and the event queue. For example pygame.event.get() retrieves events from the event queue, pygame.event.poll() gets a single event from the queue and pygame.event.peek() tests to see if there are any event types on the queue.
• pygame.draw The draw module is used to draw simple shapes onto a Surface. For example, it provides functions for drawing a rectangle (pygame.draw.rect), a polygon, a circle, an ellipse, a line etc.
• pygame.font The font module is used to create and render TrueType fonts into a new Surface object. Most of the features associated with fonts are sup- ported by the pygame.font.Font class. Free standing module functions allow the module to be initialised and shutdown, plus functions to access fonts such as pygame.font.get_fonts() which provides a list of the currently available fonts.
• pygame.image This module allows images to be saved and loaded. Note that images are loaded into a Surface object (there is no Image class unlike many other GUI oriented frameworks).
• pygame.joystick The joystick module provides the Joystick object and several supporting functions. These can be used for interacting with joysticks, gamepads and trackballs.
• pygame.key This module provides support for working with inputs from the keyboard. This allows the input keys to be obtained and modiﬁer keys (such as Control and Shift) to be identiﬁed. It also allows the approach to repeating keys to be speciﬁed.
• pygame.mouse This module provides facilities for working with mouse input such as obtaining the current mouse position, the state of mouse buttons as well as the image to use for the mouse.
• pygame.time This is the pygame module for managing timing within a game. It provides the pygame.time.Clock class that can be used to track time.

Chapter 13

StarshipMeteors pygame

13.1 Creating a Spaceship Game

In this chapter we will create a game in which you pilot a starship through a ﬁeld of meteors. The longer you play the game the larger the number of meteors you will encounter. A typical display from the game is shown below for a Apple Mac and a Windows PC:

We will implement several classes to represent the entities within the game. Using classes is not a required way to implement a game and it should be noted that many developers avoid the use of classes. However, using a class allows data associated with an object within the game to be maintained in one place; it also simpliﬁes the creation of multiple instances of the same object (such as the meteors) within the game.

The classes and their relationships are shown below:

This diagram shows that the Starship and Meteor classes will extend a class called GameObject.

In turn it also shows that the Game has a 1:1 relationship with the Starship class. That is the Game holds a reference to one Starship and in turn the starship holds a single reference back to the Game.

In contrast the Game has a 1 to many relationship with the Meteor class. That is the Game object holds references to many Meteors and each Meteor holds a reference back to the single Game object.

13.2 The Main Game Class

The ﬁrst class we will look at will be the Game class itself.

The Game class will hold the list of meteors and the starship as well as the main game playing loop.

It will also initialise the main window display (for example by setting the size and the caption of the window).

In this case we will store the display surface returned by the pygame.dis- play.set_mode() function in an attribute of the Game object called dis- play_surface. This is because we will need to use it later on to display the starship and the meteors.

We will also hold onto an instance of the pygame.time.Clock() class that we will use to set the frame rate each time round the main game playing while loop. The basic framework of our game is shown below; this listing provides the basic Game class and the main method that will launch the game. The game also deﬁnes three global constants that will be used to deﬁne the frame refresh rate and the size of the display.

The main play() method of the Game class has a loop that will continue until the user selects to quit the game. They can do this in one of two ways, either by pressing the ‘q’ key (represented by the event.key K_q) or by clicking on the window close button. In either case these events are picked up in the main event processing for loop within the main while loop method.

If the user does not want to quit the game then the display is updated (refreshed) and then the clock.tick() (or frame) rate is set.

When the user selects to quit the game then the main while loop is terminated

(the is_running flag is set to False) and the pygame.quit() method is called to shut down pygame.

At the moment this not a very interactive game as it does not do anything except allow the user to quit. In the next section we will add in behaviour that will allow us to display the space ship within the display.

13.3 The GameObject Class

The GameObject class deﬁnes three methods:

The load_image() method can be used to load an image to be used to visually represent the speciﬁc type of game object. The method then uses the width and height of the image to deﬁne the width and height of the game object.

The rect() method returns a rectangle representing the current area used by the game object on the underlying drawing surface. This differs from the images own rect() which is not related to the location of the game object on the underlying surface. Rects are very useful for comparing the location of one object with another (for example when determining if a collision has occurred).

The draw() method draws the GameObjects’ image onto the display_- surface held by the game using the GameObjects current x and y coordinates. It can be overridden by subclasses if they wish to be drawn in a different way.

The code for the GameObject class is presented below:

class GameObject:

The GameObject class is directly extended by the Starship class and the

Meteor class.

Currently there are only two types of game elements, the starship and the meteors; but this could be extended in future to planets, comets, shooting stars etc.

13.4 Displaying the Starship

The human player of this game will control a starship that can be moved around the display.

The Starship will be represented by an instance of the class Starship. This class will extend the GameObject class that holds common behaviours for any type of element that is represented within the game.

The Starship class deﬁnes its own init () method that takes a reference to the game that the starship is part of. This initialisation method sets the initial starting location of the Starship as half the width of the display for the x coordinate and the display height minus 40 for the y coordinate (this gives a bit of a buffer before the end of the screen). It then uses the load_image() method from the GameObject parent class to load the image to be used to represent the Starship. This is held in a ﬁle called starship.png. For the moment we will leave the Starship class as it is (however we will return to this class so that we can make it into a movable object in the next section).

The current version of the Starship class is given below:

In the Game class we will now add a line to the init () method to initialise the Starship object. This line is:

# Set up the starship

self.starship = Starship(self)

We will also add a line to the main while loop within the play() method just before we refresh the display. This line will call the draw() method on the starship object:

# Draw the starship self.starship.draw()

This will have the effect of drawing the starship onto the windows drawing

surface in the background before the display is refreshed.

When we now run this version of the StarshipMeteor game we now see the Starship in the display:

Of course at the moment the starship does not move; but we will address that in the next section.

13.5 Moving the Spaceship

We want to be able to move the Starship about within the bounds of the display screen.

To do this we need to change the starships x and y coordinates in response to the user pressing various keys.

We will use the arrow keys to move up and down the screen or to the left or right of the screen. To do this we will deﬁne four methods within the Starship class; these methods will move the starship up, down, left and right etc.

This version of the Starship class deﬁnes the various move methods. These methods use a new global value STARSHIP_SPEED to determine how far and how fast the Starship moves. If you want to change the speed that the Starship moves then you can change this global value.

Depending upon the direction intended we will need to modify either the x or y coordinate of the Starship.

If the starship moves to the left then the x coordinate is reduced by

STARSHIP_SPEED,

if it moves to the right then the x coordinate is increased by

STARSHIP_SPEED,

in turn if the Starship moves up the screen then the y coordinate is decremented by STARSHIP_SPEED,

but if it moves down the screen then the y coordinate is increased by

STARSHIP_SPEED.

Of course we do not want our Starship to fly off the edge of the screen and so a test must be made to see if it has reached the boundaries of the screen. Thus tests are made to see if the x or y values have gone below Zero or above the DISPLAY_WIDTH or DISPLAY_HEIGHT values. If any of these conditions are met then the x or y values are reset to an appropriate default.

We can now use these methods with player input. This player input will indicate the direction that the player wants to move the Starship. As we are using the left, right, up and down arrow keys for this we can extend the event processing loop that we have already deﬁned for the main game playing loop. As with the letter q, the event keys are preﬁxed by the letter K and an underbar, but this time the keys are named K_LEFT, K_RIGHT, K_UP and K_DOWN.

When one of these keys is pressed then we will call the appropriate move

method on the starship object already held by the Game object.

However, we are not quite ﬁnished. If we try and run this version of the program we will get a trail of Starships drawn across the screen; for example:

The problem is that we are redrawing the starship at a different position; but the previous image is still present.

We now have two choices one is to merely ﬁll the whole screen with black; effectively hiding anything that has been drawn so far; or alternatively we could just draw over the area used by the previous image position. Which approach is adopted depends on the particular scenario represented by your game. As we will have a lot of meteors on the screen once we have added them; the easiest option is to over- write everything on the screen before redrawing the starship. We will therefore add the following line:

# Clear the screen of current contents

self.display_surface.ﬁll(BACKGROUND)

This line is added just before we draw the Starship within the main game playing

while loop.

Now when we move the Starship the old image is removed before we draw the new image:

13.6 Adding a Meteor Class

The Meteor class will also be a subclass of the GameObject class. However, it will only provide a move_down() method rather than the variety of move methods of the Starship.

It will also need to have a random starting x coordinate so that when a meteor is added to the game its starting position will vary. This random position can be generated using the random.randint() function using a value between 0 and the width of the drawing surface. The meteor will also start at the top of the screen so will have a different initial y coordinate to the Starship. Finally, we also want our meteors to have different speeds; this can be another random number between 1 and some speciﬁed maximum meteor speed.

To support these we need to add random to the modules being imported and deﬁne several new global values, for example:

import pygame, random INITIAL_METEOR_Y_LOCATION = 10

MAX_METEOR_SPEED = 5

We can now deﬁne the Meteor class:

class Meteor(GameObject):

“”” represents a meteor in the game “””

def init (self, game): self.game = game

self.x = random.randint(0, DISPLAY_WIDTH) self.y = INITIAL_METEOR_Y_LOCATION

self.speed = random.randint(1, MAX_METEOR_SPEED) self.load_image(‘meteor.png’)

def move_down(self):

“”” Move the meteor down the screen “””

self.y = self.y + self.speed

if self.y > DISPLAY_HEIGHT: self.y = 5

‘)’

def str (self):

return ‘Meteor(‘ + str(self.x) + ‘, ‘ + str(self.y) +

The init () method for the Meteor class has the same steps as the Starship; the difference is that the x coordinate and the speed are randomly generated. The image used for the Meteor is also different as it is ‘meteor.png’.

We have also implemented a move_down() method. This is essentially the same as the Starships move_down().

Note that at this point we could create a subclass of GameObject called MoveableGameObject (which extends GameObject) and push the move operations up into that class and have the Meteor and Starship classes extend that class. However we don’t really want to allow meteors to move just anywhere on the screen.

We can now add the meteors to the Game class. We will add a new global value to indicate the number of initial meteors in the game:

INITIAL_NUMBER_OF_METEORS = 8

Next we will initialise a new attribute for the Game class that will hold a list of Meteors. We will use a list here as we want to increase the number of meteors as the game progresses.

To make this process easy we will use a list comprehension which allows a for

loop to run with the results of an expression captured by the list:

# Set up meteors

self.meteors = [Meteor(self) for _ in range(0, INITIAL_NUMBER_OF_METEORS)]

We now have a list of meteors that need to be displayed. We thus need to update the while loop of the play() method to draw not only the starship but also all the meteors:

# Draw the meteors and the starship

self.starship.draw()

for meteor in self.meteors: meteor.draw()

The end result is that a set of meteor objects are created at random starting locations across the top of the screen:

13.7 Moving the Meteors

We now want to be able to move the meteors down the screen so that the Starship has some objects to avoid.

We can do this very easily as we have already implemented a move_down() method in the Meteor class. We therefore only need to add a for loop to the main game playing while loop that will move all the meteors. For example:

# Move the Meteors

for meteor in self.meteors: meteor.move_down()

This can be added after the event processing for loop and before the screen is refreshed/redrawn or updated.

Now when we run the game the meteors move and the player can navigate the Starship between the falling meteors.

13.8 Identifying a Collision

At the moment the game will play for ever as there is no end state and no attempt to identify if a Starship has collided with a meteor.

We can add Meteor/Starship collision detection using PyGame Rects. As mentioned in the last chapter a Rect is a PyGame class used to represent rect- angular coordinates. It is particularly useful as the pygame.Rect class provides several collision detection methods that can be used to test if one rectangle (or point) is inside another rectangle. We can therefore use one of the methods to test if the rectangle around the Starship intersects with any of the rectangles around the Meteors.

The GameObject class already provides a method rect() that will return a Rect object representing the objects’ current rectangle with respect to the drawing surface (essentially the box around the object representing its location on the screen).
Thus we can write a collision detection method for the Game class using the
GameObject generated rects and the Rect class colliderect() method:
def _check_for_collision(self):

“”” Checks to see if any of the meteors have collided with the starship “””
result = False
for meteor in self.meteors:
if self.starship.rect().colliderect(meteor.rect()): result = True
break return result
Note that we have followed the convention here of preceding the method name with an underbar indicating that this method should be considered private to the class. It should therefore never be called by anything outside of the Game class. This convention is deﬁned in PEP 8 (Python Enhancement Proposal) but is not enforced by the language.
We can now use this method in the main while loop of the game to check for a collision:

# Check to see if a meteor has hit the ship
if self._check_for_collision(): starship_collided = True

This code snippet also introduces a new local variable starship_collided. We will initially set this to False and is another condition under which the main game playing while loop will terminate:

is_running = True
starship_collided = False

# Main game playing Loop
while is_running and not starship_collided:
Thus the game playing loop will terminate if the user selects to quit or if the starship collides with a meteor.

13.9 Identifying a Win

We currently have a way to loose the game but we don’t have a way to win the game! However, we want the player to be able to win the game by surviving for a speciﬁed period of time. We could represent this with a timer of some sort. However, in our case we will represent it as a speciﬁc number of cycles of the main game playing loop. If the player survives for this number of cycles then they have won. For example:

# See if the player has won

if cycle_count == MAX_NUMBER_OF_CYCLES: print(‘WINNER!’)

break

In this case a message is printed out stating that the player won and then the main game playing loop is terminated (using the break statement).

The MAX_NUMBER_OF_CYCLES global value can be set as appropriate, for example:

MAX_NUMBER_OF_CYCLES = 1000

13.10 Increasing the Number of Meteors

We could leave the game as it is at this point, as it is now possible to win or loose the game. However, there are a few things that can be easily added that will enhance the game playing experience. One of these is to increase the number of Meteors on the screen making it harder as the game progresses.

We can do this using a NEW_METEOR_CYCLE_INTERVAL.

NEW_METEOR_CYCLE_INTERVAL = 40

When this interval is reached we can add a new Meteor to the list of current Meteors; it will then be automatically drawn by the Game class. For example:

# Determine if new meteors should be added

if cycle_count % NEW_METEOR_CYCLE_INTERVAL == 0: self.meteors.append(Meteor(self))

Now every NEW_METEOR_CYCLE_INTERVAL another meteor will be added at a random x coordinate to the game.

13.11 Pausing the Game

Another feature that many games have is the ability to pause the game. This can be easily added by monitoring for a pause key (this could be the letter p represented by the event_key pygame.K_p). When this is pressed the game could be paused until the key is pressed again.

The pause operation can be implemented as a method _pause() that will consume all events until the appropriate key is pressed. For example:

In this method the outer while loop will loop until the paused local variable is set too False. This only happens when the ‘p’ key is pressed. The break after the statement setting paused to False ensures that the inner for loop is terminated allowing the outer while loop to check the value of paused and terminate.

The _pause() method can be invoked during the game playing cycle by monitoring for the ‘p’ key within the event for loop and calling the _pause() method from there:

elif event.key == pygame.K_p: self._pause()

Note that again we have indicated that we don’t expect the _pause() method to be called from outside the game by preﬁxing the method name with an underbar (‘_’).

13.12 Displaying the Game Over Message

PyGame does not come with an easy way of creating a popup dialog box to display messages such as ‘You Won’; or ‘You Lost’ which is why we have used print statements so far. However, we could use a GUI framework such as wxPython to do this or we could display a message on the display surface to indicate whether the player has won or lost.

We can display a message on the display surface using the pygame.font. Font class. This can be used to create a Font object that can be rendered onto a surface that can be displayed onto the main display surface.

We can therefore add a method _display_message() to the Game class that can be used to display appropriate messages:

def _display_message(self, message):
“”” Displays a message to the user on the screen “””
print(message)
text_font = pygame.font.Font(‘freesansbold.ttf’, 48)
text_surface = text_font.render(message, True, BLUE, WHITE)
text_rectangle = text_surface.get_rect()
text_rectangle.center = (DISPLAY_WIDTH / 2,
DISPLAY_HEIGHT / 2)
self.display_surface.fill(WHITE) self.display_surface.blit(text_surface, text_rectangle)

Again the leading underbar in the method name indicates that it should not be called from outside the Game class.
We can now modify the main loop such that appropriate messages are displayed
to the user, for example:

# Check to see if a meteor has hit the ship
if self._check_for_collision(): starship_collided = True
self._display_message(‘Collision: Game Over’)

The result of the above code being run when a collision occurs is shown below:

13.13 The StarshipMeteors Game

The complete listing for the ﬁnal version of the StarshipMeteors game is given below:

import pygame, random, time FRAME_REFRESH_RATE = 30

DISPLAY_WIDTH = 600

DISPLAY_HEIGHT = 400

WHITE = (255, 255, 255)

BACKGROUND = (0, 0, 0)

INITIAL_METEOR_Y_LOCATION = 10

INITIAL_NUMBER_OF_METEORS = 8

MAX_METEOR_SPEED = 5

STARSHIP_SPEED = 10

MAX_NUMBER_OF_CYCLES = 1000

NEW_METEOR_CYCLE_INTERVAL = 40


class GameObject:

def load_image(self, filename):
self.image = pygame.image.load(filename).convert() self.width = self.image.get_width()
self.height = self.image.get_height()

def rect(self):
""" Generates a rectangle representing the objects location
and dimensions """
return pygame.Rect(self.x, self.y, self.width, self.height)

def draw(self):
""" draw the game object at the current x, y coordinates """
self.game.display_surface.blit(self.image, (self.x,
self.y))

class Starship(GameObject):
""" Represents a starship"""

def  init (self, game): self.game = game
self.x = DISPLAY_WIDTH / 2 self.y = DISPLAY_HEIGHT - 40
self.load_image('starship.png')

def move_right(self):
""" moves the starship right across the screen """
self.x = self.x + STARSHIP_SPEED
if self.x + self.width > DISPLAY_WIDTH: self.x = DISPLAY_WIDTH - self.width

def move_left(self):
""" Move the starship left across the screen """
self.x = self.x - STARSHIP_SPEED
if self.x < 0: self.x = 0

def move_up(self):
""" Move the starship up the screen """
self.y = self.y - STARSHIP_SPEED
if self.y < 0: self.y = 0

def move_down(self):
""" Move the starship down the screen """
self.y = self.y + STARSHIP_SPEED
if self.y + self.height > DISPLAY_HEIGHT:

self.y = DISPLAY_HEIGHT - self.height

')'
 

def  str  (self):
return 'Starship(' + str(self.x) + ', ' + str(self.y) +
 

class Meteor(GameObject):
""" represents a meteor in the game """

def  init (self, game): self.game = game
self.x = random.randint(0, DISPLAY_WIDTH) self.y = INITIAL_METEOR_Y_LOCATION
self.speed = random.randint(1, MAX_METEOR_SPEED) self.load_image('meteor.png')

def move_down(self):
""" Move the meteor down the screen """
self.y = self.y + self.speed
if self.y > DISPLAY_HEIGHT: self.y = 5
')'
 

def  str  (self):
return 'Meteor(' + str(self.x) + ', ' + str(self.y) +
 

class Game:
""" Represents the game itself, holds the main game playing loop """

def  init (self): pygame.init()
# Set up the display
self.display_surface = pygame.display.set_mode((DISPLAY_WIDTH, DISPLAY_HEIGHT))
pygame.display.set_caption('Starship Meteors') 
# Used for timing within the program. self.clock = pygame.time.Clock()
# Set up the starship self.starship = Starship(self) 
# Set up meteors
self.meteors = [Meteor(self) for _ in range(0, INITIAL_NUMBER_OF_METEORS)]

def _check_for_collision(self):
""" Checks to see if any of the meteors have collided with the starship """
result = False
for meteor in self.meteors:
if self.starship.rect().colliderect(meteor.rect()): result = True

break return result

def _display_message(self, message):
""" Displays a message to the user on the screen """ text_font = pygame.font.Font('freesansbold.ttf', 48) text_surface = text_font.render(message, True, BLUE,
 
WHITE)
 
text_rectangle = text_surface.get_rect() text_rectangle.center = (DISPLAY_WIDTH / 2,
 
DISPLAY_HEIGHT / 2)
self.display_surface.fill(WHITE) self.display_surface.blit(text_surface, text_rectangle)

def _pause(self): paused = True while paused:
for event in pygame.event.get():
if event.type == pygame.KEYDOWN:
if event.key == pygame.K_p: paused = False
break

def play(self): is_running = True
starship_collided = False
cycle_count = 0

# Main game playing Loop
while is_running and not starship_collided:
# Indicates how many times the main game loop has
 
been run
 
cycle_count += 1

# See if the player has won
if cycle_count == MAX_NUMBER_OF_CYCLES: self._display_message('WINNER!') break

# Work out what the user wants to do
for event in pygame.event.get():
if event.type == pygame.QUIT: is_running = False
elif event.type == pygame.KEYDOWN:
# Check to see which key is pressed
if event.key == pygame.K_RIGHT:
# Right arrow key has been pressed # move the player right self.starship.move_right()
elif event.key == pygame.K_LEFT:
# Left arrow has been pressed
 


# move the player left
self.starship.move_left()
elif event.key == pygame.K_UP: self.starship.move_up()
elif event.key == pygame.K_DOWN: self.starship.move_down()
elif event.key == pygame.K_p: self._pause()
elif event.key == pygame.K_q: is_running = False

# Move the Meteors
for meteor in self.meteors: meteor.move_down()

# Clear the screen of current contents
self.display_surface.fill(BACKGROUND)

# Draw the meteors and the starship
self.starship.draw()
for meteor in self.meteors: meteor.draw()

# Check to see if a meteor has hit the ship
if self._check_for_collision(): starship_collided = True
self._display_message('Collision: Game Over')

# Determine if new mateors should be added
if cycle_count % NEW_METEOR_CYCLE_INTERVAL == 0: self.meteors.append(Meteor(self))

# Update the display
pygame.display.update()
 


frames per once)
 

# Defines the frame rate. The number is number of # second. Should be called once per frame (but only self.clock.tick(FRAME_REFRESH_RATE)
 
time.sleep(1)
# Let pygame shutdown gracefully
pygame.quit()

def main():
print('Starting Game') game = Game() game.play() print('Game Over')
if  name  == ' main ': main()

Chapter 14

Introduction to Testing

14.1 Introduction

This chapter considers the different types of tests that you might want to perform with the systems you develop in Python. It also introduces Test Driven Development.

14.2 Types of Testing

There are at least two ways of thinking about testing:

It is the process of executing a program with the intent of ﬁnding errors/bugs (see Glenford Myers, The Art of Software Testing).
It is a process used to establish that software components fulﬁl the requirements identiﬁed for them, that is that they do what they are supposed to do.

These two aspects of testing tend to have been emphasised at different points in the software lifecycle. Error Testing is an intrinsic part of the development process, and an increasing emphasis is being placed on making testing a central part of software development (see Test Driven Development).

It should be noted that it is extremely difﬁcult—and in many cases impossible— to prove that software works and is completely error free. The fact that a set of tests ﬁnds no defects does not prove that the software is error-free. ‘Absence of evidence is not evidence of absence!’. This was discussed in the late 1960s and early 1970s by Dijkstra and can be summarised as:

Testing shows the presence, not the absence of bugs

Testing to establish that software components fulﬁl their contract involves checking operations against their requirements. Although this does happen at development time, it forms a major part of Quality Assurance (QA) and User Acceptance testing. It should be noted that with the advent of Test-Driven Development, the emphasis on testing against requirements during development has become signiﬁcantly higher.

There are of course many other aspects to testing, for example, Performance Testing which identiﬁes how a system will perform as various factors that affect that system change. For example, as the number of concurrent requests increase, as the number of processors used by the underlying hardware changes, as the size of the database grows etc.

However you view testing, the more testing applied to a system the higher the level of conﬁdence that the system will work as required.

14.3 What Should Be Tested?

An interesting question is ‘What aspects of your software system should be subject to testing?’.

In general, anything that is repeatable should be subject to formal (and ideally automated) testing. This includes (but is not limited to):

The build process for all technologies involved.
The deployment process to all platforms under consideration.
The installation process for all runtime environments.
The upgrade process for all supported versions (if appropriate).
The performance of the system/servers as loads increase.
The stability for systems that must run for any period of time (e.g. 24 x 7 systems).
The backup processes.
The security of the system.
The recovery ability of the system on failure.
The functionality of the system.
The integrity of the system.

Notice that only the last two of the above list might be what is commonly considered areas that would be subject to testing. However, to ensure the quality of the system under consideration, all of the above are relevant. In fact, testing should cover all aspects of the software development lifecycle and not just the QA phase. During requirements gathering testing is the process of looking for missing or ambiguous requirements. During this phase consideration should also be made with regard to how the overall requirements will be tested, in the ﬁnal software system.

Test planning should also look at all aspects of the software under test for func- tionality, usability, legal compliance, conformance to regulatory constraints, secu- rity, performance, availability, resilience, etc. Testing should be driven by the need to identify and reduce risk.

14.4 Testing Software Systems

As indicated above there are a number of different types of testing that are commonly used within industry. These types are:

Unit Testing, which is used to verify the behaviour of individual components.
Integration Testing that tests that when individual components are combined together to provide higher-level functional units, that the combination of the units operates appropriately.
Regression Testing. When new components are added to a system, or existing components are changed, it is necessary to verify that the new functionality does not break any existing functionality. Such testing is known as Regression Testing.
Performance Testing is used to ensure that the systems’ performance is as required and, within the design parameters, and is able to scale as utilisation increases.
Stability Testing represents a style of testing which attempts to simulate system operation over an extended period of time. For example, for a online shopping application that is expected to be up and running 24 x 7 a stability test might ensure that with an average load that the system can indeed run 24 hours a day for 7 days a week.

Security Testing ensures that access to the system is controlled appropriately given the requirements. For example, for an online shopping system there may be different security requirements depending upon whether you are browsing the store, purchasing some products or maintaining the product catalogue.
Usability Testing which may be performed by a specialist usability group and may involved ﬁlming users while they use the system.
System Testing validates that the system as a whole actually meets the user requirements and conforms to required application integrity.
User Acceptance Testing is a form of user oriented testing where users conﬁrm that the system does and behaves in the way they expect.
Installation, Deployment and Upgrade Testing. These three types of testing validate that a system can be installed and deployed appropriate including any upgrade processes that may be required.
Smoke Tests used to check that the core elements of a large system operate correctly. They can typically be run quickly and in a faction of the time taken to run the full system tests.

Key testing approaches are discussed in the remainder of this section.

14.4.1 Unit Testing

A unit can be as small as a single function or as large as a subsystem but typically is a class, object, self-contained library (API) or web page.

By looking at a small self-contained component an extensive set of tests can be developed to exercise the deﬁned requirements and functionality of the unit.

Unit testing typically follows a white box approach, (also called Glass Box or Structural testing), where the testing utilizes knowledge and understanding of the code and its structure, rather than just its interface (which is known as the black box approach).

In white box testing, test coverage is measured by the number of code paths that have been tested. The goal in unit testing is to provide 100% coverage: to exercise every instruction, all sides of each logical branch, all called objects, handling of all data structures, normal and abnormal termination of all loops etc. Of course this may not always be possible but it is a goal that should be aimed for. Many auto- mated test tools will include a code coverage measure so that you are aware of how much of your code has been exercised by any given set of tests.

Unit Testing is almost always automated—there are many tools to help with this, perhaps the best-known being the xUnit family of test frameworks such as JUnit for Java and PyUnit for Python. The framework allows developers to:

focus on testing the unit,
simulate data or results from calling another unit (representative good and bad results),

create data driven tests for maximum flexibility and repeatability,
rely on mock objects that represent elements outside the unit that it must interact with.

Having the tests automated means that they can be run frequently, at the very least after initial development and after each change that affects the unit.

Once conﬁdence is established in the correct functioning of one unit, developers can then use it to help test other units with which it interfaces, forming larger units that can also be unit tested or, as the scale gets larger, put through Integration Testing.

14.4.2 Integration Testing

Integration testing is where several units (or modules) are brought together to be tested as an entity in their own right. Typically, integration testing aims to ensure that modules interact correctly and the individual unit developers have interpreted the requirements in a consistent manner.

An integrated set of modules can be treated as a unit and unit tested in much the same way as the constituent modules, but usually working at a “higher” level of functionality. Integration testing is the intermediate stage between unit testing and full system testing.

Therefore, integration testing focuses on the interaction between two or more units to make sure that those units work together successfully and appropriately. Such testing is typically conducted from the bottom up but may also be conducted top down using mocks or stubs to represented called or calling functions. An important point to note is that you should not aim to test everything together at once (so called Big Bang testing) as it is more difﬁcult to isolate bugs in order that they can be rectiﬁed. This is why it is more common to ﬁnd that integration testing has been performed in a bottom up style.

14.4.3 System Testing

System Testing aims to validate that the combination of all the modules, units, data, installation, conﬁguration etc. operates appropriately and meets the requirements speciﬁed for the whole system. Testing the system has a whole typically involves testing the top most functionality or behaviours of the system. Such Behaviour Based testing often involves end users and other stake holders who are less tech- nical. To support such tests a range of technologies have evolved that allow a more English style for test descriptions. This style of testing can be used as part of the requirements gathering process and can lead to a Behaviour Driven Development (BDD) process. The Python module pytest-bdd provides a BDD style extension to the core pytest framework.

14.4.4 Installation/Upgrade Testing

Installation testing is the testing of full, partial or upgrade install processes. It also validates that the installation and transition software needed to move to the new release for the product is functioning properly. Typically, it

veriﬁes that the software may be completely uninstalled through its back-out process.
determines what ﬁles are added, changed or deleted on the hardware on which the program was installed.
determines whether any other programs on the hardware are affected by the new software that has been installed.
determines whether the software installs and operates properly on all hardware platforms and operating systems that it is supposed to work on.

14.4.5 Smoke Tests

A smoke test is a test or suite of tests designed to verify that the fundamentals of the system work. Smoke tests may be run against a new deployment or a patched deployment in order to verify that the installation performs well enough to justify further testing. Failure to pass a smoke test would halt any further testing until the smoke tests pass. The name derives from the early days of electronics: If a device began to smoke after it was powered on, testers knew that there was no point in testing it further. For software technologies, the advantages of performing smoke tests include:

Smoke tests are often automated and standardised from one build to another.
Because smoke tests validate things that are expected to work, when they fail it is usually an indication that something fundamental has gone wrong (the wrong version of a library has been used) or that a new build has introduced a bug into core aspects of the system.
If a system is built daily, it should be smoke tested daily.
It will be necessary to periodically add to the smoke tests as new functionality is added to the system.

14.5 Automating Testing

The actual way in which tests are written and executed needs careful consideration. In general, we wish to automate as much of the testing process as is possible as this makes it easy to run the tests and also ensures not only that all tests are run but that they are run in the same way each time. In addition, once an automated test is set up it will typically be quicker to re-run that automated test than to manually repeat a series of tests. However, not all of the features of a system can be easily tested via an automated test tool and in some cases the physical environment may make it hard to automate tests.

Typically, most unit testing is automated and most acceptance testing is manual. You will also need to decide which forms of testing must take place. Most software projects should have unit testing, integration testing, system testing and acceptance testing as a necessary requirement. Not all projects will implement performance or stability testing, but you should be careful about omitting any stage of testing and be sure it is not applicable.

14.6 Test Driven Development

Test Driven Development (or TDD) is a development technique whereby devel- opers write test cases before they write any implementation code. The tests thus drive or dictate the code that is developed. The implementation only provides as much functionality as is required to pass the test and thus the tests act as a speci- ﬁcation of what the code does (and some argue that the tests are thus part of that speciﬁcation and provide documentation of what the system is capable of).

TDD has the beneﬁt that as tests must be written ﬁrst, there are always a set of tests available to perform unit, integration, regression testing etc. This is good as developers can ﬁnd that writing tests and maintaining tests is boring and of less interest than the actual code itself and thus put less emphasis into the testing regime than might be desirable. TDD encourages, and indeed requires, that developers maintain an exhaustive set of repeatable tests and that those tests are developed to the same quality and standards as the main body of code.

There are three rules of TDD as deﬁned by Robert Martin, these are:

You are not allowed to write any production code unless it is to make a failing unit test pass
You are not allowed to write any more of a unit test than is sufﬁcient to fail; and compilation failures are failures
You are not allowed to write any more production code than is sufﬁcient to pass the one failing unit test.

This leads to the TDD cycle described in the next section.

14.6.1 The TDD Cycle

There is a cycle to development when working in a TDD manner. The shortest form of this cycle is the TDD mantra:

Red / Green / Refactor

Which relates to the unit testing suite of tools where it is possible to write a unit test. Within tools such as PyCharm, when you run a pyunit or pytest test a Test View is shown with Red indicating that a test failed or Green indicating that the test passed. Hence Red/Green, in other words write the test and let it fail, then implement the code to ensure it passes. The last part of this mantra is Refactor which indicates once you have it working make the code cleaner, better, ﬁtter by Refactoring it. Refactoring is the process by which the behaviour of the system is not changed but the implementation is altered to improve it.

The full TDD cycle is shown by the following diagram which highlights the test

ﬁrst approach of TDD:

The TDD mantra can be seen in the TDD cycle that is shown above and described in more detail below:

Write a single test.
Run the test and see it fail.
Implement just enough code to get the test to pass.
Run the test and see it pass.
Refactor for clarity and deal with any issue of reuse etc.
Repeat for next test.

14.6.2 Test Complexity

The aim is to strive for simplicity in all that you do within TDD. Thus, you write a test that fails, then do just enough to make that test pass (but no more). Then you refactor the implementation code (that is change the internals of the unit under test) to improve the code base. You continue to do this until all the functionality for a unit has been completed. In terms of each test, you should again strive for simplicity with each test only testing one thing with only a single assertion per test (although this is the subject of a lot of debate within the TDD world).

14.6.3 Refactoring

The emphasis on refactoring within TDD makes it more than just testing or Test First Development. This focus on refactoring is really a focus on (re)design and incremental improvement. The tests provide the speciﬁcation of what is needed as well as the veriﬁcation that existing behaviour is maintained, but refactoring leads to better design software. Thus, without refactoring TDD is not TDD!

14.7 Design for Testability

Testability has a number of facets

Conﬁgurability. Set up the object under test to an appropriate conﬁguration for the test
Controllability. Control the input (and internal state)
Observability. Observe its output
Veriﬁability. That we can verify that output in an appropriate manner.

14.7.1 Testability Rules of Thumb

If you cannot test code then change it so that you can!

If your code is difﬁcult to validate then change it so that it isn’t!

Only one concrete class should be tested per Unit test and then Mock the Rest!

If you code is hard to reconﬁgure to work with Mocks then make it so that you code can use Mocks!

Design your code for testability!

Chapter 15

PyTest Testing Framework

15.1 Introduction

There are several testing frameworks available for Python, although only one, unittest comes as part of the typical Python installation. Typical libraries include Unit test, (which is available within the Python distribution by default) and PyTest.

In this chapter we will look at PyTest and how it can be used to write unit tests in

Python for both functions and classes.

15.2 What Is PyTest?

PyTest is a testing library for Python; it is currently one of the most popular Python testing libraries (others include unittest and doctest). PyTest can be used for various levels of testing, although its most common application is as a unit testing framework. It is also often used as a testing framework within a TDD based development project. In fact, it is used by Mozilla and Dropbox as their Python testing framework.

PyTest offers a large number of features and great flexibility in how tests are written and in how set up behaviour is deﬁned. It automatically ﬁnds test based on naming conventions and can be easily integrated into a range of editors and IDEs including PyCharm.

15.3 Setting Up PyTest

You will probably need to set up PyTest so that you can use it from within your environment. If you are using the PyCharm editor, then you will need to add the PyTest module to the current PyCharm project and tell PyCharm that you want to use PyTest to run all tests for you.

15.4 A Simple PyTest Example

Something to Test

To be able to explore PyTest we ﬁrst need something to test; we will therefore deﬁne a simple Calculator class. The calculator keeps a running total of the operations performed; it allows a new value to be set and then this value can be added to, or subtracted from, that accumulated total.

class Calculator:
def  init (self): self.current = 0
self.total = 0

def set(self, value): self.current = value

def add(self):
self.total += self.current

def sub(self):
self.total -= self.current

def total(self):
return self.total

Save this class into a ﬁle called calculator.py.

Writing a Test

We will now create a very simple PyTest unit test for our Calculator class. This test will be deﬁned in a class called test_calculator.py.

You will need to import the calculator class we wrote above into your

test_calculator.py ﬁle (remember each ﬁle is a module in Python).

The exact import statement will depend on where you placed the calculator ﬁle relative to the test class. In this case the two ﬁles are both in the same directory and so we can write:

from calculator import Calculator

We will now deﬁne a test, the test should be pre-ﬁxed with test_ for PyTest to

ﬁnd them. In fact PyTest uses several conventions to ﬁnd tests, which are:

Search for test_*.py or *_test.py ﬁles.
From those ﬁles, collect test items:
test_preﬁxed test functions,
test_preﬁxed test methods inside Test preﬁxed test classes (without an init method).

Note that we keep test ﬁles and the ﬁles containing the code to be tested separate; indeed in many cases they are kept in different directory structures. This means that there is not chance of developers accidentally using tests in production code etc.

Now we will add to the ﬁle a function that deﬁnes a test. We will call the function test_add_one; it needs to start with test_ due to the above con- vention. However, we have tried to make the rest of the function name descriptive, so that its clear what it is testing. The function deﬁnition is given below:

from calculator import Calculator

def test_add_one(): calc = Calculator() calc.set(1) calc.add()
assert calc.total == 1

The test function creates a new instance of the Calculator class and then calls several methods on it; to set up the value to add, then the call to the add() method itself etc.

The ﬁnal part of the test is the assertion. The assert veriﬁes that the behaviour of the calculator is as expected. The PyTest assert statement works out what is being tested and what it should do with the result—including adding information to be added to a test run report. It avoids the need to have to learn a load of assertSomething type methods (unlike some other testing frameworks).

Note that a test without an assertion is not a test; i.e. it does not test anything. Many IDEs provide direct support for testing frameworks including PyCharm. For example, PyCharm will now detect that you have written a function with an assert statement in it and add a Run Test icon to the grey area to the left of the editor. This can be seen in the following picture where a green arrow has been added at line 4; this is the ‘Run Test’ button:

The developer can click on the green arrow to run the test. They will then be presented with the Run menu that is preconﬁgured to use PyTest for you:

If the developer now selects the Run option; this will use the PyTest runner to execute the test and collect information about what happened and present it in a PyTest output view at the bottom of the IDE:

Here you can see a tree in the left-hand panel that currently holds the one test deﬁned in the test_calculator.py ﬁle. This tree shows whether tests have passed or failed. In this case we have a green tick showing that the test passed.

To the right of this tree is the main output panel which shows the results of running the tests. In this case it shows that PyTest ran only one test and that this was the test_add_one test which was deﬁned in test_calculator.py and that 1 test passed.

If you now change the assertion in the test to check to see that the result is 0 the test will fail. When run, the IDE display will update accordingly.

The tree in the left-hand pane now shows the test as failed while the right-hand pane provides detailed information about the test that failed including where in the test the failed assertion was deﬁned. This is very helpful when trying to debug test failures.

15.5 Working with PyTest

Testing Functions

We can test standalone functions as well as classes using PyTest. For example, given the function increment below (which merely adds one to any number passed into it):

def increment(x):

return x + 1

We can write a PyTest test for this as follows:

def test_increment_integer_3():

assert increment(3) == 4

The only real difference is that we have not had to make an instance of a class:

Organising Tests

Tests can be grouped together into one or more ﬁles; PyTest will search for all ﬁles following the naming convention (ﬁle names that either start or end with ‘test’) in speciﬁed locations:

If no arguments are speciﬁed when PyTest is run then the search for suitably named test ﬁles starts from the testpaths environment variable (if conﬁg- ured) or the current directory. Alternatively, command line arguments can be used in any combination of directories or ﬁlenames etc.

PyTest will recursively search down into sub directories, unless they match norecursedirs environment variable.
In those directories, it will search for ﬁles that match the naming conven- tions test_*.py or *_test.py ﬁles.

Tests can also be arranged within test ﬁles into Test classes. Using test classes can be helpful in grouping tests together and managing the setup and tear down behaviours of separate groups of tests. However, the same effect can be achieved by separating the tests relating to different functions or classes into different ﬁles.

Test Fixtures

It is not uncommon to need to run some behaviour before or after each test or indeed before or after a group of tests. Such behaviours are deﬁned within what is commonly known as test ﬁxtures.

We can add speciﬁc code to run:

at the beginning and end of a test class module of test code (setup_module/ teardown_module)
at the beginning and end of a test class (setup_class/teardown_class) or using the alternate style of the class level ﬁxtures (setup/teardown)
before and after a test function call (setup_function/teardown_function)
before and after a test method call (setup_method/teardown_method)

To illustrate why we might use a ﬁxture, let us expand our Calculator test:

def test_initial_value():
 calc = Calculator() 
assert calc.total == 0

def test_add_one(): 
calc = Calculator() 
calc.set(1) 
calc.add()
assert calc.total == 1

def test_subtract_one(): 
calc = Calculator() 
calc.set(1) 
calc.sub()
assert calc.total == -1

def test_add_one_and_one(): 
calc = Calculator() 
calc.set(1)
calc.add() 
calc.set(1) 
calc.add()
assert calc.total == 2

We now have four tests to run (we could go further but this is enough for now). One of the issues with this set of tests is that we have repeated the creation of the Calculator object at the start of each test. While this is not a problem in itself it does result in duplicated code and the possibility of future issues in terms of maintenance if we want to change the way a calculator is created. It may also not be

as efﬁcient as reusing the Calculator object for each test.

We can however, deﬁne a ﬁxture that can be run before each individual test function is executed. To do this we will write a new function and use the pytest.ﬁxture decorator on that function. This marks the function as being special and that it can be used as a ﬁxture on an individual function.

Functions that require the ﬁxture should accept a reference to the ﬁxture as an argument to the individual test function. For example, for a test to accept a ﬁxture called calculator; it should have an argument with the ﬁxture name, i.e. calculator. This name can then be used to access the object returned. This is illustrated below:

import pytest
from calculator import Calculator

@pytest.fixture def calculator():
"""Returns a Calculator instance"""
return Calculator()

def test_initial_value(calculator):
assert calculator.total == 0

def test_add_one(calculator): 
calculator.set(1) 
calculator.add()
assert calculator.total == 1

def test_subtract_one(calculator): 
calculator.set(1) 
calculator.sub()
assert calculator.total == -1

def test_add_one_and_one(calculator):
calculator.set(1) 
calculator.add() 
calculator.set(1) 
calculator.add()
assert calculator.total == 2

In the above code, each of the test functions accepts the calculator ﬁxture that is used to instantiate the Calculator object. We have therefore de–dupli- cated our code; there is now only one piece of code that deﬁnes how a calculator object should be created for our tests. Note each test is supplied with a completely new instance of the Calculator object; there is therefore no chance of one test impacting on another test.

It is also considered good practice to add a docstring to your ﬁxtures as we have done above. This is because PyTest can produce a list of all ﬁxtures available along with their docstrings. From the command line this is done using:

> pytest fixtures

The PyTest ﬁxtures can be applied to functions (as above), classes, modules, packages or sessions. The scope of a ﬁxture can be indicated via the (optional) scope parameter to the ﬁxture decorator. The default is “function” which is why we did not need to specify anything above. The scope determines at what point a ﬁxture should be run. For example, a ﬁxture with ‘session’ scope will be run once for the test session, a ﬁxture with module scope will be run once for the module (that is the ﬁxture and anything it generates will be shared across all tests in the current module), a ﬁxture with class scope indicates a ﬁxture that is run for each new instance of a test class created etc.

Another parameter to the ﬁxture decorator is autouse which if set to True will activate the ﬁxture for all tests that can see it. If it is set to False (which is the default) then an explicit reference in a test function (or method etc.) is required to activate the ﬁxture.

If we add some additional ﬁxtures to our tests we can see when they are run:

import pytest
from calculator import Calculator

@pytest.fixture(scope='session', 
autouse=True) def session_scope_fixture():
print('session_scope_fixture')

@pytest.fixture(scope='module', autouse=True)
def module_scope_fixture(): 
print('module_scope_fixture')
@pytest.fixture(scope='class', autouse=True)
def class_scope_fixture(): 
print('class_scope_fixture')
@pytest.fixture def calculator():
"""Returns a Calculator instance""" 
print('calculator fixture')
return Calculator()
 

def test_initial_value(calculator):
assert calculator.total == 0
def test_add_one(calculator): 
calculator.set(1) 
calculator.add()
assert calculator.total == 1

def test_subtract_one(calculator): 
calculator.set(1) 
calculator.sub()
assert calculator.total == -1
def test_add_one_and_one(calculator): 
calculator.set(1) 
calculator.add() 
calculator.set(1) 
calculator.add()
assert calculator.total == 2

If we run this version of the tests, then the output shows when the various

ﬁxtures are run:

session_scope_fixture module_scope_fixture class_scope_fixture calculator fixture

.class_scope_fixture calculator fixture

Note that higher scoped ﬁxtures are instantiated ﬁrst.

15.6 Parameterised Tests

One common requirement of a test to run the same tests multiple times with several different input values. This can greatly reduce the number of tests that must be deﬁned. Such tests are referred to as parametrised tests; with the parameter values for the test speciﬁed using the @pytest.mark.parametrize decorator.

@pytest.mark.parametrize decorator.
@pytest.mark.parametrize('input1,input2,expected', [ (3, 1, 4),
(3, 2, 5),
])
def test_calculator_add_operation(calculator, input1, input2, expected):
calculator.set(input1) 
calculator.add() 
calculator.set(input2) 
calculator.add()
assert calculator.total == expected

This illustrates setting up a parametrised test for the Calculator in which two input values are added together and compared with the expected result. Note that the parameters are named in the decorator and then a list of tuples is used to deﬁne the values to be used for the parameters. In this case the test_ calculator_add_operation will be run two passing in 3, 1 and 4 and then passing in 3, 2 and 5 for the parameters input1, input2 and expected respectively.

Testing for Exceptions

You can write tests that verify that an exception was raised. This is useful as testing negative behaviour is as important as testing positive behaviour. For example, we might want to verify that a particular exception is raised when we attempt to withdraw money from a bank account which will take us over our overdraft limit. To verify the presence of an exception in PyTest use the with statement and pytest.raises. This is a context manager that will verify on exit that the

speciﬁed exception was raised. It is used as follows:

with pytest.raises(accounts.BalanceError): current_account.withdraw(200.0)

Ignoring Tests

In some cases, it is useful to write a test for functionality that has not yet been implemented; this may be to ensure that the test is not forgotten or because it helps to document what the item under test should do. However, if the test is run then the test suite as a whole will fail because the test is running against behaviour that has yet to be written.

One way to address this problem is to decorate a test with the @pytest.- mark.skip decorator:

@pytest.mark.skip(reason=‘not implemented yet’) def test_calculator_multiply(calculator):

calculator.multiply(2, 3)

assert calculator.total == 6

This indicates that PyTest should record the presence of the test but should not try to execute it. PyTest will then note that the test was skipped, for example in PyCharm this is shown using a circle with a line through it.

It is generally considered best practice to provide a reason why the test has been skipped so that it is easier to track. This information is also available when PyTest skips the test:

Chapter 16

Mocking for Testing

16.1 Introduction

Testing software systems is not an easy thing to do; the functions, objects, methods etc. That are involved in any program can be complex things in their own right. In many cases they depend on and interact with other functions, methods and objects; very few functions and methods operate in isolation. Thus the success of failure of a function or method or the overall state of an object is dependent on other program elements.

However, in general it is a lot easier to test a single unit in isolation rather than to test it as part of a larger more complex system. For example, let us take a Python class as a single unit to be tested. If we can test this class on its own we only have to take into account the state of the classes object and the behaviour deﬁned for the class when writing our test and determining appropriate outcomes.

However, if that class interacts with external systems such as external services, databases, third party software, data sources etc. Then the testing process becomes more complex:

It may now be necessary to verify data updates made to the database, or information sent to a remote service etc. to conﬁrm that the operation of a class’s object is correct. This makes not only the software being tested more complex but it also makes the tests themselves more complex. This means that there is greater chance that the test will fail, that the tests will contain bugs or issues themselves and that the test will be harder for someone to understand and maintain. Thus a common objective when writing unit tests or subsystem tests is to be able to test elements/ units in isolation.

The question is how to do this when a function or method relies on other elements?

The key to decoupling functions, methods and objects from other program or system elements is to use mocks. These mocks can be used to decouple one object rom another, one function from another and one system from another; thereby simplifying the testing environment. These mocks are only intended to be used for testing purposes, for example the above scenario could be simpliﬁed by mocking out each of the external systems as shown below:

Mocking is not a Python speciﬁc concept and there are many mocking libraries available for may different languages. However, in this chapter we will be focussing on the unites.mock library which has been part of the standard Python distribution since Python 3.3.

16.2 Why Mock?

A useful ﬁrst question to consider with regard to mocking, in software testing, is ‘Why mock?’. That is, why bother with the concept of a mock in the ﬁrst place; why not test with the real thing?

There are several answers to this, some of which are discussed below:

Testing in isolation is easier. As mentioned in the introduction, testing a unit (whether that is a class, a function, a module etc.) is easier in isolation then when dependent on external classes, functions, modules etc.

The real thing is not available. In many cases it is necessary to mock out part of a system or an interface to another system because the real thing is just not available. This could be for several reasons including that it has not been developed yet. In the natural course of software development some parts of a system are likely to be developed and ready for testing before other parts. If one part relies on another part for some element of its operation then the system that is not yet available can be mocked out. In other situations the development team or test team may not have access to the real thing. This may because it is only available within a production context. For example, if a software development house is developing one subsystem it may not have access to another subsystem as it is proprietary and only accessible once the software has been deployed within the client organisation.

Real elements can be time consuming. We want our tests to run as quickly as possible and certainly within a Continuous Integration (CI) environment we want them to run fast enough that we can repeatedly test a system throughout the day. In some situations the real thing may take a signiﬁcant amount of time to process the test scenario. As we want to test our own code we may not be worried about whether a system outside of our control operates correctly or not (at least at this level of testing; it may still be a concern for integration and system testing). We can therefore improve the response times of our tests if we mock out the real system and replace it with a mock that provides much faster response times (possibly because it uses canned responses).

The real thing takes time to set up. In a Continuous Integration (CI) environment, new builds of a system are regularly and repeatedly tested (for example whenever a change is made to their codebase). In such situations it may be necessary to conﬁgure and deploy the ﬁnal system to a suitable environment to perform appropriate tests. If an external system is time consuming to conﬁgure, deploy and initialise it may be more effective to mock that system out.

Difﬁcult to emulate certain situations. It can be difﬁcult within a test scenario to emulate speciﬁc situations. These situations are often related to error or excep- tional circumstances that should never happen within a correctly functioning environment. However, it may well be necessary to validate that if such a situation does occur, then the software can deal with that scenario. If these scanners are related to how external (the unit under test) system fail or operate incorrectly then it may be necessary to mock out these systems to be able to generate the scenarios. We want repeatable tests. By their very nature when you run a test you either want it to pass or fail each time it is run with the same inputs. You certainly do not want tests that pass sometimes and fail other times. This mean that there is no conﬁdence in the tests and people often start ignoring failed tests. This situation can

happen if the data provided by systems that a test depends on do not supply repeatable data. This can happen for several different reason but a common cause is because they return real data. Such real data may be subject to change, for example consider a system that uses a data feed for the current exchange rate between funds and dollars. If the associated test conﬁrms that a trade when priced in dollars is correctly converted to funds using the current exchange rate then that test is likely to generate a different result every time it is run. In this situation it would lie better to mock out the current exchange rate service so that a ﬁxed/known exchange rate is used.

The Real System is not reliable enough. In some cases the real system may not be reliable enough itself to allow for repeatable tests.

The Real System may not allow tests to be repeated. Finally, the real system may not allow tests to be easily repeated. For example, a test which involves lodging a trade for a certain number of IBM shares with a Trade Order management system may not allow that trade, with those shares, for that customer to be run several times (as it would then appear to be multiple trades). However, for the purposes of testing we may want to test submitting such a trade in multiple different scenarios, multiple times. It may therefore be necessary to mock out the real Order Management System so that such tests can be written.

16.3 What Is Mocking?

The previous section gave several reasons to use mocks; the next thing to consider then is what is a mock?

Mocks, including mock functions, methods and mock objects are things that:

Possess the same interface as the real thing, whether they are mock functions, methods or whole objects. They thus take the same range and types of parameters and return similar information using similar types.
Deﬁne behaviour that in some way represents/mimics real exemplar behaviour but typically in very controlled ways. This behaviour may be hard coed, may really on a set of rules or simpliﬁed behaviour; may be very simplistic or quiet sophisticated in its own right.

They thus emulate the real system and from outside of the mock may actually appear to be the real system.

In many cases the term mock is used to cover a range of different ways in which the real thing can be emulated; each type of mock has its own characteristics. It is therefore useful to distinguish the different types of mocks as this can help deter- mine the style of mock to be adopted in a particular test situation.

The are different types of Mock including:

Test Stubs. A test stub is typically a hand coded function, method or object used for testing purposes. The behaviour implemented by a test stub may rep- resent a limited sub set of the functionality of the real thing.
Fakes. Fakes typically provide addition functionality compared with a Test Stub. Fakes may be considered to be a test speciﬁc version of the real thing, such as an in memory database used for testing rather than the real database. Such Fakes typically still have some limitations on their functionality, for example when the tests are terminated all data is purged from the in memory database rather than stored permanently on disk.
Autogenerated Test Mocks. These are typically generated automatically using a supporting framework. As part of the set up of the test the expectations associated with the test mock. These expectations may specify the results to return for speciﬁc inputs as well as whether the test mock was called etc.
Test Mock Spy. If we are testing a particular unit and it returns the correct result we might decided that we do not need to consider the internal behaviour of the unit. However, it is common to want to conﬁrm that the test mock was invoked in the ay we expected. This helps verify the internal behaviour of the unit under test. This can be done using a test mock spy. Such a test mock records how many times it was called and what the parameters used where (as well as other information). The test can then interrogate the test mock to validate that it was invoked as expected/as many times as expected/with the correct parameters etc.

16.4 Common Mocking Framework Concepts

As has been mentioned there are several mocking frameworks around for not only Python but other languages such as Java, C# and Scala etc. All of these frameworks have a common core behaviour. This behaviour allows a mock function, method or object to be created based on the interface presented by the real thing. Of course unlike languages such as C# and Java Python does not have a formal interface concept; however this does not stop the mocking framework from still using the same idea.

In general once a mock has been created it is possible to deﬁne how that mock should appear to behave; in general this involves specifying the return result to use for a function or method. It is also possible to verify that the mock has been invoked as expected with the parameters expected.

The actual mock can be added to a test or a set of tests either programmatically or via some form of decorator. In either case for the duration of the test the mock will be used instead of the real thing.

Assertions can then be used to verify the results returned by the unit under test while mock speciﬁc methods are typically used to verify (spy on) the methods deﬁned on the mock.

16.5 Mocking Frameworks for Python

Due to Python’s dynamic nature it is well suited to the construction of mock functions, methods and objects. In fact there are several widely used mocking frameworks available for Python including:

unittest.mock The unittest.mock (included in the Python distribution from Python 3.3 onwards). This is the default mocking library provided with Python for creating mock objects in Python tests.
pymox This is a widely used making framework. It is an open source frame- work and has a more complete set of facilities for enforcing the interface of a mocked class.
Mocktest This is another popular mocking framework. It has its own DSL (Domain Speciﬁc Language) to support mocking and a wide set of expectation matching behaviour for mock objects.

In the remainder of this chapter we will focus on the unittest.mock library as it is provided as part of the standard Python distribution.

16.6 The unittest.mock Library

The standard Python mocking library is the unittest.mock library. It has been included in the standard Python distribution since Python 3.3 and provides a simple way to deﬁne mocks for unit tests.

The key to the unittest.mock library is the Mock class and its subclass MagicMock. Mock and MagicMock objects can be used to mock functions, methods and even whole classes. These mock objects can have canned responses deﬁned so that when they are involved by the unit under test they will respond appropriately. Existing objects can also have attributes or individual methods mocked allowing an object to be tested with a known state and speciﬁed behaviour. To make it easy to work with mock objects, the library provides the @unittest.mock.patch() decorator. This decorator can be used to replace real functions and objects with mock instances. The function behind the decorator can also be used as a context manager allowing it to be used in with-as state-

ments providing for ﬁne grained control over the scope of the mock if required.

16.6.1 Mock and Magic Mock Classes

The unittest.mock library provides the Mock class and the MagicMock class. The Mock class is the base class for mock objects. The MagicMock class is a subclass of the Mock class. It is called the MagicMock class as it provides default implementations for several magic method such as . len (), .

str (), and . iter ().

As a simple example consider the following class to be tested:

class SomeClass():

def _hidden_method(self): return 0

def public_method(self, x):

return self.hidden_method() + x

This class deﬁnes two methods; one is intended as part of the public interface of the class (the public_method()) and one it intended only for internal or private use (the _hidden_method()). Notice that the hidden method uses the convention of preceding its name by an underbar (‘_’).

Let us assume that we wish to test the behaviour of the public_method()

and want to mock out the _hidden_method().

We can do this by writing a test that will create a mock object and use this in place of the real _hidden_method(). We could probably use either the Mock class or the MagicMock class for this; however due to the additional functionality provided by the MagicMock class it is common practice to use that class. We will therefore do the same.

The test to be created will be deﬁned within a method within a test class. The names of the test method and the test class are by convention descriptive and thus will describe what is being tested, for example:

from unittest.mock import *

from unittest import TestCase
from unittest import main

class test_SomeClass_public_interface(TestCase):

def test_public_method(self): test_object = SomeClass()
# Set up canned response on mock method
test_object._hidden_method = MagicMock(name =
'hidden_method')
test_object._hidden_method.return_value = 10
# Test the object
result = test_object.public_method(5) self.assertEqual(15, result, 'return value from
public_method incorrect')

In this case note that the class being tested is instantiated ﬁrst. The MagicMock is then instantiated and assigned to the name of the method to be mocked. This in effect replaces that method for the test_object. The MagicMock. The MagicMock object is given a name as this helps with treating any issues in the report generated by the unites framework. Following this the canned response from the mock version of the _hidden_method() is deﬁned; it will always return the value 10.

At this point we have set up the mock to be used for the test and are now ready to run the test. This is done in the next line where the public_method() is called on the test_object with the parameter 5. The result is then stored.

The test then validates the result to ensure that it is correct; i.e. that the returned value is 15.

Although this is a very simple example it illustrates how a method can be mocked out using the MagicMock class.

16.6.2 The Patchers

The unittest.mock.patch(), unittest.mock.patch.object() and unittest.patch.dict() decorators can be used to simplify the creation of mock objects.

The patch decorator takes a target for the patch and returns a MagicMock object in its place. It can be used as a TastCase method or class decorator. As a class decorator it decorates each test method in the class automatically. It can also be used as a context manager via the with and with-as statements.
The patch.object decorator can be provided with either two or three arguments. When given three arguments it will replace the object to be patched, with a mock for the given attribute/method name. When given two arguments the object to be patched is given a default MagicMock object for the speciﬁed attribute/function.
The patch.dict decorator patches a dictionary or dictionary like object.

For example, we can rewrite the example presented in the previous section using the @patch.object decorator to provides the mock object for the _hid- den_method() (it returns a MagicMock linked to SomeClass):

class test_SomeClass_public_interface(TestCase):

@patch.object(SomeClass, ‘_hidden_method’)

def test_public_method(self, mock_method):

# Set up canned response mock_method.return_value = 10 # Create object to be tested test_object = SomeClass()

result = test_object.public_method(5) self.assertEqual(15, result, ‘return value from

public_method incorrect’)

In the above code the _hidden_method() is replaced with a mock version for SomeClass within the test_public_method() method. Note that the mock version of the method is passed in as a parameter to the test method so that the canned response can be speciﬁed.

You can also use the @patch() decorator to mock a function from a module. For example, given some external module with a function api_call, we can mock that function out using the @patch() decorator:

@patch(‘external_module.api_call’)

def test_some_func(self, mock_api_call):

This uses patch() as a decorator and passed the target object’s path. The target path was ‘external_module.api_call’ which consists of the module name and the function to mock.

16.6.3 Mocking Returned Objects

In the examples looked at so far the results returned from the mock functions or methods have been simple integers. However, in some cases the returned values must themselves be mocked as the real system would return a complex object with multiple attributes and methods.

The following example uses a MagicMock object to represent an object returned from a mocked function. This object has two attributes, one is a response code and the other is a JSON string. JSON stands for the JavaScript Object Notation and is a commonly used format in web services.

import external_module

from unittest.mock import *

from unittest import TestCase 
from unittest import main 
import json

def some_func():
# Calls out to external API - which we want to mock
response = external_module.api_call()
return responseclass test_some_func_calling_api(TestCase):

class test_some_func_calling_api(TestCase):

@patch('external_module.api_call') 
def test_some_func(self, mock_api_call):
# Sets up mock version of api_call
mock_api_call.return_value = MagicMock(status_code=200,
response=json.dumps({'key':'value'}))
# Calls some_func() that calls the (mock) api_call() function
result = some_func()
# Check that the result returned from some_func() is what was expected
self.assertEqual(result.status_code, 200, "returned status code is not 200")
self.assertEqual(result.response, '{"key": "value"}', "response JSON incorrect")

In this example the function being tested is some_func() but some_func() calls out to the mocked function external_module.api_call(). This mocked function returns a MagicMock object with a pre-speciﬁed status_code and response. The assertions then validate that the object returned by some_func() contains the correct status code and response.

16.6.4 Validating Mocks Have Been Called

Using unittest.mock it is possible to validate that a mocked function or method was called appropriately using assert_called(), assert_- called_with() or assert_called_once_with() depending on whether the function takes parameters or not.

The following version of the test_some_func_with_params() test method veriﬁes that the mock api_call() function was called with the correct parameter.

@patch('external_module.api_call_with_param') 
def test_some_func_with_param(self, mock_api_call):
# Sets up mock version of api_call
mock_api_call.return_value = MagicMock(status_code=200, 
response=json.dumps({'age': '23'}))
result = some_func_with_param('Phoebe')

# Check result returned from some_func() is what was expected
self.assertEqual(result.response, '{age": "23"}', 'JSON result incorrect')
# Verify that the mock_api_call was called with the correct params

mock_api_call.api_call_with_param.assert_called_with('Phoebe')
If we wished to validate that it had only been called once we could use the
assert_called_once_with() method.

16.7 Mock and MagicMock Usage

16.7.1 Naming Your Mocks

It can be useful to give your mocks a name. The name is used when the mock appears in test failure messages. The name is also propagated to attributes or methods of the mock:

mock = MagicMock(name=‘foo’)

16.7.2 Mock Classes

As well as mocking an individual method on a class it is possible to mock a whole class. This is done by providing the patch() decorator with the name of the class to patch (with no named attribute/method). In this case the while class is replaced by a MagicMock object. You must then specify how that class should behave.

In this example the people.Person class has been mocked out. This class has a method calculate_pay() which is being mocked here. The Payroll class has a method generate_payslip() that expects to be given a Person object. It then uses the information provided by the person objects calculate_pay() method to generate the string returned by the generate_payslip() method.

16.7.3 Attributes on Mock Classes

Attributes on a mock object can be easily deﬁned, for example if we want to set an attribute on a mock object then we can just assign a value to the attribute:

import people

from unittest.mock import *

from unittest import TestCase

class MyTest(TestCase):

@patch('people.Person')

def test_one(self, MockPerson): 

self.assertIs(people.Person, MockPerson) 

instance = MockPerson.return_value instance.age = 24

instance.name = 'Adam'

self.assertEqual(24, instance.age, 'age incorrect') 

self.assertEqual('Adam', instance.name, 'name incorrect')

In this case the attribute age and name have been added to the mock instance of the people. Person class.

If the attribute itself needs to be a mock object then all that is required is to assign a MagicMock (or Mock) object to that attribute:

instance. address = MagicMock(name=‘Address’)

16.7.4 Mocking Constants

It is very easy to mock out a constant; this can be done using the @patch() decorator and proving the name of the constant and the new value to use. This value can be a literal value such as 42 or ‘Hello’ or it can be a mock object itself (such as a MagicMock object). For example:

@patch(‘mymodule.MAX_COUNT’, 10)

def test_something(self):

# Test can now use mymodule.MAX_COUNT

16.7.5 Mocking Properties

It is also possible to mock Python properties. This is done again using the @patch decorator but using the unittest.mock.PropertyMock class and the new_callable parameter. For example:

@patch('mymoule.Car.wheels', new_callable=mock.PropertyMock) 

def test_some_property(self, mock_wheels):

mock_wheels.return_value = 6 # Rest of test method

16.7.6 Raising Exceptions with Mocks

A very useful attribute that can be speciﬁed when a mock object is created is the side_effect. If you set this to an exception class or instance then the exception will be raised when the mock is called, for example:

mock = Mock(side_effect=Exception(‘Boom!’)) mock()

This will result in the Exception being raised when the mock() is invoked.

16.7.7 Applying Patch to Every Test Method

If you want to mock out something for every test in a test class then you can decorate the whole class rather than each individual method. The effect of deco- rating the class is that the patch will be automatically applied to all test methods in the class (i.e. To all methods starting with the word ‘test’). For example:

import people

from unittest.mock import *

from unittest import TestCase

from unittest import main

@patch('people.Person') class MyTest(TestCase):

def test_one(self, MockPerson): 
self.assertIs(people.Person, MockPerson)

def test_two(self, MockSomeClass): 
self.assertIs(people.Person, MockSomeClass)

def do_something(self):

return 'something'

In the above test class, the tests test_one and test_two are supplied with the mock version of the Person class. However the do_something() method is not affected.

16.7.8 Using Patch as a Context Manager

The patch function can be used as a context manager. This gives ﬁne grained control over the scope of the mock object.

In the following example the the test_one() method contains a with-as statement that we used to patch (mock) the person class as MockPerson. This mock class is only available within the with-as statement.

import people

from unittest.mock import * 
from unittest import TestCase 
from unittest import main

class MyTest(TestCase):

def test_one(self):

with patch('people.Person') as MockPerson: 
self.assertIs(people.Person, MockPerson) 
instance = MockPerson.return_value 
instance.calculate_pay.return_value = 250.0 
payroll = people.Payroll()
result = payroll.generate_payslip(instance) 
self.assertEqual('You earned 250.0', result,

'payslip incorrect')

16.8 Mock Where You Use It

The most common error made by people using the unittest.mock library is mocking in the wrong place. The rule is that you must mock out where you are going to use it; or to put it another way you must always mock the real thing where it is imported into, not where it’s imported from.

16.9 Patch Order Issues

It is possible to have multiple patch decorators on a test method. However, the order in which you deﬁne the patch decorators is signiﬁcant. The key to under- standing what the order should be is to work backwards so that when the mocks are passed into the test method they are presented to the right parameters. For example:

@patch('mymodule.sys') 
@patch('mymodule.os') 
@patch('mymodule.os.path') 

def test_something(self,

mock_os_path, 
mock_os, 
mock_sys):

# The rest of the test method

Notice that the last patch’s mock is passed into the second parameter passed to the test_something() method (self is the ﬁrst parameter to all methods). In turn the ﬁrst patch’s mock is passed into the last parameter. Thus the mocks are passed into the test method in the reverse order to that which they are deﬁned in.

16.10 How Many Mocks?

An interesting question to consider is how many mocks should you use per test?

This is the subject or a lot of debate within the software testing community. The general rules of thumb around this topic are given below, however it should be borne in mind that these are guidelines rather than hard and fast rules.

Avoid more than 2 or 3 mocks per test. You should avoid more than 2–3 mocks as the mocks themselves the get harder to manage. Many also consider that if you need more then 2–3 mocks per test then there are probably some underlying design issues that need to be considered. For example, if you are testing a Python class then that class may have too many dependencies. Alternatively the class may have too many responsibilities and should be broken down into several independent classes; each with a distinct responsibility. Another cause might be that the class’s behaviour may not be encapsulated enough and that you are allowing other elements to interact with the class in more informal ways (i.e. The interface between the class and other elements is not clean/exploit enough). The result is that it may be necessary to refactor your class before progressing with your development and testing.
Only Mock you Nearest Neighbour. You should only ever mock your nearest neighbour whether that is a function, method or object. You should try to avoid mocking dependencies of dependencies. If you ﬁnd yourself doing this then it will become harder to conﬁgure, maintain, understand and develop. It is also increasingly likely that you are testing the mocks rather than your own function, method or class.

16.11 Mocking Considerations

The following provide some rules of thumb to consider when using mocks with your tests:

Don’t over mock—if you do then you can end up just testing the mocks themselves.
Decide what to mock, typical examples of what to mock include those elements that are not yet available, those elements that are not by default repeatable (such as live data feeds) or those elements of the system that are time consuming or complex.
Decide where to mock such as the interfaces for the unit under test. You want to test the unit so any interface it has with another system, function, class might be a candidate for a mock.
Decide when to mock so that you can determine the boundaries for the test.
Decide how you will implement your mocks. For example you need to con- sider which mocking framework(s) you will use or how to mock larger com- ponents such as a database.

Chapter 17

Introduction to Files, Paths and IO

17.1 Introduction

The operating system is a critical part of any computer systems. It is comprised of elements that manage the processes that run on the CPU, how memory is utilised and managed, how peripheral devices are used (such as printers and scanners), it allows the computer system to communicate with other systems and it also provide support for the ﬁle system used.

The File System allows programs to permanently store data. This data can then be retrieved by applications at a later date; potentially after the whole computer has been shut down and restarted.

The File Management System is responsible for managing the creation, access and modiﬁcation of the long term storage of data in ﬁles.

This data may be stored locally or remotely on disks, tapes, DVD drives, USB drives etc.

Although this was not always the case; most modern operating systems organise ﬁles into a hierarchical structure, usually in the form of an inverted tree. For example in the following diagram the root of the directory structure is shown as ‘/’. This root directory holds six subdirectories. In turn the Users subdirectory holds 3 further directories and so on:

Each ﬁle is contained within a directory (also known as a folder on some operating systems such as Windows). A directory can hold zero or more ﬁles and zero or more directories. For any give directory there are relationships with other directories as shown below for the directory jhunt:

The root directory is the starting point for the hierarchical directory tree structure. A child directory of a given directory is known as a subdirectory. The directory that holds the given directory is known as the parent directory. At any one time, the directory within which the program or user is currently working, is known as the current working directory.

A user or a program can move around this directory structure as required. To do this the user can typically either issue a series of commands at a terminal or command window. Such as cd to change directory or pwd to print the working directory. Alternatively Graphical User Interfaces (GUIs) to operating systems usually include some form of ﬁle manager application that allows a user to view the ﬁle structure in terms of a tree. The Finder program for the Mac is shown below with a tree structure displayed for a pycharmprojects directory. A similar view is also presented for the Windows Explorer program.

17.2 File Attributes

A ﬁle will have a set of attributes associated with it such as the date that it was created, the date it was last updated/modiﬁed, how large the ﬁle is etc. It will also typically have an attribute indicating who the owner of the ﬁle is. This may be the creator of the ﬁle; however the ownership of a ﬁle can be changed either from the command line or through the GUI interface. For example, on Linux and Mac OS X the command chown can be used to change the ﬁle ownership.

It can also have other attributes which indicate who can read, write or execute

the ﬁle. In Unix style systems (such as Linux and Mac OS X) these access rights can be speciﬁed for the ﬁle owner, for the group that the ﬁle is associated with and for all other users.

The ﬁle owner can have rights speciﬁed for reading, writing and executing a ﬁle. These are usually represented by the symbols ‘r’, ‘w’ and ‘x’ respectively. For example the following uses the symbolic notation associated with Unix ﬁles and indicates that the ﬁle owner is allowed to read, write and execute a ﬁle:

Here the ﬁrst dash is left blank as it is to do with special ﬁles (or directories), then the next set of three characters represent the permissions for the owner, the fol- lowing set of three the permissions for all other users. As this example has rwx in

the ﬁrst group of three characters this indicates that the user can read ‘r’, write ‘w’ and execute ‘x’ the ﬁle. However the next six characters are all dashes indicating that the group and all other users cannot access the ﬁle at all.

The group that a ﬁle belongs to is a group that can have any number of users as members. A member of the group will have the access rights as indicated by the group settings on the ﬁle. As for the owner of a ﬁle these can be to read, write or execute the ﬁle. For example, if group members are allowed to read and execute a ﬁle, then this would be shown using the symbolic notation as:

Now this example indicates that only members of the group can read and execute the ﬁle; note that group members cannot write the ﬁle (they therefore cannot modify the ﬁle).

If a user is not the owner of a ﬁle, nor a member of the group that the ﬁle is part of, then their access rights are in the ‘everyone else’ category. Again this category can have read, write or execute permissions. For example, using the symbolic notation, if all users can read the ﬁle but are not able to do anything else, then this would be shown as:

Of course a ﬁle can mix the above permissions together, so that an owner may be allowed to read, write and execute a ﬁle, the group may be able to read and execute the ﬁle but all other users can only read the ﬁle. This would be shown as:

In addition to the symbolic notation there is also a numeric notation that is used with Unix style systems. The numeric notation uses three digits to represent the permissions. Each of the three rightmost digits represents a different component of the permissions: owner, group, and others.

Each of these digits is the sum of its component bits in the binary numeral system. As a result, speciﬁc bits add to the sum as it is represented by a numeral:

The read bit adds 4 to its total (in binary 100),
The write bit adds 2 to its total (in binary 010), and
The execute bit adds 1 to its total (in binary 001).
This the following symbolic notations can be represented by an equivalent numeric notation:

Symbolic notation	Numeric notation	Meaning
rwx—–	0700	Read, write, and execute only for owner
-rwxrwx—	0770	Read, write, and execute for owner and group
-rwxrwxrwx	0777	Read, write, and execute for owner, group and others

Directories have similar attributes and access rights to ﬁles. For example, the following symbolic notation indicates that a directory (indicated by the ‘d’) has read and execute permissions for the directory owner and for the group. Other users cannot access this directory:

The permissions associated with a ﬁle or directory can be changed either using a command from a terminal or command window (such as chmod which is used to modify the permissions associated with a ﬁle or directory) or interactively using the ﬁle explorer style tool.

17.3 Paths

A path is a particular combination of directories that can lead to a speciﬁc sub directory or ﬁle.

This concept is important as Unix/Linux/Max OS X and Windows ﬁle systems represent an inverted tree of directories and ﬁles., It is thus important to be able to uniquely reference locations with the tree.

For example, in the following diagram the path /Users/jhunt/work- spaces/pycharmprojects/furtherpython/chapter2 is highlighted:

A path may be absolute or relative. An absolute path is one which provides a complete sequence of directories from the root of the ﬁle system to a speciﬁc sub directory or ﬁle.

A relative path provides a sequence from the current working directory to a particular subdirectory or ﬁle.

The absolute path will work wherever a program or user is currently located within the directory tree. However, a relative path may only be relevant in a speciﬁc location.

For example, in the following diagram, the relative path pycharmprojects/ furtherpython/chapter2 is only meaningful relative to the directory workspaces:

Note that an absolute path starts from the root directory (represented by ‘/’) where as a relative path starts from a particular subdirectory (such as pychamprojects).

17.4 File Input/Output

File Input/Output (often just referred to as File I/O) involves reading and writing data to and from ﬁles. The data being written can be in different formats.

For example a common format used in Unix/Linux and Windows systems is the ASCII text format. The ASCII format (or American Standard Code for Information Interchange) is a set of codes that represent various characters that is widely used by operating systems. The following table illustrates some of the ASCII character codes and what they represent:

Decimal code	Character	Meaning
42	*	Asterisk
43	+	Plus
48	0	Zero
49	1	One
50	2	Two
51	3	Three
65	A	Uppercase A
66	B	Uppercase B
67	C	Uppercase C
68	D	Uppercase D

Decimal code	Character	Meaning
97	a	Lowercase a
98	b	Lowercase b
99	c	Lowercase c
100	d	Lowercase d

ASCII is a very useful format to use for text ﬁles as they can be read by a wide range of editors and browsers. These editors and browsers make it very easy to create human readable ﬁles. However, programming languages such as Python often use a different set of character encodings such as a Unicode character encoding (such as UTF-8). Unicode is another standard for representing characters using various codes. Unicode encoding systems offer a wider range of possible character encodings than ASCII, for example the latest version of Unicode in May 2019, Unicode 12.1, contains a repertoire of 137,994 characters covering 150 modern and historic scripts, as well as multiple symbol sets and emojis.

However, this means that it can be necessary to translate ASCII into Unicode (e.g. UTF-8) and vice versa when reading and writing ASCII ﬁles in Python.

Another option is to use a binary format for data in a ﬁle. The advantage of using binary data is that there is little or no translation required from the internal repre- sentation of the data used in the Python program into the format stored in the ﬁle. It is also often more concise than an equivalent ASCII format and it is quicker for a program to read and write and takes up less disk space etc. However, the down side of a binary format is that it is not in an easily human readable format. It may also be difﬁcult for other programs, particularly those written in other programming lan- guages such as Java or C#, to read the data in the ﬁles.

17.5 Sequential Access Versus Random Access

Data can be read from (or indeed written to) a ﬁle either sequentially or via a random access approach.

Sequential access to data in a ﬁle means that the program reads (or writes) data to a ﬁle sequentially, starting at the beginning of a ﬁle and processing the data an item at a time until the end of the ﬁle is reached. The read process only ever moves forward and only to the next item of data to read.

Random Access to a data ﬁle means that the program can read (or write) data anywhere into the ﬁle at any time. That is the program can position itself at a particular point in the ﬁle (or rather a pointer can be positioned within the ﬁle) and it can then start to read (or write) at that point. If it is reading then it will read the next data item relative to the pointer rather than the start of the ﬁle. If it is writing data then it will write data from that point rather than at the end of the ﬁle. If there is already data at that point in the ﬁle then it will be over written. This type of access is also known as Direct Access as the computer program needs to know where the data is stored within the ﬁle and thus goes directly to that location for the data. In some cases the location of the data is recorded in an index and thus is also known as indexed access.

Sequential ﬁle access has advantages when a program needs to access information in the same order each time the data is read. It is also is faster to read or write all the data sequentially than via direct access as there is no need to move the ﬁle pointer around.

Random access ﬁles however are more flexible as data does not need to be written or read in the order in which it is obtained. It is also possible to jump to just the location of the data required and read that data (rather than needing to sequentially read through all the data to ﬁnd the data items of interest).

17.6 Files and I/O in Python

In the remainder of this section of the book we will explore the basic facilities provided for reading and writing ﬁles in Python. We will also look at the underlying streams model for ﬁle I/O. After this we will explore the widely used CSV and Excel ﬁle formats and libraries available to support those. This section concludes by exploring the Regular Expression facilities in Python. While this last topic is not strictly part of ﬁle I/O it is often used to parse data read from ﬁles to screen out unwanted information.

Chapter 18

Reading and Writing Files

18.1 Introduction

Reading data from and writing data to a ﬁle is very common within many programs. Python provides a large amount of support for working with ﬁles of various types. This chapter introduces you to the core ﬁle IO functionality in Python.

18.2 Obtaining References to Files

Reading from, and writing to, text ﬁles in Python is relatively straightforward. The built in open() function creates a ﬁle object for you that you can use to read and/ or write data from and/ or to a ﬁle.

The function requires as a minimum the name of the ﬁle you want to work with. Optionally you can specify the access mode (e.g. read, write, append etc.). If you do not specify a mode then the ﬁle is open in read-only mode. You can also specify whether you want the interactions with the ﬁle to be buffered which can improve

performance by grouping data reads together.

The syntax for the open() function is

file_object = open(file_name, access_mode, buffering)

Where

ﬁle_name indicates the ﬁle to be accessed.
access_mode The access_mode determines the mode in which the ﬁle is to be opened, i.e. read, write, append, etc. A complete list of possible values is given below in the table. This is an optional parameter and the default ﬁle access mode is read (r).

buffering If the buffering value is set to 0, no buffering takes place. If the buffering value is 1, line buffering is performed while accessing a ﬁle.

The access_mode values are given in the following table.

Mode	Description
r	Opens a ﬁle for reading only. The ﬁle pointer is placed at the beginning of the ﬁle. This is the default mode
rb	Opens a ﬁle for reading only in binary format. The ﬁle pointer is placed at the beginning of the ﬁle. This is the default mode
r+	Opens a ﬁle for both reading and writing. The ﬁle pointer placed at the beginning of the ﬁle
rb+	Opens a ﬁle for both reading and writing in binary format. The ﬁle pointer placed at the beginning of the ﬁle
w	Opens a ﬁle for writing only. Overwrites the ﬁle if the ﬁle exists. If the ﬁle does not exist, creates a new ﬁle for writing
wb	Opens a ﬁle for writing only in binary format. Overwrites the ﬁle if the ﬁle exists. If the ﬁle does not exist, creates a new ﬁle for writing
w+	Opens a ﬁle for both writing and reading. Overwrites the existing ﬁle if the ﬁle exists. If the ﬁle does not exist, creates a new ﬁle for reading and writing
wb+	Opens a ﬁle for both writing and reading in binary format. Overwrites the existing ﬁle if the ﬁle exists. If the ﬁle does not exist, creates a new ﬁle for reading and writing
a	Opens a ﬁle for appending. The ﬁle pointer is at the end of the ﬁle if the ﬁle exists. That is, the ﬁle is in the append mode. If the ﬁle does not exist, it creates a new ﬁle for writing
ab	Opens a ﬁle for appending in binary format. The ﬁle pointer is at the end of the ﬁle if the ﬁle exists. That is, the ﬁle is in the append mode. If the ﬁle does not exist, it creates a new ﬁle for writing
a+	Opens a ﬁle for both appending and reading. The ﬁle pointer is at the end of the ﬁle if the ﬁle exists. The ﬁle opens in the append mode. If the ﬁle does not exist, it creates a new ﬁle for reading and writing
ab+	Opens a ﬁle for both appending and reading in binary format. The ﬁle pointer is at the end of the ﬁle if the ﬁle exists. The ﬁle opens in the append mode. If the ﬁle does not exist, it creates a new ﬁle for reading and writing

The ﬁle object itself has several useful attributes such as

ﬁle.closed returns True if the ﬁle has been closed (can no longer be accessed because the close() method has been called on it).
ﬁle.mode returns the access mode with which the ﬁle was opened.
ﬁle.name The name of the ﬁle.

The ﬁle.close() method is used to close the ﬁle once you have ﬁnished with it. This will flush any unwritten information to the ﬁle (this may occur because of buffering) and will close the reference from the ﬁle object to the actual underlying operating system ﬁle. This is important to do as leaving a reference to a ﬁle open can cause problems in larger applications as typically there are only a certain number of ﬁle references possible at one time and over a long period of time these

may all be used up resulting in future errors being thrown as ﬁles can no longer be opened.

The following short code snippet illustrates the above ideas:

file = open(‘myfile.txt’, ‘r+’) print(‘file.name:‘, file.name) print(‘file.closed:‘, file.closed) print(‘file.mode:‘, file.mode) file.close()

print(‘file.closed now:‘, file.closed)

The output from this is:

file.name: myfile.txt file.closed: False file.mode: r+ file.closed now: True

18.3 Reading Files

Of course, having set up a ﬁle object we want to be able to either access the contents of the ﬁle or write data to that ﬁle (or do both). Reading data from a text ﬁle is supported by the read(), readline() and readlines() methods:

The read() method This method will return the entire contents of the ﬁle as a single string.
The readline() method reads the next line of text from a ﬁle. It returns all the text on one line up to and including the newline character. It can be used to read a ﬁle a line at a time.
The readlines() method returns a list of all the lines in a ﬁle, where each item of the list represents a single line.

Note that once you have read some text from a ﬁle using one of the above operations then that line is not read again. Thus using readlines() would result in a further readlines() returning an empty list whatever the contents of the ﬁle.

The following illustrates using the readlines() method to read all the text in a text ﬁle into a program and then print each line out in turn:

file = open(‘myfile.txt‘, ‘r‘)

lines = file.readlines() for line in lines:

print(line, end=”) file.close()

Notice that within the for loop we have indicated to the print function that we want the end character to be ” rather than a newline; this is because the line string already possesses the newline character read from the ﬁle.

18.4 File Contents Iteration

As suggested by the previous example; it is very common to want to process the contents of a ﬁle one line at a time. In fact Python makes this extremely easy by making the ﬁle object support iteration. File iteration accesses each line in the ﬁle and makes that line available to the for loop. We can therefore write:

file = open(‘myfile.txt‘, ‘r‘)

for line in file:

print(line, end=”) file.close()

It is also possible to use the list comprehension to provide a very concise way to load and process lines in a ﬁle into a list. It is similar to the effect of readlines() but we are now able to pre-process the data before creating the list:

file = open(‘myfile.txt‘, ‘r‘)

lines = [line.upper() for line in file] file.close()

print(lines)

18.5 Writing Data to Files

Writing a string to a ﬁle is supported by the write() method. Of course, the ﬁle object we create must have an access mode that allows writing (such as ‘w’). Note that the write method does not add a newline character (represented as ‘\n’) to the end of the string—you must do this manually.

An example short program to write a text ﬁle is given below:

print(‘Writing file‘)

f = open(‘my-new-file.txt‘, ‘w‘) f.write(‘Hello from Python!!\n‘) f.write(‘Working with files is easy…\n‘) f.write(‘It is cool …\n‘)

f.close()

This creates a new ﬁle called my-new-ﬁle.txt. It then writes three strings to the ﬁle each with a newline character on the end; it then closes the ﬁle.

The effect of this is to create a new ﬁle called myﬁle.txt with three lines in it:

18.6 Using Files and with Statements

Like several other types where it is important to shut down resources; the ﬁle object class implements the Context Manager Protocol and thus can be used with the with statement. It is therefore common to write code that will open a ﬁle using the with as structure thus ensuring that the ﬁle will be closed when the block of code is ﬁnished with, for example:

with open(‘my-new-file.txt’, ‘r’) as f:

lines = file.readlines()

for line in lines: print(line, end=”)

18.7 The Fileinput Module

In some situations, you may need to read the input from several ﬁles in one go. You could do this by opening each ﬁle independently and then reading the contents and appending that contents to a list etc. However, this is a common enough require- ment that the ﬁleinput module provides a function ﬁleinput.input() that can take a list of ﬁles and treat all the ﬁles as a single input signiﬁcantly simplifying this process, for example:

with fileinput.input(files=(‘spam.txt‘, ‘eggs.txt‘)) as f:

for line in f: process(line)

Features provided by the ﬁleinput module include

Return the name of the ﬁle currently being read.
Return the integer “ﬁle descriptor” for the current ﬁle.
Return the cumulative line number of the line that has just been read.Return the line number in the current ﬁle. Before the ﬁrst line has been read this returns 0.
A boolean function that indicates if the current line just read is the ﬁrst line of its ﬁle

Some of these are illustrated below:

with fileinput.input(files=(‘textfile1.txt‘, ‘textfile2.txt‘)) as f:

line = f.readline() print(‘f.filename():‘, f.filename())

print(‘f.isfirstline():‘, f.isfirstline()) print(‘f.lineno():‘, f.lineno())

print(‘f.filelineno():‘, f.filelineno())

for line in f: print(line, end=”)

18.8 Renaming Files

A ﬁle can be renamed using the os.rename() function. This function takes two arguments, the current ﬁlename and the new ﬁlename. It is part of the Python os module which provides methods that can be used to perform a range of ﬁle-processing operations (such as renaming a ﬁle). To use the module, you will ﬁrst need to import it. An example of using the rename function is given below:

import os

os.rename(‘myfileoriginalname.txt‘,’ myfilenewname.txt‘)

18.9 Deleting Files

A ﬁle can be deleted using the os.remove() method. This method deletes the ﬁle speciﬁed by the ﬁlename passed to it. Again, it is part of the os module and therefore this must be imported ﬁrst:

import os

os.remove(‘somefilename.txt’)

18.10 Random Access Files

All the examples presented so far suggest that ﬁles are accessed sequentially, with the ﬁrst line read before the second and so on. Although this is (probably) the most common approach it is not the only approach supported by Python; it is also possible to use a random-access approach to the contents within a ﬁle.

To understand the idea of random ﬁle access it is useful to understand that we can maintain a pointer into a ﬁle to indicate where we are in that ﬁle in terms of reading or writing data. Before anything is read from a ﬁle the pointer is before the beginning of the ﬁle and reading the ﬁrst line of text would for example, advance the point to the start of the second line in the ﬁle etc. This idea is illustrated below:

When randomly accessing the contents of a ﬁle the programmer manually moves the pointer to the location required and reads or writes text relative to that pointer. This means that they can move around in the ﬁle reading and writing data.

The random-access aspect of a ﬁle is provided by the seek method of the ﬁle object:

ﬁle.seek (offset, whence) this method determines where the next read or write operation (depending on the mode used in the open() call) takes place.

In the above the offset parameter indicates the position of the read/ write pointer within the ﬁle. The move can also be forwards or backwards (represented by a negative offset).

The optional whence parameter indicates where the offset is relative to. The values used for whence are:

0 indicates that the offset is relative to start of ﬁle (the default).
1 means that the offset is relative to the current pointer position.
2 indicates the offset is relative to end of ﬁle.

Thus, we can move the pointer to a position relative to the start of the ﬁle, to the end of the ﬁle, or to the current position.

For example, in the following sample code we create a new text ﬁle and write a set of characters into that ﬁle. At this point the pointer is positioned after the ‘z’ in the ﬁle. However, we then use seek() to move the point to the 10th character in the ﬁle and now write ‘Hello’, next we reposition the pointer to the 6th character in the ﬁle and write out ‘BOO’. We then close the ﬁle. Finally, we read all the lines from the ﬁle using a with as statement and the open() function and from this we will see that the text is the ﬁle is now abcdefBOOjHELLOpqrstuvwxyz:

f = open(‘text.txt‘, ‘w’) f.write(‘abcdefghijklmnopqrstuvwxyz\n’) f.seek(10,0)

f.write(‘HELLO‘)

f.seek(6, 0)

f.write (‘BOO‘) f.close()

with open(‘text.txt‘, ‘r‘) as f:

for line in f: print(line, end=”)

18.11 Directories

Both Unix like systems and Windows operating systems are hierarchical structures comprising directories and ﬁles. The os module has several functions that can help with creating, removing and altering directories. These include:

mkdir() This function is used to create a directory, it takes the name of the directory to create as a parameter. If the directory already exists FileExistsError is raised.
- chdir() This function can be used to change the current working directory. This is the directory that the application will read from/ write to by default.
- getcwd() This function returns a string representing the name of the current working directory.
- rmdir() This function is used to remove/ delete a directory. It takes the name of the directory to delete as a parameter.
- listdir() This function returns a list containing the names of the entries in the directory speciﬁed as a parameter to the function (if no name is given the current directory is used).

A simple example illustrates the use of some of these functions is given below:

import os

print(‘os.getcwd(:‘, os.getcwd()) print(‘List contents of directory‘) print(os.listdir())

print(‘Create mydir‘) os.mkdir(‘mydir‘)

print(‘List the updated contents of directory‘) print(os.listdir())

print(‘Change into mydir directory‘) os.chdir(‘mydir‘) print(‘os.getcwd(:‘, os.getcwd())

print(‘Change back to parent directory‘) os.chdir(‘..‘)

print(‘os.getcwd(:‘, os.getcwd()) print(‘Remove mydir directory‘) os.rmdir(‘mydir‘)

print(‘List the resulting contents of directory‘) print(os.listdir())

Note that ‘..’ is a short hand for the parent directory of the current directory and

‘.’ is short hand for the current directory.

An example of the type of output generated by this program for a speciﬁc set up on a Mac is given below:

os.getcwd(:

/Users/Shared/workspaces/pycharm/pythonintro/textfiles List contents of directory

[‘my-new-file.txt’, ‘myfile.txt’, ‘textfile1.txt’, ‘textfile2.txt’]

Create mydir

List the updated contents of directory

[‘my-new-file.txt’, ‘myfile.txt’, ‘textfile1.txt’, ‘textfile2.txt’, ‘mydir’]

Change into mydir directory os.getcwd(:

/Users/Shared/workspaces/pycharm/pythonintro/textfiles/mydir

Change back to parent directory os.getcwd(:

/Users/Shared/workspaces/pycharm/pythonintro/textfiles

Remove mydir directory

List the resulting contents of directory

[‘my-new-file.txt’, ‘myfile.txt’, ‘textfile1.txt’, ‘textfile2.txt’]

18.12 Temporary Files

During the execution of many applications it may be necessary to create a tem- porary ﬁle that will be created at one point and deleted before the application ﬁnishes. It is of course possible to manage such temporary ﬁles yourself however, the tempﬁle module provides a range of facilities to simplify the creation and management of these temporary ﬁles.

Within the tempﬁle module TemporaryFile, NamedTemporaryFile, TemporaryDirectory, and SpooledTemporaryFile are high-level ﬁle objects which provide automatic cleanup of temporary ﬁles and directories. These objects implement the Context Manager Protocol.

The tempﬁle module also provides the lower-level function mkstemp() and mkdtemp() that can be used to create temporary ﬁles that require the developer to management them and delete them at an appropriate time.

The high-level feature for the tempﬁle module are:

TemporaryFile(mode=‘w+b’) Return an anonymous ﬁle-like object that can be used as a temporary storage area. On completion of the managed context (via a with statement) or destruction of the ﬁle object, the temporary ﬁle will be removed from the ﬁlesystem. Note that by default all data is written to the temporary ﬁle in binary format which is generally more efﬁcient.
- NamedTemporaryFile(mode=‘w+b’) This function operates exactly as TemporaryFile() does, except that the ﬁle has s visible name in the ﬁle system.
- SpooledTemporaryFile(max_size=0, mode=‘w+b’) This function operates exactly as TemporaryFile() does, except that data is spooled in memory until the ﬁle size exceeds max_size, or until the ﬁle’s ﬁleno () method is called, at which point the contents are written to disk and oper- ation proceeds as with TemporaryFile().
- TemporaryDirectory(sufﬁx=None, preﬁx=None, dir=None) This function creates a temporary directory. On completion of the context or destruction of the temporary directory object the newly created temporary directory and all its contents are removed from the ﬁlesystem.

The lower level functions include:

mkstemp() Creates a temporary ﬁle that is only readable or writable by the user who created it.
mkdtemp() Creates a temporary directory. The directory is readable, writable, and searchable only by the creating user ID.
gettempdir() Return the name of the directory used for temporary ﬁles. This deﬁnes the default value for the default temporary directory to be used with the other functions in this module.

An example of using the TemporaryFile function is given below. This code imports the tempﬁle module then prints out the default directory used for

temporary ﬁles. It then creates a TemporaryFile object and prints its name and mode (the default mode is binary but for this example we have overwritten this so that plain text is used). We have then written a line to the ﬁle. Using seek we are repositioning ourselves at the start of the ﬁle and then reading the line we have just written.

import tempfile

print(‘tempfile.gettempdir():‘, tempfile.gettempdir()) temp = tempfile.TemporaryFile(‘w+‘) print(‘temp.name:‘, temp.name)

print(‘temp.mode:‘, temp.mode) temp.write(‘Hello world!‘) temp.seek(0)

line = temp.readline() print(‘line:‘, line)

The output from this when run on an Apple Mac is:

tempfile.gettempdir():

/var/folders/6n/8nrnt9f93pn66ypg9s5dq8y80000gn/T temp.name: 4

temp.mode: w+ line: Hello world!

Note that the ﬁle name is ‘4’ and that the temporary directory is not a meaningful name!

18.13 Working with Paths

The pathlib module provides a set of classes representing ﬁlesystem paths; that is paths through the hierarchy of directories and ﬁles within an operating systems ﬁle structure. It was introduced in Python 3.4. The core class in this module is the Path class.

A Path object is useful because it provides operations that allow you to manipulate and manage the path to a ﬁle or directory. The Path class also repli- cates some of the operations available from the os module (such as mkdir, rename and rmdir) which means that it is not necessary to work directly with the os module.

A path object is created using the Path constructor function; this function actually returns a speciﬁc type of Path depending on the type of operating system being used such as a WindowsPath or a PosixPath (for Unix style systems).

The Path() constructor takes the path to create for example ‘D:/mydir’ (on Windows) or ‘/Users/user1/mydir’ on a Mac or ‘/var/temp’ on Linux etc.

You can then use several different methods on the Path object to obtain infor- mation about the path such as:

exists() returns True of False depending on whether the path points to an existing ﬁle or directory.
- is_dir() returns True if the path points to a directory. False if it refer- ences a ﬁle. False is also returned if the path does not exist.
- is_ﬁle() returns True of the path points to a ﬁle, it returns False if the path does not exist or the path references a directory.
- absolute() A Path object is considered absolute if it has both a root and (if appropriate) a drive.
- is_absolute() returns a Boolean value indicating whether the Path is absolute or not.

An example of using some of these methods is given below:

from pathlib import Path

print(‘Create Path object for current directory‘) p = Path(‘.‘)

print(‘p:‘, p) print(‘p.exists():‘, p.exists())

print(‘p.is_dir():‘, p.is_dir())

print(‘p.is_file():‘, p.is_file())

print(‘p.absolute():‘, p.absolute())

Sample output produced by this code snippet is:

Create Path object for current directory p: .

p.exists(): True p.is_dir(): True p.is_file(): False p.absolute():

/Users/Shared/workspaces/pycharm/pythonintro/textfiles

There are also several methods on the Path class that can be used to create and remove directories and ﬁles such as:

mkdir() is used to create a directory path if it does not exist. If the path already exists, then a FileExistsError is raised.
- rmdir() remove this directory; the directory must be empty otherwise an error will be raised.

rename(target) rename this ﬁle or directory to the given target.
- unlink() removes the ﬁle referenced by the path object.
- joinpath(*other) appends elements to the path object e.g. path.joinpath(‘/ temp’).
- with_name(new_name) return a new path object with the name changed.
- The ‘/’ operator can also be used to create new path objects from existing paths for example path/ ‘test’/ ‘output’ which would append the directories test and out to the path object.

Two Path class methods can be used to obtain path objects representing key directories such as the current working directory (the directory the program is logically in at that point) and the home directory of the user running the program:

Path.cwd() return a new path object representing the current directory.
Path.home() return a new path object representing the user’s home directory.

An example using several of the above features is given below. This example obtains a path object representing the current working directory and then appends ‘text’ to this. The result path object is then checked to see if the path exists (on the computer running the program), assuming that the path does not exist it is created and the exists() method is rerun.

p = Path.cwd()

print(‘Set up new directory‘) newdir = p / ‘test’

print(‘Check to see if newdir exists‘) print(‘newdir.exists():‘, newdir.exists()) print(‘Create new dir‘)

newdir.mkdir()

print(‘newdir.exists():‘, newdir.exists())

The effect of creating the directory can be seen in the output:

Set up new directory

Check to see if newdir exists newdir.exists(): False Create new dir newdir.exists(): True

A very useful method in the Path object is the glob(pattern) method. This method returns all elements within the path that meet the pattern speciﬁed.

For example path.glob(‘*.py’) will return all the ﬁles ending .py within the current path.

Note that ‘**/*.py’ would indicate the current directory and any sub directory. For example, the following code will return all ﬁles where the ﬁle name ends with ‘.txt’ for a given path:

print(‘-‘ * 10)

for file in path.glob(‘*.txt’): print(‘file:‘, file)

print(‘–‘ * 10)

An example of the output generated by this code is:

file: my-new-file.txt file: myfile.txt file: textfile1.txt file: textfile2.txt

Paths that reference a ﬁle can also be used to read and write data to that ﬁle. For example the open() method can be used to open a ﬁle that by default allows a ﬁle to be read:

open(mode=‘r’) this can be used to open the ﬁle referenced by the path object.

This is used below to read the contents of a ﬁle a line at a time (note that with as statement is used here to ensure that the ﬁle represented by the Path is closed):

p = Path(‘mytext.txt‘)

with p.open() as f: print(f.readline())

However, there are also some high-level methods available that allow you to easily write data to a ﬁle or read data from a ﬁle. These include the Path methods write_text and read_text methods:

write_text(data) opens the ﬁle pointed to in text mode and writes the data to it and then closes the ﬁle.
read_text() opens the ﬁle in read mode, reads the text and closes the ﬁle; it then returns the contents of the ﬁle as a string.

These are used below

dir = Path(‘./test’) print(‘Create new file‘) newfile = dir / ‘text.txt’ print(‘Write some text to file‘)

newfile.write_text(‘Hello Python World!’) print(‘Read the text back again’) print(newfile.read_text())

print(‘Remove the file‘) newfile.unlink()

Which generates the following output:

Create new file

Write some text to file Read the text back again Hello Python World!

Remove the file

Chapter 19

Stream IO

19.1 Introduction

In this chapter we will explore the Stream I/O model that under pins the way in which data is read from and written to data sources and sinks. One example of a data source or sink is a ﬁle but another might be a byte array.

This model is actually what sits underneath the ﬁle access mechanisms discussed in the previous chapter.

It is not actually necessary to understand this model to be able to read and write data to and from a ﬁle, however in some situations it is useful to have an under- standing of this model so that you can modify the default behaviour when necessary.

The remainder of this chapter ﬁrst introduces the Stream model, discusses Python streams in general and then presents the classes provided by Python. It then considers what is the actual effect of using the open() function presented in the last chapter.

19.2 What is a Stream?

Streams are objects which serve as sources or sinks of data. At ﬁrst this concept can seem a bit strange. The easiest way to think of a stream is as a conduit of data flowing from or into a pool. Some streams read data straight from the “source of the data” and some streams read data from other streams. These latter streams then do some “useful” processing of the data such as converting the raw data into a speciﬁc format. The following ﬁgure illustrates this idea.

In the above ﬁgure the initial FileIO stream reads raw data from the actual data source (in this case a ﬁle). The BufferedReader then buffers the data reading process for efﬁciency. Finally the TextIOWrapper handles string encoding; that is it converts strings from the typical ASCII representation used in a ﬁle into the internal representation used by Python (which uses Unicode).

You might ask at this point why have a streams model at all; after all we read and wrote data to ﬁles without needing to know about streams in the last chapter? The answer is that a stream can read or write data to or from a source of data rather than just from a ﬁle. Of course a ﬁle can be a source of data but so can a socket, a pipe, a string, a web service etc. It is therefore a more flexible data I/O model.

19.3 Python Streams

The Python io module provides Python’s main facilities for dealing with data input and output. There are three main types of input/output these are text I/O, binary I/O and raw I/. O. These categories can be used with various types of data source/sinks. Whatever the category, each concrete stream can have a number of properties such as being read-only, write-only or read-write. It can also support sequential access or random access depending on the nature of the underlying data sink. For example, reading data from a socket or pipe is inherently sequential whereas reading data from a ﬁle can be performed sequentially or via a random-access approach.

Whichever stream is used however, they are aware of the type of data they can process. For example, attempting to supply a string to a binary write-only stream will raise a TypeError. As indeed will presenting binary data to a text stream etc.

As suggested by this there are a number of different types of stream provided by

the Python io module and some of these are presented below:

The abstract IOBase class is at the root of the stream IO class hierarchy. Below this class are stream classes for unbuffered and buffered IO and for text oriented IO.

19.4 IOBase

This is the abstract base class for all I/O stream classes. The class provides many abstract methods that subclasses will need to implement.

The IOBase class (and its subclasses) all support the iterator protocol. This means that an IOBase object (or an object of a subclass) can iterate over the input data from the underling stream.

IOBase also implements the Context Manager Protocol and therefore it can be used with the with and with-as statements.

The IOBase class deﬁnes a core set of methods and attributes including:

close() flush and close the stream.
- closed an attribute indicating whether the stream is closed.
- ﬂush() flush the write buffer of the stream if applicable.
- readable() returns True if the stream can be read from.
- readline(size=-1) return a line from the stream. If size is speciﬁed at most size bytes will be read.
- readline(hint=-1) read a list of lines. If hint is speciﬁed then it is used to control the number of lines read.
- seek(offset[, whence]) This method moves the current the stream position/pointer to the given offset. The meaning of the offset depends on the whence parameter. The default value for whence is SEEK_SET.
- SEEK_SET or 0: seek from the start of the stream (the default); offset must either be a number returned by TextIOBase.tell(), or zero. Any other offset value produces undeﬁned behaviour.
- SEEK_CUR or 1: “seek” to the current position; offset must be zero, which is a no-operation (all other values are unsupported).
- SEEK_END or 2: seek to the end of the stream; offset must be zero (all other values are unsupported).

seekable() does the stream support seek().
- tell() return the current stream position/pointer.
- writeable() returns true if data can be written to the stream.
- writelines(lines) write a list of lines to the stream.

19.5 Raw IO/UnBuffered IO Classes

Raw IO or unbuffered IO is provided by the RawIOBase and FileIO classes.

RawIOBase This class is a subclass of IOBase and is the base class for raw binary (aka unbuffered) I/O. Raw binary I/O typically provides low-level access to an underlying OS device or API, and does not try to encapsulate it in high-level primitives (this is the responsibility of the Buffered I/O and Text I/O classes that can wrap a raw I/O stream). The class adds methods such as:

read(size=-1) This method reads up to size bytes from the stream and returns them. If size is unspeciﬁed or -1 then all available bytes are read.
- readall() This method reads and returns all available bytes within the stream.
- readint(b) This method reads the bytes in the stream into a pre-allocated, writable bytes-like object b (e.g. into a byte array). It returns the number of bytes read.
- write(b) This method writes the data provided by b (a bytes -like object such as a byte array) into the underlying raw stream.

FileIO The FileIO class represents a raw unbuffered binary IO stream linked to an operating system level ﬁle. When the FileIO class is instantiated it can be given a ﬁle name and the mode (such as ‘r’ or ‘w’ etc.). It can also be given a flag to indicate whether the ﬁle descriptor associated with the underlying OS level ﬁle should be closed or not.

This class is used for the low-level reading of binary data and is at the heart of all ﬁle oriented data access (although it is often wrapped by another stream such as a buffered reader or writer).

19.6 Binary IO/Buffered IO Classes

Binary IO aka Buffered IO is a ﬁlter stream that wraps a lower level RawIOBase stream (such as a FileIO stream). The classes implementing buffered IO all extend the BufferedIOBase class and are:

BufferedReader When reading data from this object, a larger amount of data may be requested from the underlying raw stream, and kept in an internal buffer. The buffered data can then be returned directly on subsequent reads.

BufferedWriter When writing to this object, data is normally placed into an internal buffer. The buffer will be written out to the underlying RawIOBase object under various conditions, including:

when the buffer gets too small for all pending data;
when ﬂush() is called;
when the BufferedWriter object is closed or destroyed.

BufferedRandom A buffered interface to random access streams. It sup- ports seek() and tell() functionality.

BufferedRWPair A buffered I/O object combining two unidirectional

RawIOBase objects – one readable, the other writeable—into a single bidirectional endpoint.

Each of the above classes wrap a lower level byte oriented stream class such as the io.FileIO class, for example:

f = io.FileIO(‘data.dat’) br = io.BufferedReader(f) print(br.read())

This allows data in the form of bytes to be read from the ﬁle ‘data.dat’. You can of course also read data from a different source, such as an in memory BytesIO object:

binary_stream_from_file = io.BufferedReader(io.BytesIO(b‘starship.png’)) bytes = binary_stream_from_file.read(4) print(bytes)

In this example the data is read from the BytesIO object by the BufferedReader. The read() method is then used to read the ﬁrst 4 bytes, the output is:

Note the ‘b’ in front of both the string ‘starship.png’ and the result ‘star’. This indicates that the string literal should become a bytes literal in Python 3. Bytes literals are always preﬁxed with ‘b’ or ‘B’; they produce an instance of the bytes type instead of the str type. They may only contain ASCII characters.

The operations supported by buffered streams include, for reading:

peek(n) return up to n bytes of data without advancing the stream pointer. The number of bytes returned may be less or more than requested depending on the amount of data available.
read(n) return n bytes of data as bytes, if n is not supplied (or is negative) the read all available data.
readl(n) read up to n bytes of data using a single call on the raw data stream.

The operations supported by buffered writers include:

write(bytes) writes the bytes-like data and returns the number of bytes written.
ﬂush() This method forces the bytes held in the buffer into the raw stream.

19.7 Text Stream Classes

The text stream classes are the TextIOBase class and its two subclasses

TextIOWrapper and StringIO.

TextIOBase This is the root class for all Text Stream classes. It provides a character and line based interface to Stream I/O. This class provides several additional methods to that deﬁned in its parent class:

read(size=-1) This method will return at most size characters from the stream as a single string. If size is negative or None, it will read all remaining data.
- readline(size=-1) This method will return a string representing the current line (up to a newline or the end of the data whichever comes ﬁrst). If the stream is already at EOF, an empty string is returned. If size is speciﬁed, at most size characters will be read.
- seek(offset, [, whence]) change the stream position/pointer by the speciﬁed offset. The optional whence parameter indicates where the seek should start from:
  - SEEK_SET or 0: (the default) seek from the start of the stream.
  - SEEK_CUR or 1: seek to the current position; offset must be zero, which is a no-operation.
  - SEEK_END or 2: seek to the end of the stream; offset must be zero.
- tell() Returns the current stream position/pointer as an opaque number. The number does not usually represent a number of bytes in the underlying binary storage.
- write(s) This method will write the string s to the stream and return the number of characters written.

TextIOWrapper. This is a buffered text stream that wraps a buffered binary stream and is a direct subclass of TextIOBase. When a TextIOWrapper is created there are a range of options available to control its behaviour:

io.TextIOWrapper(buffer, encoding=None, errors=None, newline=No ne, line_buffering=False, write_through=False)

Where

buffer is the buffered binary stream.
encoding represents the text encoding used such as UTF-8.
errors deﬁnes the error handling policy such as strict or ignore.
newline controls how line endings are handled for example should they be ignored (None) or represented as a linefeed, carriage return or a newline/carriage return etc.
line_buffering if True then ﬂush() is implied when a call to write contains a newline character or a carriage return.
write_through if True then a call to write is guaranteed not to be buffered.

The TextIOWrapper is wrapped around a lower level binary buffered I/O stream, for example:

f = io.FileIO(‘data.txt‘) br = io.BufferedReader(f)

text_stream = io.TextIOWrapper(br, ‘utf-8‘)

StringIO This is an in memory stream for text I/O. The initial value of the buffer held by the StringIO object can be provided when the instance is created, for example:

in_memory_text_stream = io.StringIO(‘to be or not to be that is the question’)

print(‘in_memory_text_stream’, in_memory_text_stream) print(in_memory_text_stream.getvalue()) in_memory_text_stream.close()

This generates:

in_memory_text_stream <_io.StringIO object at 0x10fdfaee8> to be or not to be that is the question

Note that the underlying buffer (represented by the string passed into the

StringIO instance) is discarded when the close() method is called.

The getvalue() method returns a string containing the entire contents of the buffer. If it is called after the stream was closed then an error is generated.

19.8 Stream Properties

It is possible to query a stream to determine what types of operations it supports. This can be done using the readable(), seekable() and writeable() methods. For example:

f = io.FileIO(‘myfile.txt’) br = io.BufferedReader(f)

text_stream = io.TextIOWrapper(br, encoding=‘utf-8’)

print(‘text_stream’, text_stream) print(‘text_stream.readable():’, text_stream.readable()) print(‘text_stream.seekable()’, text_stream.seekable()) print(‘text_stream.writeable()’, text_stream.writable())

text_stream.close()

The output from this code snippet is:

text_stream <_io.TextIOWrapper name=’myfile.txt’ encoding=’utf- 8′>

text_stream.readable(): True text_stream.seekable() True text_stream.writeable() False

19.9 Closing Streams

All opened streams must be closed. However, you can close the top level stream and this will automatically close lower level streams, for example:

f = io.FileIO(‘data.txt’) br = io.BufferedReader(f)

text_stream = io.TextIOWrapper(br, ‘utf-8‘) print(text_stream.read()) text_stream.close()

19.10 Returning to the open() Function

If streams are so good then why don’t you use them all the time? Well actually in Python 3 you do! The core open function (and indeed the io.open() function) both return a stream object. The actual type of object returned depends on the ﬁle mode speciﬁed, whether buffering is being used etc. For example:

import io

# Text stream

f1 = open(‘myfile.txt’, mode=‘r’, encoding=‘utf-8’) print(f1)

# Binary IO aka Buffered IO

f2 = open(‘myfile.dat’, mode=‘rb’) print(f2)

f3 = open(‘myfile.dat’, mode=‘wb’) print(f3)

# Raw IO aka Unbufferedf IO

f4 = open(‘starship.png’, mode=‘rb’, buffering=0) print(f4)

When this short example is run the output is:

<_io.TextIOWrapper name=’myfile.txt’ mode=’r’ encoding=’utf-8′>

<_io.BufferedReader name=’myfile.dat’>

<_io.BufferedWriter name=’myfile.dat’>

<_io.FileIO name=’starship.png’ mode=’rb’ closefd=True>

As you can see from the output, four different types of object have been returned from the open() function. The ﬁrst is a TextIOWrapper, the second a BufferedReader, the third a BufferedWriter and the ﬁnal one is a FileIO object. This reflects the differences in the parameters passed into the open (0 function. For example, f1 references a io.TextIOWrapper because it must encode (convert) the input text into Unicode using the UTF-8 encoding scheme. While f2 holds a io.BufferedReader because the mode indicates that we want to read binary data while f3 holds a io.BufferedWriter because the mode used indicates we want to write binary data. The ﬁnal call to open returns a FileIO because we have indicated that we do not want to buffer the data and thus we can use the lowest level of stream object.

In general the following rules are applied to determine the type of object returned based on the modes and encoding speciﬁed:

Class	mode	Buffering
FileIO	binary	no
BufferedReader	‘rb’	yes
BufferedWriter	‘wb’	yes
BufferedRandom	‘rb+’ ‘wb+’ ‘ab+’	yes
TextIOWrapper	Any text	yes

Note that not all mode combinations make sense and thus some combinations will generate an error.

In general you don’t therefore need to worry about which stream you are using or what that stream does; not least because all the streams extend the IOBase class and thus have a common set of methods and attributes.

However, it is useful to understand the implications of what you are doing so that you can make better informed choices. For example, binary streams (that do less processing) are faster than Unicode oriented streams that must convert from ASCII into Unicode.

Also understanding the role of streams in Input and Output can also allow you to change the source and destination of data without needing to re-write the whole of your application. You can thus use a ﬁle or stdin for testing and a socket for reading data in production.

Chapter 20

Working with CSV Files

20.1 Introduction

This chapter introduces a module that supports the generation of CSV (or Comma Separated Values) ﬁles.

20.2 CSV Files

The CSV (Comma Separated Values) format is the most common import and export format for spreadsheets and databases. However, CSV is not a precise standard with multiple different applications having different conventions and speciﬁc standards. The Python csv module implements classes to read and write tabular data in CSV format. As part of this it supports the concept of a dialect which is a CSV format used by a speciﬁc application or suite of programs, for example, it supports

an Excel dialect.

This allows programmers to say, “write this data in the format preferred by Excel,” or “read data from this ﬁle which was generated by Excel,” without knowing the precise details of the CSV format used by Excel.

Programmers can also describe the CSV dialects understood by other applica- tions or deﬁne their own special-purpose CSV dialects.

The csv module provides a range of functions including csv.reader (csvﬁle, dialect=’excel’, **fmtparams) Returns a reader object which will iterate over lines in the given csvﬁle. An optional dialect parameter can be given. This may be an instance of a subclass of the Dialect class or one of the strings returned by the list_dialects() function. The other optional fmtparams keyword arguments can be given to override individual formatting parameters in the current dialect.

csv.writer (csvﬁle, dialect=’excel’, **fmtparams) Returns a writer object responsible for converting the user’s data into delimited strings on the given csvﬁle. An optional dialect parameter provided. The fmtparams keyword arguments can be given to override individual formatting parameters in the current dialect.
csv.list_dialects() Return the names of all registered dialects. For example on a Mac OS X the default list of dialects is [‘excel’, ‘excel-tab’, ‘unix’].

20.1.1 The CSV Writer Class

A CSV Writer is obtained from the csv.writer() function. The csvwriter

supports two methods used to write data to the CSV ﬁle:

csvwriter.writerow(row) Write the row parameter to the writer’s ﬁle object, formatted according to the current dialect.
csvwriter.writerows(rows) Write all elements in rows (an iterable of row objects as described above) to the writer’s ﬁle object, formatted according to the current dialect.
Writer objects also have the following public attribute:
csvwriter.dialect A read-only description of the dialect in use by the writer.

The following program illustrates a simple use of the csv module which creates a ﬁle called sample.csv.

As we have not speciﬁed a dialect, the default ‘excel’ dialect will be used. The writerow() method is used to write each comma separate list of strings to the CSV ﬁle.

print(‘Crearting CSV file’)

with open(‘sample.csv’, ‘w’, newline=”) as csvfile: writer = csv.writer(csvfile) writer.writerow([‘She Loves You’, ‘Sept 1963’])

writer.writerow([‘I Want to Hold Your Hand’, ‘Dec 1963’]) writer.writerow([‘Cant Buy Me Love’, ‘Apr 1964’]) writer.writerow([‘A Hard Days Night’, ‘July 1964’])

The resulting ﬁle can be viewed as shown below:

However, as it is a CSV ﬁle, we can also open it in Excel:

20.1.2 The CSV Reader Class

A CSV Reader object is obtained from the csv.reader() function. It imple- ments the iteration protocol.

If a csv reader object is used with a for loop then each time round the loop it supplies the next row from the CSV ﬁle as a list, parsed according to the current CSV dialect.

Reader objects also have the following public attributes:

csvreader.dialect A read-only description of the dialect in use by the parser.
csvreader.line_num The number of lines read from the source iterator. This is not the same as the number of records returned, as records can span multiple lines.

The following provides a very simple example of reading a CSV ﬁle using a csv reader object:

print(‘Starting to read csv file’)

with open(‘sample.csv’, newline=”) as csvfile:

reader = csv.reader(csvfile) for row in reader:

print(*row, sep=’, ‘) print(‘Done Reading’)

The output from this program, based on the sample.csv ﬁle created earlier is:

Starting to read csv file She Loves You, Sept 1963

I Want to Hold Your Hand, Dec 1963 Cant Buy Me Love, Apr 1964

A Hard Days Night, July 1964 Done Reading

20.1.3 The CSV DictWriter Class

In many cases the ﬁrst row of a CSV ﬁle contains a set of names (or keys) that deﬁne the ﬁelds within the rest of the CSV. That is the ﬁrst row gives meaning to the columns and the data held in the rest of the CSV ﬁle. It is therefore very useful to capture this information and to structure the data written to a CSV ﬁle or loaded from a CSV ﬁle based on the keys in the ﬁrst row.

The csv.DictWriter returns an object that can be used to write values into the CSV ﬁle based on the use of such named columns. The ﬁle to be used with the DictWriter is provided when the class is instantiated.

import csv

with open(‘names.csv’, ‘w’, newline=”) as csvfile: fieldnames = [‘first_name’, ‘last_name’, ‘result’] writer = csv.DictWriter(csvfile, fieldnames=fieldnames) writer.writeheader()

writer.writerow({‘first_name’: ‘John’,

‘last_name’: ‘Smith’, ‘result’ : 54})

writer.writerow({‘first_name’: ‘Jane’,

‘last_name’: ‘Lewis’, ‘result’: 63})

writer.writerow({‘first_name’: ‘Chris’,

‘last_name’: ‘Davies’, ‘result’ : 72})

Note that when the DictWriter is created a list of the keys must be provided that are used for the columns in the CSV ﬁle.

The method writeheader() is then used to write the header row out to the CSV ﬁle.

The method writerow() takes a dictionary object that has keys based on the keys deﬁned for the DictWriter. These are then used to write data out to the CSV (note the order of the keys in the dictionary is not important).

In the above example code the result of this is that a new ﬁle called names.csv

is created which can be opened in Excel:

Of course, as this is a CSV ﬁle it can also be opened in a plain text editor as well.

20.1.1 The CSV DictReader Class

As well as the csv.DictWriter there is a csv.DictReader. The ﬁle to be used with the DictReader is provided when the class is instantiated. As with the DictReader the DictWriter class takes a list of keys used to deﬁne the columns in the CSV ﬁle. If the headings to be used for the ﬁrst row can be provided although this is optional (if a set of keys are not provided, then the values in the ﬁrst row of the CSV ﬁle will be used as the ﬁeldnames).

The DictReader class provides several useful features including the ﬁeldnames property that contains a list of the keys/headings for the CSV ﬁle as deﬁned by the ﬁrst row of the ﬁle.

The DictReader class also implements the iteration protocol and thus it can be used in a for loop in which each row (after the ﬁrst row) is returned in turn as a dictionary. The dictionary object representing each row can then be used to access each column value based on the keys deﬁned in the ﬁrst row.

An example is shown below for the CSV ﬁle created earlier:

import csv

print(‘Starting to read dict CSV example’)

with open(‘names.csv’, newline=”) as csvfile:

reader = csv.DictReader(csvfile) for heading in reader.fieldnames:

print(heading, end=’ ‘)

print(‘\n ‘)

for row in reader:

print(row[‘first_name’], row[‘last_name’], row[‘result’])

print(‘Done’)

This generates the following output:

Starting to read dict CSV example first_name last_name result

John Smith 54

Jane Lewis 63

Chris Davies 72 Done

Chapter 21

Working with Excel Files

21.1 Introduction

This chapter introduces the openpyxl module that can be used when working with Excel ﬁles. Excel is a software application developed by Microsoft that allows users to work with spreadsheets. It is a very widely used tool and ﬁles using the Excel ﬁle format are commonly encountered within many organisations. It is in effect the industry standard for spreadsheets and as such is a very useful tool to have in the developers toolbox.

21.2 Excel Files

Although CSV ﬁles are a convenient and simple way to handle data; it is very common to need to be able to read or write Excel ﬁles directly. To this end there are several libraries available in Python for this purpose. One widely used library is the OpenPyXL library. This library was originally written to support access to Excel 2010 ﬁles. It is an open source project and is well documented.

The OpenPyXL library provides facilities for

reading and writing Excel workbooks,
creating/accessing Excel worksheets,
creating Excel formulas,
creating graphs (with support from additional modules).

As OpenPyXL is not part of the standard Python distribution you will need to install the library yourself using a tool such as Anaconda or pip (e.g. pip install openpyxl). Alternatively, if you are using PyCharm you will be able to add the OpenPyXL library to your project.

21.3 The Openpyxl. Workbook Class

The key element in the OpenPyXL library is the Workbook class. This can be imported from the module:

from openpyxl import Workbook

A new instance of the (in memory) Workbook can be created using the Workbook class (note at this point it is purely a structure within the Python program and must be saved before an actual Excel ﬁle is created).

wb = Workbook()

21.4 The Openpyxl. WorkSheet Objects

A workbook is always created with at least one worksheet. You can get hold of the currently active worksheet using the Workbook.active property:

ws = wb.active

You can create additional worksheets using the workbooks’ create_sheet () method:

ws = wb.create_sheet(‘Mysheet‘)

You can access or update the title of the worksheet using the title property:

ws.title = ‘New Title‘

The background colour of the tab holding this title is white by default. You can change this providing an RRGGBB colour code to the worksheet. sheet_properties.tabColor attribute, for example:

ws.sheet_properties.tabColor = “1072BA“

21.5 Working with Cells

It is possible to access the cells within a worksheet. A cell can be accessed directly as keys on the worksheet, for example:

ws[‘A1‘] = 42

cell = ws[‘A1‘]

This returns a cell object; you can obtain the value of the cell using the value property, for example

print(cell.value)

There is also the Worksheet.cell() method. This provides access to cells using row and column notation:

d = ws.cell(row=4, column=2, value=10)

A row of values can also be added at the current position within the Excel ﬁle using append:

ws.append([1, 2, 3])

This will add a row to the Excel ﬁle containing 1, 2, and 3. Ranges of cells can be accessed using slicing:

cell_range = ws[‘A1‘:’C2‘]

Ranges of rows or columns can also be obtained:

col = ws[‘C‘] col_range = ws[‘C:D‘] row10 = ws[10] row_range = ws[5:10]

The value of a cell can also be an Excel formula such as

ws[‘A3‘] = ‘=SUM(A1, A2)‘

A workbook is actually only a structure in memory; it must be saved to a ﬁle for permanent storage. These workbooks can be saved using the save() method. This method takes a ﬁlename and writes the Workbook out in Excel format.

workbook = Workbook()

…

workbook.save(‘balances.xlsx‘)

21.6 Sample Excel File Creation Application

The following simple application creates a Workbook with two worksheets. It also contains a simple Excel formula that sums the values held in to other cells:

from openpyxl import Workbook

def main():

print(‘Starting Write Excel Example with openPyXL’)

workbook = Workbook()

# Get the current active worksheet

ws = workbook.active ws.title = ‘my worksheet’

ws.sheet_properties.tabColor = ‘1072BA’

ws[‘A1’] = 42

ws[‘A2’] = 12

ws[‘A3’] = ‘=SUM(A1, A2)’

ws2 = workbook.create_sheet(title=‘my other sheet’) ws2[‘A1’] = 3.42

ws2.append([1, 2, 3]) ws2.cell(column=2, row=1, value=15)

workbook.save(‘sample.xlsx’) print(‘Done Write Excel Example’)

if name == ‘ main ‘: main()

The Excel ﬁle generated from this can be viewed in Excel as shown below:

21.7 Loading a Workbook from an Excel File

Of course, in many cases it is necessary not just to create Excel ﬁles for data export but also to import data from an existing Excel ﬁle. This can be done using the OpenPyXL load_workbook() function. This function opens the speciﬁed Excel ﬁle (in read only mode by default) and returns a Workbook object.

from openpyxl import load_workbook

workbook = load_workbook(filename=’sample.xlsx’)

You can now access a list of sheets, their names, obtain the currently active sheet etc. using properties provided by the workbook object:

workbook.active returns the active worksheet object.
workbook.sheetnames returns the names (strings) of the worksheets in this workbook.
workbook.worksheets returns a list of worksheet objects.

The following sample application reads the Excel ﬁle created earlier in this chapter:

from openpyxl import load_workbook

def main():

print(‘Starting reading Excel file using openPyXL’)

workbook = load_workbook(filename=‘sample.xlsx’) print(workbook.active) print(workbook.sheetnames) print(workbook.worksheets)

print(‘-‘ * 10)

ws = workbook[‘my worksheet’] print(ws[‘A1’])

print(ws[‘A1’].value)

print(ws[‘A2’].value)

print(ws[‘A3’].value)

print(‘-‘ * 10)

for sheet in workbook: print(sheet.title)

print(‘-‘ * 10)

cell_range = ws[‘A1’:‘A3’]

for cell in cell_range: print(cell[0].value)

print(‘-‘ * 10)

print(‘Finished reading Excel file using openPyXL’) if name == ‘ main ‘:

main()

The output from this application is illustrated below:

Starting reading Excel file using openPyXL

[‘my worksheet’, ‘my other sheet’]

[<Worksheet “my worksheet”>, <Worksheet “my other sheet”>]

<Cell ‘my worksheet’.A1> 42

=SUM(A1, A2)

my worksheet my other sheet

=SUM(A1, A2)

Finished reading Excel file using openPyXL

Chapter 22

Regular Expressions in Python

22.1 Introduction

Regular Expression are a very powerful way of processing text while looking for recurring patterns; they are often used with data held in plain text ﬁles (such as log ﬁles), CSV ﬁles as well as Excel ﬁles. This chapter introduces regular expressions, discusses the syntax used to deﬁne a regular expression pattern and presents the Python re module and its use.

22.2 What Are Regular Expressions?

A Regular Expression (also known as a regex or even just re) is a sequence of characters (letters, numbers and special characters) that form a pattern that can be used to search text to see if that text contains sequences of characters that match the pattern.

For example, you might have a pattern deﬁned as three characters followed by three numbers. This pattern could be used to look for such a pattern in other strings. Thus, the following strings either match (or contain) this pattern or they do not:

Abc123	Matches the pattern
A123A	Does not match the pattern
123AAA	Does not match the pattern

Regular Expression are very widely used for ﬁnding information in ﬁles, for example

ﬁnding all lines in a log ﬁle associated with a speciﬁc user or a speciﬁc operation,
for validating input such as checking that a string is a valid email address or postcode/ZIP code etc.

Support for Regular Expressions is wide spread within programming languages such as Java, C#, PHP and particularly Perl. Python is no exception and has the built-in module re (as well as additional third-party modules) that support Regular Expressions.

22.1 Regular Expression Patterns

You can deﬁne a regular expression pattern using any ASCII character or number. Thus, the string ‘John’ can be used to deﬁne a regex pattern that can be used to match any other string that contains the characters ‘J’, ‘o’, ‘h’, ‘n’. Thus each of the following strings will match this pattern:

‘John Hunt’
‘John Jones’
‘Andrew John Smith’
‘Mary Helen John’
‘John John John’
‘I am going to visit the John’
‘I once saw a ﬁlm by John Wayne’

But the following strings would not match the pattern:

‘Jon Davies’ in this case because the spelling of John is different.
‘john williams’ in this case because the capital J does not match the lowercase j.
‘David James’ in this case because the string does not contain the string John!

Regular expressions (regexs) use special characters to allow more complex patterns to be described. For example, we can use the special characters ‘[]’ to deﬁne a set of characters that can match. For example, if we want to indicate that the J may be a capital or a lower-case letter then we can write ‘[Jj]’—this indicates that either ‘J’ or ‘j’ can match the ﬁrst.

[Jj]ohn—this states that the pattern starts with either a capital J or a lowercase j followed by ‘ohn’.

Now both ‘john williams’ and ‘John Williams’ will match this regex pattern.

Regular Expression Patterns 259

22.3.1 Pattern Metacharacters

There are several special characters (often referred to as metacharacters) that have a speciﬁc meaning within a regex pattern, these are listed in the following table:

Character	Description	Example
[]	A set of characters	[a-d] characters in the sequence ‘a’ to ‘d’
\	Indicates a special sequence (can also be used to escape special characters)	‘\d’ indicates the character should be an integer
.	Any character with the exception of the newline character	‘J.hn’ indicates that there can be any character after the ‘J’ and before the ‘h’
^	Indicates a string must start with the following pattern	“^hello” indicates the string must start with ‘hello’
$	Indicates a string must end with the preceding pattern	“world$” indicates the string must end with ‘world’
*	Zero or more occurrences of the preceding pattern	“Python*” indicates we are looking for zero or more times Python is in a string
+	One or more occurrences of preceding pattern	“info+” indicates that we must ﬁnd info in the string at least once
?	Indicates zero or 1 occurrences of the preceding pattern	“john?” indicates zero or one instances of the ‘John’
{}	Exactly the speciﬁed number of occurrences	“John{3}” this indicates we expect to see the ‘John’ in the string three times. “X{1,2}” indicates that there can be one or two Xs next to each other in the string
\|	Either or	“True\|OK” indicates we are looking for either True or OK
()	Groups together a regular expression; you can then apply another operator to the whole group	“(abc\|xyz){2}” indicates that we are looking for the string abc or xyz repeated twice

22.3.2 Special Sequences

A special sequence is a combination of a ‘\’ (backslash) followed by a character combination which then has a special meaning. The following table lists the common special sequences used in Regular Expressions:

Sequence	Description	Example
\A	Returns a match if the following characters are at the beginning of the string	“\AThe” must start with ‘The’
\b	Returns a match where the speciﬁed characters are at the beginning or at the end of a word	“\bon” or “on\b” indicates a string must start or end with ‘on’
\B	Indicates that the following characters must be present in a string but not at the start (or at the end) of a word	r”\Bon” or r”on\B” must not start or end with ‘on’
\d	Returns a match where the string contains digits (numbers from 0–9)	“\d”
\D	Returns a match where the string DOES NOT contain digits	“\D”
\s	Returns a match where the string contains a white space character	“\s*”
\S	Returns a match where the string DOES NOT contain a white space character	“\S”
\w	Returns a match where the string contains any word characters (characters from a to Z, digits from 0–9, and the underscore _ character)	“\w”
\W	Returns a match where the string DOES NOT contain any word characters	“\W”
\Z	Returns a match if the following characters are present at the end of the string	“Hunt\Z”

22.3.3 Sets

A set is a sequence of characters inside a pair of square brackets which have speciﬁc meanings. The following table provides some examples.

Set	Description
[jeh]	Returns a match where one of the speciﬁed characters (j, e or h) are present
[a–x]	Returns a match for any lower-case character, alphabetically between a and x
[^zxc]	Returns a match for any character EXCEPT z, x and c
[0123]	Returns a match where any of the speciﬁed digits (0, 1, 2, or 3) are present
[0–9]	Returns a match for any digit between 0 and 9
[0–9][0–9]	Returns a match for any two-digit numbers from 00 and 99
[a–zA–Z]	Returns a match for any character alphabetically between a and z or A and Z

The Python re Module 261

22.2 The Python re Module

The Python re module is the built-in module provided by Python for working with Regular Expressions.

You might also like to examine the third-party regex module, which is backwards compatible with the default re module but provides additional functionality.

22.3 Working with Python Regular Expressions

22.3.1 Using Raw Strings

An important point to note about many of the strings used to deﬁne the regular expression patterns is that they are preceded by an ‘r’ for example r’/bin/sh$’.

The ‘r’ before the string indicates that the string should be treated as a raw

string.

A raw string is a Python string in which all characters are treated as exactly that; individual characters. It means that backslash (‘\’) is treated as a literal character rather than as a special character that is used to escape the next character.

For example, in a standard string ‘\n’ is treated as a special character repre- senting a newline, thus if we wrote the following:

s = ‘Hello \n world‘ print(s)

We will get as output:

Hello World

However, if we preﬁx the string with an ‘r’ then we are telling Python to treat it as a raw string. For example:

s = r’Hello \n world‘ print(s)

The output is now

Hello \n world

This is important for regular expression as characters such as backslash (‘\’) are used within patterns to have a special regular expression meaning and thus we do not want Python to process them in the normal way.

22.3.2 Simple Example

The following simple Python program illustrates the basic use of the re module. It is necessary to import the re module before you can use it.

import re

text1 = ‘john williams‘ pattern = ‘[Jj]ohn‘

print(‘looking in‘, text1, ‘for the pattern‘, pattern)

if re.search(pattern, text1): print(‘Match has been found‘)

When this program is run, we get the following output:

looking in john williams for the pattern [Jj]ohn Match has been found

If we look at the code, we can see that the string that we are examining contains ‘john williams’ and that the pattern used with this string indicates that we are looking for a sequence of ‘J’ or ‘j’ followed by ‘ohn’. To perform this test we use the re. search() function passing the regex pattern, and the text to test, as parameters. This function returns either None (which is taken as meaning False by the If statement) or a Match Object (which always has a Boolean value of True). As of course ‘john’ at the start of text1 does match the pattern, the re.search() function returns a match object and we see the ‘Match has been found’ message is printed out.

Both the Match object and search() method will be described in more detail below; however, this short program illustrates the basic operation of a Regular Expression.

22.3.3 The Match Object

Match objects are returned by the search() and match() functions.

They always have a boolean value of True.

The functions match() and search() return None when there is no match and a Match object when a match is found. It is therefore possible to use a match object with an if statement:

import re

match = re.search(pattern, string)

if match:

process(match)

Match objects support a range of methods and attributes including:

match.re The regular expression object whose match() or search() method produced this match instance.
match.string The string passed to match() or search().
match.start([group]) / match.end([group]) Return the indices of the start and end of the substring matched by group.
match.group() returns the part of the string where there was a match.

22.3.4 The search() Function

The search() function searches the string for a match, and returns a Match object if there is a match. The signature of the function is:

re.search(pattern, string, flags=0)

The meaning of the parameters are:

pattern this is the regular expression pattern to be used in the matching process.
string this is the string to be searched.
ﬂags these (optional) flags can be used to modify the operation of the search.

The re module deﬁnes a set of flags (or indicators) that can be used to indicate any optional behaviours associated with the pattern. These flags include:

Flag	Description
re.IGNORECASE	Performs case-insensitive matching
re.LOCALE	Interprets words according to the current locale. This interpretation affects the alphabetic group (\w and \W), as well as word boundary behavior(\b and \B)
re.MULTILINE	Makes $ match the end of a line (not just the end of the string) and makes ^ match the start of any line (not just the start of the string)
re.DOTALL	Makes a period (dot) match any character, including a newline
re.UNICODE	Interprets letters according to the Unicode character set. This flag affects the behavior of \w, \W, \b, \B
re.VERBOSE	Ignores whitespace within the pattern (except inside a set [] or when escaped by a backslash) and treats unescaped # as a comment marker

If there is more than one match, only the ﬁrst occurrence of the match will be returned:

import re

line1 = ‘The price is 23.55‘ containsIntegers = r’\d+‘

if re.search(containsIntegers, line1): print(‘Line 1 contains an integer‘)

else:

print(‘Line 1 does not contain an integer‘)

In this case the output is

Line 1 contains an integer

Another example of using the search() function is given below. In this case the pattern to look for deﬁnes three alternative strings (that is the string must contain either Beatles, Adele or Gorillaz):

import re

# Alternative words

music = r’Beatles|Adele|Gorillaz‘ request = ‘Play some Adele‘

if re.search(music, request): print(‘Set Fire to the Rain‘)

else:

print(‘No Adele Available‘)

In this case we generate the output:

Set Fire to the Rain

22.3.5 The match() Function

This function attempts to match a regular expression pattern at the beginning of a string. The signature of this function is given below:

re.match(pattern, string, flags=0)

The parameters are:

pattern this is the regular expression to be matched.
string this is the string to be searched.
ﬂags modiﬁer flags that can be used.

The re.match() function returns a Match object on success, None on failure.

22.3.6 The Difference Between Matching and Searching

Python offers two different primitive operations based on regular expressions:

match() checks for a match only at the beginning of the string,
search() checks for a match anywhere in the string.

22.3.7 The ﬁndall() Function

The ﬁndall() function returns a list containing all matches. The signature of this function is:

re.findall(pattern, string, flags=0)

This function returns all non-overlapping matches of pattern in string, as a list of strings.

The string is scanned left-to-right, and matches are returned in the order found. If one or more groups are present in the pattern, then a list of groups is returned; this will be a list of tuples if the pattern has more than one group. If no matches are found, an empty list is returned.

An example of using the ﬁndall() function is given below. This example looks for a substring starting with two letters and followed by ‘ai’ and a single character. It is applied to a sentence and returns only the sub string ‘Spain’ and ‘plain’.

import re

str = ‘The rain in Spain stays mainly on the plain‘ results = re.findall(‘[a-zA-Z]{2}ai.‘, str) print(results)

for s in results: print(s)

The output from this program is

[‘Spain’, ‘plain’]

Spain plain

22.3.8 The ﬁnditer() Function

This function returns an iterator yielding matched objects for the regular expres- sion pattern in the string supplied. The signature for this function is:

re.finditer(pattern, string, flags=0)

The string is scanned left-to-right, and matches are returned in the order found. Empty matches are included in the result. Flags can be used to modify the matches.

22.3.9 The split() Function

The split() function returns a list where the string has been split at each match. The syntax of the split() function is

re.split(pattern, string, maxsplit=0, flags=0)

The result is to split a string by the occurrences of pattern. If capturing parentheses are used in the regular expression pattern, then the text of all groups in the pattern are also returned as part of the resulting list. If maxsplit is nonzero, at most maxsplit splits occur, and the remainder of the string is returned as the ﬁnal element of the list. Flags can again be used to modify the matches.

import re

str = ‘It was a hot summer night‘ x = re.split(‘\s‘, str)

print(x)

The output is

[‘It’, ‘was’, ‘a’, ‘hot’, ‘summer’, ‘night’]

22.3.10 The sub() Function

The sub() function replaces occurrences of the regular expression pattern in the string with the repl string.

re.sub(pattern, repl, string, max=0)

This method replaces all occurrences of the regular expression pat- tern in string with repl, substituting all occurrences unless max is provided. This method returns the modiﬁed string.

import re

pattern = ‘(England|Wales|Scotland)‘

input = ‘England for football, Wales for Rugby and Scotland for the Highland games‘

print(re.sub(pattern, ‘England‘, input ))

Which generates:

England for football, England for Rugby and England for the Highland games

You can control the number of replacements by specifying the count parameter: The following code replaces the ﬁrst 2 occurrences:

import re

pattern = ‘(England|Wales|Scotland)‘

input = ‘England for football, Wales for Rugby and Scotland for the Highland games‘

x = re.sub(pattern, ‘Wales‘, input, 2) print(x)

which produces

Wales for football, Wales for Rugby and Scotland for the Highland games

You can also ﬁnd out how many substitutions were made using the subn() function. This function returns the new string and the number of substitutions in a tuple:

import re

pattern = ‘(England|Wales|Scotland)‘

input = ‘England for football, Wales for Rugby and Scotland for the Highland games‘

print(re.subn(pattern,’Scotland‘, input ))

The output from this is:

(‘Scotland for football, Scotland for Rugby and Scotland for the Highland games’, 3)

22.3.11 The compile() Function

Most regular expression operations are available as both module-level functions (as described above) and as methods on a compiled regular expression object.

The module-level functions are typically simpliﬁed or standardised ways to use the compiled regular expression. In many cases these functions are sufﬁcient but if ﬁner grained control is required then a compiled regular expression may be used.

re.compile(pattern, flags=0)

The compile() function compiles a regular expression pattern into a regu- lar expression object, which can be used for matching using its match(), search() and other methods as described below.

The expression’s behaviour can be modiﬁed by specifying a flags value. V The statements:

prog = re.compile(pattern) result = prog.match(string)

are equivalent to

result = re.match(pattern, string)

but using re.compile() and saving the resulting regular expression object for reuse is more efﬁcient when the expression will be used several times in a single program.

Compiled regular expression objects support the following methods and attributes:

Pattern.search(string, pos, endpos) Scan through string looking for the ﬁrst location where this regular expression produces a match and return a corresponding Match object. Return None if no position in the string matches the pattern. Starting at pos if provided and ending at endpos if this is provided (otherwise process the whole string).
Pattern.match(string, pos, endpos)If zero or more characters at the beginning of string match this regular expression, return a correspond- ing match object. Return None if the string does not match the pattern. The pos and endpos are optional and specify the start and end positions within which to search.
Pattern.split(string, maxsplit = 0)Identical to the split() function, using the compiled pattern.
Pattern.ﬁndall(string[, pos[, endpos]])Similar to the ﬁndall () function, but also accepts optional pos and endpos parameters that limit the search region like for search().
Pattern.ﬁnditer(string[, pos[, endpos]])Similar to the ﬁnd- iter() function, but also accepts optional pos and endpos parameters that limit the search region like for search().
Pattern.sub(repl, string, count = 0)Identical to the sub() function, using the compiled pattern.
Pattern.subn(repl, string, count = 0)Identical to the subn() function, using the compiled pattern.
Pattern.pattern the pattern string from which the pattern object was compiled.

An example of using the compile() function is given below. The pattern to be compiled is deﬁned as containing 1 or more digits (0 to 9):

import re

line1 = ‘The price is 23.55‘ containsIntegers = r’\d+‘

rePattern = re.compile(containsIntegers)

matchLine1 = rePattern.search(line1)

if matchLine1:

print(‘Line 1 contains a number‘) else:

print(‘Line 1 does not contain a number‘)

The compiled pattern can then be used to apply methods such as search() to a speciﬁc string (in this case held in line1). The output generated by this is:

Line 1 contains a number

Of course the compiler pattern object supports a range of methods in addition to

search() as illustrated by the spilt method:

p = re.compile(r’\W+‘) s = ‘20 High Street‘ print(p.split(s))

The output from this is

[’20’, ‘High’, ‘Street’]

Chapter 23

Introduction to Databases

23.1 Introduction

There are several different types of database system in common use today including Object databases, NoSQL databases and (probably the most common) Relational Databases. This chapter focusses on Relational Databases as typiﬁed by database systems such as Oracle, Microsoft SQL Server and MySQL. The database we will use in this book is MySQL.

23.2 What Is a Database?

A database is essentially a way to store and retrieve data.

Typically, there is some form of query language used with the database to help select the information to retrieve such as SQL or Structured Query Language.

In most cases there is a structure deﬁned that is used to hold the data (although this is not true of the newer NoSQL or non-relational unstructured databases such as CouchDB or MongoDB).

In a Relational Database the data is held in tables, where the columns deﬁne the properties or attributes of the data and each row deﬁnes the actual values being held, for example:

In this diagram there is a table called students; it is being used to hold information about students attending a meeting. The table has 5 attributes (or columns) deﬁned for id, name, surname, subject and email.

In this case, the id is probably what is known as a primary key. The primary key is a property that is used to uniquely identify the student row; it cannot be omitted and must be unique (within the table). Obviously names and subjects may well be duplicated as there may be more than one student studying Animation or Games and students may have the same ﬁrst name or surname. It is probable that the email column is also unique as students probably don’t share an email address but again this may not necessarily be the case.

You might at this point wonder why the data in a Relational Database is called relational and not tables or tabular? The reason is because of a topic known as relational algebra that underpins Relational Database theory. Relational Algebra takes its name from the mathematical concept known as a relation. However, for the purposes of this chapter you don’t need to worry about this and just need to remember that data is held in tables.

23.2.1 Data Relationships

When the data held in one table has a link or relationship to data held in another table then an index or key is used to link the values in one table to another. This is illustrated below for a table of addresses and a table of people who live in that address. This shows for example, that ‘Phoebe Gates’ lives at address ‘addr2’ which is 12 Queen Street, Bristol, BS42 6YY.

This is an example of a many to one (often written as many:1) relationship; that is there are many people who can live at one address (in the above Adam Smith also lives at address ‘addr2’). In Relational Databases there can be several different types of relationship such as:

one:one where only one row in one table references one and only one row in another table. An example of a one to one relationship might be from a person to an order for a unique piece of jewellery.
one:many this is the same as the above address example, however in this case the direction of the relationship is reversed (that is to say that one address in the addresses table can reference multiple persons in the people table).
many:many This is where many rows in one table may reference many rows in a second table. For example, many students may take a particular class and a student may take many classes. This relationship usually involves an intermediate (join) table to hold the associations between the rows.

23.2.2 The Database Schema

The structure of a Relational Database is deﬁned using a Data Deﬁnition Language or Data Description Language (a DDL).

Typically, the syntax of such a language is limited to the semantics (meaning) required to deﬁne the structure of the tables. This structure is known as the database schema. Typically, the DDL has commands such as CREATE TABLE, DROP TABLE (to delete a table) and ALTER TABLE (to modify the structure of an existing table).

Many tools provided with a database allow you to deﬁne the structure of the database without getting too bound up in the syntax of the DDL; however, it is useful to be aware of it and to understand that the database can be created in this way. For example, we will use the MySQL database in this chapter. The MySQL

Workbench is a tool that allows you to work with MySQL databases to manage and query the data held within a particular database instance. For references for mySQL and the MySQL Workbench see the links at the end of this chapter.

As an example, within the MySQL Workbench we can create a new table using a menu option on a database:

Using this we can interactively deﬁne the columns that will comprise the table:

Here each column name, its type and whether it is the primary key (PK), not empty (or Not Null NN) or unique (UQ) have been speciﬁed. When the changes are applied, the tool also shows you the DDL that will be used to create the database:

When this is applied a new table is created in the database as shown below:

The tool also allows us to populate data into the table; this is done by entering data into a grid and hitting apply as shown below:

23.3 SQL and Databases

We can now use query languages to identify and return data held in the database often using speciﬁc criteria. For example, let us say we want to return all the people who have the surname Jones from the following table:

We can do this by specifying that data should be returned where the surname equals ‘Jones’; in SQL this would look like:

SELECT * FROM students where surname=’Jones’;

The above SELECT statement states that all the properties (columns or attributes) in a row in the table students are to be returned where the surname equals ‘Jones’. The result is that two rows are returned:

Note we need to specify the table we are interested in and what data we want to return (the ‘*’ after the select indicated we want all the data). If we were only interested in their ﬁrst names then we could use:

SELECT name FROM students where surname=’Jones’;
This would return only the names of the students

23.4 Data Manipulation Language

Data can also be inserted into a table or existing data in a table can be updated. This is done using the Data Manipulation Language (DML).

For example, to insert data into a table we merely need to write an INSERT SQL statement providing the values to be added and how they map to the columns in the table:

INSERT INTO ‘students’ (‘id’, ‘name’, ‘surname’, ‘subject’, ’email’) VALUES (‘6’, ‘James’, ‘Andrews’, ‘Games’, ‘ja@my.com’);

This would add the row 6 to the table students with the result that the table would now have an additional row:

Updating an existing row is a little more complicated as it is ﬁrst necessary to identify the row to be updated and then the data to modify. Thus an UPDATE statement includes a where clause to ensure the correct row is modiﬁed:

UPDATE ‘students’ SET ’email’=‘grj@my.com’ WHERE ‘id’=’2’;

The effect of this code is that the second row in the students table is modiﬁed with the new email address:

23.5 Transactions in Databases

Another important concept within a database is that of a Transaction. A Transaction represents a unit of work performed within a database management system (or similar system) against a database instance, and is independent of any other transaction.

Transactions in a database environment have two main purposes

To provide a unit of work that allows recovery from failures and keeps a database consistent even in cases of system failure, when execution stops (completely or partially). This is because either all the operations within a transaction are performed or none of them are. Thus, if one operation causes an error then all the changes being made by the transaction thus far are rolled back and none of them will have been made.
To provide isolation between programs accessing a database concurrently. This means that the work being done by one program will not interact with another programs work.

A database transaction, by deﬁnition, must be atomic, consistent, isolated and durable:

Atomic This indicates that a transaction represents an atomic unit of work; that is either all the operations in the transaction are performed or none of them are performed.
Consistent Once completed the transaction must leave the data in a consistent state with any data constraints met (such as a row in one table must not reference an non-existent row in another table in a one to many relationship etc.).
Isolated This relates to the changes being made by concurrent transactions; these changes must be isolated from each other. That is, one transaction cannot see the changes being made by another transaction until the second transaction completes and all changes are permanently saved into the database.
Durable This means that once a transaction completes then the changes it has made are permanently stored into the database (until some future transaction modiﬁes that data).

Database practitioners often refer to these properties of database transactions using the acronym ACID (for Atomic, Consistent, Isolated, Durable).

Not all databases support transactions although all commercial, production quality databases such as Oracle, Microsoft SQL Server and MySQL, do support transactions.

Chapter 24

Python DB-API

24.1 Accessing a Database from Python

The standard for accessing a database in Python is the Python DB-API. This speciﬁes a set of standard interfaces for modules that wish to allow Python to access a speciﬁc database. The standard is described in PEP 249,a PEP is a Python Enhancement Proposal.

Almost all Python database access modules adhere to this standard. This means that if you are moving from one database to another, or attempting to port a Python program from one database to another, then the APIs you encounter should be very similar (although the SQL processed by different database can also differ). There are modules available for most common databases such as MySQL, Oracle, Microsoft SQL Server etc.

24.2 The DB-API

There are several key elements to the DB_API these are:

The connect function. The connect() function that is used to connect to a database and returns a Connection Object.
Connection Objects. Within the DB-API access to a database is achieved through connection objects. These connection objects provide access to cursor objects.
Cursor objects are used to execute SQL statements on the database.
The result of an execution. These are the results that can be fetched as a sequence of sequences (such a tuple of tuples). The standard can thus be used to select, insert or update information in the database.

These elements are illustrated below:

The standard speciﬁes a set of functions and objects to be used to connect to a database. These include the connection function, the Connection Object and the Cursor object.

The above elements are described in more detail below.

24.2.1 The Connect Function

The connection function is deﬁned as:

connect(parameters…)

It is used to make the initial connection to the database. The connection returns a Connection Object. The parameters required by the connection function are data- base dependent.

24.2.2 The Connection Object

The Connection Object is returned by the connect() function. The Connection object provides several methods including:

close() used to close the connection once you no longer need it. The con- nection will be unusable from this point onwards.
- commit() used to commit a pending transaction.

rollback() used to rollback all the changes made to the database since the last transaction commit (optional as not all databases provide transaction support).
- cursor() returns a new Cursor object to use with the connection.

24.2.3 The Cursor Object

The Cursor object is returned from the connection.cusor() method. A Cursor Object represents a database cursor, which is used to manage the context of a fetch operation or the execution of a database command. Cursors support a variety of attributes and methods:

cursor.execute(operation, parameters) Prepare and execute a database operation (such as a query statement or an update command). Parameters may be provided as a sequence or mapping and will be bound to variables in the operation. Variables are speciﬁed in a database-speciﬁc notation.
cursor.rowcount a read-only attribute providing the number of rows that the last cursor.execute() call returned (for select style statements) or affected (for update or insert style statements).
cursor.description a read only attribute providing information on the columns present in any results returned from a SELECT operation.
cursor.close() closes the cursor. From this point on the cursor will not be usable.

In addition, the Cursor object also provides several fetch style methods. These methods are used to return the results of a database query. The data returned is made up of a sequence of sequences (such as a tuple of tuples) where each inner sequence represents a single row returned by the SELECT statement. The fetch methods deﬁned by the standard are:

cursor.fetchone() Fetch the next row of a query result set, returning a single sequence, or None when no more data is available.
- cursor.fetchall() Fetch all (remaining) rows of a query result, returning them as a sequence of sequences.
- cursor.fetchman(size) Fetch the next set of rows of a query result, returning a sequence of sequences (e.g. a tuple of tuples). An empty sequence is returned when no more rows are available. The number of rows to fetch per call is speciﬁed by the parameter.

24.2.4 Mappings from Database Types to Python Types

The DB-API standard also speciﬁes a set of mappings from the types used in a database to the types used in Python. For a full listing see the DB-API standard itself but the key mappings include:

Date(year, month, day)	Represents a database date
Time(hour, minute, second)	Represents a time database value
Timestamp(year, month, day, hour, minute, second)	Holds a database time stamp value
String	Used to represent string like database data (such as VARCHARs)

24.2.5 Generating Errors

The standard also speciﬁes a set of Exceptions that can be thrown in different situations. These are presented below and in the following table:

The above diagram illustrates the inheritance hierarchy for the errors and warning associated with the standard. Note that the DB-API Warning and Error both extend the Exception class from standard Python; however, depending on the speciﬁc implementation there may be one or more additional classes in the hier- archy between these classes. For example, in the PyMySQL module there is a

MySQLError class that extends Exception and is then extended by both Warning and Error.

Also note that Warning and Error have no relationship with each other. This is because Warnings are not considered Errors and thus have a separate class hierarchies. However, the Error is the root class for all database Error classes.

A description of each Warning or Error class is provided below.

Warning	Used to warn of issues such as data truncations during inserting, etc.
Error	The base class of all other error exceptions
InterfaceError	Exception raised for errors that are related to the database interface rather than the database itself
DatabaseError	Exception raised for errors that are related to the database
DataError	Exception raised for errors that are due to problems with the data such as division by zero, numeric value out of range, etc.
OperationalError	Exception raised for errors that are related to the database’s operation and not necessarily under the control of the programmer, e.g. an unexpected disconnect occurs, etc.
IntegrityError	Exception raised when the relational integrity of the database is affected
InternalError	Exception raised when the database encounters an internal error, e.g. the cursor is not valid anymore, the transaction is out of sync, etc.
ProgrammingError	Exception raised for programming errors, e.g. table not found, syntax error in the SQL statement, wrong number of parameters speciﬁed, etc.
NotSupportedError	Exception raised in case a method or database API was used which is not supported by the database, e.g. requesting a .rollback() on a connection that does not support transactions or has transactions turned off

24.2.6 Row Descriptions

The Cursor object has an attribute description that provides a sequence of sequences; each sub sequence provides a description of one of the attributes of the data returned by a SELECT statement. The sequence describing the attribute is made up of up to seven items, these include:

name representing the name of the attribute,
type_code which indicates what Python type this attribute has been mapped to,
display_size the size used to display the attribute,
internal_size the size used internally to represent the value,

precision if a real numeric value the precision supported by the attribute,
- scale indicates the scale of the attribute,
- null_ok this indicates whether null values are acceptable for this attribute.

The ﬁrst two items (name and type_code) are mandatory, the other ﬁve are optional and are set to None if no meaningful values can be provided.

24.3 Transactions in PyMySQL

Transactions are managed in PyMySQL via the database connection object. This object provides the following method:

connection.commit() this causes the current transaction to commit all the changes made permanently to the database. A new transaction is then started.
connection.rollback() this causes all changes that have been made so far (but not permanently stored into the database i.e. Not committed) to be removed. A new transaction is then started.

The standard does not specify how a database interface should manage turning on and off transaction (not least because not all databases support transactions). However, MySQL does support transactions and can work in two modes; one supports the use of transactions as already described; the other uses an auto commit mode. In auto commit mode each command sent to the database (whether a SELECT statement or an INSERT/UPDATE statement) is treated as an independent transaction and any changes are automatically committed at the end of the state- ment. This auto commit mode can be turned on in PyMySQL using:

connection.autocommit(True) turn on autocommit (False to turn off auto commit which is the default).

Other associated methods include

connection.get_autocommit() which returns a boolean indicating whether auto commit is turned on or not.
connection.begin() to explicitly begin a new transaction.

Chapter 24

Python DB-API

Accessing a Database from Python
The standard for accessing a database in Python is the Python DB-API. This
specifies a set of standard interfaces for modules that wish to allow Python to access
a specific database. The standard is described in PEP 249 ,a PEP is a Python Enhancement Proposal.
Almost all Python database access modules adhere to this standard. This means
that if you are moving from one database to another, or attempting to port a Python
program from one database to another, then the APIs you encounter should be very
similar (although the SQL processed by different database can also differ). There are
modules available for most common databases such as MySQL, Oracle,
Microsoft SQL Server etc.
24.2 The DB-API
There are several key elements to the DB_API these are:

The connect function. The connect() function that is used to connect to a
database and returns a Connection Object.
Connection Objects. Within the DB-API access to a database is achieved
through connection objects. These connection objects provide access to cursor
objects.
Cursor objects are used to execute SQL statements on the database.
The result of an execution. These are the results that can be fetched as a
sequence of sequences (such a tuple of tuples). The standard can thus be used to
select, insert or update information in the database.

The standard specifies a set of functions and objects to be used to connect to a
database. These include the connection function, the Connection Object and the
Cursor object.
The above elements are described in more detail below.