Strings and Text

Introduction

Strings are just the data type for text. So far, you’ve used strings but without learning much about the details. Now it’s time to dive into them, learning what makes strings tick, and mastering some of the powerful string manipulation tools you have at your disposal.

This chapter has benefitted from the Python String Cook Book and Jake VanderPlas’ Python Data Science Handbook.

Note that there are more powerful methods for working with strings called regular expressions but these will be covered in a different chapter.

Creating Strings

We’ve created strings in passing earlier in the book, but didn’t discuss the details. First, you can create a string using either single quotes (') or double quotes ("). It’s good to be consistent in this, even if it doesn’t matter which you use, but automatic code formatters tend to prefer ". If you have a quote inside a string, use ' within it.

string_one = "This is a string"
string_two = (
    "If I want to include a 'quote' inside a string, I use double quotes on the outside"
)

Strings are of type str:

type(string_one)

Strings in Python can be indexed, so we can get certain characters out by using square brackets to say which positions we would like.

var = "banana"

var[:3]

The usual slicing tricks that apply to lists work for strings too, i.e. the positions you want to get can be retrieved using the var[start:stop:step] syntax. Here’s an example of getting every other character from the string starting from the 2nd position.

var[1::2]

Note that strings, like tuples such as (1, 2, 3) but unlike lists such as [1, 2, 3], are immutable. This means commands like var[0] = "B" will result in an error. If you want to change a single character, you will have to replace the entire string. In this example, the command to do that would be var = "Banana".

Like lists, you can find the length of a string using len():

len(var)

You can concatenate strings using the + operator:

string_one + ". " + string_two + "."

Note that we added extra characters so that the phrase made sense. Another way of achieving the same end that scales to many words or phrases more efficiently (if you have them in a list) is:

". ".join([string_one, string_two])

Three useful functions to know about are upper(), lower(), and title(). Let’s see what they do

var = "input TEXT"
var_list = [var.upper(), var.lower(), var.title()]
print(var_list)

Note that there are many built-in functions for using strings in Python, you can find a comprehensive list here.

Exercise

Reverse the string "gnirts desrever a si sihT" using indexing operations.

While we’re using print(), it has a few tricks. If we have a list, we can print out entries with a given separator:

print(*var_list, sep="; and \n")

(We’ll find out more about what ‘’ does shortly.) To turn variables of other kinds into strings, use the str() function, for example

(
    "A boolean is either "
    + str(True)
    + " or "
    + str(False)
    + ", there are only "
    + str(2)
    + " options."
)

In this example two boolean variables and one integer variable were converted to strings. str() generally makes an intelligent guess at how you’d like to convert your non-string type variable into a string type. You can pass a variable or a literal value to str().

f-strings

The example above is quite verbose. Another way of combining strings with variables is via f-strings. A simple f-string looks like this:

variable = 15.32399
print(f"You scored {variable}")

This is similar to calling str on variable and using + for concatenation but much shorter to write. You can add expressions to f-strings too:

print(f"You scored {variable**2}")

This also works with functions; after all **2 is just a function with its own special syntax.

In this example, the score number that came out had a lot of (probably) uninteresting decimal places. So how do we polish the printed output? You can pass more inforation to the f-string to get the output formatted just the way you want. Let’s say we wanted two decimal places and a sign (although you always write + in the formatting, the sign comes out as + or - depending on the value):

print(f"You scored {variable:+.2f}")

There are a whole range of formatting options for numbers as shown in the following table:

Number	Format	Output	Description
15.32347	{:.2f}	15.32	Format float 2 decimal places
15.32347	{:+.2f}	+15.32	Format float 2 decimal places with sign
-1	{:+.2f}	-1.00	Format float 2 decimal places with sign
15.32347	{:.0f}	15	Format float with no decimal places
3	{:0>2d}	03	Pad number with zeros (left padding, width 2)
3	{:*<4d}	3***	Pad number with *’s (right padding, width 4)
13	{:*<4d}	13**	Pad number with *’s (right padding, width 4)
1000000	{:,}	1,000,000	Number format with comma separator
0.25	{:.1%}	25.0%	Format percentage
1000000000	{:.2e}	1.00e+09	Exponent notation
12	{:10d}	12	Right aligned (default, width 10)
12	{:<10d}	12	Left aligned (width 10)
12	{:^10d}	12	Center aligned (width 10)

As well as using this page interactively through the Colab and Binder links at the top of the page, or downloading this page and using it on your own computer, you can play around with some of these options over at this link.

Special Characters and How to Escape Strings

Python has a string module that comes with some useful built-in strings and characters. For example

import string

string.punctuation

gives you all of the punctuation,

string.ascii_letters

returns all of the basic letters in the ‘ASCII’ encoding (with .ascii_lowercase and .ascii_uppercase variants), and

string.digits

gives you the numbers from 0 to 9. Finally, though less impressive visually, string.whitespace gives a string containing all of the different (there is more than one!) types of whitespace.

There are other special characters around; in fact, we already met the most famous of them: “” for new line. To actually print “” we have to ‘escape’ the backward slash by adding another backward slash:

print("Here is a \n new line")
print("Here is an \\n escaped new line ")

The table below shows the most important escape commands:

Code	Result
`\'`	Single Quote (useful if using `'` for strings)
`\"`	Double Quote (useful if using `"` for strings)
`\\`	Backslash
`\n`	New Line
`\r`	Carriage Return
`\t`	Tab

Here’s a more complicated example:

print("a\tb\nA\tB")

Raw Strings

Strings prefixed with r such as r’…’ and r”…” are called raw strings and treat backslashes as literal characters rather than special characters.

print(r"a\tb\nA\tB")

Cleaning Text

You often want to make changes to the text you’re working with. In this section, we’ll look at the various options to do this.

Replacing sub-strings

A common text task is to replace a substring within a longer string. Let’s say you have a string variable var. You can use .replace(old_text, new_text) to do this.

"Value is objective".replace("objective", "subjective")

As with any variable of a specific type (here, string), this would also work with variables:

text = "Value is objective"
old_substr = "objective"
new_substr = "subjective"
text.replace(old_substr, new_substr)

Note that .replace() performs an exact replace and so is case-sensitive.

Replacing characters with translate

A character is an individual entry within a string, like the ‘l’ in ‘equilibrium’. You can always count the number of characters in a string variable called var by using len(var). A very fast method for replacing individual characters in a string is str.translate().

Replacing characters is extremely useful in certain situations, most commonly when you wish to remote all punctuation prior to doing other text analysis. You can use the built-in string.punctuation for this.

Let’s see how to use it to remove all of the vowels from some text. With apologies to economist Lisa Cook, we’ll use the abstract from Cook (2011) as the text we’ll modify and we’ll first create a dictionary of translations of vowels to nothing, i.e. "".

example_text = "Much recent work has focused on the influence of social capital on innovative outcomes. Little research has been done on disadvantaged groups who were often restricted from participation in social networks that provide information necessary for invention and innovation. Unique new data on African American inventors and patentees between 1843 and 1930 permit an empirical investigation of the relation between social capital and economic outcomes. I find that African Americans used both traditional, i.e., occupation-based, and nontraditional, i.e., civic, networks to maximize inventive output and that laws constraining social-capital formation are most negatively correlated with economically important inventive activity."
vowels = "aeiou"
translation_dict = {x: "" for x in vowels}
translation_dict

Now we turn our dictionary into a string translator and apply it to our text:

translator = example_text.maketrans(translation_dict)
example_text.translate(translator)

Exercise

Use translate to replace all puncuation from the following sentence with spaces: “The well-known story I told at the conferences [about hypocondria] in Boston, New York, Philadelphia,…and Richmond went as follows: It amused people who knew Tommy to hear this; however, it distressed Suzi when Tommy (1982–2019) asked, "How can I find out who yelled, ‘Fire!’ in the theater?" and then didn’t wait to hear Missy give the answer—‘Dick Tracy.’”

Generally, str.translate is very fast at replacing individual characters in strings. But you can also do it using a list comprehension and a join() of the resulting list, like so:

"".join(
    [
        ch
        for ch in "Example. string. with- excess_ [punctuation]/,"
        if ch not in string.punctuation
    ]
)

Splitting strings

If you want to split a string at a certain position, there are two quick ways to do it. The first is to use indexing methods, which work well if you know at which position you want to split text, eg

"This is a sentence and we will split it at character 18"[:18]

Next up we can use the built-in split() function, which returns a list of places where a given sub-string occurs:

"This is a sentence. And another sentence. And a third sentence".split(".")

Note that the character used to split the string is removed from the resulting list of strings. Let’s see an example with a string used for splitting instead of a single character:

"This is a sentence. And another sentence. And a third sentence".split("sentence")

A useful extra function to know about is splitlines(), which splits a string at line breaks and returns the split parts as a list.

count and find

Let’s do some simple counting of words within text using str.count(). Let’s use the first verse of Elizabeth Bishop’s sestina ‘A Miracle for Breakfast’ for our text.

text = "At six o'clock we were waiting for coffee, \n waiting for coffee and the charitable crumb \n that was going to be served from a certain balcony \n --like kings of old, or like a miracle. \n It was still dark. One foot of the sun \n steadied itself on a long ripple in the river."
word = "coffee"
print(f'The word "{word}" appears {text.count(word)} times.')

Meanwhile, find() returns the position where a particular word or character occurs.

text.find(word)

We can check this using the number we get and some string indexing:

text[text.find(word) : text.find(word) + len(word)]

But this isn’t the only place where the word ‘coffee’ appears. If we want to find the last occurrence, it’s

text.rfind(word)

Working with Multiple Strings

We’ve seen how to work with individual strings. But often we want to work with a group of strings, otherwise known as a corpus, that is a collection of texts. It could be a collection of words, sentences, paragraphs, or some domain-based grouping (eg job descriptions). Just like any other Python object, you can put strings into a list (or other iterable).

And, fortunately, many of the methods that we have seen deployed on a single string can be straightforwardly scaled up to hundreds, thousands, or millions of strings using pandas or other tools. This scaling up is achieved via vectorisation, in analogy with going from a single value (a scalar) to multiple values in a list (a vector).

As a very minimal example, here is capitalisation of names vectorised using a list comprehension:

[name.capitalize() for name in ["ada", "adam", "elinor", "grace", "jean"]]

A pandas series can be used in place of a list. Let’s create the series first:

import pandas as pd

dfs = pd.Series(
    ["ada lovelace", "adam smith", "elinor ostrom", "grace hopper", "jean bartik"],
    dtype="string",
)
dfs

Now we use the syntax series.str.function to change the text series:

dfs.str.title()

If we had a data frame and not a series, the syntax would change to refer just to the column of interest like so:

df = pd.DataFrame(dfs, columns=["names"])
df["names"].str.title()

The table below shows a non-exhaustive list of the string methods that are available in pandas.

Function (preceded by `.str.`)	What it does
`len()`	Length of string.
`lower()`	Put string in lower case.
`upper()`	Put string in upper case.
`capitalize()`	Put string in leading upper case.
`swapcase()`	Swap cases in a string.
`translate()`	Returns a copy of the string in which each character has been mapped through a given translation table.
`ljust()`	Left pad a string (default is to pad with spaces)
`rjust()`	Right pad a string (default is to pad with spaces)
`center()`	Pad such that string appears in centre (default is to pad with spaces)
`zfill()`	Pad with zeros
`strip()`	Strip out leading and trailing whitespace
`rstrip()`	Strip out trailing whitespace
`lstrip()`	Strip out leading whitespace
`find()`	Return the lowest index in the data where a substring appears
`split()`	Split the string using a passed substring as the delimiter
`isupper()`	Check whether string is upper case
`isdigit()`	Check whether string is composed of digits
`islower()`	Check whether string is lower case
`startswith()`	Check whether string starts with a given sub-string

Regular expressions can also be scaled up with pandas. The below table shows vectorised regular expressions.

Function	What it does
`match()`	Call `re.match()` on each element, returning a boolean.
`extract()`	Call `re.match()` on each element, returning matched groups as strings.
`findall()`	Call `re.findall()` on each element
`replace()`	Replace occurrences of pattern with some other string
`contains()`	Call `re.search()` on each element, returning a boolean
`count()`	Count occurrences of pattern
`split()`	Equivalent to `str.split()`, but accepts regexes
`rsplit()`	Equivalent to `str.rsplit()`, but accepts regexes

Let’s see a couple of these in action. First, splitting on a given sub-string:

df["names"].str.split(" ")

It’s fairly common that you want to split out strings and save the results to new columns in your data frame. You can specify a (max) number of splits via the n= kwarg and you can get the columns using expand

df["names"].str.split(" ", n=2, expand=True)

Exercise

Using vectorised operations, create a new column with the index position where the first vowel occurs for each row in the names column.

Here’s an example of using a regex function with pandas:

df["names"].str.extract("(\w+)", expand=False)

There are a few more vectorised string operations that are useful.

Method	Description
`get()`	Index each element
`slice()`	Slice each element
`slice_replace()`	Replace slice in each element with passed value
`cat()`	Concatenate strings
`repeat()`	Repeat values
`normalize()`	Return Unicode form of string
`pad()`	Add whitespace to left, right, or both sides of strings
`wrap()`	Split long strings into lines with length less than a given width
`join()`	Join strings in each element of the Series with passed separator
`get_dummies()`	extract dummy variables as a data frame

The get() and slice() methods give access to elements of the lists returned by split(). Here’s an example that combines split() and get():

df["names"].str.split().str.get(-1)

If we have a column with tags split by a symbol, we can use the get_dummies() function to split it out. For example, let’s create a data frame with a single column that mixes subject and nationality tags:

df = pd.DataFrame(
    {
        "names": [
            "ada lovelace",
            "adam smith",
            "elinor ostrom",
            "grace hopper",
            "jean bartik",
        ],
        "tags": ["uk; cs", "uk; econ", "usa; econ", "usa; cs", "usa; cs"],
    }
)
df

If we now use str.get_dummies() and split on ; we can get a data frame of dummies.

df["tags"].str.get_dummies(";")

Reading Text In

Text file

If you have just a plain text file, you can read it in like so:

fname = 'book.txt'
with open(fname, encoding='utf-8') as f:
    text_of_book = f.read()

You can also read a text file directly into a pandas data frame using

df = pd.read_csv('book.txt', delimiter = "\n")

In the above, the delimiter for different rows of the data frame is set as “”, which means new line, but you could use whatever delimiter you prefer.

Exercise

Download the file ‘smith_won.txt’ using this link (use right-click and save as). Then read the text in using pandas.

CSV file

CSV files are already split into rows. By far the easiest way to read in csv files is using pandas,

df = pd.read_csv('book.csv')

Remember that pandas can read many other file types too.