Webscraping and APIs

Introduction

This chapter will show you how to work with online data that is either obtained from webpages via webscraping or more directly over the internet via an API. An important principle is always to use an API if one is available as this is designed to pass information directly into your Python session and will save you a lot of effort.

Prerequisites

You will need to install the pandas and polars package for this chapter. We’ll use seaborn too, which you should already have installed. You will also need to install the beautifulsoup, pandas-datareader, and wbgapi packages in your terminal using uv add beautifulsoup4, and uv add wbgapi respectively. We’ll also use two built-in packages, textwrap and requests.

To kick off, let’s import some of the packages we need (it’s always good practice to import the packages you need at the top of a script or notebook).

import textwrap

import lets_plot as lp
import pandas as pd
import polars as pl
import requests
from bs4 import BeautifulSoup

Extracting Data from Files on the Internet using polars

It’s easy to read data from the internet once you have the url and file type. Here, for instance, is an example that reads in the ‘storms’ dataset, which is stored as a CSV file in a URL (we’ll only grab the first 10 rows):

pl.read_csv(
    "https://vincentarelbundock.github.io/Rdatasets/csv/dplyr/storms.csv", n_rows=10
)

Obtaining data using APIs

Using an API (application programming interface) is another way to draw down information from the interweb. Their just a way for one tool, say Python, to speak to another tool, say a server, and usefully exchange information. The classic use case would be to post a request for data that fits a certain query via an API and to get a download of that data back in return. (You should always preferentially use an API over webscraping a site.)

Because they are designed to work with any tool, you don’t actually need a programming language to interact with an API, it’s just a lot easier if you do.

Note

An API key is needed in order to access some APIs. Sometimes all you need to do is register with site, in other cases you may have to pay for access.

To see this, let’s directly use an API to get some time series data. We will make the call out to the internet using the requests package.

An API has an ‘endpoint’, the base url, and then a URL that encodes the question. Let’s see an example with the ONS API for which the endpoint is “https://api.beta.ons.gov.uk/v1/”. The rest of the API has the form ‘data?uri=’ and then the long ID of both the timeseries (jp9z) and then the dataset (LMS), which is vacancies in the UK services sector.

The data that are returned by APIs are typically in JSON format, which looks a lot like a nested Python dictionary and its entries can be accessed in the same way–this is what is happening when getting the series’ title in the example below. JSON is not good for analysis, so we’ll use polars to put the data into shape.

url = "https://api.beta.ons.gov.uk/v1/data?uri=/employmentandlabourmarket/peopleinwork/employmentandemployeetypes/timeseries/jp9z/lms/previous/v108"

# Get the data from the ONS API:
json_data = requests.get(url).json()

title = json_data["description"]["title"]

# Convert dates using string operations
df = (
    pl.DataFrame(json_data["months"])
    .with_columns(
        [
            # Add day to make it a valid date string
            (pl.col("date") + "-01").str.to_date(format="%Y %b-%d").alias("date"),
            pl.col("value").cast(pl.Float64).alias("value"),
        ]
    )
    .drop_nulls("date")
    .sort("date")
)


# Initialize the library
lp.LetsPlot.setup_html()

# Create plot using the alias
chart = (
    lp.ggplot(df, lp.aes(x="date", y="value"))
    + lp.geom_line(size=2.0, color="steelblue")
    + lp.ggtitle(title)
    + lp.ylim(0, df["value"].max() * 1.2)
    + lp.theme_classic()
)

chart

We’ve talked about reading APIs. You can also create your own to serve up data, models, whatever you like! This is an advanced topic and we won’t cover it; but if you do need to, the simplest way is to use Fast API. You can find some short video tutorials for Fast API here.

Accessing World Bank Data with wbgapi

While APIs can be accessed directly using tools like requests, some specialized libraries make working with structured datasets much easier. One such example is wbgapi, which provides a convenient interface for accessing World Bank data.

Let’s look at an example using World Bank data on CO₂-equivalent emissions per capita:

# World Bank CO2 equivalent emissions (metric tons per capita)
# https://data.worldbank.org/indicator/EN.GHG.ALL.PC.CE.AR5
# country and region codes at http://api.worldbank.org/v2/country
import wbgapi as wb

indicator_code = "EN.GHG.ALL.PC.CE.AR5"

df = (
    pl.from_pandas(
        wb.data.DataFrame(
            indicator_code,
            ["USA", "CHN", "IND", "EAS", "ECS"],
            time=range(2019, 2020),
            labels=True,
        ).reset_index()
    )
    .rename({"Country": "country"})
    .with_columns(pl.col("country").map_elements(lambda x: textwrap.fill(x, 10)))
    .sort(indicator_code, descending=True)
)

df.head()
lp.LetsPlot.setup_html()

country_order = df["country"].to_list()

plot = (
    lp.ggplot(df, lp.aes(x="country", y=indicator_code))
    + lp.geom_bar(lp.aes(fill="country"), color="black", alpha=0.8, stat="identity")
    + lp.scale_x_discrete(limits=country_order)
    + lp.scale_fill_discrete()
    + lp.theme_minimal()
    + lp.theme(legend_position="none")
    + lp.ggsize(600, 400)
    + lp.labs(
        subtitle="Greenhouse gases (CO2-equivalent metric tons per capita, 2019)",
        title="The USA leads the world on per-capita emissions",
        y="",
    )
)

plot.show()

The Eurostat SDMX API

Sometimes it’s convenient to use APIs directly. The Eurostat API provides access to a massive repository of European statistical data using the SDMX (Statistical Data and Metadata eXchange) standard. While Eurostat offers multiple formats, using the SDMX-ML (XML) format via the sdmx1 library allows us to pull structured data into the Python ecosystem with high precision.

Key to using the Eurostat API is understanding the Data Structure Definition (DSD). Every dataset is essentially a multidimensional “cube” where each dimension (like Geography, Unit, or Frequency) has specific codes.

To find the exact codes you need:

The Data Browser: Browse the Eurostat Data Navigation Tree. Once you find a table (e.g., “HICP - monthly data”), the “Dataset Code” (like prc_hicp_manr) is shown in brackets.

Positional Keys: Eurostat’s REST API expects a “key string” where codes are placed in a specific order separated by dots (e.g., Freq.Unit.Item.Geo). If you know the order, you can “slice” the data cube directly.

Let’s see an example of this in action. We want to see the Harmonised Index of Consumer Prices (HICP)—specifically the annual rate of change for all items—for Germany and France. We will use the resource prc_hicp_manr, requesting Monthly frequency (M), the Annual Rate of Change unit (RCH_A), and the “All-items” classification (CP00).

import sdmx
import polars as pl

# Tell sdmx we want ESTAT data
client = sdmx.Client('ESTAT')

# 2. Build the URL-style positional key
# Format: [Freq].[Unit].[Coicop].[Geo]
# We use '+' to join multiple countries (DE and FR)
resource_id = 'prc_hicp_manr'
key_string = 'M.RCH_A.CP00.DE+FR'

# 3. Fetch the data directly
# 'startPeriod' limits the timeline to recent data
response = client.data(
    resource_id=resource_id,
    key=key_string,
    params={'startPeriod': '2024-01'}
)

# 4. Convert the SDMX-ML response to a Polars DataFrame
# We bridge through Pandas as sdmx1 is optimized for it
df_pd = sdmx.to_pandas(response).to_frame(name='value').reset_index()
df = pl.from_pandas(df_pd)

print(df.head())
TIME_PERIOD geo unit freq coicop value
0 2024-01 DE RCH_A M CP00 3.1
1 2024-02 DE RCH_A M CP00 2.7
2 2024-03 DE RCH_A M CP00 2.3
3 2024-04 DE RCH_A M CP00 2.4
4 2024-05 DE RCH_A M CP00 2.8

Great that worked! We have data in a nice tidy format.

Other Useful APIs

  • There is a regularly updated list of APIs over at this public APIs repo on github. It doesn’t have an economics section (yet), but it has a LOT of other APIs.
  • Berkeley Library maintains a list of economics APIs that is well worth looking through.
  • NASDAQ Data Link, which has a great deal of financial data.
  • DBnomics: publicly-available economic data provided by national and international statistical institutions, but also by researchers and private companies.

Webscraping

Webscraping is a way of grabbing information from the internet that was intended to be displayed in a browser. But it should only be used as a last resort, and only then when permitted by the terms and conditions of a website.

If you’re getting data from the internet, it’s much better to use an API whenever you can: grabbing information in a structure way is exactly why APIs exist. APIs should also be more stable than websites, which may change frequently. Typically, if an organisation is happy for you to grab their data, they will have made an API expressly for that purpose. It’s pretty rare that there’s a major website which does permit webscraping but which doesn’t have an API; for these websites, if they don’t have an API, chances scraping is against their terms and conditions. Those terms and conditions may be enforceable by law (different rules in different countries here, and you really need legal advice if it’s not unambiguous as to whether you can scrape or not.)

There are other reasons why webscraping is not so good; for example, if you need a back-run then it might be offered through an API but not shown on the webpage. (Or it might not be available at all, in which case it’s best to get in touch with the organisation or check out WaybackMachine in case they took snapshots).

So this book is pretty down on webscraping as there’s almost always a better solution. However, there are times when it is useful.

If you do find yourself in a scraping situation, be really sure to check that’s legally allowed and also that you are not violating the website’s robots.txt rules: this is a special file on almost every website that sets out what’s fair play to crawl (conditional on legality) and what robots should not go poking around in.

In Python, you are spoiled for choice when it comes to webscraping. There are five very strong libraries that cover a real range of user styles and needs: requests, lxml, beautifulsoup, selenium, and *scrapy**.

For quick and simple webscraping, my usual combo would requests, which does little more than go and grab the HTML of a webpage, and beautifulsoup, which then helps you to navigate the structure of the page and pull out what you’re actually interested in. For dynamic webpages that use javascript rather than just HTML, you’ll need selenium. To scale up and hit thousands of webpages in an efficient way, you might try scrapy, which can work with the other tools and handle multiple sessions, and all other kinds of bells and whistles… it’s actually a “web scraping framework”.

It’s always helpful to see coding in practice, so that’s what we’ll do now, but note that we’ll be skipping over a lot of important detail such as user agents, being ‘polite’ with your scraping requests, being efficient with caching and crawling.

In lieu of a better example, let’s scrape the research page of http://aeturrell.com/

url = "http://aeturrell.com/research"
page = requests.get(url)
page.text[:300]

Okay, what just happened? We asked requests to grab the HTML of the webpage and then printed the first 300 characters of the text that it found.

Let’s now parse this into something humans can read (or can read more easily) using beautifulsoup:

soup = BeautifulSoup(page.text, "html.parser")
print(soup.prettify()[60000:60500])

Now we see more structure of the page and even some HTML tags such as ‘title’ and ‘link’. Now we come to the data extraction part: say we want to pull out every paragraph of text, we can use beautifulsoup to skim down the HTML structure and pull out only those parts with the paragraph tag (‘p’).

# Get all paragraphs
all_paras = soup.find_all("p")
# Just show one of the paras
all_paras[1]

Although this paragraph isn’t too bad, you can make this more readable by stripping out HTML tags altogether with the .text method:

all_paras[1].text

Now let’s say we didn’t care about most of the page, we only wanted to get hold of the names of projects. For this we need to identify the tag type of the element we’re interested in, in this case ‘div’, and it’s class type, in this case “project-name”. We do it like this (and show nice text in the process):

projects = soup.find_all("div", class_="project-content listing-pub-info")
projects = [x.text.strip() for x in projects]
projects

Hooray! We managed to get the information we wanted: all we needed to know was the right tags. A good tip for finding the tags of the info you want is to look at in your browser (eg Google Chrome) and then right-click on the bit you’re interested in, then hit ‘Inspect’. This will show you the HTML element of the bit of the page you clicked on.

That’s almost it for this very, very brief introduction to webscraping. We’ll just see one more thing: how to iterate over multiple pages.

Imagine we had a root webpage such as “www.codingforeconomists.com” which had subpages such as “www.codingforeconomists.com/page=1”, “www.codingforeconomists.com/page=2”, and so on. One need only iterate create the HTML strings to pass into a function that scrapes each one and return the relevant data, eg for the first 50 pages, and with a function called scraper(), one might run

start, stop = 0, 50
root_url = "www.codingforeconomists.com/page="
info_on_pages = [scraper(root_url + str(i)) for i in range(start, stop)]

That’s all we’ll cover here but remember we’ve barely scraped the surface of this big, complex topic. If you want to read about an application, it’s hard not to recommend the paper on webscraping that has undoubtedly change the world the most, and very likely has affected your own life in numerous ways: “The PageRank Citation Ranking: Bringing Order to the Web” by Page, Brin, Motwani and Winograd. For a more in-depth example of webscraping, check out realpython’s tutorial.

Webscraping Tables

There are times when you don’t need to scrape an entire webpage; you simply want the structured data from a specific table. While Polars is a high-performance data engine, it focuses on strict data formats (like Parquet or CSV) and does not natively include an HTML parser. However, we can easily bridge this gap by using Pandas to fetch the table and then converting it into a Polars DataFrame.

We will read data from ‘https://webscraper.io/test-sites/tables’ using pd.read_html(). This function scans the webpage and returns a list of all tables it finds as DataFrames. To target a specific table, we use the match= keyword argument with text that uniquely appears in the table we want—in this case, “First Name”.

Once captured, we convert the result to Polars using pl.from_pandas() to take advantage of Polars’ superior query performance and expression API.

import polars as pl

pd_list = pd.read_html("https://webscraper.io/test-sites/tables", match="First Name")
# Retrieve first entry from list of data frames
df = pl.from_pandas(pd_list[0])

print(df.head())

This gives us the table neatly loaded into a polars data frame ready for further use.

If you get a ‘403’ error, it means that the website has blocked pandas because it can see that you are engaged in web scraping. This is because some people web scrape irresponsibly, or because websites have provided other, preferred ways for you to obtain the data, eg via a download of the whole thing (think Wikipedia) or through an API. (If you really need to, you can often get around the 403 error though.)