Andrew B. Collier / @datawookie

Firefox version 146 is out! What updates are relevant to web scraping (specifically features that apply to Linux users)?

Chrome 143 includes a long list of updates. This post focuses on the changes that are relevant to web scraping in a dynamic, browser-driven environment (the kinds of sites where you rely on Playwright, Puppeteer or Selenium to execute JavaScript, interact with the DOM and extract rendered content). If you’re using simple HTTP requests to scrape static pages these changes won’t affect you. But for anyone running large-scale automation or dealing with modern, JavaScript-heavy websites, a few of the updates in Chrome 143 are worth paying attention to.

A common pattern when working with Camoufox in particular and Playwright in general is to use a context manager to handle the browser lifecycle. It’s clean, safe and ensures everything is torn down properly when your script completes. However, sometimes it’s not possible to wrestle all of your code into a context manager. There’s an alternative though.

It’s been a while since I last used the Tor Browser. So long, in fact, that it’s not even installed on my machine. Let’s fix that.

There’s a skulk of updates in the new 145 version of Firefox. I’ll only be looking at changes that are relevant to web scraping.

I’ve recently been given early access to a service that provides data on job listings published by a wide range of companies. The dataset offers a near real-time view of hiring activity, broken down at the company level. This is a potentially valuable signal for tracking labour market trends, gauging corporate growth or powering job intelligence tools.

Downloading content from SharePoint can be tricky. It might appear that a Microsoft login is required. You might attempt to automate the login process but run into other challenges.

If you are lucky though it might not be all that hard.

For the purpose of illustration I’ll document the process I went through for downloading a CSV document. The URL for the document is stored as URL in a module called const.py.

I’ve noticed that some of my scraper tests are significantly slower than others. On closer examination I discovered that most of the delay is being incurred when BeautifulSoup is parsing HTML. And a significant proportion of that time is spent checking character encoding.

The paper The Ten-Year Cycle in Numbers of the Lynx in Canada by Charles Elton and Mary Nicholson is a foundational work in population ecology and wildlife dynamics. They documented the regular, roughly ten-year population cycle of the Canadian Lynx (Lynx canadensis), observed in fur trade records from the Hudson’s Bay Company (HBC).

My scrapers often run in a serverless environment. If I’m using Camoufox then that needs to be baked into my Docker image too.

Playwright launches a browser. And browsers can be resource hungry beasts.

I often run Playwright on small, resource constrained virtual machines or in a serverless environment. These normally don’t have a lot of memory or disk space. Running out of either of these resources will cause Playwright (and potentially other processes) to fall over.

Is it possible to prune Playwright so that it plays better in a resource constrained environment? Let’s see.

The distances of Hasler kayak races for various divisions are nominally 4, 8 and 12 miles. However, the actual distances vary to some degree from one race venue to another. This makes it difficult to compare race times across different races. Using data from Paddle UK I attempt to estimate the actual distances.

Sometimes you’ll want to initiate a Selenium or Playwright session with an existing set of cookies. My approach to this is to retrieve those cookies using a browser and save them to a file so that I can easily load them into my script.

Sometimes a site will work fine with Selenium or Playwright until you try headless mode. Then it might fling up some anti-both mechanism. Or just stop responding altogether. Fortunately there are some simple things that you can do to work around this.

These are the approaches that I usually take.

Notes to self on using pytest.

TODO: Combine this with the “Starter” post.

I previously looked at the NetNut proxies. This post reviews the Webshare proxy service.

In the previous post we considered a few approaches to testing a Selenium web scraper. Now we’ll do the same for web scrapers using Playwright.

In previous posts we considered a few approaches for testing scrapers targeting static sites. Sometimes you won’t be able to get away with these static tools and you’ll be forced to use browser automation. In this post I’ll look at some options for testing a Selenium web scraper.

A common web crawler requirement is to iterate over a paginated list of links, following each link to retrieve detailed data. For example:

• iterating over a list of profiles on LinkedIn and then retrieving employment history from each profile;
• traversing a list of products on Amazon and then retrieving reviews for each product; or
• navigating the list of items on BBC News and retrieving the full articles.

What if your text data is contaminated with Unicode characters and HTML entities? Ideally you want your persisted data to be pristine. Metaphorically it should be prêt à manger (ready to eat). In principle I also want my text to be as simple as possible: ASCII characters, nothing else. This is sometimes achievable without the loss of too much information.

JSON-LD (JavaScript Object Notation for Linked Data) is a lightweight, flexible and standardised format intended to provide context and meaning to the data on a webpage. It’s easy and convenient for both humans and machines to read and write.

The previous post in this series considered the mocking capabilities in the unittest package. Now we’ll look at what it offers for patching.

Previous posts in this series used the responses and vcr packages to mock HTTP responses. Now we’re going to look at the capabilities for mocking in the unittest package, which is part of the Python Standard Library. Relative to responses and vcr this functionality is rather low-level. There’s more work required, but as a result there’s potential for greater control.

In the previous post I used the responses package to mock HTTP responses, producing tests that were quick and stable. Now I’ll look at an alternative approach to mocking using VCR.py.

As mentioned in the introduction to web scraper testing, unit tests should be self-contained and not involve direct access to the target website. The responses package allows you to easily mock the responses returned by a website, so it’s well suited to the job. The package is stable and well documented.

Site evolution. DOM drift. Selector decay. XPath extinction. Scraper rot. CAPTCHA catastrophe. Anti-bot apocalypse.

Inevitably even a carefully crafted web scraper will fail because the target site has changed in some way. Regular systematic testing is vital to ensure that you don’t lose valuable data.

The Zyte API implements session management, which makes it possible to emulate a browser session when interacting with a site via the API.

In a previous post I looked at various ways to use the Zyte API to retrieve web content. Now I’m going to delve into options for managing cookies via the Zyte API.

Zyte is a data extraction platform, useful for web scraping and data processing at scale. It’s intended to simplify data collection and, based on my experience certainly does!

Quick notes on the process of installing the CPLEX optimiser.

Quick notes on the process of installing the MOSEK optimiser.

Pyomo is another flexible Open Source optimisation modelling language for Python. It can be used to define, solve, and analyse a wide range of optimisation problems, including Linear Programming (LP) and Mixed-Integer Programming (MIP), nonlinear programming (NLP), and differential equations.

📢 The book Hands-On Mathematical Optimization with Python (available free online) is an excellent resource on optimisation with Python and Pyomo.

CVXPY is a powerful, Open Source optimization modelling library for Python. It provides an interface for defining, solving, and analysing a wide range of convex optimization problems, including Linear Programming (LP), Quadratic Programming (QP), Second-Order Cone Programming (SOCP), and Semidefinite Programming (SDP).

SciPy is a general-purpose scientific computing library for Python, with an optimize module for optimisation.

We will be considering two types of optimisation problems: sequential optimisation and global optimisation. These approaches can be applied to the same problem but will generally yield distinctly different results. Depending on your objective one or the other might be the best fit for your problem.

I’m evaluating optimisation systems for application to a large scale solar energy optimisation project. My primary concerns are with efficiency, flexibility and usability. Ideally I’d like to evaluate all of them on a single, well defined problem. And, furthermore, that problem should at least resemble the solar energy project.

In a previous post I looked at the HTTP request headers used to manage browser caching. In this post I’ll look at a real world example. It’s a rather deep dive into something that’s actually quite simple. However, I find it helpful for my understanding to pick things apart and understand how all of the components fit together.

In this post I’ll be testing the proxy service provided by NetNut. For a bit of context take a look at my What is a Proxy? post.

A proxy is a server or software that acts as an intermediary between a client (often a web browser) and one or more servers, typically on the internet. Proxies are used for a variety of purposes, including improving security, enhancing privacy, managing network traffic, and bypassing restrictions.

I recently migrated this blog from GitLab Pages to Vercel. There were two main reasons for the move:

1. The blog was taking too long to build on GitLab Pages, which hindered efficient updates and added unnecessary delays to my workflow. Admittedly, this was partially my own doing since my build process was far too complicated.
2. I want to have greater control over redirects (specifically the ability to redirect URLs that didn’t end in a slash to ones that did, which was apparently important for SEO purposes).

For a side project I needed to scrape data for the NYSE Composite Index going back as far as possible.

In a previous post I looked at retrieving a list of assets from the Alpaca API using the {alpacar} R package. Now we’ll explore how to retrieve historical and current price data.

How to list assets available to trade via the Alpaca API using the {alpacar} R package.

The {alpacar} package for R is a wrapper around the Alpaca API. API documentation can be found here. In this introductory post I show how to install and load the package, then authenticate with the API and retrieve account information.

A few days ago I wrote about a scraper for gathering economic calendar data. Well, I’m back again to write about another aspect of the same project: acquiring earnings calendar data.

Avoiding data duplication is a persistent challenge with acquiring data from websites or APIs. You can try to brute force it: pull the data again and then compare it locally to establish whether it’s fresh or stale. But there are other approaches that, if supported, can make this a lot simpler.

If you use Selenium for browser automation then at some stage you are likely to need to download a file by clicking a button or link on a website. Sometimes this just works. Other times it doesn’t.

I needed an offline copy of an economic calendar with all of the major international economic events. After grubbing around the internet I found the Economic Calendar on Myfxbook which had everything that I needed.

A few months ago I listened to an episode on the Founder’s Journal podcast that reviewed an essay, The Opportunity Cost of Everything, by Jack Raines. If you haven’t read it, then I suggest you invest 10 minutes in doing so. It will be time well spent.