Introduction​

The Archive.org Wayback Machine is pretty awe inspiring. It's been archiving web pages since 1996 and has amassed 284 billion page captures and over 15 petabytes of raw data. Many of these are sites that are no longer online and their content would have been otherwise lost to time. For sites that are still around, it can be absolutely fascinating to watch how they've evolved over the years.

Take Reddit for instance. You can go back in time and watch it grow from this...

to this...

to this...

with about 192 thousand other stops along the way. That's just absolutely incredible to me (if you agree then please consider donating to help them keep doing what they do).

That's all well-and-good but the thing about 192 thousand web captures, let alone 284 billion, is that that's just way too much for one person to sift through by hand. There's a lot of interesting data there, but if you want to actually do something with it then you'll need some sort of scraper to collect the data from the Wayback Machine.

What might one call such a thing? Hmmm, I dunno. Maybe a...

That's what I called my own reusable middleware and command-line utility at least (original, right?).

In this article, I'll walk you through the process of writing it using python and Scrapy. I should mention that scraping archived pages from the Wayback Machine isn't exactly a new idea- the official Scrapy docs list scraping cached copies of pages under "Common Practices"- but I'm going to try to put a little bit of a twist on it.

If all you wanted to do was fetch a current or historical snapshot then you could just write something like

def waybackify_url(url, closest_timestamp='2017'):
    return f'https://web.archive.org/web/{closest_timestamp}/{url}'

and "waybackify" your URLs before crawling them. That's great if you just want to avoid rate limits and bans but it sure doesn't make for much of a web scraping tutorial! Aside from that, it doesn't really help much if you're interested in how a page changes over time. For example, I wanted to look at all of the available Hacker News snapshots when I was writing the Reverse Engineering the Hacker News Ranking Algorithm.

The most obvious solution in cases like this is to write a spider that crawls the archive index pages

and extracts the timestamps from the URLs. That's definitely doable but it gets slightly more complicated than it seems at first and if you're putting this logic in your spider then chances are that reusing the code won't be trivial. Scraping time series data is a fairly general problem and so it would be nice if the code be reused with minimal modifications to spider code.

A more natural way to approach the problem is to write middleware that does the dirty work and integrates easily with existing code. I had basically already done this myself but hadn't quite cleaned it up enough that I would feel comfortable open sourcing it. After recently writing Advanced Web Scraping and receiving a lot of really positive feedback, I realized that others might find it informative if I did a little walkthrough of how I developed it and made the code available.

Just talking about middleware can get a little bit boring so I'll frame the discussion within the context of trying to analyze time series data from Reddit. The analysis there won't be particularly deep but there was something pretty interesting that popped out of the data. It should be more than enough of a starting point to do some internet archaeology of your own!

A Simple Reddit Spider​

Let's begin by throwing together a very simple spider that just grabs the titles and scores of the stories on the front page of Reddit. We'll do that right after we get the boilerplate out of the way by setting up a virtualenv, installing Scrapy, and scaffolding out a default Scrapy project.

mkdir ~/scrapers/reddit
cd ~/scrapers/reddit
virtualenv env
. env/bin/activate
pip install scrapy
scrapy startproject reddit_scraper
cd reddit_scraper

If any of that stuff doesn't make sense to you then you might want to go check out The Scrapy Tutorial or The Advanced Web Scraping Tutorial (though you can probably follow along fine just knowing that that sets up a project scaffold).

Now we can add a basic spider to reddit_scraper/spiders/reddit_spider.py

from datetime import datetime as dt
import scrapy

class RedditSpider(scrapy.Spider):
    name = 'reddit'

    def start_requests(self):
        yield scrapy.Request('http://reddit.com')

    def parse(self, response):
        items = []
        for div in response.css('div.sitetable div.thing'):
            try:
                title = div.css('p.title a::text').extract_first()
                votes_div = div.css('div.score.unvoted')
                votes = votes_div.css('::attr(title)').extract_first()
                votes = votes or votes_div.css('::text').extract_first()

                items.append({'title': title, 'votes': int(votes)})
            except:
                pass

        if len(items) > 0:
            timestamp = response.meta['wayback_machine_time'].timestamp()
            return {'timestamp': timestamp, 'items': items}

This spider is about as simple as they come: it starts at http://reddit.com and doesn't crawl anywhere else from there. It uses a few CSS selectors to pull out the title and votes for each story on the front page and then attaches a timestamp to them. If we run the spider with

scrapy crawl reddit -o snapshots.jl

then it will produce an uglified version of something like

{
  "timestamp": 1491171571.881031,
  "items": [{
      "title": "Evidence that WSJ used FAKE screenshots",
      "votes": 32459
    }, {
      "title": "Expand the canvas? [MOD APPROVED]",
      "votes": 16305
    }, {
      "title": "My sister's cat spazzing out in his new cat tree!",
      "votes": 15815
    }, etc.
  ]
}

in snapshots.jl.

To track changes over time, we could now set up a cron job to run our scraper at regular intervals. That's a great way to collect frequently changing data but- much like in the movie Primer- you can't go back further than when you first turned it on. That's where the Wayback Machine comes in.

The Wayback Machine is basically a much more complicated spider that is saving the entire HTML content of each snapshot. If we can feed the historical HTML snapshots into our spider and attach the correct timestamps then it will effectively be as though we were running our scraper at those points in time.

The point that I'm trying to make here isn't the obvious one that we can use the HTML snapshots to extract the historical data. It's that if we connect the snapshots to our spider in the right way then the spider should be none the wiser and things should just work. This is in contrast to the other approach we discussed where our spider would need to be aware of the archive index pages, urls, etc.

Developing the Middleware​

Writing a Scrapy Downloader Middleware is generally where you'll end up whenever you need to intercept requests and responses to modify or replace them. Downloader middleware classes implement process_request(request, spider) and process_response(request, response, spider) methods that have a lot of freedom in what they can do. Let's start piecing together the middleware in reddit_scraper/middlewares.py and it should hopefully become clear exactly how much you can accomplish with this freedom.

We'll first add the basic initialization that loads our WAYBACK_MACHINE_TIME_RANGE setting and saves the crawler.

import json
from datetime import datetime as dt

from scrapy import Request
from scrapy.http import Response
from scrapy.exceptions import IgnoreRequest

class UnhandledIgnoreRequest(IgnoreRequest):
    pass

class WaybackMachine:
    cdx_url_template = ('http://web.archive.org/cdx/search/cdx?url={url}'
                    '&output=json&fl=timestamp,original,statuscode,digest')
    snapshot_url_template = 'http://web.archive.org/web/{timestamp}id_/{original}'

    def __init__(self, crawler):
        self.crawler = crawler

        # read the settings
        self.time_range = crawler.settings.get('WAYBACK_MACHINE_TIME_RANGE')

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler)

There are a few other things in here but they aren't doing anything yet and we'll get to them shortly (so just ignore them for now). To actually turn this on, we'll have to also add a couple of settings to reddit_scraper/settings.py.

# enable the middleware
DOWNLOADER_MIDDLEWARES = {
    'reddit_scraper.middlewares.WaybackMachine': 50,
}

# only consider snapshots during the year of 2016
WAYBACK_MACHINE_TIME_RANGE = (20160101000000, 20170101000000)

# be bad but not too bad
ROBOTSTXT_OBEY = False
DOWNLOAD_DELAY = 5

Our middleware is now enabled but we need to implement request and response processing to actually make it useful. The request processing is the simpler of the two: we'll let any web.archive.org requests through without modification and for everything else we'll construct a request to the Wayback Machine's public CDX Server API.

    def process_request(self, request, spider):
        # let any web.archive.org requests pass through
        if request.url.find('http://web.archive.org/') == 0:
            return

        # otherwise request a CDX listing of available snapshots
        return self.build_cdx_request(request)

    def build_cdx_request(self, request):
        cdx_url = self.cdx_url_template.format(url=request.url)
        cdx_request = Request(cdx_url)
        cdx_request.meta['original_request'] = request
        cdx_request.meta['wayback_machine_cdx_request'] = True
        return cdx_request

Returning a new request aborts the current request processing and sends the new request into the downloader pipeline. The new request passes through our WaybackMachine middleware unscathed this time because the URL starts with http://web.archive.org/. It (hopefully) makes it all the way CDX server which will provide what is basically the computer-friendly version of the archive index pages. The CDX server will specifically return a JSON file including the timestamp, URL, and statuscode of each snapshot request as well as a hash of the snapshot content. That will look something like

[["timestamp","original","statuscode","digest"],
["20020718215101", "http://reddit.com:80/", "200", "VNG6YBPFVMBWJPRETPQX45QEHDHXFOFD"],
["20020802023739", "http://reddit.com:80/", "200", "VNG6YBPFVMBWJPRETPQX45QEHDHXFOFD"],
["20020923101504", "http://reddit.com:80/", "200", "VNG6YBPFVMBWJPRETPQX45QEHDHXFOFD"],
etc.
]

and, like all responses, will make its way back through the downloader middleware. Our spider wouldn't know what to do with it though so we need to intercept the response and prevent it from making it that far. We can do this by implementing process_response(request, response, spider).

    def process_response(self, request, response, spider):
        meta = request.meta

        # parse CDX requests and schedule future snapshot requests
        if meta.get('wayback_machine_cdx_request'):
            snapshot_requests = self.build_snapshot_requests(response, meta)

            # schedule all of the snapshots
            for snapshot_request in snapshot_requests:
                self.crawler.engine.schedule(snapshot_request, spider)

            # abort this request
            raise UnhandledIgnoreRequest

        # clean up snapshot responses
        if meta.get('original_request'):
            return response.replace(url=meta['original_request'].url)

        return response

We start out by grabbing the meta information on the request (you may have already noticed that we had attached some to our CDX request in build_cdx_request(request)). This meta information is then used to determine whether this is a response to a CDX request; if it is then we parse it to construct requests for the individual snapshots, schedule these with the Scrapy engine, and then abort the request by throwing an unhandled error that we defined earlier. The last little bit of code there is to make the response URL match that of the original request for the snapshot requests so that the spider doesn't have to know about where the snapshot responses actually came from.

The final piece of the puzzle is to implement the code for actually parsing the CDX responses and building the snapshot requests.

    def build_snapshot_requests(self, response, meta):
        # parse the CDX snapshot data
        data = json.loads(response.text)
        keys, rows = data[0], data[1:]
        def build_dict(row):
            new_dict = {}
            for i, key in enumerate(keys):
                new_dict[key] = row[i]
            return new_dict
        snapshots = list(map(build_dict, rows))

        # construct the requests
        snapshot_requests = []
        for snapshot in snapshots:
            # ignore snapshots outside of the time range
            if not (self.time_range[0] < int(snapshot['timestamp']) < self.time_range[1]):
                continue

            # update the url to point to the snapshot
            url = self.snapshot_url_template.format(**snapshot)
            original_request = meta['original_request']
            snapshot_request = original_request.replace(url=url)

            # attach extension specify metadata to the request
            snapshot_request.meta.update({
                'original_request': original_request,
                'wayback_machine_url': snapshot_request.url,
                'wayback_machine_time': dt.strptime(snapshot['timestamp'], '%Y%m%d%H%M%S'),
            })

            snapshot_requests.append(snapshot_request)

        return snapshot_requests

You can see here that our snapshot requests are built using original_request as a base. This means that they still have the original callbacks attached and are otherwise identical except for the extra meta data that we attach (and the temporarily different url property). Additionally, our snapshot_url_template uses a lesser known feature of the Wayback Machine API that allows us to get the original raw page content instead of the one with modified links and added content. After we switch response.url back to original_request.url in process_response(request, response, spider), the response will only be distinguishable from one coming from the original server in that it has the additional wayback_machine_url and wayback_machine_datetime meta data attached. This will all come in handy when it comes time to integrate the middleware with our spider as we'll do momentarily.

Putting It All Together​

We designed our middleware in such a way that pretty much any existing spider should "just work" and now we get to reap the benefits of that. No modifications of any generated requests are required and the only evidence that the responses were fetched from archive.org should be the additional meta data and some minor header differences. Indeed, we could run our scraper again now and it would successfully parse all of the available snapshots. The only problem is that the timestamps would be wrong because we're populating them with datetime.datetime.now().

To remedy this, we simply replace

        return {
            'timestamp': dt.now().timestamp(),
            'items': items,
        }

with

        return {
            'timestamp': response.meta['wayback_machine_time'].timestamp(),
            'items': items,
        }

in reddit_scraper/spiders/reddit_spider.py. Running the crawler with scrapy crawl reddit_scraper -o snapshots.jl should now yield items for every front page snapshot from 2016!

We now have the data in a nice structured format and can finally get to the fun part. Let's start with loading the JSON Lines file back into python

from datetime import datetime as dt
import json
import numpy as np

# load in the data
times, median_scores = [], []
with open('snapshots.jl', 'r') as f:
    for line in f:
        row = json.loads(line)
        scores= [item['votes'] for item in row['items']]
        time = dt.utcfromtimestamp(row['timestamp'])
        times.append(time)
        median_votes.append(np.median(scores))

# plot the data
fig = plt.figure(figsize=single_figsize)
ax = fig.add_subplot(1, 1, 1)
ax.plot(times, median_scores, 'o', ms=1)

We're computing the median scores here instead of the average scores in order to suppress the noise of unusually popular stories. The median score should be fairly representative of some convolution of Reddit site traffic and their scoring algorithm. Now let's take a look at how the median scores change over time by plotting them.

import matplotlib.pyplot as plt

# label the months
xticks = [dt(2016, i + 1, 18) for i in range(12)]
xticklabels = [date.strftime('%b') for date in xticks]
ax.set_xticklabels('')
ax.set_xticks(xticks, minor=True)
ax.set_xticklabels(xticklabels, minor=True)

# title and axis labels
ax.set_title('Reddit Front Page Stories - 2016')
ax.set_ylabel('Median Score')

# format and save it
fig.tight_layout()
fig.savefig('reddit-front-page-stories-2016.png')

Things are fairly stable with a small upwards trend until we hit December and the median scores go crazy! Let's zoom in a little bit so we can see more clearly when this happened.

It looks like the median scores abruptly doubled on December 7th and the variations within each day also became more pronounced. It's pretty clear that there was some algorithm change at Reddit that took place on that date. This was also a significant enough change that it would likely be noticed by regular users.

Let's Google 'Reddit vote scores "December 7th, 20016"' and see who else noticed it. One of the first results is Reddit overhauls upvote algorithm to thwart cheaters and show the site's true scale which links to a Reddit self-post by an admin named KeyserSosa from December 6th, 2016. Here is an abridged excerpt from that post.

In the 11 years that Reddit has been around, we've accumulated a lot of rules in our vote tallying as a way to mitigate cheating and brigading on posts and comments Here's a rough schematic of what the code looks like without revealing any trade secrets or compromising the integrity of the algorithm. Many of these rules are still quite useful, but there are a few whose primary impact has been to sometimes artificially deflate scores on the site.

...Very soon (think hours, not days), we're going to cut the scores over to be reflective of these new and updated tallies...

TL;DR voting is confusing, we cleaned up some outdated rules on voting, and we're updating the vote scores to be reflective of what they actually are. Scores are increasing by a lot.

And that's exactly what we saw in the data. The data lets us even go a bit further and see that the new rules were relatively constant from the 7th up until the 16th or 17th when the median scores seem to fluctuate between the 5k-20k range and the 30-40k range. It's hard to say exactly why this was happening from just the plots we've generated so far; maybe they were trying out new rule variations or maybe specific thresholds were being reached. If we were to dig in a little bit deeper and track individual story trajectories then we could probably make some more specific guesses (this is sadly outside the scope of this article however).

In addition to the scores, we also scraped the story titles. Let's cycle through all of the stories again and pick out the most highly rated ones.

class TopStories:
    def __init__(self, N=10):
        self.stories = []
        self.N = N

    def add_story(self, new_story):
        # update any existing story with the higher score
        for story in self.stories:
            if story['title'] == new_story['title']:
                story['votes'] = max(story['votes'], new_story['votes'])
                return

        # insert a story in it's proper position
        for i in range(len(self.stories))[::-1]:
            if new_story['votes'] > self.stories[i]['votes']:
                if i == 0 or self.stories[i - 1]['votes'] > new_story['votes']:
                    self.stories.insert(i, new_story)
                    if len(self.stories) > self.N:
                        self.stories.pop()
                    return
            else:
                break

        # otherwise add it to the end of necessary
        if len(self.stories) < self.N:
                self.stories.append(new_story)

# load in the data
top_stories = TopStories()
with open('snapshots.jl', 'r') as f:
    for line in f:
        row = json.loads(line)
        time = dt.utcfromtimestamp(row['timestamp'])
        for item in row['items']:
            top_stories.add_story(item)

Well, now I kind of wish that we had scraped the URLs too so I could include the links. Here's 1 dad reflex 2 children at least, it's pretty impressive.

There's a ton more that we could do here if we extracted a bit more data, but hopefully this is enough to give you a test for how easy it is to mess around with the data once we have it.

Wrap Up​

Well, I hope that you enjoyed our little foray into internet archaeology here. There are countless possibilities for what you can do with time series data that you scrape from the Wayback Machine and what we've done here barely scrapes the surface. You could use the data that we scraped here to apply the techniques developed in Reverse Engineering the Hacker News Ranking Algorithm to Reddit or you could scrape other sites to find historical product prices, review scores, or anything else you're curious about or need to run your business.

I would love to here about whatever analysis you might undertake so feel free to reach out at evan@intoli.com (that's doubly true if you're looking for someone to help your business solve their data needs!). If you do plan to actually use the middleware that we developed then please check out the full code on the Scrapy Wayback Machine GitHub repo. I skipped over some important error handling and edge cases to simplify the code a bit during this tutorial and you'll really want those in production. The repo contains those additions as well as a really useful command-line utility for scraping pages without custom parsing (which may be useful if you want to parse them in a language other than python).

Finally, please don't forget to donate to archive.org if you're scraping data from their servers. They provide an awesome public server and scraping consumes a lot of their resouces. Throwing them a few bucks goes a long way in helping them provide the services that they do!

Introduction​

I wouldn't really consider web scraping one of my hobbies or anything but I guess I sort of do a lot of it. It just seems like many of the things that I work on require me to get my hands on data that isn't available any other way. I need to do static analysis of games for Intoli and so I scrape the Google Play Store to find new ones and download the apks. The Pointy Ball extension requires aggregating fantasy football projections from various sites and the easiest way was to write a scraper. When I think about it, I've probably written about 40-50 scrapers. I'm not quite at the point where I'm lying to my family about how many terabytes of data I'm hoarding away... but I'm close.

I've tried out x-ray/cheerio, nokogiri, and a few others but I always come back to my personal favorite: scrapy. In my opinion, scrapy is an excellent piece of software. I don't throw such unequivocal praise around lightly but it feels incredibly intuitive and has a great learning curve.

You can read The Scrapy Tutorial and have your first scraper running within minutes. Then, when you need to do something more complicated, you'll most likely find that there's a built in and well documented way to do it. There's a lot of power built in but the framework is structured so that it stays out of your way until you need it. When you finally do need something that isn't there by default, say a Bloom filter for deduplication because you're visiting too many URLs to store in memory, then it's usually as simple as subclassing one of the components and making a few small changes. Everything just feels so easy and that's really a hallmark of good software design in my book.

I've toyed with the idea of writing an advanced scrapy tutorial for a while now. Something that would give me a chance to show off some of its extensibility while also addressing realistic challenges that come up in practice. As much as I've wanted to do this, I just wasn't able to get past the fact that it seemed like a decidely dick move to publish something that could conceivably result in someone's servers getting hammered with bot traffic.

I can sleep pretty well at night scraping sites that actively try to prevent scraping as long as I follow a few basic rules. Namely, I keep my request rate comparable to what it would be if I were browsing by hand and I don't do anything distasteful with the data. That makes running a scraper basically indistinguishable from collecting data manually in any ways that matter. Even if I were to personally follow these rules, it would still feel like a step too far to do a how-to guide for a specific site that people might actually want to scrape.

And so it remained just a vague idea in my head until I encountered a torrent site called Zipru. It has multiple mechanisms in place that require advanced scraping techniques but its robots.txt file allows scraping. Furthermore, there is no reason to scrape it. It has a public API that can be used to get all of the same data. If you're interested in getting torrent data then just use the API; it's great for that.

In the rest of this article, I'll walk you through writing a scraper that can handle captchas and various other challenges that we'll encounter on the Zipru site. The code won't work exactly as written because Zipru isn't a real site but the techniques employed are broadly applicable to real-world scraping and the code is otherwise complete. I'm going to assume that you have basic familiarity with python but I'll try to keep this accessible to someone with little to no knowledge of scrapy. If things are going too fast at first then take a few minutes to read The Scrapy Tutorial which covers the introductory stuff in much more depth.

Setting Up the Project​

We'll work within a virtualenv which lets us encapsulate our dependencies a bit. Let's start by setting up a virtualenv in ~/scrapers/zipru and installing scrapy.

mkdir ~/scrapers/zipru
cd ~/scrapers/zipru
virtualenv env
. env/bin/activate
pip install scrapy

The terminal that you ran those in will now be configured to use the local virtualenv. If you open another terminal then you'll need to run . ~/scrapers/zipru/env/bin/active again (otherwise you may get errors about commands or modules not being found).

You can now create a new project scaffold by running

scrapy startproject zipru_scraper

which will create the following directory structure.

└── zipru_scraper
    ├── zipru_scraper
    │   ├── __init__.py
    │   ├── items.py
    │   ├── middlewares.py
    │   ├── pipelines.py
    │   ├── settings.py
    │   └── spiders
    │       └── __init__.py
    └── scrapy.cfg

Most of these files aren't actually used at all by default, they just suggest a sane way to structure our code. From now on, you should think of ~/scrapers/zipru/zipru_scraper as the top-level directory of the project. That's where any scrapy commands should be run and is also the root of any relative paths.

Adding a Basic Spider​

We'll now need to add a spider in order to make our scraper actually do anything. A spider is the part of a scrapy scraper that handles parsing documents to find new URLs to scrape and data to extract. I'm going to lean pretty heavily on the default Spider implementation to minimize the amount of code that we'll have to write. Things might seem a little automagical here but much less so if you check out the documentation.

First, create a file named zipru_scraper/spiders/zipru_spider.py with the following contents.

import scrapy

class ZipruSpider(scrapy.Spider):
    name = 'zipru'
    start_urls = ['http://zipru.to/torrents.php?category=TV']

Our spider inherits from scrapy.Spider which provides a start_requests() method that will go through start_urls and use them to begin our search. We've provided a single URL in start_urls that points to the TV listings. They look something like this.

At the top there, you can see that there are links to other pages. We'll want our scraper to follow those links and parse them as well. To do that, we'll first need to identify the links and find out where they point.

The DOM inspector can be a huge help at this stage. If you were to right click on one of these page links and look at it in the inspector then you would see that the links to other listing pages look like this

<a href="/torrents.php?...page=2" title="page 2">2</a>
<a href="/torrents.php?...page=3" title="page 3">3</a>
<a href="/torrents.php?...page=4" title="page 4">4</a>

Next we'll need to construct selector expressions for these links. There are certain types of searches that seem like a better fit for either css or xpath selectors and so I generally tend to mix and chain them somewhat freely. I highly recommend learning xpath if you don't know it, but it's unfortunately a bit beyond the scope of this tutorial. I personally find it to be pretty indispensible for scraping, web UI testing, and even just web development in general. I'll stick with css selectors here though because they're probably more familiar to most people.

To select these page links we can look for <a> tags with "page" in the title using a[title ~= page] as a css selector. If you press ctrl-f in the DOM inspector then you'll find that you can use this css expression as a search query (this works for xpath too!). Doing so lets you cycle through and see all of the matches. This is a good way to check that an expression works but also isn't so vague that it matches other things unintentionally. Our page link selector satisfies both of those criteria.

To tell our spider how to find these other pages, we'll add a parse(response) method to ZipruSpider like so

    def parse(self, response):
        # proceed to other pages of the listings
        for page_url in response.css('a[title ~= page]::attr(href)').extract():
            page_url = response.urljoin(page_url)
            yield scrapy.Request(url=page_url, callback=self.parse)

When we start scraping, the URL that we added to start_urls will automatically be fetched and the response fed into this parse(response) method. Our code then finds all of the links to other listing pages and yields new requests which are attached to the same parse(response) callback. These requests will be turned into response objects and then fed back into parse(response) so long as the URLs haven't already been processed (thanks to the dupe filter).

Our scraper can already find and request all of the different listing pages but we still need to extract some actual data to make this useful. The torrent listings sit in a <table> with class="list2at" and then each individual listing is within a <tr> with class="lista2". Each of these rows in turn contains 8 <td> tags that correspond to "Category", "File", "Added", "Size", "Seeders", "Leechers", "Comments", and "Uploaders". It's probably easiest to just see the other details in code, so here's our updated parse(response) method.

    def parse(self, response):
        # proceed to other pages of the listings
        for page_url in response.xpath('//a[contains(@title, "page ")]/@href').extract():
            page_url = response.urljoin(page_url)
            yield scrapy.Request(url=page_url, callback=self.parse)

        # extract the torrent items
        for tr in response.css('table.lista2t tr.lista2'):
            tds = tr.css('td')
            link = tds[1].css('a')[0]
            yield {
                'title' : link.css('::attr(title)').extract_first(),
                'url' : response.urljoin(link.css('::attr(href)').extract_first()),
                'date' : tds[2].css('::text').extract_first(),
                'size' : tds[3].css('::text').extract_first(),
                'seeders': int(tds[4].css('::text').extract_first()),
                'leechers': int(tds[5].css('::text').extract_first()),
                'uploader': tds[7].css('::text').extract_first(),
            }

Our parse(response) method now also yields dictionaries which will automatically be differentiated from the requests based on their type. Each dictionary will be interpreted as an item and included as part of our scraper's data output.

We would be done right now if we were just scraping most websites. We could just run

scrapy crawl zipru -o torrents.jl

and a few minutes later we would have a nice JSON Lines formatted torrents.jl file with all of our torrent data. Instead we get this (along with a lot of other stuff)

[scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
[scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
[scrapy.core.engine] DEBUG: Crawled (403) <GET http://zipru.to/robots.txt> (referer: None) ['partial']
[scrapy.core.engine] DEBUG: Crawled (403) <GET http://zipru.to/torrents.php?category=TV> (referer: None) ['partial']
[scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 http://zipru.to/torrents.php?category=TV>: HTTP status code is not handled or not allowed
[scrapy.core.engine] INFO: Closing spider (finished)

Drats! We're going to have to be a little more clever to get our data that we could totally just get from the public API and would never actually scrape.

The Easy Problem​

Our first request gets a 403 response that's ignored and then everything shuts down because we only seeded the crawl with one URL. The same request works fine in a web browser, even in incognito mode with no session history, so this has to be caused by some difference in the request headers. We could use tcpdump to compare the headers of the two requests but there's a common culprit here that we should check first: the user agent.

Scrapy identifies as "Scrapy/1.3.3 (+http://scrapy.org)" by default and some servers might block this or even whitelist a limited number of user agents. You can find lists of the most common user agents online and using one of these is often enough to get around basic anti-scraping measures. Pick your favorite and then open up zipru_scraper/settings.py and replace

# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'zipru_scraper (+http://www.yourdomain.com)'

with

USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36'

You might notice that the default scrapy settings did a little bit of scrape-shaming there. Opinions differ on the matter but I personally think it's OK to identify as a common web browser if your scraper acts like somebody using a common web browser. So let's slow down the response rate a little bit by also adding

CONCURRENT_REQUESTS = 1
DOWNLOAD_DELAY = 5

which will create a somewhat realistic browsing pattern thanks to the AutoThrottle extension. Our scraper will also respect robots.txt by default so we're really on our best behavior.

Now running the scraper again with scrapy crawl zipru -o torrents.jl should produce

[scrapy.core.engine] DEBUG: Crawled (200) <GET http://zipru.to/robots.txt> (referer: None)
[scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET http://zipru.to/threat_defense.php?defense=1&r=78213556> from <GET http://zipru.to/torrents.php?category=TV>
[scrapy.core.engine] DEBUG: Crawled (200) <GET http://zipru.to/threat_defense.php?defense=1&r=78213556> (referer: None) ['partial']
[scrapy.core.engine] INFO: Closing spider (finished)

That's real progress! We got two 200 statuses and a 302 that the downloader middleware knew how to handle. Unfortunately, that 302 pointed us towards a somewhat ominous sounding threat_defense.php. Unsurprisingly, the spider found nothing good there and the crawl terminated.

Downloader Middleware​

It will be helpful to learn a bit about how requests and responses are handled in scrapy before we dig into the bigger problems that we're facing. When we created our basic spider, we produced scrapy.Request objects and then these were somehow turned into scrapy.Response objects corresponding to responses from the server. A big part of that "somehow" is downloader middleware.

Downloader middlewares inherit from scrapy.downloadermiddlewares.DownloaderMiddleware and implement both process_request(request, spider) and process_response(request, response, spider) methods. You can probably guess what those do from their names. There are actually a whole bunch of these middlewares enabled by default. Here's what the standard configuration looks like (you can of course disable things, add things, or rearrange things).

DOWNLOADER_MIDDLEWARES_BASE = {
    'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware': 100,
    'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware': 300,
    'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware': 350,
    'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware': 400,
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': 500,
    'scrapy.downloadermiddlewares.retry.RetryMiddleware': 550,
    'scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware': 560,
    'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware': 580,
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 590,
    'scrapy.downloadermiddlewares.redirect.RedirectMiddleware': 600,
    'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': 700,
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 750,
    'scrapy.downloadermiddlewares.stats.DownloaderStats': 850,
    'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware': 900,
}

As a request makes its way out to a server, it bubbles through the process_request(request, spider) method of each of these middlewares. This happens in sequential numerical order such that the RobotsTxtMiddleware processes the request first and the HttpCacheMiddleware processes it last. Then once a response has been generated it bubbles back through the process_response(request, response, spider) methods of any enabled middlewares. This happens in reverse order this time so the higher numbers are always closer to the server and the lower numbers are always closer to the spider.

One particularly simple middleware is the CookiesMiddleware. It basically checks the Set-Cookie header on incoming responses and persists the cookies. Then when a response is on its way out it sets the Cookie header appropriately so they're included on outgoing requests. It's a little more complicated than that because of expirations and stuff but you get the idea.

Another fairly basic one is the RedirectMiddleware which handles, wait for it... 3XX redirects. This one lets any non-3XX status code responses happily bubble through but what if there is a redirect? The only way that it can figure out how the server responds to the redirect URL is to create a new request, so that's exactly what it does. When the process_response(request, response, spider) method returns a request object instead of a response then the current response is dropped and everything starts over with the new request. That's how the RedirectMiddleware handles the redirects and it's a feature that we'll be using shortly.

If it was surprising at all to you that there are so many downloader middlewares enabled by default then you might be interested in checking out the Architecture Overview. There's actually kind of a lot of other stuff going on but, again, one of the great things about scrapy is that you don't have to know anything about most of it. Just like you didn't even need to know that downloader middlewares existed to write a functional spider, you don't need to know about these other parts to write a functional downloader middleware.

The Hard Problem(s)​

Getting back to our scraper, we found that we were being redirected to some threat_defense.php?defense=1&... URL instead of receiving the page that we were looking for. When we visit this page in the browser, we see something like this for a few seconds

before getting redirected to a threat_defense.php?defense=2&... page that looks more like this

A look at the source of the first page shows that there is some javascript code responsible for constructing a special redirect URL and also for manually constructing browser cookies. If we're going to get through this then we'll have to handle both of these tasks.

Then, of course, we also have to solve the captcha and submit the answer. If we happen to get it wrong then we sometimes redirect to another captcha page and other times we end up on a page that looks like this

where we need to click on the "Click here" link to start the whole redirect cycle over. Piece of cake, right?

All of our problems sort of stem from that initial 302 redirect and so a natural place to handle them is within a customized version of the redirect middleware. We want our middleware to act like the normal redirect middleware in all cases except for when there's a 302 to the threat_defense.php page. When it does encounter that special 302, we want it to bypass all of this threat defense stuff, attach the access cookies to the session, and finally re-request the original page. If we can pull that off then our spider doesn't have to know about any of this business and requests will "just work."

So open up zipru_scraper/middlewares.py and replace the contents with

import os, tempfile, time, sys, logging
logger = logging.getLogger(__name__)

import dryscrape
import pytesseract
from PIL import Image

from scrapy.downloadermiddlewares.redirect import RedirectMiddleware

class ThreatDefenceRedirectMiddleware(RedirectMiddleware):
    def _redirect(self, redirected, request, spider, reason):
        # act normally if this isn't a threat defense redirect
        if not self.is_threat_defense_url(redirected.url):
            return super()._redirect(redirected, request, spider, reason)

        logger.debug(f'Zipru threat defense triggered for {request.url}')
        request.cookies = self.bypass_threat_defense(redirected.url)
        request.dont_filter = True # prevents the original link being marked a dupe
        return request

    def is_threat_defense_url(self, url):
        return '://zipru.to/threat_defense.php' in url

You'll notice that we're subclassing RedirectMiddleware instead of DownloaderMiddleware directly. This allows us to reuse most of the built in redirect handling and insert our code into _redirect(redirected, request, spider, reason) which is only called from process_response(request, response, spider) once a redirect request has been constructed. We just defer to the super-class implementation here for standard redirects but the special threat defense redirects get handled differently. We haven't implemented bypass_threat_defense(url) yet but we can see that it should return the access cookies which will be attached to the original request and that the original request will then be reprocessed.

To enable our new middleware we'll need to add the following to zipru_scraper/settings.py.

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.redirect.RedirectMiddleware': None,
    'zipru_scraper.middlewares.ThreatDefenceRedirectMiddleware': 600,
}

This disables the default redirect middleware and plugs ours in at the exact same position in the middleware stack. We'll also have to install a few additional packages that we're importing but not actually using yet.

pip install dryscrape # headless webkit
pip install Pillow # image processing
pip install pytesseract # OCR

Note that all three of these are packages with external dependencies that pip can't handle. If you run into errors then you may need to visit the dryscrape, Pillow, and pytesseract installation guides to follow platform specific instructions.

Our middleware should be functioning in place of the standard redirect middleware behavior now; we just need to implement bypass_thread_defense(url). We could parse the javascript to get the variables that we need and recreate the logic in python but that seems pretty fragile and is a lot of work. Let's take the easier, though perhaps clunkier, approach of using a headless webkit instance. There are a few different options but I personally like dryscrape (which we already installed).

First off, let's initialize a dryscrape session in our middleware constructor.

    def __init__(self, settings):
        super().__init__(settings)

        # start xvfb to support headless scraping
        if 'linux' in sys.platform:
            dryscrape.start_xvfb()

        self.dryscrape_session = dryscrape.Session(base_url='http://zipru.to')

You can think of this session as a single browser tab that does all of the stuff that a browser would typically do (e.g. fetch external resources, execute scripts). We can navigate to new URLs in the tab, click on things, enter text into inputs, and all sorts of other things. Scrapy supports concurrent requests and item processing but the response processing is single threaded. This means that we can use this single dryscrape session without having to worry about being thread safe.

So now let's sketch out the basic logic of bypassing the threat defense.

    def bypass_threat_defense(self, url=None):
        # only navigate if any explicit url is provided
        if url:
            self.dryscrape_session.visit(url)

        # solve the captcha if there is one
        captcha_images = self.dryscrape_session.css('img[src *= captcha]')
        if len(captcha_images) > 0:
            return self.solve_captcha(captcha_images[0])

        # click on any explicit retry links
        retry_links = self.dryscrape_session.css('a[href *= threat_defense]')
        if len(retry_links) > 0:
            return self.bypass_threat_defense(retry_links[0].get_attr('href'))

        # otherwise, we're on a redirect page so wait for the redirect and try again
        self.wait_for_redirect()
        return self.bypass_threat_defense()

    def wait_for_redirect(self, url = None, wait = 0.1, timeout=10):
        url = url or self.dryscrape_session.url()
        for i in range(int(timeout//wait)):
            time.sleep(wait)
            if self.dryscrape_session.url() != url:
                return self.dryscrape_session.url()
        logger.error(f'Maybe {self.dryscrape_session.url()} isn\'t a redirect URL?')
        raise Exception('Timed out on the zipru redirect page.')

This handles all of the different cases that we encountered in the browser and does exactly what a human would do in each of them. The action taken at any given point only depends on the current page so this approach handles the variations in sequences somewhat gracefully.

The one last piece of the puzzle is to actually solve the captcha. There are captcha solving services out there with APIs that you can use in a pinch, but this captcha is simple enough that we can just solve it using OCR. Using pytesseract for the OCR, we can finally add our solve_captcha(img) method and complete the bypass_threat_defense() functionality.

    def solve_captcha(self, img, width=1280, height=800):
        # take a screenshot of the page
        self.dryscrape_session.set_viewport_size(width, height)
        filename = tempfile.mktemp('.png')
        self.dryscrape_session.render(filename, width, height)

        # inject javascript to find the bounds of the captcha
        js = 'document.querySelector("img[src *= captcha]").getBoundingClientRect()'
        rect = self.dryscrape_session.eval_script(js)
        box = (int(rect['left']), int(rect['top']), int(rect['right']), int(rect['bottom']))

        # solve the captcha in the screenshot
        image = Image.open(filename)
        os.unlink(filename)
        captcha_image = image.crop(box)
        captcha = pytesseract.image_to_string(captcha_image)
        logger.debug(f'Solved the Zipru captcha: "{captcha}"')

        # submit the captcha
        input = self.dryscrape_session.xpath('//input[@id = "solve_string"]')[0]
        input.set(captcha)
        button = self.dryscrape_session.xpath('//button[@id = "button_submit"]')[0]
        url = self.dryscrape_session.url()
        button.click()

        # try again if it we redirect to a threat defense URL
        if self.is_threat_defense_url(self.wait_for_redirect(url)):
            return self.bypass_threat_defense()

        # otherwise return the cookies as a dict
        cookies = {}
        for cookie_string in self.dryscrape_session.cookies():
            if 'domain=zipru.to' in cookie_string:
                key, value = cookie_string.split(';')[0].split('=')
                cookies[key] = value
        return cookies

You can see that if the captcha solving fails for some reason that this delegates back to the bypass_threat_defense() method. This grants us multiple captcha attempts where necessary because we can always keep bouncing around through the verification process until we get one right.

This should be enough to get our scraper working but instead it gets caught in an infinite loop.

[scrapy.core.engine] DEBUG: Crawled (200) <GET http://zipru.to/robots.txt> (referer: None)
[zipru_scraper.middlewares] DEBUG: Zipru threat defense triggered for http://zipru.to/torrents.php?category=TV
[zipru_scraper.middlewares] DEBUG: Solved the Zipru captcha: "UJM39"
[zipru_scraper.middlewares] DEBUG: Zipru threat defense triggered for http://zipru.to/torrents.php?category=TV
[zipru_scraper.middlewares] DEBUG: Solved the Zipru captcha: "TQ9OG"
[zipru_scraper.middlewares] DEBUG: Zipru threat defense triggered for http://zipru.to/torrents.php?category=TV
[zipru_scraper.middlewares] DEBUG: Solved the Zipru captcha: "KH9A8"
...

It at least looks like our middleware is successfully solving the captcha and then reissuing the request. The problem is that the new request is triggering the threat defense again. My first thought was that I had some bug in how I was parsing or attaching the cookies but I triple checked this and the code is fine. This is another of those "the only things that could possibly be different are the headers" situation.

The headers for scrapy and dryscrape are obviously both bypassing the initial filter that triggers 403 responses because we're not getting any 403 responses. This must somehow be caused by the fact that their headers are different. My guess is that one of the encrypted access cookies includes a hash of the complete headers and that a request will trigger the threat defense if it doesn't match. The intention here might be to help prevent somebody from just copying the cookies from their browser into a scraper but it also just adds one more thing that you need to get around.

So let's specify our headers explicitly in zipru_scraper/settings.py like so.

DEFAULT_REQUEST_HEADERS = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'User-Agent': USER_AGENT,
    'Connection': 'Keep-Alive',
    'Accept-Encoding': 'gzip, deflate',
    'Accept-Language': 'en-US,*',
}

Note that we're explicitly adding the User-Agent header here to USER_AGENT which we defined earlier. This was already being added automatically by the user agent middleware but having all of these in one place makes it easier to duplicate the headers in dryscrape. We can do that by modifying our ThreatDefenceRedirectMiddleware initializer like so.

def __init__(self, settings):
    super().__init__(settings)

    # start xvfb to support headless scraping
    if 'linux' in sys.platform:
        dryscrape.start_xvfb()

    self.dryscrape_session = dryscrape.Session(base_url='http://zipru.to')
    for key, value in settings['DEFAULT_REQUEST_HEADERS'].items():
        # seems to be a bug with how webkit-server handles accept-encoding
        if key.lower() != 'accept-encoding':
            self.dryscrape_session.set_header(key, value)

Now, when we run our scraper again with scrapy crawl zipru -o torrents.jl we see a steady stream of scraped items and our torrents.jl file records it all. We've successfully gotten around all of the threat defense mechanisms!

Wrap it Up​

We've walked through the process of writing a scraper that can overcome four distinct threat defense mechanisms:

1. User agent filtering.
2. Obfuscated javascript redirects.
3. Captchas.
4. Header consistency checks.

Our target website Zipru may have been fictional but these are all real anti-scraping techniques that you'll encounter on real sites. Hopefully you'll find the approach we took useful in your own scraping adventures.

My previous post accidentally had "(2010)" added to the title by a moderator and then users flagged the story because of this and it was automatically penalized after hitting a hidden flag threshold. It sounds like the other submissions were penalized due to excessive flagging and there has been some interesting discussion as to whether some users abuse this and possible solutions.

I published an article on Friday titled Reverse Engineering the Hacker News Ranking Algorithm and posted it on Hacker News. Now, I know that it's always a crapshoot whether or not something makes it to the front page, but I had a hunch that this article had what it takes. It was directly relevant to the Hacker News audience in multiple ways, told a fairly interesting analysis story, and provided source code and data to make it easy for readers to try things out themselves. At the very least, I figured that if I could get enough attention to initially get on to the front page then the article would be fairly well-received.

You can imagine then that I was pretty happy when the story popped onto the front page after 45 minutes or so. Traffic started flowing in and I was looking forward to having some substantial discussion in the comments section. Then all of a sudden it was just gone. It didn't just fall off the front page, it was like it completely disappeared from the rankings. A few minutes later @awsoutage left a comment noticing the same thing that I did

and, later on, so did @mos_basik

I have to admit that I found it a bit comforting that I wasn't the only one who thought this all seemed a bit fishy.

I was basically rolling in Hacker News data after working on my last article so it seemed pretty natural to take a quick look and see how the ranking of my post changed over time. The data backed up what @awsoutage, @mos_bsik, and I had all noticed.

The story made it to the front page, started getting upvoted at a fairly high rate, and then disappeared within minutes. For comparison, take a look at the trajectories of some more typical stories that made it to the front page.

These seem to make a lot more sense; stories rise up to some peak position and then slowly drift down through the back pages over the course of days until they're gone. Flagging and other factors affect these trajectories but they still fall off somewhat continuously. There's also clearly some sort of auto-penalty that kicks in once stories are 15 hours old and other minor things but nothing nearly as extreme as disappearing from the rankings entirely.

My guess here is that moderator intervention was responsible for my post's disappearance. It was on the front page for a grand total of 9 minutes, received 9 upvotes in that time, and then received an additional 11 upvotes after disappearing as people finished reading the article. Maybe "Reverse Engineering" in the title made a moderator worry that it was going to discuss how the spam prevention mechanisms worked? That's the only thing I can think of even though that would be an unfounded fear. I had purposely avoided analyzing the spam prevention mechanisms and limited the analysis to the components of the algorithm that were already publicly available from posts by @pg and the Arc source code releases.

In any case, this occurrence seemed like a natural opportunity to extend my previous analysis. I now strongly suspect that there's a mechanism for moderators to remove stories from the top stories section or, at the very least, attach a factor similar to the lightweight-factor* discussed in my last post that is so extreme that it effectively does the same thing.

The data signature of moderator intervention is relatively simple. Basically, if a story has a normal trajectory on the front page for a while and then instantaneously makes a huge jump downwards or disappears completely then it suggests that a moderator may have manually adjusted the story. To make this a little more concrete let's limit it to stories that got at least 10 votes, dropped from the front page by over 100 spots in less than 30 seconds, and weren't marked "dead", "dupe", or "flagged" (because those might organically result in the same behavior). Applying this filter to the data lets us pick out the trajectories of stories that were doing well before being severely penalized.

As you can see, my post has some company here. This has happened to 26 out of 1213 front page stories over the last couple of weeks or so. That's about 2.1% so it's not a particularly common occurrence but it is happening on a daily basis.

Now let's take a closer look at the stories that have moderator fingerprints on them. Are they spam, are they critical of Y Combinator companies, or what?

A few other posts seem a bit off-topic, some may be a bit light on content, but for the most part I wouldn't think twice if I saw any of these stories on the front page. I mean... they were all actually upvoted to the front page after all. Some of them are the types of stories that tend to invite flagging and might have been removed because of that (either manually or automatically).

This is most likely the case for the political posts. They're usually fairly contentious and the moderators have expressed distaste for an overabundance of them (e.g. the Political Detox Week attempted a few months ago). I wouldn't be surprised to see those removed, especially when the comment sections turn toxic (as they often do).

There are also a few articles that are thinly veiled affiliate spam. For example, the ShelfJoy links about books that Aaron Swartz and David Bowie loved fall into this category. They get called out in the comments for being very low-effort lists of Amazon affiliate links.

I have a hard time imagining others triggering any sort of hidden flag threshold. I obviously think that Reverse Engineering the Hacker News Ranking Algorithm falls into that category, but what about "Python Fire"? A new, useful open source project that was met with glowing praise in the comments? Take a look at the first sentences of some of the comments: "Convenient.", "Some libraries just seem obvious in retrospect.", "Looks great!", "Looks pretty good.", and "Looks fantastic.". That just really doesn't seem like a recipe for heavy flagging.

The "Investor to Airbnb CEO" post is the only one on the list that pertains to a Y Combinator company. I read through the comments to get a feel for the tone and how much controversy there was. As a whole, they're fairly critical of the CEO. Take a look at the top comment.

I find this to be a very respectful comment that highlights the positive role that Y Combinator plays in the startup community while also explaining why that translates into it having some responsibility for taking a stance against unethical behavior. That said, it's clearly critical of how Y Combinator has responded to controversies regarding the ethics of their companies.

There was another part of the comment that I found a bit chilling though: "I am looking forward to see[ing] constructive debate around this topic in the comments." Did a moderator really read that and then disappear the story? This might just be an unfortunate coincidence caused by some flagging threshold but the pattern looks strikingly similar to that of my own post. It was on the front page for about 20 minutes, was getting rapidly upvoted, and then suddenly dropped hundreds of positions in the rankings.

The penalty was then reversed shortly after but it was off the front page at that point. The submission never got a fraction of the attention that it would have received otherwise (though it did briefly rise back on to the front page after this).

I'm very curious to hear some other people's opinions on this. Were these stories removed automatically or was there moderator intervention? If you think it was the moderators then do their intentions make sense to you? I don't want to jinx things but... I am looking forward to seeing constructive debate around this topic in the comments.

Today happened to be the first day in my life that I have ever had even a shadow of a doubt about the truth of this statement. I woke up this morning and immediately started thinking about how much better life would be if we could all just agree to switch to Daylight Savings Time year round and let Standard Time go the way of Bramble Cay melomys. We would get an extra hour of daylight in the winter, I wouldn't have had to wake up an hour early, and, of course, so many lives would be saved.*

Then a thought occurred to me: this is a factoid that I picked up on the playground in elementary school. Not that that necessarily means it's wrong but the playground is also where I learned that gum stays in your stomach for seven years, the Great Wall of China is the only man-made structure visible from space, and that this:

is redeemable for a free Tootsie Pop (photo credit: regan76). I still haven't gotten over my disappointment after visiting the Great Wall.

In light of this, I decided to seek out a more trustworthy source and a quick googling found many articles supporting the claim. Their references painted a more blurry picture however. The effects of daylight and daylight saving time on US pedestrian fatalities and motor vehicle occupant fatalities is one of the most cited of papers on the topics. It concludes that 171 pedestrian fatalities and 195 vehicle occupant fatalities are caused each year by the time switch. These deaths are not caused by sleepy drivers in the spring though, they're caused by reduced sunlight in the winter.

This is apparently a popular topic in Finland as well where Daylight Saving Time Transitions and Road Traffic Accidents found no evidence of increased traffic accidents and another paper found no increase in work-related accidents. There are some papers that claim an effect, such as Fatal accidents following changes in daylight savings time: the American experience., but as that paper itself points out: many of these analyses have conflicting results.

At this point, I decided that I should do a little analysis of my own. I started looking around for US mortality datasets and quickly found that the data situation is complicated. There's an understandable concern about protecting individuals' privacy and so recent data tends to be either not publicly available or obfuscated in various ways so that identification is impossible. Unfortunately, removing the date of death is one of those ways.

The most recent usable data that I could get my hands on were the FMWK304 and GMWK305 mortality tables from the National Vital Statistics System. These include the total deaths per day from 1999-2007 segmented by cause of death. I focused on death by motor vehicle accidents for both occupants and pedestrians.

I aggregated the data across the available years to see the average number of deaths per day for the week before, the week of, and the week after the DST transition.

Then I considered the average of the week before and the week after as a baseline and subtracted this off from the week of the transition to find the excess deaths.

We would expect a large excess on Monday in particular if the original premise of tired commuters causing accidents were true. We do see an excess of about $6$ on Monday but the errors are so large that it's indistinguishable from zero.

The statistical uncertainties are even larger for the pedestrian deaths because they're less frequent

and the excess deaths

are again fairly indistinguishable from zero. There's actually a deficit observed on Monday but that's meaningless in the face of the large statistical error.

We can also aggregate the deaths over the entire week instead of dividing it up by day. This could possibly improve the signal to noise ratio if there are continued effects throughout the week. The excesses for the week of the transition are $33 \pm 51$ vehicle occupant deaths and $5.6 \pm 11$ pedestrian deaths. All that we could really do with these would be to put upper limits on the excesses; we can't exclude zero in either case.

So there you have it: there isn't really much evidence that the Daylight Savings Time transition results in any deaths due to tired driving. Is it weird that I'm a little disappointed? It's certainly possible, maybe even likely, that it does but the nine years of data that we looked at simply don't have the statistical resolving power to prove it. You could decrease the errors a little bit by coming up with a more sophisticated model for the baseline subtraction but that would only change the statistical errors by a factor of $\sqrt{\frac 2 3} \approx 0.82$. Another factor of $\sqrt{\frac{9}{37}} \approx 0.49$ could be achieved by getting access to the non-public National Death Index data stretching back to 1980, but I would be hesitant to use data any older than that because there have been tremendous gains in vehicle safety since then. That would knock the statistical errors on the excess vehicle occupant deaths for the week down to about $21$ which may or may not be enough to resolve a signal.

Overall, I would judge this factoid as much less apparently true than I expected it to be. It might be possible to discern a definitive signal with access to the non-public data but I have a feeling that it would still be fairly ambiguous. The statistical errors are such that the problem is highly susceptible to p-hacking and that's probably why you'll find papers using different model assumptions and coming to conflicting conclusions.

It might not be a killer but can we at least all agree that Standard Time sucks?

Introduction​

Articles occasionally pop up on Hacker News that analyze historical data relating to posts and comments on the site. Some of the analyses have been quite interesting but they almost universally focus on either basic metrics, content analysis, or how to get on the front page. Every time I read one of those articles, it always gets me wondering about what the same data could tell us about how Hacker News actually works.

There are a lot of questions one could ask but one of the most obvious is: what determines the position of stories on the front page? I find this to be a particularly interesting question, not because I actually care about the answer, but because it feels like the data should be able to tell us the answer. We could of course pick some different models and use the historical data to fit and validate them... but this isn't what I mean. I just have this feeling that the data can actually tell us the answer in a more direct way than global optimization.

What follows is an exploration of how we can use the data to learn about how the algorithm works. It shouldn't be confused with an attempt to find the best predictor for the front page rank, there are better ways to do that. My main goal was to tease out the ranking algorithm from the data in a simple and elegant fashion. This made it a little more interesting as an endeavor and hopefully makes it a more interesting read as well.

When You Make Assumptions​

Anybody who reads Hacker News with any regularity probably has some rough but intuitive understanding of how the ranking algorithm works. If you've ever seen a story and thought "jeez, this must have been flagged a lot to be this far down the page" then you were basically predicting where it should appear and then noticing a discrepancy from that. Here are a few examples of my guess-timations of where a story might appear on the front page.

I doubt that anybody is going to start calling my psychic hotline after seeing me make these guesses but it's actually pretty interesting that we can make even rough predictions. I think that this implies that we have some underlying beliefs about how the ranking algorithm works. Some of these might be things that we just feel must be true while others are more heavily supported by observations. When I approach a new problem, I like to start by clarifying any implicit assumptions and then seeing if I can use them to build a more structured approach.

One very general assumption that I made when coming up with those predictions was that the rank is primarily determined by a story's vote total and its age which we'll denote with $v$ and $t$ respectively. I know that there are contributions from flagging, @dang boosting stories, spam detection, etc. but I expect these to be somewhat secondary in most cases and we don't have access to their corresponding variables anyway. Another broad assumption is that $v$ and $t$ are used to calculate some numerical score and that the stories are ranked by sorting their scores. We'll represent this score function as $\mathcal{F}(t, v)$.

Having a general framework for how the ranking works, we can move to thinking about $\mathcal{F}(t, v)$ itself. Let's start with how the score changes with age. Here's how I classify some guesses regarding the age dependence of the score.

There are two main things I consider when judging these different curves. First off, it doesn't really make sense for the score to get higher as a story gets older; you don't see stories from a few months ago hopping back onto the front page for no reason. I'm assuming that the score must decrease as $t$ decreases and for a fixed $v$. Mathematically, this is equivalent to saying that the partial derivative $\frac{\partial \mathcal F}{\partial t} < 0$.

The second assumption is more subtle: small differences in age make less of a difference for large $t$. The difference between ages of one hour and eleven hours will have a huge impact on the score while the difference between one year and one year plus ten hours should be pretty negligible. This means that the score curves upwards as a function of $t$ and therefore $\frac{\partial^2 \mathcal F}{\partial t^2} > 0$.

We can apply similar reasoning to the dependence of $\mathcal F(t, v)$ on the number of votes $v$. I expect more votes to always mean a higher score so we can conclude that $\frac{\partial \mathcal F(t, v)}{\partial v} > 0$. My first guess would be that $\mathcal F(t, v)$ is proportional to $v$ but I could imagine somebody wanting to slightly suppress the growth for very large $v$ because stories that are on the front page for longer will get seen by more people and will therefore get even more votes. There's a multiplicative effect here and you could counteract this by introducing downwards curvature with something like $\mathcal F(t, v) \propto v^{\frac 1 2}$. Any functions that suppress this effect will have $\frac{\partial^2 \mathcal F}{\partial v^2} < 0$ while functions with $\frac{\partial^2 \mathcal F}{\partial v^2} > 0$ would actually exacerbate it. We'll assume that $\frac{\partial^2 \mathcal F}{\partial v^2} \le 0$ to allow for suppression but not enhancement of the effect.

I should probably mention now that these second derivative limits are a little bit over-simplified. It's easy to see that multiplying the score function by a positive constant won't affect the relative rank of stories but there are actually a much broader set of transformations that also won't affect it. Given a score function $\mathcal F(t, v)$, you can apply any monotonically increasing transformation and get out another score function that has the same lexicographical sorting order. So if $\mathcal F(t, v) \propto v$ then $\mathcal F(t, v)^2$ will have upward curvature as a function of $v$ and violate our assumption that it shouldn't. You can similarly apply transformations that would cause the $\frac{\partial^2 \mathcal F}{\partial t^2} > 0$ assumption to be violated.

What's really going on here is that those limits need to be constructed with appropriate scaling factors in order to be invariant under all allowable transformations. I'm tempted to explain this more but it would add a lot of complexity to the discussion without really changing anything in the end. I think that the best balance to strike here is to just be aware that there's a little more to the story but that there should exist a score function that still satisfies our constraints.

Working in Observations​

So to summarize the last section, these are what we're going to accept as absolute truth:

1. $\mathcal{F}(t, v)$ - The score function used to rank stories.
2. $\frac{\partial \mathcal F}{\partial t} < 0$ - It decreases over time.
3. $\frac{\partial^2 \mathcal F}{\partial t^2} > 0$ - It curves upwards over time.
4. $\frac{\partial \mathcal F}{\partial v} > 0$ - It increases with votes.
5. $\frac{\partial^2 \mathcal F}{\partial v^2} \le 0$ - It curves downwards with votes.

Now let's see what they can buy us.

Anything further is going to need to be tied to actual observations on how stories are ranked. Fortunately, this is something that we can easily observe by visiting the website. Say that we observe two stories on the front page and Story 1 appears in a better position than Story 2. This tells us that Story 1 has a higher score than Story 2 or, more formally, $\mathcal{F}(t\_1, v\_1) > \mathcal{F}(t\_2, v\_2)$.

We now need to work in our assumptions somehow in order to make this statement more useful. If you're a little rusty on your calculus then this chart might help to understand what we're about to do.

What's plotted here is $\mathcal F(t, v)$ for some value of $v$, essentially reducing it to a one-dimensional function. The red line shows the tangent curve which just barely kisses $\mathcal F(t, v)$ at $t=t\_1$ and has a slope of $\frac{\partial \mathcal F}{\partial t} \biggr\rvert\_{t=t\_{1}}$. The arrow points to $(t\_2, \mathcal F(t\_1, v) + (t\_2 - t\_1) \frac{\partial \mathcal F}{\partial t} \biggr\rvert\_{t=t\_{1}})$ and it's noted that this point is always below $(t\_2, \mathcal F(t\_2, v))$. This would not always be true if the score could curve downwards with time but this is forbidden by Assumption 2 and so we can safely conclude that $\mathcal F(t\_1, v) + (t\_2 - t\_1) \frac{\partial \mathcal F}{\partial t} \biggr\rvert\_{t=t\_1} < \mathcal F(t\_2, v)$. Plugging this back into our relationship between the scores of the two observed stories results in

$\mathcal F(t\_1, v\_1) > \mathcal F(t\_1, v\_2) + (t\_2 - t\_1) \frac{\partial \mathcal F}{\partial t} \biggr\rvert\_{t=t\_1,v=v\_2}$

A similar inequality for the extrapolation as a function of $v$ follows from Assumption 5: $\mathcal F(t, v\_2) + (v\_1 - v\_2) \frac{\partial \mathcal F}{\partial v} \biggr\rvert\_{v=v\_2} \ge \mathcal F(t, v\_1)$. The direction of the inequality is opposite here because $g(v)$ curves downwards instead of upwards but otherwise it's the exact same concept. We can now plug this into the left hand side to get

$\mathcal F(t\_1, v\_2) + (v\_1 - v\_2) \frac{\partial \mathcal F}{\partial v} \biggr\rvert\_{t=t\_1,v=v\_2} > \mathcal F(t\_1, v\_2) + (t\_2 - t\_1) \frac{\partial \mathcal F}{\partial t} \biggr\rvert\_{t=t\_1,v=v\_2}$

which simplifies to

$(v\_1 - v\_2) \frac{\partial \mathcal F}{\partial v} \biggr\rvert\_{t=t\_1,v=v\_2} > (t\_2 - t\_1) \frac{\partial \mathcal F}{\partial t} \biggr\rvert\_{t=t\_1,v=v\_2}$

OK, that's basically it! We now have an equation that translates our assumptions into a simple relationship that can be used to determine $\mathcal F(t, v)$. Each observation further constrains a differential equation that can then be solved to find $\mathcal F(t, v)$. This could be done directly but things will be much easier if we introduce an additional assumption here that the relative rate of score decay as a function of $t$ is independent of $v$. That makes the differential equation separable and so there exists a solution of the form $\mathcal F(t, v) = f(t)g(v)$. This will just make it easier to visualize and discuss, it's otherwise not a necessary assumption.

Substituting in this separable form of the score function gives us

$-\frac{f(t\_1)}{f^\prime(t\_1)} > \frac{(t\_2 - t\_1)}{(v\_2 - v\_1)} \frac{g(v\_2)}{g^\prime(v\_2)}$

when $v\_1 > v\_2$ or with a $<$ sign instead of the $>$ sign when $v\_2 > v\_1$. Note that we also implicitly used Assumptions 2 and 4 here to swap the equality sign appropriately when dividing or multiplying by negative quantities.

The general approach that we'll now take is to choose an ansatz, or reasonable guess, for $g(v)$ and then use that to determine an estimate for $f(t)$. We can then turn around and use this estimate of $f(t)$ to figure out a better guess for $g(v)$. If we repeat this process then things should hopefully settle down to satisfactory functions for $g(v)$ and $f(t)$. This lets us deal with differential equations of a single variable for simplicity.

Let's use $g(v)=v$ as our initial ansatz. It's simple, satisfies our assumptions, and is probably what I would have picked if I had programmed Hacker News. Given this ansatz we find that our inequality simplifies to

$-\frac{f(t\_1)}{f^\prime(t\_1)} > v_2\frac{(t\_2 - t\_1)}{(v\_2 - v\_1)}$

where the direction of the inequality again depends on whether or not $v\_1 > v\_2$. To avoid writing that left hand side over and over let's define $\tau(t) \equiv \frac{f(t)}{f\prime(t)}$ which then reduces our bound equation to

$\tau(t\_1) > v\_2\frac{(t\_2 - t\_1)}{(v\_2 - v\_1)}$

Let's take a minute to step back and see what this equation is really telling us. To simplify things a bit, we'll briefly consider the case of exponential decay $f(t)=e^{-t/\tau\_0}$ where larger values of $\tau\_0$ mean that stories fall off the front page more slowly. This isn't an assumption that we'll use in the analysis; we'll just use it to develop some intuition about what the bounds mean. We can then see that $\tau(t) = \frac{f(t)}{f\prime(t)} = \tau\_0$ so our inequalities are going to be directly constraining $\tau\_0$ regardless of $t$. The exact nature of the constraint is going to depend on whether $v\_1>v\_2$ and whether or not $t\_1>t\_2$.

If $v\_1 > v\_2$ then we'll be putting a lower bound on $\tau\_0$. If $t\_1$ is less than $t\_2$ then this bound will be negative which doesn't tell us much because $\tau\_0$ has to be positive for exponential decay. This makes a lot of sense because a newer story with more votes has to have a higher score given Assumptions 2 and 4. Seeing that just confirms our assumptions, it doesn't tell us anything further about $f(t)$. It gets more interesting when the top story is older than the lower story. That puts a positive lower bound on $\tau\_0$ which means that if old stories were to fall off more quickly than the bound allows then Story 1 would have already fallen below Story 2.

On the other hand, if $v\_1 < v\_2$ then we'll be putting an upper bound on $\tau\_0$. If $t\_1$ is less than $t\_2$ then this bound will be negative which would mean that $f^\prime(t) > 0$, violating Assumption 2. Indeed, if Assumptions 2 and 4 hold then we would expect to never see the higher scored story be both older and have less votes so this issue should never come up. $t\_1>t\_2$ is the more interesting case because it means that the top story is newer but has less votes. This means that Story 2 must have started falling off with a small enough value of $\tau\_0$ in order to come in second even though it has more votes.

Hopefully those examples make the bounds that we're calculating a little more intuitive. The lower bounds eliminate faster decays while the higher bounds eliminate slower ones. The decay doesn't need to be a constant value though, it can of course depend on $t$. If it does, then we'll need to compute this bounds for different values of $t$ to get a more complete picture.

The Good​

Now it's just a matter of actually crunching some data to see what we find. I put together a dataset of front page snapshots from 2007 to 2017 that includes the vote total, age, and relative position of each story to use for this analysis. The stories are limited to those that link to external URLs in case self-posts are treated differently. You can grab the dataset, as well as the analysis code, from this GitHub repository if you would like to play along at home.

Let's start out by just histogramming the upper and lower bounds that we observe. Each entry here is a bound on $\tau(t)$ obtained by evaluating the inequality that we derived earlier for two observed stories.

We can already see how these two histograms can work together to constrain $\tau(t)$. Let's combine them and plot the number of bounds that a given value of $\tau(t)$ would violate as a function of $t$.

The points on this plot show the value of $\tau(t)$ for each $t$ bin that violates the fewest bounds. The line represents the $\tau(t) \approx \tau\_0 + \tau\_1 t$ fit that minimizes the total number of bounds that are violated. To my eye, this seems like a pretty reasonable fit and I don't think that we could accurately constrain any higher order terms given the available data.

This differential equation described by this parameterization of $\tau(t)$ has a very simple solution

$f(t)=(\tau\_0 + \tau\_1 t)^{-\frac{1}{\tau\_1}}$

Interestingly, you can see that the limit of this quantity as $\tau\_1 \to 0$ is proportional to, and therefore equivalent to in terms of ranking, $e^{-t / \tau\_0}$. This is the function we played around with when trying to develop some intuition about what the bounds mean and we found that $\tau(t)$ would be constrained to a constant value of $\tau\_0$. We can think of the solution we found here as a generalization of exponential decay that adds another parameter to make the score fall off more slowly for larger values of $t$.

This plot of the bounds violations encompasses what I was talking about when I said that the data could tell us how the algorithm works. From just glancing at the figure, it's obvious that the score decays as a power law with age and you can even pull out rough estimates for the parameters by eye: $\tau\_0$ is about 2 hours and $\tau\_1$ is about 0.75. There's just something about that I find extremely satisfying.

Perhaps I'm getting ahead of myself a bit though... we said before that $g(v)=v$ was just a guess and that we would need to revisit it after we used it to constrain $f(t)$. Let's flip it around now and use our newly determined function $f(t)$ to constrain $\frac{g(v)}{g^\prime(v)}$ which we'll denote $\nu(v)$.

The line again represents a minimization of the total bound violations, this time parameterized as $\nu(v)=\nu\_0 + \nu\_1 v$. The fit parameters are $\nu\_0=-0.41 \pm 0.31$ and $\nu\_1=1.091 \pm 0.069$ where the errors are only statistical and were determined by bootstrapping the set of bounds on $\nu(v)$. These are both consistent with our ansatz of $\nu\_0=0$ and $\nu\_1=1$ which confirms that our ansatz was fairly accurate.

We could now try to iterate further using this solution but things are complicated by the fact that an arbitrary scaling factor can be applied to $\nu\_1$ and $\tau\_1$ without effecting the ranking properties of the score function. Unfortunately, the main thing that this would accomplish would be to let these parameters drift indiscriminately. If we had found non-zero curvature for $\nu(v)$ then we could have fixed the linear term to $1$ and then used the iterative process to determine the higher order terms. The only thing that we would be constraining in this linear case would be $\nu\_0$ and it doesn't seem like we have the resolving power to differentiate between $0$ and $-1$ so there isn't much point in doing that.

Ultimately we conclude that the score function, or at least an equivalent score function is

$\mathcal F(t, v) = \frac{v}{(\frac{\tau\_0}{\tau\_1} + t)^{\frac 1 \tau\_1}}$

with values of $\tau\_0 = 1.95 \pm 0.16$ and $\tau\_1 = 0.688 \pm 0.033$ extracted from the fit.

The Bad​

I mentioned in the previous section that I put together a dataset of stories from 2007-2017. What I didn't mention is that I was only showing you an analysis of data from 2007 and 2008. That's partially because I wanted you to see this approach succeed before seeing it fail but it's also because it's a little easier to understand this next figure after walking through the analysis for a single time period.

This figure shows the number of bound violations for different values of $\tau(t)$ within a window of $6 < t < 8$ for each three month period from mid 2007 until present. The columns are also normalized to give a consistent picture even though the amount of data varies over time.

It's clear that $\tau(t)$ dropped significantly around the first quarter of 2009 which tells us that the score function was adjusted so that stories would drop off more quickly. It seems like maybe there were changes around mid-2010 and mid-2011 as well but it's hard to tell exactly because the amount of data is also changing and this could just be shifts in the quantity of noise. What's unmistakable, however, is that our valley of $\tau(t)$ values with relatively few bound violations gets demolished in mid-2014. This sort of change is what we would expect if our assumptions get seriously broken (e.g. by heavy-duty vote fuzzing).

I just want to take a second to emphasize that it's really cool that the general approach we've taken allows us to visually see when these changes in the algorithm occur as well as their nature. That said, these algorithm changes unfortunately do mark a breakdown in some of our assumptions and that means that any results we pull out will become less accurate. Our approach is somewhat robust against factors like flagging and vote fuzzing: both of these will basically add noise to the upper and lower bound values which will cancel to leading order when determining the best values for $\tau(t)$ or $\nu(v)$. If additional factors become large enough that the lower and upper bounds overlap a lot though then the extracted fit parameters may be seriously biased. Just by eye, it's clear that this will be the case for mid-2014 through present and there's a good chance that it's also the case for 2009 through mid-2014.

Let's look briefly at what happens in the data after mid-2014. The $\tau(t)$ constraints actually aren't that bad...

You could almost imagine how deconvolving that image would give back our nice clean valley but it's not really possible to put too much faith in the extracted parameters. You can definitely rule out exponential behavior though and you can probably reasonably constrain $\tau\_1$ to within a factor of 50% or so.

Where things get really bad is when we use the extracted $\tau(t)$ function to constrain $\nu(v)$.

The extracted value for $\nu\_1$ is $0.52 \pm 0.01$ which isn't consistent with our ansatz at all (and the first order terms agreement can't be improved via iteration). What's most likely happening here is that there are just way more upper bounds than lower bounds and the blurring effect introduced by vote fuzzing, story promotion, etc. are causing these to overwhelm the lower bounds and push the preferred $\nu(v)$ down significantly. If $g(v)$ is still equal to $v$ then maybe we can kinda sorta use our $\tau(t)$ constraints to determine $\mathcal F(t, v)$, but we really can't do that with the same level of confidence that we did for the data from 2007-2008.

The situation for the 2009-2014 data has the same problem to a much, much lesser extent. This article already has more than enough figures so I won't show the plots here but they're included in the GitHub repository if you would like to see them. The extracted value for $\nu\_1$ here is $0.81 \pm 0.02$ which is much better than in the 2014-2017 but still demonstrates some violation of our assumptions. Despite that, the valley is clearly defined and you can much more clearly see how it was shifted down. I would actually be willing to somewhat trust the $\tau(t)$ parameterization in this case while also recognizing that there are definitely additional factors at play.

An interesting thing to note here is that both of these later time periods suggest $v\_0=-1$ rather than $0$. It's a little hard to trust this coming from data where our assumptions clearly break down but with the 2007-2009 data being inconclusive and the 2009-2014 data looking not-actually-that-bad then it at least suggests the possibility of $g(v)=v-1$ where the nitial automatic self-vote doesn't count.

(the ugly)​

So you've seen the good and you've seen the bad. Now it's time to see... code written in Arc. I kid 'cause I love.

Anyway, there exists a sort of Bizarro Hacker News out there where everyone really loves lisp. OK, I guess that that applies to the real Hacker News too... but everybody there specifically loves Arc. For those of you who aren't familiar: Arc is a dialect of Lisp developed by Paul Graham. The Arc Language Forum is a place for people to post and discuss stuff relating to Arc and it uses- or at least used at some point- basically the same code as Hacker News. More importantly, this code was included as part of the Arc distribution.

The first major Arc release was that of Arc 2 in February 2008 and it included the source for Hacker News in news.arc. The file is dated September 2006 though and presumably the file had been in use for the previous year before being released. The portion of the code relevant to the ranking algorithm is

; Ranking

; Votes divided by the age in hours to the gravityth power.
; Would be interesting to scale gravity in a slider.

(= gravity* 1.4 timebase* 120 front-threshold* 1)

(def frontpage-rank (s (o gravity gravity*))
  (/ (- (realscore s) 1)
     (expt (/ (+ (item-age s) timebase*) 60) gravity)))

(def realscore (i) (- i!score i!sockvotes))

(def item-age (i) (minutes-since i!time))

which, if you aren't familiar with lisp, might be more comprehensible rewritten in python as

def frontpage_rank(story, gravity=1.4, timebase=120):
    effective_score = story.score - story.sockvotes - 1
    return effective_score / ((timebase + story.age) / 60)**gravity)

This is also equivalent to a score function

$\mathcal F(t, v) = \frac{v - 1}{(\frac{\tau\_0}{\tau\_1} + t)^{\frac 1 \tau\_1}}$

where $\tau\_1 = \frac{1}{1.4} \approx 0.714$ and $\tau\_0 = \frac{120}{60}\tau\_1 = 1.43$. Note that the initial self-vote is subtracted off so the $\nu\_0=-1$ values that we got from the later time periods were likely correct (the early data was inconclusive). The vote total also subtracts off "sockvotes" which are presumably votes connected to spam accounts or vote rings. Other than that, our initial assumptions hold quite well and it's unsurprising that we had such success with the early data.

Now that we know what the actual score function was, we can compare it to what we extracted using our differential equation approach. To make things more interesting, let's also include the parameters extracted using global optimization over the data for two different cost functions. One is the Euclidean distance such that if the front page ordering is observed as $[1, 2, 3, 4]$ but predicted as $[1, 4, 3, 2]$ then the cost is $\sqrt{(1-1)^2 + (2-4)^2 + (3-3)^2 + (4-2)^2}=2$. The second is the Levenshtein distance applied to the same vectors. I won't go into the details here but the Levenshtein distance will basically punish a story the same amount for being a little or a lot out of order (e.g. $[4, 1, 2, 3]$ and $[1, 2, 4, 3]$ are equivalently bad predictions).

We can see that the differential equation approach, or Diff-EQ for short, outperforms both global optimizations in this case. All three methods over-estimate $\tau\_0$ but Diff-EQ least significantly and barely outside of what would be reasonably consistent given the statistical error. The methods all slightly underestimate $\tau\_1$ as well but Diff-EQ gives the closest estimate and is totally consistent given the statistical error. It's also worth emphasizing again that the Diff-EQ approach told us what functional form to use while the other methods tell us very little about whether different functional forms would perform better.

So the early days were all sunshine and rainbows but you'll remember that things started to go downhill a bit in early 2009. It turns out that Arc 3 was released in May 2009 lining up perfectly with where we predicted that the algorithm changes. The updated version of news.arc does indeed update the algorithm as follows.

(= gravity* 1.8 timebase* 120 front-threshold* 1
   nourl-factor* .4 lightweight-factor* .3 )

(def frontpage-rank (s (o scorefn realscore) (o gravity gravity*))
  (* (/ (let base (- (scorefn s) 1)
          (if (> base 0) (expt base .8) base))
        (expt (/ (+ (item-age s) timebase*) 60) gravity))
     (if (no (in s!type 'story 'poll))  .5
         (blank s!url)                  nourl-factor*
         (lightweight s)                (min lightweight-factor*
                                             (contro-factor s))
                                        (contro-factor s))))

(def contro-factor (s)
  (aif (check (visible-family nil s) [> _ 20])
    (min 1 (expt (/ (realscore s) it) 2))
    1))

(def realscore (i) (- i!score i!sockvotes))

(disktable lightweights* (+ newsdir* "lightweights"))

(def lightweight (s)
  (or s!dead
      (mem 'rally s!keys)  ; title is a rallying cry
      (mem 'image s!keys)  ; post is mainly image(s)
      (lightweights* (sitename s!url))
      (lightweight-url s!url)))

(defmemo lightweight-url (url)
  (in (downcase (last (tokens url #\.))) "png" "jpg" "jpeg"))

(def item-age (i) (minutes-since i!time))