Paginating your hapi api

February 10, 2017Gar

Most APIs will eventually run up against the problem of paginating data. Sending entire data sets in one request is simply too expensive for both client and server. I am going to show you an easy way to paginate your data in hapi, in a way that is easy to use in your client code.

hapi-pagination

To add pagination to your API, the first step is to install the hapi-pagination module. It will automatically add page and limit parameters to the routes you want paginated, and is very customizable to suit your particular needs.

Here's the config I ended up going with

{
    "meta": {
      "location": "header",
    },
    "routes": {
      "include": ["/cats"]
    }
  }
}

I set the location to header because this is the easiest way to add pagination to an existing API. If you were starting from scratch you could also design it to return pagination data as part of the response body. The header option will return links to the paginated data in the Link header, which I will explain in a moment.

route setup

Examples here will be using muckraker to query the database. It uses sql with parameter substitutions, and I think it should be easy enough to translate over to whatever you're using for database queries.

const server = new Hapi.Server(Config.hapi);
const Muckraker = require('muckraker');
const db = new Muckraker({/* put your db config here*/});
server.bind({ db }); //Allows route handlers to access the database through this.db

Let's start with a basic route that returns the contents of a database table

server.route({ method: 'GET', path: '/cats', config: {
  description: 'Pagination example',
  plugins: {
    pagination: {
      enabled: true
    }
  },
  validate: {
    query: {
      limit: Joi.number().default(10).min(1).max(100), //Set a sensible default and max page size
      page: Joi.number().positive() //Make sure they can't give a page number that would create a negative offset
    }
  },
  handler: function (request, reply) {

    const params = Object.assign({}, request.query);
    params.offset = (params.page - 1) * params.limit;
    const result = this.db.cats.count(params).then((cat_count) => {

      request.totalCount = cat_count.count;
      return this.db.cats.all(params);
    });

    return reply(result);
  }
});

Here are the sql files for the two database queries:

--cats_one_count.sql
select count(*)::integer as count from cats

--cats_all.sql
select * from cats order by name limit ${limit} offset ${offset}

Don't forget to order before you limit and offset or you'll get unpredictable results!

You could pass in more parameters to both queries if you wanted. Muckraker ignores extra parameters so you can send the same params to both queries. I like doing it that way because it lessens the likelihood I'm sending different where clauses to either query.

This is all you need to do at the most basic level to add pagination to your route.

client setup

The next step is adding pagination awareness to your client. What we will be doing is parsing the Link header and acting accordingly. There are a LOT of client side code patterns out there. I am going to try to keep the examples here a little bit abstract because I don't want to get bogged down in domain-specific approaches.

First, we need to parse the Link header when we parse the response from the server. I am using the approach that the node github library uses here

Add this somewhere in the code that parses the response from the API:

const link = response.headers.link; //Link header from the response, may look different in your framework
const links = {};

if (link) {
  link.replace(/<([^>]*)>;\s*rel="([\w]*)\"/g, function(m, uri, type) {
    links[type] = uri;
  });
}

this.links = links; //Set a links attribute on this collection, may look different in your framework

The idea is that now you have a links attribute you can check to see if there is more data to get from the server. If there is a next page then this.links.next will exist, and if there is a previous page this.links.prev will exist. You can use these attributes to determine whether or not to display next and prev navigation options to the user, or simply to query them to fetch all the data to the client at once.

For now let's go forward as if we wanted to pass the pagination on to the end user. I like to decorate my collections with a few methods for pagination like this.

next: function () {

  if (this.links.next) {
    this.fetch({ url: this.links.next, reset: true }); //Backbone style fetch, may look different in your framework
  }
},
prev: function () {

  if (this.links.prev) {
    this.fetch({ url: this.links.prev, reset: true });  //Backbone style fetch, may look different in your framework
  }
}

What's going on here is that we're checking the links object we built when we parsed the response from the server, and using it to fetch the next or previous page of data. You could also conditionally show pagination buttons based on the presence of this.links.next and this.links.prev

There is a lot more you could do with this, but we already have a very basic end-to-end pagination approach for both API and client.

&yet

The Blog