# Scraping with Puppeteer

> Learn web scraping with Puppeteer by building a JavaScript job board that scrapes remote jobs from a site and stores them in a database to display later.

Author: [Flavio Copes](https://flaviocopes.com/about/) | Published: 2021-08-22 | Topics: [Tools](https://flaviocopes.com/tags/tools/) | Canonical: https://flaviocopes.com/puppeteer-scraping/

In this post we're going to create a "[JavaScript](https://flaviocopes.com/javascript/) Job Board" that aggregates remote jobs for JavaScript developers.

Here are the steps to complete our project:

1. Create a [Node.js](https://flaviocopes.com/nodejs/) scraper built using [Puppeteer](https://flaviocopes.com/puppeteer/) that fetches jobs from the remoteok.io website
2. Store the jobs into a database
3. Create a Node.js application to display those jobs on our own website

> Word of caution: I am using this website just as an example. I am not recommending you to scrape it because it has an official API if you want to use its data. It's just to explain how Puppeteer works with a website that everybody knows, and to show you how you can use it in practice.

Let's go!

## Create a scraper for JavaScript jobs
We're going to scrape JavaScript jobs from remoteok.io, a great remote jobs website.

The website hosts many different kinds of jobs. JavaScript jobs are listed under the "JavaScript" tag, and they are all available, at the time of writing, in this page: https://remoteok.io/remote-javascript-jobs

I said "at the time of writing" because this is an important realization: the website might change at any time. We are not guaranteed anything. With scraping, any change to the website might halt our application from working. It's not an API, which is sort of a contract between 2 parties.

So, from my experience, scraping applications require more maintenance. But sometimes we have no other choice to do a specific task, so they are still a valid tool at our disposal.

## Set up Puppeteer
Let's start by creating a new folder, and inside the folder run

```bash
npm init -y
```

Then install Puppeteer using

```bash
npm install puppeteer
```

Now create an `app.js` file. At the top, require the `puppeteer` library we just installed:

```js
const puppeteer = require("puppeteer")
```

then we can use the `launch()` method to create a browser instance:

```js
;(async () => {
  const browser = await puppeteer.launch({ headless: false })
})()
```

We passed the `{ headless: false }` configuration object to show Chrome while Puppeteer is performing its operations, so we can see what's happening, which is helpful while building the application.

Next we can use the `newPage()` method on the `browser` object to get the `page` object, and we call the `goto()` method on the `page` object to load the JavaScript jobs page:

```js
const puppeteer = require('puppeteer')

;(async () => {
  const browser = await puppeteer.launch()
  const page = await browser.newPage()
  await page.goto("https://remoteok.io/remote-javascript-jobs
")
})()
```

Now run `node app.js` from the terminal, and a Chromium instance will start, loading the page we told it to load:

![Chromium browser window showing RemoteOK JavaScript jobs page loaded by Puppeteer with automated testing notification](https://flaviocopes.com/images/puppeteer-scraping/Screen_Shot_2020-09-06_at_08.56.43.png)

## Get the jobs from the page
Now we need to figure out a way to get the jobs details from the page.

To do so, we're goingt o use the `page.evaluate()` function that Puppeteer gives us.

Inside its callback function we basically transition into the browser, so we can use the `document` object that now points to the page DOM, even though the code will run in a Node.js environment. That's a bit of a magic part performed by Puppeteer.

Inside this callback function we can't print anything to the console, because that would be printed to the browser console, not the Node.js terminal.

What we can do is we can return an object from it, so we have access to it as the value returned by `page.evaluate()`:

```js
const puppeteer = require('puppeteer')

;(async () => {
  const browser = await puppeteer.launch({ headless: false })
  const page = await browser.newPage()
  await page.goto('https://remoteok.io/remote-javascript-jobs')

  /* Run javascript inside the page */
  const data = await page.evaluate(() => {
    return ....something...
  })

  console.log(data)
  await browser.close()
})()
```

Inside that function we first create an empty array, that we'll fill with the values we want to return.

We find each job, which is wrapped in a `tr` HTML element with the `job` class, then we get data from each job using `querySelector()` and `getAttribute()`:

```js
/* Run javascript inside the page */
const data = await page.evaluate(() => {
  const list = []
  const items = document.querySelectorAll("tr.job")

  for (const item of items) {
    list.push({
      company: item.querySelector(".company h3").innerHTML,
      position: item.querySelector(".company h2").innerHTML,
      link: "https://remoteok.io" + item.getAttribute("data-href"),
    })
  }

  return list
})
```

I found which were the exact selectors to be used by looking at the page source using the browser devtools:

![Browser developer tools showing HTML structure of job listings with tr.job elements and CSS selectors highlighted](https://flaviocopes.com/images/puppeteer-scraping/Screen_Shot_2020-09-06_at_10.37.04.png)

Here's the full source code:

```js
const puppeteer = require("puppeteer")

;(async () => {
  const browser = await puppeteer.launch({ headless: false })
  const page = await browser.newPage()
  await page.goto("https://remoteok.io/remote-javascript-jobs")

  /* Run javascript inside the page */
  const data = await page.evaluate(() => {
    const list = []
    const items = document.querySelectorAll("tr.job")

    for (const item of items) {
      list.push({
        company: item.querySelector(".company h3").innerHTML,
        position: item.querySelector(".company h2").innerHTML,
        link: "https://remoteok.io" + item.getAttribute("data-href"),
      })
    }

    return list
  })

  console.log(data)
  await browser.close()
})()
```

If you run this, you will get back an array of objects, each containing the job details:

![Terminal output showing scraped job data as JavaScript objects with company, position, and link properties](https://flaviocopes.com/images/puppeteer-scraping/Screen_Shot_2020-09-06_at_10.35.28.png)

## Store jobs in the database
Now we're ready to store this data into a local database.

We will run the Puppeteer script once in a while, removing all the jobs stored, first, and then re-filling the database with the new ones found.

We'll use [MongoDB](https://flaviocopes.com/mongodb/). From the terminal, run:

```bash
npm install mongodb
```

Then in `app.js` we add this logic to initialize the `jobs` database, and a `jobs` collection inside it:

```js
const puppeteer = require("puppeteer")
const mongo = require("mongodb").MongoClient

const url = "mongodb://localhost:27017"
let db, jobs

mongo.connect(
  url,
  {
    useNewUrlParser: true,
    useUnifiedTopology: true,
  },
  (err, client) => {
    if (err) {
      console.error(err)
      return
    }
    db = client.db("jobs")
    jobs = db.collection("jobs")

    //....
  }
)
```

Now we put the code we had that did the scraping into this function, where we have that `//....` comment. This will make the code run after we connected to MongoDB:

```js
const puppeteer = require("puppeteer")
const mongo = require("mongodb").MongoClient

const url = "mongodb://localhost:27017"
let db, jobs

mongo.connect(
  url,
  {
    useNewUrlParser: true,
    useUnifiedTopology: true,
  },
  (err, client) => {
    if (err) {
      console.error(err)
      return
    }
    db = client.db("jobs")
    jobs = db.collection("jobs")
    ;(async () => {
      const browser = await puppeteer.launch({ headless: false })
      const page = await browser.newPage()
      await page.goto("https://remoteok.io/remote-javascript-jobs")

      /* Run javascript inside the page */
      const data = await page.evaluate(() => {
        const list = []
        const items = document.querySelectorAll("tr.job")

        for (const item of items) {
          list.push({
            company: item.querySelector(".company h3").innerHTML,
            position: item.querySelector(".company h2").innerHTML,
            link: "https://remoteok.io" + item.getAttribute("data-href"),
          })
        }

        return list
      })

      console.log(data)
      jobs.deleteMany({})
      jobs.insertMany(data)
      await browser.close()
    })()
  }
)
```

At the end of the function I added

```js
jobs.deleteMany({})
jobs.insertMany(data)
```

to first clear the MongoDB table, and then insert the array we have.

Now if you try running `node app.js` again, and you inspect the MongoDB database content with the terminal console or an app like [TablePlus](https://tableplus.com/), you will see the data being present:

![MongoDB database interface showing jobs collection with stored scraped data including company names and job links](https://flaviocopes.com/images/puppeteer-scraping/Screen_Shot_2020-09-06_at_10.56.39.png)

Cool! We can now set up a cron job or any other automation to run this application every day or every 6 hours to always have fresh data.

## Create the Node.js app to visualize the jobs
Now what we need is a way to visualize those jobs. We need an application.

We'll build a Node.js app based on express and server-side templates with Pug.

Create a new folder, and inside it run `npm init -y`.

Then install Express, MongoDB and Pug:

```bash
npm install express mongodb pug
```

We first initialize Express:

```js
const express = require("express")
const path = require("path")

const app = express()
app.set("view engine", "pug")
app.set("views", path.join(__dirname, "."))

app.get("/", (req, res) => {
  //...
})

app.listen(3000, () => console.log("Server ready"))
```

Then we initialize MongoDB, and we get the jobs data into the `jobs` array:

```js
const express = require("express")
const path = require("path")

const app = express()
app.set("view engine", "pug")
app.set("views", path.join(__dirname, "."))

const mongo = require("mongodb").MongoClient

const url = "mongodb://localhost:27017"
let db, jobsCollection, jobs

mongo.connect(
  url,
  {
    useNewUrlParser: true,
    useUnifiedTopology: true,
  },
  (err, client) => {
    if (err) {
      console.error(err)
      return
    }
    db = client.db("jobs")
    jobsCollection = db.collection("jobs")
    jobsCollection.find({}).toArray((err, data) => {
      jobs = data
    })
  }
)

app.get("/", (req, res) => {
  //...
})

app.listen(3000, () => console.log("Server ready"))
```

Most of this code is same as the code we used in the Puppeteer script to insert the data. The difference is that now we use `find()` to get the data from the database:

```js
jobsCollection.find({}).toArray((err, data) => {
  jobs = data
})
```

Finally we render a Pug template when the user hits the `/` endpoint:

```js
app.get("/", (req, res) => {
  res.render("index", {
    jobs,
  })
})
```

Here's the full `app.js` file:

```js
const express = require("express")
const path = require("path")

const app = express()
app.set("view engine", "pug")
app.set("views", path.join(__dirname, "."))

const mongo = require("mongodb").MongoClient

const url = "mongodb://localhost:27017"
let db, jobsCollection, jobs

mongo.connect(
  url,
  {
    useNewUrlParser: true,
    useUnifiedTopology: true,
  },
  (err, client) => {
    if (err) {
      console.error(err)
      return
    }
    db = client.db("jobs")
    jobsCollection = db.collection("jobs")
    jobsCollection.find({}).toArray((err, data) => {
      jobs = data
    })
  }
)

app.get("/", (req, res) => {
  res.render("index", {
    jobs,
  })
})

app.listen(3000, () => console.log("Server ready"))
```

The `index.pug` file, hosted in the same folder as `app.js`, will iterate on the jobs array to print the details we stored:

```pug
html
  body
    each job in jobs
      p
      | #{job.company}
      br
      a(href=`${job.link}`) #{job.position}
```

Here's the result:

![Node.js web application displaying JavaScript job listings with company names and clickable job position links](https://flaviocopes.com/images/puppeteer-scraping/Screen_Shot_2020-09-06_at_11.27.20.png)