Control a browser game with Arduino and a joystick via Johnny Five

In this project we’re going to use a Joystick electronics device, and we’re going to hook it up to the platform game we created in the post “How to create a game with Phaser.js”

The goal is to move the player in the game

Using the Joystick.

Sounds fun, right?

We’ll do it using Johnny Five. In our Node.js app, we’ll connect to the device, and we’ll create a Websockets server. Before starting, I highly recommend you to read the Johnny Five Tutorial.

The browser client will connect to this Websockets server and we’ll stream the left/right/still/up events to handle the player movements.

Let’s go!

The Joystick

This is the Joystick component we’re going to use in this project.

It’s just like the ones you can find on real-world device, like a Playstation controller:

It has 5 pins: GND, +5V (VCC), X, Y, and SW.

X and Y are the coordinates of the joystick.

X is an analog output, and signals the movement on the joystick on the X axis.

Y is the same, for the Y axis:

Taking the joystick as it’s pictured above, with the pins on the left, these will be the measures:

X is -1 when full left and 1 when at full right.

Y is -1 when reaches the top, 1 at the bottom.

The SW pin is a digital output that’s activated when the joystick is pressed (it can be pressed), but we will not use it.

Connect 4 wires to the joystick:

Connect to the Arduino

I use an Arduino Uno board clone and I’ll connect pin #1 and #2 to GND and +5V.

Pin #3 (x) goes to A0 and pin #4 (y) goes to A1:

Connect the Arduino to the computer via USB port now, we’re going to work on the Node.js Johnny Five program in the next lesson.

Make sure you already load the StandardFirmataPlus program on the Arduino board using the Arduino IDE, like I explain in the Johnny Five Tutorial

The Johnny Five Node.js app

Let’s initialize our Johnny Five app.

Create a joystick.js file.

Add

const { Board, Joystick } = require("johnny-five")

on top.

Initialize a board, and add board ready event:

const { Board, Joystick } = require("johnny-five")
const board = new Board()

board.on("ready", () => {
  //ready!
})

Now initialise a new joystick object, telling it which input pins we’re going to use for input X and Y:

const { Board, Joystick } = require("johnny-five")
const board = new Board()

board.on("ready", () => {
  const joystick = new Joystick({
    pins: ["A0", "A1"],
  })
})

Now we can start listening on the change event on the joystick object. When a change happens, we can get the x and y coordinates by referencing this.x and this.y, like this:

joystick.on("change", function () {
  console.log("x : ", this.x)
  console.log("y : ", this.y)
})

Here’s the full code in action:

const { Board, Joystick } = require("johnny-five")
const board = new Board()

board.on("ready", () => {
  const joystick = new Joystick({
    pins: ["A0", "A1"],
  })

  joystick.on("change", function () {
    console.log("x : ", this.x)
    console.log("y : ", this.y)
  })
})

If you start this program using node joystick.js you’ll see lots of values being printed to the console when you move the joystick around:

Great!

Now let’s try to make more sense out of this data.

The position 0 is where the joystick is still, untouched.

I want to detect the “right” movement when X is higher than 0.5, and “left” when X is lower than -0.5.

Same for the Y axis. In other words, I want to only trigger a movement when the joystick goes outside a certain gray zone:

This is how we can do this:

const { Board, Joystick } = require("johnny-five")
const board = new Board()

board.on("ready", () => {
  const joystick = new Joystick({
    pins: ["A0", "A1"],
  })

  joystick.on("change", function () {
    if (this.x > 0.5) {
      console.log("right")
    }
    if (this.x < -0.5) {
      console.log("left")
    }
    if (this.x > -0.5 && this.x < 0.5) {
      console.log("still")
    }
    if (this.y > 0.5) {
      console.log("down")
    }
    if (this.y < -0.5) {
      console.log("up")
    }
  })
})

Try running the program, and you’ll see the left/right/up/down/still words being printed instead of the actual coordinates numbers:

Create a Websockets Server

Now the question is this: how can we communicate between the game running in the browser, and our hardware project?

Since the app is working locally, the idea I had was to creata a Websockets connection between the browser and the Node.js Johnny Five process.

The Node.js app will be the Websockets server, and the browser will connect to it.

Then, the server will send messages when the joystick is moved.

Let’s work on the Websockets server now.

First install the ws npm package:

npm install ws

When the board is ready, we initialize the joystick object and a new Websockets server process. I’ll use port 8085, as a :

const { Board, Joystick } = require('johnny-five')
const board = new Board()

board.on('ready', () => {
  const joystick = new Joystick({
    pins: ['A0', 'A1']
  })
  const WebSocket = require('ws')
  const wss = new WebSocket.Server({ port: 8085 })
})

Next, we add an event listener that provides a callback that is fired when a client is connected to the Websockets server:

wss.on('connection', ws => {

})

Inside here, we’ll start listening for the change event on the joystick object, like we did in the last lesson, and in addition to printing to the console, we’ll send a message to the client using the ws.send() method:

const { Board, Joystick } = require('johnny-five')
const board = new Board()

board.on('ready', () => {
  const joystick = new Joystick({
    pins: ['A0', 'A1']
  })

  const WebSocket = require('ws')
  const wss = new WebSocket.Server({ port: 8085 })

  wss.on('connection', ws => {
    ws.on('message', message => {
      console.log(`Received message => ${message}`)
    })
    
    joystick.on('change', function() {
      if (this.x > 0.5) {
        console.log('->')
        ws.send('right')
      }
      if (this.x < -0.5) {
        console.log('<-')
        ws.send('left')
      }
      if (this.x > -0.5 && this.x < 0.5 ) {
        console.log('still')
        ws.send('still')
      }
      if (this.y > 0.5) {
        console.log('down')
      }
      if (this.y < -0.5) {
        console.log('up')
        ws.send('jump')
      }
    })
  })
})

Connect to the Websockets server from the game

If you are not familiar with the platform game I built in the Games level, check it out first.

The game is all built in one single app.js file, which uses the Phaser.js framework.

We have 2 main functions: create() and update().

At the end of the create() function, I’m going to connect to the Websockets server:

const url = 'ws://localhost:8085'
connection = new WebSocket(url)

connection.onerror = error => {
  console.error(error)
}

We also need to initialize let connection at the top of the file, outside the function, because we’re going to reference that variable in the update() function, too.

I hardcoded the URL to the Websockets server, because it’s a local server and this would not work outside of a single, local scenario.

The joystick is one, after all. But it’s great to test things out.

If there’s any error during the connection, we’d see an error here.

In the update() function, now we have this code:

function update() {
  if (cursors.left.isDown) {
    player.setVelocityX(-160)
    player.anims.play('left')
  } else if (cursors.right.isDown) {
    player.setVelocityX(160)
    player.anims.play('right')
  } else {
    player.setVelocityX(0)
    player.anims.play('still')
  }

  if (cursors.up.isDown && player.body.touching.down) {
    player.setVelocityY(-330)
  }
}

I’m going to change this all, because instead of controlling the player movement using the keyboard, we’re going to use the joystick.

We’ll listen to the message event on the connection object:

function update() {
  connection.onmessage = e => {

  }
}

The e object passed to the callback stands for “event” and we can get the data sent by the server on its data property:

function update() {
  connection.onmessage = e => {
    console.log(e.data)
  }
}

Now we can detect the message sent, and we can move the player accordingly:

connection.onmessage = e => {
  if (e.data === 'left') {
    player.setVelocityX(-160)
    player.anims.play('left')
  }
  if (e.data === 'right') {
    player.setVelocityX(160)
    player.anims.play('right')
  }
  if (e.data === 'still') {
    player.setVelocityX(0)
    player.anims.play('still')
  }
  if (e.data === 'jump' && player.body.touching.down) {
    player.setVelocityY(-330)
  }
}

That’s it! Now our joystick will move the player on the screen!

Alternative using WebUSB

The Node.js server with Websockets is a good cross-browser way to solve the connection problem.

Another way is to use WebUSB, a technology that’s only available on Chromium-based browsers like Chrome, Edge and others.

Using this, we can make pages detect a device and they can talk to it directly.

To do so, we must ask the user to perform an action, like pressing a “Connect” button like the one I added to the game index.html file:

<!DOCTYPE html>
<html>
  <head>
    <script src="./dist/app.js"></script>
  </head>
  <body>
    <button id="connect">Connect</button>
  </body>
</html>

(the rest of the game is appended to the body in a canvas tag automatically)

This time I used an Arduino MKR WiFi 1010 device because WebUSB does not support the Arduino Uno board, for technical reasons.

I connected the joystick to that board, using the same wiring we used in the previous lessons:

To configure the Arduino to play well with WebUSB I recommend you to read https://webusb.github.io/arduino/.

Here is the Arduino sketch, written in the Arduino Language (C++) this time, instead of Johnny Five:

#include <WebUSB.h>
WebUSB WebUSBSerial(1 /* http:// */, "localhost:3000"); //provide a hint at what page to load

#define Serial WebUSBSerial

const int xpin = 0;
const int ypin = 1;

void loop() {
  if (Serial) {

    int x = analogRead(xpin);
    int y = analogRead(ypin);

    bool still = false;

    if (x > 768) {
      still = false;
      Serial.println('R');
    }
    if (x < 256) {
      still = false;
      Serial.println('L');
    }
    if (x > 256 && x < 768 ) {
      if (!still) {
        still = true;
        Serial.println('S');
      }
    }
    if (y < 256) {
      Serial.println('J');
    }
  }
}

It is very similar to the program we built in Node.js previously.

This time however we communicate a letter to the Web page using the WebUSBSerial interface defined by WebUSB.

We send a letter: R for right, L for left, S for still, and J for jump.

I added a still boolean to prevent the S letter to being sent too many times unnecessarily. In this way we only send it once when we return to a still state.

On the Web page side, I added a serial.js file that is used to abstract a lot of the low level code we don’t need to worry about. I found it in the WebUSB Arduino demos https://webusb.github.io/arduino/demos/serial.js and I adapted it to be an ES Module file, by removing the IIFE (immediately-invoked function) and added an export default serial at the end:

const serial = {}

serial.getPorts = function () {
  return navigator.usb.getDevices().then((devices) => {
    return devices.map((device) => new serial.Port(device))
  })
}

serial.requestPort = function () {
  const filters = [
    { vendorId: 0x2341, productId: 0x8036 }, // Arduino Leonardo
    { vendorId: 0x2341, productId: 0x8037 }, // Arduino Micro
    { vendorId: 0x2341, productId: 0x804d }, // Arduino/Genuino Zero
    { vendorId: 0x2341, productId: 0x804e }, // Arduino/Genuino MKR1000
    { vendorId: 0x2341, productId: 0x804f }, // Arduino MKRZERO
    { vendorId: 0x2341, productId: 0x8050 }, // Arduino MKR FOX 1200
    { vendorId: 0x2341, productId: 0x8052 }, // Arduino MKR GSM 1400
    { vendorId: 0x2341, productId: 0x8053 }, // Arduino MKR WAN 1300
    { vendorId: 0x2341, productId: 0x8054 }, // Arduino MKR WiFi 1010
    { vendorId: 0x2341, productId: 0x8055 }, // Arduino MKR NB 1500
    { vendorId: 0x2341, productId: 0x8056 }, // Arduino MKR Vidor 4000
    { vendorId: 0x2341, productId: 0x8057 }, // Arduino NANO 33 IoT
    { vendorId: 0x239a }, // Adafruit Boards!
  ]
  return navigator.usb
    .requestDevice({ filters: filters })
    .then((device) => new serial.Port(device))
}

serial.Port = function (device) {
  this.device_ = device
  this.interfaceNumber_ = 2 // original interface number of WebUSB Arduino demo
  this.endpointIn_ = 5 // original in endpoint ID of WebUSB Arduino demo
  this.endpointOut_ = 4 // original out endpoint ID of WebUSB Arduino demo
}

serial.Port.prototype.connect = function () {
  let readLoop = () => {
    this.device_.transferIn(this.endpointIn_, 64).then(
      (result) => {
        this.onReceive(result.data)
        readLoop()
      },
      (error) => {
        this.onReceiveError(error)
      }
    )
  }

  return this.device_
    .open()
    .then(() => {
      if (this.device_.configuration === null) {
        return this.device_.selectConfiguration(1)
      }
    })
    .then(() => {
      var configurationInterfaces = this.device_.configuration.interfaces
      configurationInterfaces.forEach((element) => {
        element.alternates.forEach((elementalt) => {
          if (elementalt.interfaceClass == 0xff) {
            this.interfaceNumber_ = element.interfaceNumber
            elementalt.endpoints.forEach((elementendpoint) => {
              if (elementendpoint.direction == "out") {
                this.endpointOut_ = elementendpoint.endpointNumber
              }
              if (elementendpoint.direction == "in") {
                this.endpointIn_ = elementendpoint.endpointNumber
              }
            })
          }
        })
      })
    })
    .then(() => this.device_.claimInterface(this.interfaceNumber_))
    .then(() => this.device_.selectAlternateInterface(this.interfaceNumber_, 0))
    .then(() =>
      this.device_.controlTransferOut({
        requestType: "class",
        recipient: "interface",
        request: 0x22,
        value: 0x01,
        index: this.interfaceNumber_,
      })
    )
    .then(() => {
      readLoop()
    })
}

serial.Port.prototype.disconnect = function () {
  return this.device_
    .controlTransferOut({
      requestType: "class",
      recipient: "interface",
      request: 0x22,
      value: 0x00,
      index: this.interfaceNumber_,
    })
    .then(() => this.device_.close())
}

serial.Port.prototype.send = function (data) {
  return this.device_.transferOut(this.endpointOut_, data)
}

export default serial

In app.js I include this file at the top:

import serial from "./serial.js"

and since now we have a button, I added a DOMContentLoaded event listener:

document.addEventListener('DOMContentLoaded', () => {

}

I wrap everything I had in the file, except the import statements, into it.

Next I add a reference to the Connect button:

let connectButton = document.querySelector("#connect")

and a port variable that we’ll later initialize:

let port

At the end of the create() function, we immediately ask for the devices to the serial object, and we call connect() if one device is already paired and connected

I add a click event listener on the Connect button. When we click that button, we’ll request the connection to a device.

connectButton.addEventListener("click", () => {
  if (port) {
    port.disconnect()
    connectButton.textContent = "Connect"
    port = null
  } else {
    serial.requestPort().then((selectedPort) => {
      port = selectedPort
      port.connect().then(() => {
        connectButton.remove()
      })
    })
  }
})

Our request must be in a user-initiated event, like a click, otherwise the browser will not do anything and deny our action.

When the user presses the button the browser asks the permission to connect:

And once the permission is granted, we can control the game using the Joystick!

Here is the new update() function that will handle receiving the data from the device. We have a function attached to the onReceive property of the port object. This will fire when there’s a new message coming in, and we’ll handle the letters communicated to us:

function update() {
  if (port) {
    port.onReceive = (data) => {
      let textDecoder = new TextDecoder()
      let key = textDecoder.decode(data)

      if (key === "L") {
        player.setVelocityX(-160)
        player.anims.play("left")
      }
      if (key === "R") {
        player.setVelocityX(160)
        player.anims.play("right")
      }
      if (key === "S") {
        player.setVelocityX(0)
        player.anims.play("still")
      }
      if (key === "J" && player.body.touching.down) {
        player.setVelocityY(-330)
      }
    }
    port.onReceiveError = (error) => {
      console.error(error)
    }
  }
}

That’s it! We can now play the game like before, when we used Websockets, except now we don’t need an external Node.js server - the connection happens directly between the browser and the device.