Simple Jarvis using Android Things

Simple Jarvis using Android Things

How to build a simple AI assistant based IoT

After getting an Android Things kit at Google Developer Day India, I have been very excited at getting into IoT and building with Android Things. Here I am finally messing around with it.

Hardware

Some of the supported kits are shown below. I am using the NXP Pico i.MX7D for this article. Raspberry Pi 3 and the Intel Edison are also some of the other supported boards with more manufacturers working to add support.

Setup

This consists of basically two steps:

  • Connect the hardware components

  • Installing Android Things on the hardware

Connect the Hardware

Having max to max fixed a few broken sockets and fixed a few broken stuff at home, building this was very exciting for me.

I followed the following instructions but I only attached the display to the board as I only wanted to mess around with some simple stuff before going into more detailed setups.

I sadly didn’t have a WiFi antenna provided with my kit and had to connect to the internet through an ethernet cable. Also a Rainbow Hat wasn’t provided and hence I decided to make just simple experiments with the kit. Once all the hardware was connected and I powered the kit, it looked like this.

My final setupMy final setup

Installing Android Things

After the hardware is all connected you have to install Android Things onto the board. I suggest the Android Things Setup Utility.

The following guide will help setup the kit. I will also provide a short brief into installing Android Things.

Once you run the executable file, the below are the setups that you encounter.

Once I was done installing Android Things on the board, I didn’t setup the WiFi as I didn’t have an antenna. I hence used an ethernet cable to connect to the internet. Once it was set up my kit looked like this.

Building and Deploying

Create a new project

Creating a new Android Things project is very straightforward.

  • Start a new project

  • Select Android Things as the desired form factor

  • In my case, I wanted a UI so I selected “Android Things Empty Activity”

  • Click “Next” and my project was ready

Project Configuration

Since the project creation tools set everything up for you, you don’t have to worry about what went on under the hood to work with Android Things. However, I was curious anyways and noticed a few things.

The Android Things dependency was automatically added to the build.gradle file.

compileOnly 'com.google.android.things:androidthings:+'

The manifest had 2 items of interest

  • <uses-library android:name="com.google.android.things />

  • an intent filter to handle IOT_LAUNCHER

The full explanation for these can be found here. In short, the IOT_LAUNCHERcategory on the intent filter indicates which activity should be launched when run on an IOT device, and the uses-library declaration makes the Android Things library available at runtime.

Building Hello World

From here, building a Hello World UI application for Android Things was no different than for a phone/tablet once the initial project setup was finished.

I kept this very simple and added a single button that, when tapped, showed a “Hello World” toast.

Once the code was ready, I deployed the project like I would to a phone/tablet and my new app was running on my personally constructed Android Things hardware.

The code can be found on GitHub here:

Google Assistant Project

I further decided to make a Google Assistant application which is one of the sample projects. For this project you will require:

  • The previous setup

  • Standard mobile earphones which have a microphone

Google does provide a DIY kit which can be used with the raspberry pi to do the same. You should be able to hook up a button, an external microphone and speaker to the kit and use that with the below project to build a smart assistant. I made the following project with the above simple setup and a standard mobile earphone.

Setup Project to work with Google Assistant API

First clone the following repository onto your device. It has been modified to run for a standard line input/output using a microphone.

Configure Credentials

Enable the following activity controls in the Google Account you plan to use with the Assistant:

  • Web & App Activity

Make sure you check the box that says “Include Chrome browsing history and activity from websites and apps that use Google services”

  • Device Information

  • Voice & Audio Activity

Once done with that:

  • In the Cloud Platform Console, go to the Projects page. Select an existing project or create a new project.

  • Enable the Google Assistant API on the project you selected.

  • Click Enable.

  • Create an OAuth Client ID

  • Click Other (select Other UI and User data if prompted) and give the client ID a name.

  • On the OAuth consent screen tab, give the product a name (don’t use “Google” in the name).

  • Click Create. A dialog box appears that shows you a client ID and secret. (No need to remember or save this, just close the dialog.)

  • Click ⬇ (at the far right of screen) for the client ID to download the client secret JSON file (client_secret_NNNN.json or client_id.json).

  • Open a terminal with python support and install the google-auth-oauthlib. The command is as follows:

  • In your virtual environment, run the command:

pip install google-auth-oauthlib[tool]
  • Navigate to your top-level project directory.

  • Copy the client_secret_NNNN.json file over to your projects root directory, then run Google’s oauthlib tool to generate the credentials.

google-oauthlib-tool --client-secrets client_secret_NNNN.json \ --credentials app/src/main/res/raw/credentials.json \ --scope https://www.googleapis.com/auth/assistant-sdk-prototype \ --save
  • Now Open the code in android studios. Sync the gradle and then deploy like usual. On the first install permissions have been granted to the application. Restart the application

  • Now the application will run properly. Press the button on the screen and your device will start listening to your voice.

Your voice query gets sent over via the Google Assistant API, and you get a response almost instantaneously. Neat! Since the service is linked to your Google account, you can even tell it to add events to your Calendar and access other information. Ask for a joke, find out tomorrow’s weather, and when you run out of ideas, check out this comprehensive list.

Extra Material

Dave Smith has a great, high-level overview of the Android Things platform. This is a wonderful place to start if you have no idea what Android Things is or want to explain it to someone else.

Rebecca Franks and Wayne Piekarski gave some really nice talks at GDD events last year

Conclusion

I hope this helps people with Android Things supported kits to build a simple Google Assistant application. Again, this has been a lot of fun and it’s great to see how easy it is these days to sprinkle some AI into any project. I still haven’t fully explored the capabilities of my DIY Google Assistant yet, but I think there’s a lot of potential for interesting projects here.

I would also like to extend my sincere gratitude to Nate Ebel and w4ilun for their previous works on this topic which helped me get this far.

I really hope that you find this article useful. I invite you to participate in the discussion in the comments below, I’m always interested to know your thoughts and happy to answer any questions you might have in your mind.

Thanks for reading! :)

P.S Do feel free to connect with me on LinkedIn

Did you find this article valuable?

Support Rohit Jacob Mathew by becoming a sponsor. Any amount is appreciated!