Create a Speech to Text Chat Room with Wit and PubNub

Today I’m going to show you how to add Speech to Text functionality to a chat room using Wit. Wit is a natural language processing API, meaning it takes raw audio input and processes it into data developers can use.

This walkthrough is also available via video. Follow along with the instructions below. Additionally, you can check out the full speech to text chat demo here.

Speech To Text Chat

First, we need a chat room.

You’ll first need to sign up for a PubNub account. Once you sign up, you can get your unique PubNub keys in the PubNub Developer Portal. Once you have, clone the GitHub repository, and enter your unique PubNub keys on the PubNub initialization, for example:

Let’s start with the Chat in 10 lines of lines of code tutorial. I’ve embedded the full example here. You can play around with the chat room by typing a message in the input field and pressing “Enter” or “Return.”

See the Pen PubNub Chat in 10 lines of Code by Ian Jennings (@ianjennings) on CodePen.

When sending your message, a publish action is made on the PubNub network using the PubNub Javascript SDK. All clients that are subscribed to the same channel will receive that message.

Here we take the value from the input box and publish it to our ‘chat’ channel.

Subscribing to the channel is just as easy. Supply the channel name and a function to call when a message is received.

Here we subscribe to the same ‘chat’ channel and append add the text to the chat output when we receive a message.

Now that we’ve got our chat example, let’s add speech to text voice recognition! We’re going to combine Wit’s Microphone example with our chat example.

Note: Microphone relies on WebRTC, which works on Chrome, Firefox and Opera right now. Safari and IE don’t support WebRTC yet.

Follow the directions on the Wit site to set up your Microphone example or skip ahead to the fun part and play with my hosted demo.

speech to text

Press the microphone button then say “send a message” and then your message. You should see the following appear below the microphone icon.

The first result is the parsed intent that you programmed Wit to recognize. The second is the message_body, the parsed message for that intent.

Note, when you use your own API key, you’ll need to train Wit to use your own intents like my “send a message” intent.

Now let’s combine the two demos to create a speech to text chat room!

All we need to do is hook up the mic.onresult results to a PubNub publish. Then, whatever message gets returned by Wit will be published to chat!

speech to text

Check out the full working demo here or browse the source.

There are all sorts of improvements that can be made to this demo. We can support more intents like “who’s here” or show who’s talking by combining more Wit and PubNub features. An expanded demo is coming soon.

Concerned siblings

Because margins collapse, the elements need not be concerned with their siblings.

Whenever you work with CSS, it is important that the properties and values you assign to a class are only relevant to that specific class and never in relation to a sibling.

For example you never want to add more margin-top to an element because it is “too close to the header.” The correct approach would be to add more margin-bottom to the header.

Character Development

A couple months ago I made a spreadsheet called “Friends.”

I listed all the people I wanted to spend time and put them in a spreadsheet. I still feel dirty about it.

image

The truth is, it solved a problem that was becoming a larger and larger issue for me last year. My friends were becoming a victim of my inability to keep up with the intense networking system in NY.

To set the stage here, there’s all sorts of funny stuff going on.

  • Personally I’ve been tricked into interviewing for jobs or “wined and dined” in hopes I would be willing to pick up a contract gig at a cheap price.
  • A friend recently told me that he couldn’t trust the advice of a mentor because he didn’t know if he was using him to orchestrate his uprising as an industry figure.
  • Another friend who is more interested in the food he’s eating than the people he meets, but feels it’s his responsibility to continue networking.
  • I’ve been given advice “do this favor” so that this person owes you something in return.

Personally I’m not interested in playing this game.

Announcing the PubNub Console for Google Chrome

Our goal at PubNub to make it as easy as possible to start developing realtime apps. Today we are happy to announce the first release of the PubNub Console for Google Chrome.

The PubNub Console is a Google Chrome extension. Once installed, it adds an additional tab especially for PubNub in Google Chrome’s developer tools. You can learn more about developer tools, including how to open the multi-tabbed window with Chrome DevTools for Google Developers.

Want to see it in action and learn more? Check out the video below:

The extension monitors PubNub traffic on the page you are inspecting. Whenever the page publishes or receives a message it shows up in the console for that channel.

PubNubConsoleChrome2

Channel tabs are added to the menu on the left when messages are published or subscribed to on the page. You can click on a channel to filter PubNub traffic for that specific channel.

It includes a few special features that makes it easy to develop with realtime messages from protocols like Websockets when using the PubNub JavaScript SDK:

  • All output is JSON formatted and color coded for improved dev & debugging happiness.
  • Channel output can be cleared by clicking “Clear Output” so you can reset the screen and start fresh.
  • If Storage & Playback is enabled on your PubNub account, you can load the previous two minutes of messages in a channel by clicking “Previous 2 minutes.” You can continue clicking this to go back in time.  The next release will add some additional timespan options for more control of stored messages.
  • Data persists through navigation so you can refresh the page to see every message PubNub generates.

Additional Chrome WebSocket Tools

If you need a more detailed view of what’s happening behind the scenes, all raw PubNub traffic is available for inspection through the “Network” tab. There is quite a lot of noise here which is why we were inspired to build the new PubNub Chrome Console for you.

PubNubChromeConsole4

If you would like to publish data to a PubNub stream, make presence calls, or experiment with PAM or SSL the PubNub Developer Console offers a fully featured way to interact with PubNub channels.

PubNubDevConsole PubNubChromeConsoleRound

Be sure to tweet @sw1tch or @PubNub with feedback and feature requests for the PubNub Console for Google Chrome. Stay tuned because we are already working on the next feature upgrades to the PubNub Console for Chrome Developers!

Hackers add Mote.io support for Google Play, Plex, and TuneIn radio as the Chrome Extension goes open source!

Support for 3 new sites

image

It’s been an exciting few weeks in the world of Mote.io. I recently open sourced the Google Chrome extension for the HackNYhackathon that happened at NYU on September 28th.

In just 24 hours, hackers at HackNY built remotes for Google Play, Plex, and TuneIn. After some code review and a couple pull requests, they were merged in and pushed live. Homebase has also been updated to reflect the newly supported sites.

If you’d like to fork the repository and take a stab at making a remote of your own, you can find the code on github here: https://github.com/ianjennings/mote.io-extension

Some words from the hackers themselves

image

I asked each of the hackers to tell me about their school, major, team, and what their experience working with the Mote.io API was like. It turns out it was a couple of their first hackathons!

I’m currently a sophomore as an ITI (Info Tech & Informatics) major at Rutgers but I’ve taken CS courses back in high school. HackNY was my first hackathon and I went just for the learning experience. My team consisted of me and a friend (Jon Wong – a junior at SIT (Stevens Institute) – ECE major) who also came along for the learning experience. I initially read a post somewhere a few weeks back stating that their friend created mote.io and would be at the hackathon. I didn’t think too much of the app at first but once we got to hackNY and I sat down to play around with it, I realized the potential this intuitively designed app had. Our initial goal was to write a remote to work with the Spotify web player (play.spotify.com) however, we couldn’t seem to get it to work and I just suggested that we write a remote for Tunein.

Your API was extremely easy to understand and utilize. After some digging through the source pages of tunein, we were able to determine which classes would seek the playback info and playback controls. It was relatively easy to program after crossing the initial threshold of understanding how everything worked. Definitely going to look to complete that Spotify remote soon.

Josh Sheng – TuneIn Radio Remote

I’m from Columbia University and worked on this with Nathan Bendich. With this being our first hackathon, we tried to build something simple that has some real world application and had a straightforward API to work from. Personally, I chose to create a remote for Google Play as it is my primary music player. The two main scenarios I had in mind for the remote were

a) Studying/reading on my bed while my computer was playing music at my desk and b) Having my computer connected to speakers at some kind of large gathering and being able to control it from across the room.

It was definitely an interesting project, as we had no experience with js or jquery, but we were able to pick up enough based on the pre-existing remotes and limited experience with css selectors. The main issue we had trouble with was having the first song play automatically, as there was no button bound to select a song. We got some much needed last-minute help from Jesse from Clef with a workaround there.

Phillip Godzin – Google Play Remote

We, Shivam Mevawala and Sameer Chauhan, are senior electrical engineering students at the Cooper Union as well as roommates. We recently bought a Google Chromecast and plugged it in to the living room television. We have a Plex Media Server set up on a computer and are able to stream it to the television, but we had no way to control it without waking back to the computer, which was in another room. That’s when we decided to make a Plex remote for Mote.io.

Shivam Mevawala – Plex Remote

Find Mote.io at Music Hack Day NYC

I’ll be presenting the Mote.io API at Music Hack Day NYC tomorrow. This is your chance to make a remote for your favorite music site with help from me personally.

Announcing the Mote.io release video!

image

After a long weekend of filming last week, I have finally finished the Mote.io launch video.

This 40 second short only took me a weekend of filming and editing. I purchased music and voiceovers, and threw the whole thing together in iMovie. Another blog post about making the video to come later.

We’ve come a long way from the first video demo I released in February 2012 when I first claimed “Mote.io will be available in app stores soon.”

Mote.io will be available in the app store soon! 

Get Mote.io for Android

Don’t fix what aint broke

When I was 13 I created a site called 22Pixels, a forum where teenagers like myself could share techniques for Photoshop. It was a simple PHPBB forum mostly focused around making “sigs,” signatures or content that would appear below every post a user made.

I got a young start as my dad gave me the tools and knowledge to put up a basic web page with FTP. He’s an old school hacker who runs Bootdisk.com. He would always always say “Don’t fix what ain’t broke.” It was his recipe for success.

Success

As I learned to program a bit better, I created a list of the best photoshop downloads. It was a simple robot that would crawl other photoshop sites, copy some text, transload an image, and add the link to the index. People could vote reddit style on what the best downloads were.

It worked very well. I became the top result for “Photoshop CS4 Brushes”. I dominated the market for people searching specifically with ts version in their query.

Then CS5 is released. I change my SEO to match CS5. This worked even better. My traffic grew steadily as I continued to scrape downloads from other websites.

Failure

In late 2011 I started Hacker League with two friends from college. Everything else got put on hold, and Hacker League took off. It had our full attention.

The story ends well for Hacker League, but not so much for 22Pixels.

See, as I pointed my attention toward Hacker League, I removed it from 22Pixels. I made two awful decisions because I was not giving 22Pixels enough attention.

I did not watch site growth closely and I was not aware of what was driving site traffic.

The Big Picture

When I logged into Google Analytics, this is what I saw:

This is a monthly view of traffic. For a site like 22Pixels this kind of information was worthless.

The site runs autonomously and since I wasn’t doing anything active to promote it, the monthly view wasn’t very helpful. I would check and make sure that the overall traffic was increasing, but never looked at it relative to the past 6 months or a year.

This was my worst mistake, I wasn’t even aware that the site was growing. This led me to make some idiotic decisions in 2012.

Things Break

The site had a”tags” feature. It was an awful hackjob that slowed down every page load considerably. As the site grew, it had a larger and larger impact on the site performance.

I wrote this off to being a problem with the growing database, not traffic. I decided to remove it because it was raising my hosting costs and I didn’t think it could possibly provide anybody with any real value.

But it turns out the search engines must have loved the unique page titles it provided. When it was removed, my traffic started slipping.

I wasn’t drilling into the stats deep enough to even know that this was providing decent traffic. I wasn’t checking them often enough to know that it had even changed anything.

The correct answer here would have been to rewrite the feature. To watch the stats. To understand the market.

Fixing What Ain’t Broke

When I changed my SEO for Photoshop CS5 I had great results.

CS6 was due to come out in May 2012, so I decided it was time to play my trick again. I updated one variable in my configuration and “CS5” was now “CS6”.

The graph above is the direct impact changing this variable had. It’s about 5,000 views a week difference. You can see, it took me 2 months to notice.

And when I changed it back, the stats pop back up. But my SEO already took the hit and it was too late.

What Aint Broke

The above is a graph for “Photoshop CS[x] brushes” where x is the photoshop version from CS1 to CS6.

It’s easy to see why this happened now. CS5 is the red line and CS6 is the yellow. It turns out that people had been searching for CS5 the entire time. I was 8 months early to changing the keyword.

Take a close look at the end of the blue graph and the beginning of the red (CS4 vs CS5). CS5 was almost immediately adopted and CS4 took huge hits.

Now look at the end of the red line versus the beginning of the yellow (CS5 vs CS6). CS5 slowly transitions into CS6.

Additionally, each release of Photoshop dilutes the market as I can only choose one version to stick with. At the end of 2012 you can see that people are still using all the versions of Photoshop CS!

Don’t fix what ain’t broke

The larger morals here are that

When you figure things out, don’t change them.

If something breaks, repair it so it’s back to how it was

You need to understand the difference between fixing, updating, and improving. You need to recognize the reasons for doing so and the risks involved.

You especially need to understand your market and how it will change, because it will.

Make something successful and keep it successful.. Don’t fix what ain’t broke. Stay the same and change what’s only nessesay. Do the research to find out what’s needed.

And remember, you can only focus your attention on so much. If you let something start to slip, it’s going to slide.

Mote.io hackers bring support for Last.fm and Twitch.tv!

Hackathon hackers have brought you two new remotes for two great sites, Last.fm and Twitch.tv. That brings us to a total of 12 Mote.io remotes – DOUBLE the number of remotes the app launched with in September.

If you’re not keeping track at home, Mote.io now supports Youtube, Hype Machine, Vimeo, Pandora, Rdio, SoundCloud, Grooveshark, Plex, TuneIn Radio, Google Play , Twitch.tv, and Last.fm! Woo!

You can get a hold of the new remotes by updating your Chrome extension:

https://chrome.google.com/webstore/detail/moteio/okkhbojknlfdoooeghbkplihbjajpecc

The new kids: Last.fm & Twitch.tv

The Last.fm remote lets you take control of streaming radio stations while the Twtich remote gives you a full couch mode experience of all the video game streams Twitch.tv has to offer.

Thanks to the awesome organizers at HackRU and Music Hack Day NYC, I was given the opportunity to present the Mote.io API at their events. Awesome hackers from each event spent the weekend building remotes for their favorite sites.

Want Mote.io at your next hackathon? Send me an email at hello@mote.io

Here’s what the Twitch.tv and Last.fm hackers had to say about developing for Mote.io:

Bryant Satterfield – Twitch.tv Remote

I remember I was in the Cave here at Rutgers browsing HN went I discovered Mote.io. After checking it out, I thought it wouldn’t be a bad idea to create a Twitch remote. I watch Twitch a lot and I figured others might find a use for it too. I talked to Vaibhav, who also had the same idea and we decided to collaborate. Our first attempt at the remote was at HackNY where I finished the main menu functionality, but nothing else as I wasted too much time watching Twitch. At HackRU, I met someone who also had the idea to do Twitch remote, Sam Sheikh. I took him on for finishing the project and helping cleaning up the old functionality. It was a very quick process and we finished the first complete version with all the necessary features. Working with Mote.io is very straightforward and thats why I like working with it. I plan to work on a remote for the Spotify Web Player soon. I started a couple of weeks ago, but haven’t had much time to work on it recently.

James Scott – Last.fm Remote

Hey. I added the remote during Music Hack Day in New York. I’m on the Last.fm webteam and thought that with the adding of video content to the Last.fm player coming up, a remote control would be the perfect compliment to the experience. Other stuff I’ve built includes the lastfm node library and boxsocialfm.com (the best domain name in the business).

Some other great news!

Before working on Mote.io I created a website called Hacker League (http://hackerleague.org) with some friends from school. We were in our senior year of college at Rutgers and hackathons were just taking off. We weren’t going to miss a single one.

We noticed that every event was using a different website for announcements, registration, and hack submission. Wouldn’t it be great if there was a site that did all of that especially for hackathons?

We launched the first version of the site at the Fall HackNY Hackathon in 2011. Just two years later Hacker League has powered hundreds of events.

Today I’m excited to announce that the company has sold to Intel.

Hacker League will operate under the Mashery brand. Mashery has been a huge supporter of the hackathon community for years and we don’t believe Hacker League could have gone to a better company.

You can read more about the acquisition here:

Thanks

I’m ecstatic that developers take their time to build remotes for Mote.io. That the hackathon community supports Hacker League. That you’re on the other end of this mailing list.

The only way to turn these projects from code into a community is with your support. I’m glad to have it.

Thanks, Ian

Mote.io now supports YouTube + Google Drive and a bonus app update!

Mote.io now supports YouTube

YouTube was the number one requested site that Mote.io users wanted supported. It was difficult to imagine how Mote.io could work with the classic YouTube.com experience. In order to do it right, I had to write a custom web app.

The site can be found here:
http://moteioyoutube-9226.onmodulus.net/

Mote.io now supports Google Drive Presentations

I thought it would be awesome if I could demo Mote.io at HackNY this year using Mote.io (more on this soon). So I built a small remote to page through slides on a Google drive presentation.

It’s still a little rough, but you can find directions about how to control a Google drive presentation here:
https://mote.io/google-drive

Mote.io app update

An app update was released a few days ago that bring some awesome upgrades to Mote.io. The first thing you’ll notice is a new app icon.

Along with the new icon are some updates to the theme. The app will also now automatically sync if you have your credentials save, so you don’t have to tap the “login” button every time.

You may have also noticed that the launch video shows remotes with Twitter and Facebook icons, but that feature never appeared in the app store version.

I’m glad to say it’s back. You can now share what you’re currently listening to on Facebook and Twitter with many of the remotes.

Really cool stuff is on the way

I’m really excited for my next blog post. Some really cool stuff came out of HackNY, but there is still lots of work to be done so I can’t talk about it just yet. Stay tuned!