Categories
Flex

New ActionScript Native Extension – Speech Recognition



Here is a Speech Native Extension that takes advantage of the Android’s speech recognition APIs.

Here is what the ANE does,

  • Provide an API to check if the feature is supported.
  • Provide an API to launch the default speech recognition UI and capture voice input.
  • Trigger events based on the success or failure of the above call.
    • In case of success, the appropriate event has information of the best match !
    • In case of failure, the appropriate event is fired.


Here are the steps required to include the ANE in your Android Project,

  • Download the ANE from here
  • In Flash Builder 4.6, open the Properties dialogue of the Project you want to include speech support in.
  • Select Build Path > Native Extensions
  • Add ANE – Select the location of the downloaded Speech.ane file
  • Navigate to Flex Build Packaging > Google Android > Native Extensions
  • Check the Package checkbox next to Speech.ane (This ANE supports only the Android Platform at the moment)
  • Apply and Close
  • Open the <ProjectName>-app.xml file
  • Scroll down until you find the Android manifest statements
  • Add the  below code just before </manifest>
    (i.e., inside the <manifest></manifest>tags)

    <application>
     <activity android:name="com.inoel.ane.speech.extensions.SpeechHandler">
     </activity>
</application>

You will now be able to access the below APIs,

Initializing and specifying prompt text

//Creating and initializing a Speech Object
//Parameter - Prompt displayed on the Native Speech Recognition UI
var speech:Speech = new Speech("Please speak something !");

Check if speech input is supported

// Returns a Boolean value
trace("Speech Supported: " + speech.isSupported);

Accept Speech Input

//Calls the method which in turn fire's the Native Speech UI for accepting speech input
speech.listen();

Listen for the following events

  • speechToTextEvent.VOICE_RECOGNIZED – Fired on successful voice recognition. Recognized String resides in the event.data attribute.
  • speechToTextEvent.VOICE_NOT_RECOGNIZED – Fired on unsuccessful attempt. A common cause for failure could be network (data) unavailability. I believe this is overcome in new Ice Cream Sandwich which does not require a network connection to recognize speech ! Includes the error string in the event.data attribute.

Download the sample project from here. Please ensure you manually add the ANE with the steps above, or you get a ‘Class not found Exception’.

If you are interested in understanding how ANEs are used in Flash Builder 4.6, have a look at the recording in my previous post, http://blog.immanuelnoel.com/2011/10/20/native-extensions-flash-builder-webinar/

I also plan to post the entire source for the ANE itself. Watch out for that !

—– UPDATE 19/02/2012 —-
Source for the ANE is posted below,
http://blog.immanuelnoel.com/2012/02/19/speech-recognition-ane-source/

Speech Recognition ANE - Source
Native Extensions - Flash Builder Webinar

By Immanuel Noel

A techie at heart. Works with Adobe at Bangalore, India. Currently do DevOps. Been a part of the ColdFusion Engineering, Flash Runtime Engineering, Flash Builder Engineering teams in the past. Was a Flash Platform Evangelist, evangelizing the Adobe Flex platform. Spoke at numerous ColdFusion / Flash and Flex tech conferences. This blog is a collection of some of my strides with technology.

More on me on the home page

106 replies on “New ActionScript Native Extension – Speech Recognition”

I’m really impressed how Native Extensions ae taking mobile AIR. Very impressed 🙂

AIR needs a little better performances, to minimize the gap between native Objective C apps and AIR apps.

Love the way this is improving 🙂

Hello Immanuel,

Thanks for your great ANE.

Would you please add the api that allows developer to manipulate language options for speech recognition(i.e. Japanese) while their android is of different version(i.e English).

best.

shilik

Shilik,
I don’t really see an (Android SDK) API support for what you ask, but there is an option to specify FREE_FORM or WEB_FORM language model (More here – See LANGUAGE_MODEL_FREE_FORM and LANGUAGE_MODEL_WEB_SEARCH). I’ll put up the source in a few days, so you can go ahead and include this in the API ! 🙂
As a default selection though, you can choose from a list of options in the Android settings menu.

By Post Author

It crashes silently on my Galaxy Tab(1).
A quick test showed that speech is supported, but then … Do you have any advice?

Immanuel, thank you for posting this. I would like to know if you have any recommendations for debugging a native extension once its running. for ex Log.d() doesn’t seem to work from DDMS in eclipse even when using the same package name,do you have any recommendations for debugging an ANE?

Sean,
Writing to log is definitely the best way to debug native code of an ANE.
The below statement works for me,
Log.v("Tag - App Name", "Custom Message");

Log.d stands for debug log, and Log.v for Verbose. Ensure you have the right option selected from the dropDown in the LogCat view, and ofcourse, the right device selected 🙂

By Post Author

Immanuel, Happy new Year!!

I would like to ask, if you have interesting in development one Text to Speech Extension.

I try but I don’t be really fit with all the process.

Let’s do it,

Best,

Ramón

Thanks for offering up this extension.
I was wondering if you could post the source code and / or elaborate on how you can get the onActivityResult method to run in an FREFunction. I’m currently trying to write an extension that requires that function’s use, and am having difficulty.

Hi Immanuel

This is really good. When you do you think you will be uploading your source? I’m interested to see how you got onActivityResult to work in the native extension. I am trying to make an ANE for a Music Picker.

Intent i = new Intent();
i.setAction(Intent.ACTION_GET_CONTENT);
i.setType(“audio/*”);
context.getActivity().startActivityForResult(Intent.createChooser(i, “Select audio file”),0);

It shows the picker, but onActivityResult is not called – so I can never get the file that was chosen. So I am very interested to see how you have solved this problem!

Mark/Andrew, I’ve been delayed in posting the source. I’ll try to post the source ASAP.
In the meantime, the only solution to your problem is creating a new activity, listen for onActivityResult in the new activity, and fire it from within your extension code.
I’ll try to post a detailed explaination in the following week.

By Post Author

Hi,

I get this error.

VerifyError: Error #1014: Class com.inoel.ane.speech::speechToTextEvent could not be found.

I am using Flex 4.6 and your source project along with ANE.

Yes,

Here is the code for your reference

<![CDATA[


]]>

Please let me know if you need any other information.

Thanks
Ankur

I believe you are pointing to a code snippet within a MXML file.
The activity tag needs to be added in the applicationname-app.xml file. Do you have that inserted as well ?

By Post Author

It worked for me now. The error was because it was not included in Flex Build Packaging.

Thanks for your help and this awesome work.

Ankur

Hi Immanuel,
I was able to check the package for android for speech.ANE. once it is done, i get the error on digital signature “The specified file doesn’t found” for C:\Certificates\Immanuel.p12.. I don’t have a certificate.. please help. Im a newbie to flex.. thanks in advance.. I use FB4.6

Hi Immanuel,
sorry to interrupt you again. I missed to create a certificate. I created a certificate there and proceed. I got the below error message.
==>”There are packaging errors/warnings. Check each native extension in the Flex Build Packaging page for each target platform.

please be advised that the Android platforms native extensions are used when running the application in the AIR Debug Launcher. They may raise errors if they are incompatible with this environment. Would you like to continue?”
I pressed OK since I’m sure that other platforms where unchecked at “Flex Build Packaging”.
OK->Process terminated unexpectedly. invalid application description: unknown namespace:http//ns.adobe.com/air/application/3.1

Process terminated unexpectedly.

invalid application descriptor: Unknown namespace: http://ns.adobe.com/air/application/3.1

Launch command details: “D:\Fx SDK\flex 4.5.1\bin\adl.exe” -runtime “D:\Fx SDK\flex 4.5.1\runtimes\air\win” -profile extendedMobileDevice -screensize 600×999:600×1024 -XscreenDPI 168 -XversionPlatform AND -extdir “C:\Users\ADMIN44\Adobe Flash Builder 4.6\.metadata\.plugins\com.adobe.flexbuilder.project.ui\ANEFiles\SpeechMobile\win32\” “C:\Users\ADMIN44\Adobe Flash Builder 4.6\SpeechMobile\bin-debug\SpeechMobile-app.xml” “C:\Users\ADMIN44\Adobe Flash Builder 4.6\SpeechMobile\bin-debug”

@Dinesh, The ANE has been built only for the Android platform, and *WILL* crash if run on a emulator / simulator. Please test on an Android device only.
Also, please ensure the below steps are followed (as indicated in the blog post above),
-> Navigate to Flex Build Packaging > Google Android > Native Extensions
-> Check the Package checkbox next to Speech.ane (This ANE supports only the Android Platform at the moment)

By Post Author

@immanuel: executed @ Google Nexus One. First, ask for adobe air to install on device. I agreed and got the below one.
Error occurred while packaging the application:

C:\Users\ADMIN44\Adobe Flash Builder 4.6\SpeechMobile\bin-debug\SpeechMobile-app.xml(234): error 103: application.android.colorDepth is an unexpected element/attribute

the air version specified:

I also tried changing

Do I have to make changes in xml file????

@immanuel: executed @ Google Nexus One. First, ask for adobe air to install on device. I agreed and got the below one.
Error occurred while packaging the application:

C:\Users\ADMIN44\Adobe Flash Builder 4.6\SpeechMobile\bin-debug\SpeechMobile-app.xml(234): error 103: application.android.colorDepth is an unexpected element/attribute

the air version specified: AIR 2.6

I also tried changing to AIR 3.1

Do I have to make changes in xml file????

@Immanuel: It works fine now. I again downloaded from the blog and ran into device without any changes. Thanks again and sorry for interrupting you often.

Hi Immanuel,

First I want to thank you for coding and sharing this ANE, it works like a charm!

When do you think you will be releasing the code? I would be very glad to see what you have done to make this work.

Also, for the project I am working on, I need to get the entire list or results from google STT and not just the first one. Having the code will allow me to add this functionality…

Thanks again,
Sefi

Immanuel,

Thanks for posting the source, It’s very well documented, and adding the functionality to get the entire list of results was a breeze.

Do you know if there is a way to use the native speech recognition API without showing it’s UI?
Some times I will want to use my own UI, sometimes I will want to completely hide the UI…

Sefi

Hi,
I am trying to use your ane file and using flash cs5.5, I was able to integrate the code and package it. However, in my application description file, I have added the extensions tag and the extensionsID as

com.inoel.ane.speech

It seems to compile without any problems. However there is absolutely no response. Not sure if voice thing is working or not. I also have added

in the descriptor file just above. I am trying to print a message to see if voice is supported or not and it does not seem to respond to either. What could be the problem. Do I need to set permissions in the descriptor file?

@Bharkan You only need to set the ‘Internet’ permission. The only other editing required in the descriptor file is to add the activity. Steps for the same are included within the post above.

Do confirm once you get it working.

By Post Author

It is working now. Thanks a bunch. However, it is on and off in my phone. It gives a message saying working but does not type the number I speak out.

Good morning,

i am experimenting with your extension. but when i execute
the command speech.listen(); the UI isn’t fired and it seems
to get in a endless loop. i added the application tag and it
returns true when i check if it is supported.
what could cause this problem?

Hello Immanuel and thank you for a lovely guide.

I’m having a problem though,
When i launch the app i see a popup saying something like “[something] is waiting for [something else]” and then the screen goes white. I can trace and i can place graphics in this white space but i don’t see the native gui showing. Anyone else experienced this?

I’m completely new to Flash Builder and Android development, but skilled in as3.

Hi, new learner here. My name is Kevin.
I have stuck in the native extension on my sms apps.
What i try to do is call my native extension code which trying to invoke the smsmanager.sendTextMessage(…) and create an notification.
Somehow it didn’t send out the sms. Can you help me on this?

Here is line in the native code :
package com.leebrimelow.notifications;

import android.app.Activity;
import android.app.Notification;
import android.app.NotificationManager;
import android.app.PendingIntent;
import android.content.Context;
import android.content.Intent;
import android.telephony.SmsManager;

import com.adobe.fre.FREContext;
import com.adobe.fre.FREFunction;
import com.adobe.fre.FREInvalidObjectException;
import com.adobe.fre.FREObject;
import com.adobe.fre.FRETypeMismatchException;
import com.adobe.fre.FREWrongThreadException;

public class NotificationFunction extends Activity implements FREFunction {

@Override
public FREObject call(FREContext context, FREObject[] args) {

String message = “”;
try {
message = args[0].getAsString();
} catch (IllegalStateException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (FRETypeMismatchException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (FREInvalidObjectException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (FREWrongThreadException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}

String ns = Context.NOTIFICATION_SERVICE;
NotificationManager mNotificationManager = (NotificationManager) context.getActivity().getSystemService(ns);

long when = System.currentTimeMillis();

Notification notification = new Notification(context.getResourceId(“drawable.notify”), message, when);

notification.defaults |= Notification.DEFAULT_SOUND;

CharSequence contentTitle = “Notification from Flash”;
CharSequence contentText = “Hello” ;//message;
Intent notificationIntent = new Intent();
PendingIntent contentIntent = PendingIntent.getActivity(context.getActivity(), 0, notificationIntent, 0);

notification.setLatestEventInfo(context.getActivity(), contentTitle, contentText, contentIntent);

final int HELLO_ID = 1;

mNotificationManager.notify(HELLO_ID, notification);

//String sms = Context.TELEPHONY_SERVICE;
//SmsManager smsManager1 = (SmsManager) context.getActivity().getSystemService(name)

SmsManager smsManager = SmsManager.getDefault();
String smsNumber = “012345678”;
String smsText = “Testing message”;
smsManager.sendTextMessage(smsNumber, null, smsText, null, null);

return null;

}

}

Hey!

I’m using Flash CS6, and it seems like the additions to the manifest tags are being stripped out when publishing. This is causing the “Access of undefined property speechToTextEvent” error to be thrown.

Do you know any ways around this?

Hi Immanuel:

Can you give us the complementary .swc so that we can get at the .ane from Flash Pro CS5.5? The included SpeechLibrary.swc appears to be to compile the .ane; as it doesn’t work from Flash Pro.

Has anyone else come up with the necessary .swc for this?

Thanks

Fixed my issue. Needed to check “Manually manage permissions and manifest additions in the application descriptor file” in the Permissions tab of the AIR for Android settings menu.

Dear Mr.Noel, thank you so much for posting this. I was able to make a test program that asked for voice, and it works. Voice recognition seems like a miracle when it works. However, i noticed that on my Galaxy Nexus phone, which just got upgraded to Jelly Bean (android 4.1) today, if I turn off WIFI, the voice recognizer doens’t work any more, but I know that my phone does have the offline voice recognition module already downloaded (a mere 22 MB for english).. is there some flag we have to change in the ANE to permit offline recognition? It would be far superior for an app not to incur data charges to do recognition…

Thanks for pointing out the issue Edward. I haven’t really had the chance to get to this. Yet to get my hands on a 4.1 device 🙂
Missing out on an important usecase though. Will try to get this sorted.

By Post Author

Dosn’t seem to work on Galaxy SII (android-version 2.3.4) / Desire HD (android-version 2.3.5).
The air app crashes in 99% of all applications tests. Sometimes directly after the app starts .. sometimes when the speech recognition popup appears (sorry! .. the app .. quit unexpectedly ..).

this is how my applicationname-app.xml file looks:
..

<![CDATA[

]]>

..

i’m using flash cs6

i’ve deleted the “<" at the beginning .. hope it will be posted now

![CDATA[
uses-permission android:name=”android.permission.INTERNET”/>
application>
activity android:name=”com.inoel.ane.speech.extensions.SpeechHandler”>
/activity>
/application>
/manifest>]]>

@73C, verified to be working on Desire HD (android-version 2.3.5). Please check the comments above on problems with the App descriptor or the usage. I’m pretty sure the problem is with incorrect usage.

By Post Author

yes your right. my fault! everything was ok with the ane-related stuff. the problem was somewhere else ( the app quits on deactivation ^^ ). so great work and thanks for that extension! and sorry for spaming with unrelated stuff but it was late that day 😉
btw. works on android 4.0.4 too ( Galaxy SII )

Hi, I am trying to use your life saver extension, but I am unable to run it on Flash CS6, it publish ok, and the issupport flag its enabled, but the event is never triggered. Any suggestion?

Hello, thank you so much for your work. I’m using Flash Pro CS6 and I can’t understand why this doesn’t work for me.

So when I publish my flash I have this error “1046: Type was not found or was not a compile-time constant: Speech.”

I added de .ane file
http://i41.tinypic.com/yorw0.jpg
and this is my .xml file
http://pastebin.com/kS7VSUeA

I also prove the .swc tip you post above. Same result 🙁

Hope you can help me

Ok, finally it’s working but I don’t know how to get the string.

Box.text=e.data; doesn’t work in Flas CS6

Any Idea?
Thanks

Hi Noel,

Thank you so much for sharing this ANE. I am new in Flash builder and I am trying to implement your’s ANE in my android APP. It is working fine but can you guide me that how can I control it to stop playing or how can I control it by saying few commands like “Play”, “Stop”, “Next”, “Back”. Right now it is going back automatically. I want it to control on my commands not automatically. Please help and let me know if you can…

Thanks & Regards,
Sanjeev

@Sanjeev
Having your app react to commands is just a matter of reading the textual representation of the voice recognized, and then performing appropriate actions, right? 🙂

By Post Author

Hi, i’m new to Flash builder and would like is there a possible to make the voice recognition able to detect mandarin instead of english?.
And do u know any ways to compare 2 speeches to see whether they are similar? 1 of the voice will be pre-recorded. The other voice will be the user saying out the word or sentence where the app is suppose to detect it.

The Sample is very fun to play ^v^

How can I hide, move or resize Prompt “Speak now” on the Native Speech Recognition UI

Hello Immanuel,

Thanks so much for this fantastic ANE. I just wanted to know weather you have tested this on KitKat. I am experiencing some trouble as the listen function doesnt seem to work on this version. Thanks again for the ANE and the help.

Regards,

Gautham

@gru_narin @ครูนรินทร์ Thailand
The Speak Now prompt cannot be hidden. If you have a way to hide the prompt with native Android APIs, go ahead, download the ANE source, and modify it as per your needs. See the previous comment for ANE source location.

By Post Author

Hi Immanuel,

It turned out the problem was not with the android 4.4 but with my xml file. Despite making the changes you have said, my flash cs6 just reverts back to what it was before. I have checked the manually change permissions options despite that flash modifies the xml file and removes the four lines of code.

Below is my XML file could you pls tell me if you notice something wrong

<!–
Usage:

To localize the description, use the following format for the description element.

English App description goes here
French App description goes here
Japanese App description goes here

To localize the name, use the following format for the name element.

English App name goes here
French App name goes here
Japanese App name goes here

–>

micCheck
1.0.0

micCheck

micCheck

micCheck.swf
standard
false
true
false
portrait
cpu
false

false
false

<![CDATA[

]]>

com.inoel.ane.speech

@Gautham I’m sorry Gautham. I’m unable to see any issues in the snippet you posted.

@Euriza Ann. Do you have any more info on the error you are facing?

@Gerry The code must work everywhere, including JellyBean, and releases beyond that. What I mean’t, was that the device must have capability to recognize and parse voice without the help of internet. The code is actually agnostic to you. So if the voice search on the Google search widget / Google Now works, this code must work as well!

By Post Author

Dear Noel,
your extension is very interesting and I would like to test.
I work in Flash CC, i have included the .ane and i have tried to export the .apk but it generates errors. I past your simple code:
var speech:Speech = new Speech(“Please speak something!”);
trace(“Speech Supported: ” + speech.isSupported);
speech.listen();

errors:
Scene 1, Layer ‘Layer 1’, Frame 1, Line 1, Column 12 1046: Type was not found or was not a compile-time constant: Speech.
Scene 1, Layer ‘Layer 1’, Frame 1, Line 1, Column 25 1180: Call to a possibly undefined method Speech.

you can help me?
thanks, Antonio

@immanuel, I have try and try but not work.

I have create a new Android project for see if work, and in the first frame:


var speech:speech; new Speech(“Please speak something !”);
trace(“Speech Supported: ” + speech.isSupported);

I have add correctly the ane file into Actionscript settings and compiled the XML file.

When i compile it return an error: Constructor functions must be instance methods.

I understand that it is simple, but i’m a designer and not a programmer 😉

what i’m wrong?

tks

hi, new user here… this is my first comment.. First of all I thank you for providing this ANE. I used Adobe Flash Professional CC and when I compile, i get this error: “Included native extension XXXXXX included a SWF version 14 which is greater than the version of the root swf of this application : 11” .. I have never seen this error before therefore I have no idea what it is saying.. How do I solve this problem?

@Immanuel i an using your ANE and my code is:-
speech = new Speech(“Hello !”);
trace(“speech is recognize : “+speech.isSupported);
speech.listen();
speech.addEventListener(speechToTextEvent.VOICE_RECOGNIZED,onRecognized);
speech.addEventListener(speechToTextEvent.VOICE_NOT_RECOGNIZED,onNotRecognized);

function onNotRecognized(event:speechToTextEvent):void
{
trace(” no recognition”);
}
function onRecognized(event:speechToTextEvent):void
{
trace(” recognized”);
trace(“Now on recognitiondata is ! “+event.data);
}
but event speechToTextEvent.VOICE_RECOGNIZED doesn’t fire and above code only trace
speech is recognize : true

I have the same problem with AnaFB,because I also used Flash Pro CS6.
How to resolve the problem.
Who can help me?

I have a couple questions. Is this also compatible with Flash Pro CC (2014) or just Flash Builder? Also in the near future will there be a AIR for iOS release?

Immanuel, thanks for the great and simple Speech ANE! Looks like you wrote this a couple years ago, and I am surprised that it took me this long to finally implement it into my apps. Again, thanks for your hard work, I am very appreciative.

Hi Immanuel Noel,

I want to know if there is a solution for creating an SWF File for generating text from anyone’s voice?
Is there any way to use Windows Speech Recognition API to create high quality Voice Recognition Desktop apps (Windows) using AS3…(I mean API of a library that is installed on windows or even any Offline APIs if there is any, not online APIs)
I know we can create air for android apps that takes advantage of the Android’s speech recognition APIs, BUT I’m looking for a desktop (Windows) one …
Waiting for your answer…
Kind Regards.

Leave a Reply

Your email address will not be published. Required fields are marked *