New ActionScript Native Extension – Speech Recognition

by

Here is a Speech Native Extension that takes advantage of the Android’s speech recognition APIs.

Here is what the ANE does,

  • Provide an API to check if the feature is supported.
  • Provide an API to launch the default speech recognition UI and capture voice input.
  • Trigger events based on the success or failure of the above call.
    • In case of success, the appropriate event has information of the best match !
    • In case of failure, the appropriate event is fired.


Here are the steps required to include the ANE in your Android Project,

  • Download the ANE from here
  • In Flash Builder 4.6, open the Properties dialogue of the Project you want to include speech support in.
  • Select Build Path > Native Extensions
  • Add ANE – Select the location of the downloaded Speech.ane file
  • Navigate to Flex Build Packaging > Google Android > Native Extensions
  • Check the Package checkbox next to Speech.ane (This ANE supports only the Android Platform at the moment)
  • Apply and Close
  • Open the <ProjectName>-app.xml file
  • Scroll down until you find the Android manifest statements
  • Add the  below code just before </manifest>
    (i.e., inside the <manifest></manifest>tags)

    <application>
     <activity android:name="com.inoel.ane.speech.extensions.SpeechHandler">
     </activity>
</application>

You will now be able to access the below APIs,

Initializing and specifying prompt text

//Creating and initializing a Speech Object
//Parameter - Prompt displayed on the Native Speech Recognition UI
var speech:Speech = new Speech("Please speak something !");

Check if speech input is supported

// Returns a Boolean value
trace("Speech Supported: " + speech.isSupported);

Accept Speech Input

//Calls the method which in turn fire's the Native Speech UI for accepting speech input
speech.listen();

Listen for the following events

  • speechToTextEvent.VOICE_RECOGNIZED – Fired on successful voice recognition. Recognized String resides in the event.data attribute.
  • speechToTextEvent.VOICE_NOT_RECOGNIZED - Fired on unsuccessful attempt. A common cause for failure could be network (data) unavailability. I believe this is overcome in new Ice Cream Sandwich which does not require a network connection to recognize speech ! Includes the error string in the event.data attribute.

Download the sample project from here. Please ensure you manually add the ANE with the steps above, or you get a ‘Class not found Exception’.

If you are interested in understanding how ANEs are used in Flash Builder 4.6, have a look at the recording in my previous post, http://blog.immanuelnoel.com/2011/10/20/native-extensions-flash-builder-webinar/

I also plan to post the entire source for the ANE itself. Watch out for that !

—– UPDATE 19/02/2012 —-
Source for the ANE is posted below,
http://blog.immanuelnoel.com/2012/02/19/speech-recognition-ane-source/

76 Responses to “New ActionScript Native Extension – Speech Recognition”

  2. Georges

    I’m really impressed how Native Extensions ae taking mobile AIR. Very impressed :)

    AIR needs a little better performances, to minimize the gap between native Objective C apps and AIR apps.

    Love the way this is improving :)

  3. Girish

    I tried running the SpeechMobile.fxp app in my FB 4.6. The compiled the app was run on my Galaxy SII. After I click the speak button it crashes with this message. http://tinypic.com/r/2hx0xv5/5 .

    Any idea why this happened? Please help

    Regards
    Girish R

  4. Ramon Helena

    Hello Immanuel,

    This look interesting, please check the Ane address is broken,

    Best,

    Ramón

  5. Immanuel Noel

    Thanks for pointing it out Girish.
    The ANE was targetted at Android 2.3.3. Fixed that. Do post in if you have any issues with it.

  8. Ramón Helena

    Thanks Immanuel, works correctly, congratulations!

    Best,

    Ramón

  9. shilik

    Hello Immanuel,

    Thanks for your great ANE.

    Would you please add the api that allows developer to manipulate language options for speech recognition(i.e. Japanese) while their android is of different version(i.e English).

    best.

    shilik

  10. Immanuel Noel

    Shilik,
    I don’t really see an (Android SDK) API support for what you ask, but there is an option to specify FREE_FORM or WEB_FORM language model (More here – See LANGUAGE_MODEL_FREE_FORM and LANGUAGE_MODEL_WEB_SEARCH). I’ll put up the source in a few days, so you can go ahead and include this in the API ! :)
    As a default selection though, you can choose from a list of options in the Android settings menu.

  11. ArneO

    It crashes silently on my Galaxy Tab(1).
    A quick test showed that speech is supported, but then … Do you have any advice?

  12. Immanuel Noel

    Arne,
    Reason for that must be the missing Activity tags in the app.xml file ! Instructions on where and how to add them, are above.

  14. Sean

    Immanuel, thank you for posting this. I would like to know if you have any recommendations for debugging a native extension once its running. for ex Log.d() doesn’t seem to work from DDMS in eclipse even when using the same package name,do you have any recommendations for debugging an ANE?

  15. Immanuel Noel

    Sean,
    Writing to log is definitely the best way to debug native code of an ANE.
    The below statement works for me,
    Log.v("Tag - App Name", "Custom Message");

    Log.d stands for debug log, and Log.v for Verbose. Ensure you have the right option selected from the dropDown in the LogCat view, and ofcourse, the right device selected :)

  16. Ramón Helena

    Immanuel, Happy new Year!!

    I would like to ask, if you have interesting in development one Text to Speech Extension.

    I try but I don’t be really fit with all the process.

    Let’s do it,

    Best,

    Ramón

  17. Andrew

    Thanks for offering up this extension.
    I was wondering if you could post the source code and / or elaborate on how you can get the onActivityResult method to run in an FREFunction. I’m currently trying to write an extension that requires that function’s use, and am having difficulty.

  18. Mark

    Hi Immanuel

    This is really good. When you do you think you will be uploading your source? I’m interested to see how you got onActivityResult to work in the native extension. I am trying to make an ANE for a Music Picker.

    Intent i = new Intent();
    i.setAction(Intent.ACTION_GET_CONTENT);
    i.setType(“audio/*”);
    context.getActivity().startActivityForResult(Intent.createChooser(i, “Select audio file”),0);

    It shows the picker, but onActivityResult is not called – so I can never get the file that was chosen. So I am very interested to see how you have solved this problem!

  20. Immanuel Noel

    Mark/Andrew, I’ve been delayed in posting the source. I’ll try to post the source ASAP.
    In the meantime, the only solution to your problem is creating a new activity, listen for onActivityResult in the new activity, and fire it from within your extension code.
    I’ll try to post a detailed explaination in the following week.

  21. Ankur Arora

    Hi,

    I get this error.

    VerifyError: Error #1014: Class com.inoel.ane.speech::speechToTextEvent could not be found.

    I am using Flex 4.6 and your source project along with ANE.

  23. Ankur Arora

    Yes,

    Here is the code for your reference

    <![CDATA[


    ]]>

    Please let me know if you need any other information.

    Thanks
    Ankur

  24. Ankur Arora

    Oh I think tags got stripped in my last post but it is there anyways.

  25. Immanuel Noel

    I believe you are pointing to a code snippet within a MXML file.
    The activity tag needs to be added in the applicationname-app.xml file. Do you have that inserted as well ?

  27. Ankur Arora

    It worked for me now. The error was because it was not included in Flex Build Packaging.

    Thanks for your help and this awesome work.

    Ankur

  28. Dinesh

    Hi Immanuel,
    I was able to check the package for android for speech.ANE. once it is done, i get the error on digital signature “The specified file doesn’t found” for C:\Certificates\Immanuel.p12.. I don’t have a certificate.. please help. Im a newbie to flex.. thanks in advance.. I use FB4.6

  30. Dinesh

    Hi Immanuel,
    sorry to interrupt you again. I missed to create a certificate. I created a certificate there and proceed. I got the below error message.
    ==>”There are packaging errors/warnings. Check each native extension in the Flex Build Packaging page for each target platform.

    please be advised that the Android platforms native extensions are used when running the application in the AIR Debug Launcher. They may raise errors if they are incompatible with this environment. Would you like to continue?”
    I pressed OK since I’m sure that other platforms where unchecked at “Flex Build Packaging”.
    OK->Process terminated unexpectedly. invalid application description: unknown namespace:http//ns.adobe.com/air/application/3.1

    Process terminated unexpectedly.

    invalid application descriptor: Unknown namespace: http://ns.adobe.com/air/application/3.1

    Launch command details: “D:\Fx SDK\flex 4.5.1\bin\adl.exe” -runtime “D:\Fx SDK\flex 4.5.1\runtimes\air\win” -profile extendedMobileDevice -screensize 600×999:600×1024 -XscreenDPI 168 -XversionPlatform AND -extdir “C:\Users\ADMIN44\Adobe Flash Builder 4.6\.metadata\.plugins\com.adobe.flexbuilder.project.ui\ANEFiles\SpeechMobile\win32\” “C:\Users\ADMIN44\Adobe Flash Builder 4.6\SpeechMobile\bin-debug\SpeechMobile-app.xml” “C:\Users\ADMIN44\Adobe Flash Builder 4.6\SpeechMobile\bin-debug”

  31. Immanuel Noel

    @Dinesh, The ANE has been built only for the Android platform, and *WILL* crash if run on a emulator / simulator. Please test on an Android device only.
    Also, please ensure the below steps are followed (as indicated in the blog post above),
    -> Navigate to Flex Build Packaging > Google Android > Native Extensions
    -> Check the Package checkbox next to Speech.ane (This ANE supports only the Android Platform at the moment)

  32. Dinesh

    @immanuel: executed @ Google Nexus One. First, ask for adobe air to install on device. I agreed and got the below one.
    Error occurred while packaging the application:

    C:\Users\ADMIN44\Adobe Flash Builder 4.6\SpeechMobile\bin-debug\SpeechMobile-app.xml(234): error 103: application.android.colorDepth is an unexpected element/attribute

    the air version specified:

    I also tried changing

    Do I have to make changes in xml file????

  33. Dinesh

    @immanuel: executed @ Google Nexus One. First, ask for adobe air to install on device. I agreed and got the below one.
    Error occurred while packaging the application:

    C:\Users\ADMIN44\Adobe Flash Builder 4.6\SpeechMobile\bin-debug\SpeechMobile-app.xml(234): error 103: application.android.colorDepth is an unexpected element/attribute

    the air version specified: AIR 2.6

    I also tried changing to AIR 3.1

    Do I have to make changes in xml file????

  34. Dinesh

    @Immanuel: It works fine now. I again downloaded from the blog and ran into device without any changes. Thanks again and sorry for interrupting you often.

  35. Sefi Ninio

    Hi Immanuel,

    First I want to thank you for coding and sharing this ANE, it works like a charm!

    When do you think you will be releasing the code? I would be very glad to see what you have done to make this work.

    Also, for the project I am working on, I need to get the entire list or results from google STT and not just the first one. Having the code will allow me to add this functionality…

    Thanks again,
    Sefi

  40. Sefi Ninio

    Immanuel,

    Thanks for posting the source, It’s very well documented, and adding the functionality to get the entire list of results was a breeze.

    Do you know if there is a way to use the native speech recognition API without showing it’s UI?
    Some times I will want to use my own UI, sometimes I will want to completely hide the UI…

    Sefi

  43. bharkan

    Hi,
    I am trying to use your ane file and using flash cs5.5, I was able to integrate the code and package it. However, in my application description file, I have added the extensions tag and the extensionsID as

    com.inoel.ane.speech

    It seems to compile without any problems. However there is absolutely no response. Not sure if voice thing is working or not. I also have added

    in the descriptor file just above. I am trying to print a message to see if voice is supported or not and it does not seem to respond to either. What could be the problem. Do I need to set permissions in the descriptor file?

  44. Immanuel Noel

    @Bharkan You only need to set the ‘Internet’ permission. The only other editing required in the descriptor file is to add the activity. Steps for the same are included within the post above.

    Do confirm once you get it working.

  45. bharkan

    It is working now. Thanks a bunch. However, it is on and off in my phone. It gives a message saying working but does not type the number I speak out.

  46. Wendy

    what i download is speech.zip instead of speech.ane. Could you please have a check? thanks

  49. Jens

    Good morning,

    i am experimenting with your extension. but when i execute
    the command speech.listen(); the UI isn’t fired and it seems
    to get in a endless loop. i added the application tag and it
    returns true when i check if it is supported.
    what could cause this problem?

  50. Thomas

    Hello Immanuel and thank you for a lovely guide.

    I’m having a problem though,
    When i launch the app i see a popup saying something like “[something] is waiting for [something else]” and then the screen goes white. I can trace and i can place graphics in this white space but i don’t see the native gui showing. Anyone else experienced this?

    I’m completely new to Flash Builder and Android development, but skilled in as3.

  51. Thomas

    Nevermind. I had the code in the wrong part of the xml.
    Thanks again for a really cool ANE!

  52. Kevin Chua

    Hi, new learner here. My name is Kevin.
    I have stuck in the native extension on my sms apps.
    What i try to do is call my native extension code which trying to invoke the smsmanager.sendTextMessage(…) and create an notification.
    Somehow it didn’t send out the sms. Can you help me on this?

    Here is line in the native code :
    package com.leebrimelow.notifications;

    import android.app.Activity;
    import android.app.Notification;
    import android.app.NotificationManager;
    import android.app.PendingIntent;
    import android.content.Context;
    import android.content.Intent;
    import android.telephony.SmsManager;

    import com.adobe.fre.FREContext;
    import com.adobe.fre.FREFunction;
    import com.adobe.fre.FREInvalidObjectException;
    import com.adobe.fre.FREObject;
    import com.adobe.fre.FRETypeMismatchException;
    import com.adobe.fre.FREWrongThreadException;

    public class NotificationFunction extends Activity implements FREFunction {

    @Override
    public FREObject call(FREContext context, FREObject[] args) {

    String message = “”;
    try {
    message = args[0].getAsString();
    } catch (IllegalStateException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
    } catch (FRETypeMismatchException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
    } catch (FREInvalidObjectException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
    } catch (FREWrongThreadException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
    }

    String ns = Context.NOTIFICATION_SERVICE;
    NotificationManager mNotificationManager = (NotificationManager) context.getActivity().getSystemService(ns);

    long when = System.currentTimeMillis();

    Notification notification = new Notification(context.getResourceId(“drawable.notify”), message, when);

    notification.defaults |= Notification.DEFAULT_SOUND;

    CharSequence contentTitle = “Notification from Flash”;
    CharSequence contentText = “Hello” ;//message;
    Intent notificationIntent = new Intent();
    PendingIntent contentIntent = PendingIntent.getActivity(context.getActivity(), 0, notificationIntent, 0);

    notification.setLatestEventInfo(context.getActivity(), contentTitle, contentText, contentIntent);

    final int HELLO_ID = 1;

    mNotificationManager.notify(HELLO_ID, notification);

    //String sms = Context.TELEPHONY_SERVICE;
    //SmsManager smsManager1 = (SmsManager) context.getActivity().getSystemService(name)

    SmsManager smsManager = SmsManager.getDefault();
    String smsNumber = “012345678″;
    String smsText = “Testing message”;
    smsManager.sendTextMessage(smsNumber, null, smsText, null, null);

    return null;

    }

    }

  53. Chris

    Hey!

    I’m using Flash CS6, and it seems like the additions to the manifest tags are being stripped out when publishing. This is causing the “Access of undefined property speechToTextEvent” error to be thrown.

    Do you know any ways around this?

  54. Bill s.

    Hi Immanuel:

    Can you give us the complementary .swc so that we can get at the .ane from Flash Pro CS5.5? The included SpeechLibrary.swc appears to be to compile the .ane; as it doesn’t work from Flash Pro.

    Has anyone else come up with the necessary .swc for this?

    Thanks

  55. Chris

    Fixed my issue. Needed to check “Manually manage permissions and manifest additions in the application descriptor file” in the Permissions tab of the AIR for Android settings menu.

  56. edward de jong

    Dear Mr.Noel, thank you so much for posting this. I was able to make a test program that asked for voice, and it works. Voice recognition seems like a miracle when it works. However, i noticed that on my Galaxy Nexus phone, which just got upgraded to Jelly Bean (android 4.1) today, if I turn off WIFI, the voice recognizer doens’t work any more, but I know that my phone does have the offline voice recognition module already downloaded (a mere 22 MB for english).. is there some flag we have to change in the ANE to permit offline recognition? It would be far superior for an app not to incur data charges to do recognition…

  57. Immanuel Noel

    Thanks for pointing out the issue Edward. I haven’t really had the chance to get to this. Yet to get my hands on a 4.1 device :)
    Missing out on an important usecase though. Will try to get this sorted.

  59. 73C

    Dosn’t seem to work on Galaxy SII (android-version 2.3.4) / Desire HD (android-version 2.3.5).
    The air app crashes in 99% of all applications tests. Sometimes directly after the app starts .. sometimes when the speech recognition popup appears (sorry! .. the app .. quit unexpectedly ..).

    this is how my applicationname-app.xml file looks:
    ..

    <![CDATA[

    ]]>

    ..

    i’m using flash cs6

  60. 73C

    i’ve deleted the “<" at the beginning .. hope it will be posted now

    ![CDATA[
    uses-permission android:name="android.permission.INTERNET"/>
    application>
    activity android:name="com.inoel.ane.speech.extensions.SpeechHandler">
    /activity>
    /application>
    /manifest>]]>

  61. Immanuel Noel

    @73C, verified to be working on Desire HD (android-version 2.3.5). Please check the comments above on problems with the App descriptor or the usage. I’m pretty sure the problem is with incorrect usage.

  62. 73C

    yes your right. my fault! everything was ok with the ane-related stuff. the problem was somewhere else ( the app quits on deactivation ^^ ). so great work and thanks for that extension! and sorry for spaming with unrelated stuff but it was late that day ;)
    btw. works on android 4.0.4 too ( Galaxy SII )

  63. Immanuel Noel

    @73C Good to know you got it working. And just got excited knowing it works on IceCream too. I hadn’t tested that. Thanks!

  64. Adan

    Hi, I am trying to use your life saver extension, but I am unable to run it on Flash CS6, it publish ok, and the issupport flag its enabled, but the event is never triggered. Any suggestion?

  66. Sumeet Basak

    Can we have speech recognition without opening the default speech recognition UI?

  67. Immanuel Noel

    I doubt that’s possible Sumeet.. Bypassing native speech recognition UI is not possible in the case of native apps as well!

  70. T

    This is seriously awesome. Got it working on the first try, and it couldn’t be any easier to use.

Leave a Reply

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>