New ActionScript Native Extension – Speech Recognition

Here is a Speech Native Extension that takes advantage of the Android’s speech recognition APIs.

Here is what the ANE does,

  • Provide an API to check if the feature is supported.
  • Provide an API to launch the default speech recognition UI and capture voice input.
  • Trigger events based on the success or failure of the above call.
    • In case of success, the appropriate event has information of the best match !
    • In case of failure, the appropriate event is fired.


Here are the steps required to include the ANE in your Android Project,

  • Download the ANE from here
  • In Flash Builder 4.6, open the Properties dialogue of the Project you want to include speech support in.
  • Select Build Path > Native Extensions
  • Add ANE – Select the location of the downloaded Speech.ane file
  • Navigate to Flex Build Packaging > Google Android > Native Extensions
  • Check the Package checkbox next to Speech.ane (This ANE supports only the Android Platform at the moment)
  • Apply and Close
  • Open the <ProjectName>-app.xml file
  • Scroll down until you find the Android manifest statements
  • Add the  below code just before </manifest>
    (i.e., inside the <manifest></manifest>tags)

    <application>
         <activity android:name="com.inoel.ane.speech.extensions.SpeechHandler">
         </activity>
    </application>
    

You will now be able to access the below APIs,

Initializing and specifying prompt text

//Creating and initializing a Speech Object
//Parameter - Prompt displayed on the Native Speech Recognition UI
var speech:Speech = new Speech("Please speak something !");

Check if speech input is supported

// Returns a Boolean value
trace("Speech Supported: " + speech.isSupported);

Accept Speech Input

//Calls the method which in turn fire's the Native Speech UI for accepting speech input
speech.listen();

Listen for the following events

  • speechToTextEvent.VOICE_RECOGNIZED – Fired on successful voice recognition. Recognized String resides in the event.data attribute.
  • speechToTextEvent.VOICE_NOT_RECOGNIZED – Fired on unsuccessful attempt. A common cause for failure could be network (data) unavailability. I believe this is overcome in new Ice Cream Sandwich which does not require a network connection to recognize speech ! Includes the error string in the event.data attribute.

Download the sample project from here. Please ensure you manually add the ANE with the steps above, or you get a ‘Class not found Exception’.

If you are interested in understanding how ANEs are used in Flash Builder 4.6, have a look at the recording in my previous post, http://blog.immanuelnoel.com/2011/10/20/native-extensions-flash-builder-webinar/

I also plan to post the entire source for the ANE itself. Watch out for that !

—– UPDATE 19/02/2012 —-
Source for the ANE is posted below,
http://blog.immanuelnoel.com/2012/02/19/speech-recognition-ane-source/

Speech Recognition ANE - Source
Native Extensions - Flash Builder Webinar



105 Responses to “New ActionScript Native Extension – Speech Recognition”

  1. I’m really impressed how Native Extensions ae taking mobile AIR. Very impressed 🙂

    AIR needs a little better performances, to minimize the gap between native Objective C apps and AIR apps.

    Love the way this is improving 🙂

  2. I tried running the SpeechMobile.fxp app in my FB 4.6. The compiled the app was run on my Galaxy SII. After I click the speak button it crashes with this message. http://tinypic.com/r/2hx0xv5/5 .

    Any idea why this happened? Please help

    Regards
    Girish R

  3. Ramon Helena

    Hello Immanuel,

    This look interesting, please check the Ane address is broken,

    Best,

    Ramón

  4. Thanks for pointing it out Girish.
    The ANE was targetted at Android 2.3.3. Fixed that. Do post in if you have any issues with it.

  5. Ramón Helena

    Thanks Immanuel, works correctly, congratulations!

    Best,

    Ramón

  6. Hello Immanuel,

    Thanks for your great ANE.

    Would you please add the api that allows developer to manipulate language options for speech recognition(i.e. Japanese) while their android is of different version(i.e English).

    best.

    shilik

  7. Shilik,
    I don’t really see an (Android SDK) API support for what you ask, but there is an option to specify FREE_FORM or WEB_FORM language model (More here – See LANGUAGE_MODEL_FREE_FORM and LANGUAGE_MODEL_WEB_SEARCH). I’ll put up the source in a few days, so you can go ahead and include this in the API ! 🙂
    As a default selection though, you can choose from a list of options in the Android settings menu.

  8. It crashes silently on my Galaxy Tab(1).
    A quick test showed that speech is supported, but then … Do you have any advice?

  9. Arne,
    Reason for that must be the missing Activity tags in the app.xml file ! Instructions on where and how to add them, are above.

  10. Immanuel, thank you for posting this. I would like to know if you have any recommendations for debugging a native extension once its running. for ex Log.d() doesn’t seem to work from DDMS in eclipse even when using the same package name,do you have any recommendations for debugging an ANE?

  11. Sean,
    Writing to log is definitely the best way to debug native code of an ANE.
    The below statement works for me,
    Log.v("Tag - App Name", "Custom Message");

    Log.d stands for debug log, and Log.v for Verbose. Ensure you have the right option selected from the dropDown in the LogCat view, and ofcourse, the right device selected 🙂

  12. Ramón Helena

    Immanuel, Happy new Year!!

    I would like to ask, if you have interesting in development one Text to Speech Extension.

    I try but I don’t be really fit with all the process.

    Let’s do it,

    Best,

    Ramón

  13. Thanks for offering up this extension.
    I was wondering if you could post the source code and / or elaborate on how you can get the onActivityResult method to run in an FREFunction. I’m currently trying to write an extension that requires that function’s use, and am having difficulty.

  14. Hi Immanuel

    This is really good. When you do you think you will be uploading your source? I’m interested to see how you got onActivityResult to work in the native extension. I am trying to make an ANE for a Music Picker.

    Intent i = new Intent();
    i.setAction(Intent.ACTION_GET_CONTENT);
    i.setType(“audio/*”);
    context.getActivity().startActivityForResult(Intent.createChooser(i, “Select audio file”),0);

    It shows the picker, but onActivityResult is not called – so I can never get the file that was chosen. So I am very interested to see how you have solved this problem!

  15. Mark/Andrew, I’ve been delayed in posting the source. I’ll try to post the source ASAP.
    In the meantime, the only solution to your problem is creating a new activity, listen for onActivityResult in the new activity, and fire it from within your extension code.
    I’ll try to post a detailed explaination in the following week.

  16. Ankur Arora

    Hi,

    I get this error.

    VerifyError: Error #1014: Class com.inoel.ane.speech::speechToTextEvent could not be found.

    I am using Flex 4.6 and your source project along with ANE.

  17. Ankur Arora

    Yes,

    Here is the code for your reference

    <![CDATA[


    ]]>

    Please let me know if you need any other information.

    Thanks
    Ankur

  18. Ankur Arora

    Oh I think tags got stripped in my last post but it is there anyways.

  19. I believe you are pointing to a code snippet within a MXML file.
    The activity tag needs to be added in the applicationname-app.xml file. Do you have that inserted as well ?

  20. Ankur Arora

    It worked for me now. The error was because it was not included in Flex Build Packaging.

    Thanks for your help and this awesome work.

    Ankur

  21. Hi Immanuel,
    I was able to check the package for android for speech.ANE. once it is done, i get the error on digital signature “The specified file doesn’t found” for C:\Certificates\Immanuel.p12.. I don’t have a certificate.. please help. Im a newbie to flex.. thanks in advance.. I use FB4.6

  22. Hi Immanuel,
    sorry to interrupt you again. I missed to create a certificate. I created a certificate there and proceed. I got the below error message.
    ==>”There are packaging errors/warnings. Check each native extension in the Flex Build Packaging page for each target platform.

    please be advised that the Android platforms native extensions are used when running the application in the AIR Debug Launcher. They may raise errors if they are incompatible with this environment. Would you like to continue?”
    I pressed OK since I’m sure that other platforms where unchecked at “Flex Build Packaging”.
    OK->Process terminated unexpectedly. invalid application description: unknown namespace:http//ns.adobe.com/air/application/3.1

    Process terminated unexpectedly.

    invalid application descriptor: Unknown namespace: http://ns.adobe.com/air/application/3.1

    Launch command details: “D:\Fx SDK\flex 4.5.1\bin\adl.exe” -runtime “D:\Fx SDK\flex 4.5.1\runtimes\air\win” -profile extendedMobileDevice -screensize 600×999:600×1024 -XscreenDPI 168 -XversionPlatform AND -extdir “C:\Users\ADMIN44\Adobe Flash Builder 4.6\.metadata\.plugins\com.adobe.flexbuilder.project.ui\ANEFiles\SpeechMobile\win32\” “C:\Users\ADMIN44\Adobe Flash Builder 4.6\SpeechMobile\bin-debug\SpeechMobile-app.xml” “C:\Users\ADMIN44\Adobe Flash Builder 4.6\SpeechMobile\bin-debug”

  23. @Dinesh, The ANE has been built only for the Android platform, and *WILL* crash if run on a emulator / simulator. Please test on an Android device only.
    Also, please ensure the below steps are followed (as indicated in the blog post above),
    -> Navigate to Flex Build Packaging > Google Android > Native Extensions
    -> Check the Package checkbox next to Speech.ane (This ANE supports only the Android Platform at the moment)

  24. @immanuel: executed @ Google Nexus One. First, ask for adobe air to install on device. I agreed and got the below one.
    Error occurred while packaging the application:

    C:\Users\ADMIN44\Adobe Flash Builder 4.6\SpeechMobile\bin-debug\SpeechMobile-app.xml(234): error 103: application.android.colorDepth is an unexpected element/attribute

    the air version specified:

    I also tried changing

    Do I have to make changes in xml file????

  25. @immanuel: executed @ Google Nexus One. First, ask for adobe air to install on device. I agreed and got the below one.
    Error occurred while packaging the application:

    C:\Users\ADMIN44\Adobe Flash Builder 4.6\SpeechMobile\bin-debug\SpeechMobile-app.xml(234): error 103: application.android.colorDepth is an unexpected element/attribute

    the air version specified: AIR 2.6

    I also tried changing to AIR 3.1

    Do I have to make changes in xml file????

  26. @Immanuel: It works fine now. I again downloaded from the blog and ran into device without any changes. Thanks again and sorry for interrupting you often.

  27. Sefi Ninio

    Hi Immanuel,

    First I want to thank you for coding and sharing this ANE, it works like a charm!

    When do you think you will be releasing the code? I would be very glad to see what you have done to make this work.

    Also, for the project I am working on, I need to get the entire list or results from google STT and not just the first one. Having the code will allow me to add this functionality…

    Thanks again,
    Sefi

  28. Sefi Ninio

    Immanuel,

    Thanks for posting the source, It’s very well documented, and adding the functionality to get the entire list of results was a breeze.

    Do you know if there is a way to use the native speech recognition API without showing it’s UI?
    Some times I will want to use my own UI, sometimes I will want to completely hide the UI…

    Sefi

  29. Hi,
    I am trying to use your ane file and using flash cs5.5, I was able to integrate the code and package it. However, in my application description file, I have added the extensions tag and the extensionsID as

    com.inoel.ane.speech

    It seems to compile without any problems. However there is absolutely no response. Not sure if voice thing is working or not. I also have added

    in the descriptor file just above. I am trying to print a message to see if voice is supported or not and it does not seem to respond to either. What could be the problem. Do I need to set permissions in the descriptor file?

  30. @Bharkan You only need to set the ‘Internet’ permission. The only other editing required in the descriptor file is to add the activity. Steps for the same are included within the post above.

    Do confirm once you get it working.

  31. It is working now. Thanks a bunch. However, it is on and off in my phone. It gives a message saying working but does not type the number I speak out.

  32. what i download is speech.zip instead of speech.ane. Could you please have a check? thanks

  33. Good morning,

    i am experimenting with your extension. but when i execute
    the command speech.listen(); the UI isn’t fired and it seems
    to get in a endless loop. i added the application tag and it
    returns true when i check if it is supported.
    what could cause this problem?

  34. Hello Immanuel and thank you for a lovely guide.

    I’m having a problem though,
    When i launch the app i see a popup saying something like “[something] is waiting for [something else]” and then the screen goes white. I can trace and i can place graphics in this white space but i don’t see the native gui showing. Anyone else experienced this?

    I’m completely new to Flash Builder and Android development, but skilled in as3.

  35. Nevermind. I had the code in the wrong part of the xml.
    Thanks again for a really cool ANE!

  36. Kevin Chua

    Hi, new learner here. My name is Kevin.
    I have stuck in the native extension on my sms apps.
    What i try to do is call my native extension code which trying to invoke the smsmanager.sendTextMessage(…) and create an notification.
    Somehow it didn’t send out the sms. Can you help me on this?

    Here is line in the native code :
    package com.leebrimelow.notifications;

    import android.app.Activity;
    import android.app.Notification;
    import android.app.NotificationManager;
    import android.app.PendingIntent;
    import android.content.Context;
    import android.content.Intent;
    import android.telephony.SmsManager;

    import com.adobe.fre.FREContext;
    import com.adobe.fre.FREFunction;
    import com.adobe.fre.FREInvalidObjectException;
    import com.adobe.fre.FREObject;
    import com.adobe.fre.FRETypeMismatchException;
    import com.adobe.fre.FREWrongThreadException;

    public class NotificationFunction extends Activity implements FREFunction {

    @Override
    public FREObject call(FREContext context, FREObject[] args) {

    String message = “”;
    try {
    message = args[0].getAsString();
    } catch (IllegalStateException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
    } catch (FRETypeMismatchException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
    } catch (FREInvalidObjectException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
    } catch (FREWrongThreadException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
    }

    String ns = Context.NOTIFICATION_SERVICE;
    NotificationManager mNotificationManager = (NotificationManager) context.getActivity().getSystemService(ns);

    long when = System.currentTimeMillis();

    Notification notification = new Notification(context.getResourceId(“drawable.notify”), message, when);

    notification.defaults |= Notification.DEFAULT_SOUND;

    CharSequence contentTitle = “Notification from Flash”;
    CharSequence contentText = “Hello” ;//message;
    Intent notificationIntent = new Intent();
    PendingIntent contentIntent = PendingIntent.getActivity(context.getActivity(), 0, notificationIntent, 0);

    notification.setLatestEventInfo(context.getActivity(), contentTitle, contentText, contentIntent);

    final int HELLO_ID = 1;

    mNotificationManager.notify(HELLO_ID, notification);

    //String sms = Context.TELEPHONY_SERVICE;
    //SmsManager smsManager1 = (SmsManager) context.getActivity().getSystemService(name)

    SmsManager smsManager = SmsManager.getDefault();
    String smsNumber = “012345678”;
    String smsText = “Testing message”;
    smsManager.sendTextMessage(smsNumber, null, smsText, null, null);

    return null;

    }

    }

  37. Hey!

    I’m using Flash CS6, and it seems like the additions to the manifest tags are being stripped out when publishing. This is causing the “Access of undefined property speechToTextEvent” error to be thrown.

    Do you know any ways around this?

  38. Bill s.

    Hi Immanuel:

    Can you give us the complementary .swc so that we can get at the .ane from Flash Pro CS5.5? The included SpeechLibrary.swc appears to be to compile the .ane; as it doesn’t work from Flash Pro.

    Has anyone else come up with the necessary .swc for this?

    Thanks

  39. Fixed my issue. Needed to check “Manually manage permissions and manifest additions in the application descriptor file” in the Permissions tab of the AIR for Android settings menu.

  40. edward de jong

    Dear Mr.Noel, thank you so much for posting this. I was able to make a test program that asked for voice, and it works. Voice recognition seems like a miracle when it works. However, i noticed that on my Galaxy Nexus phone, which just got upgraded to Jelly Bean (android 4.1) today, if I turn off WIFI, the voice recognizer doens’t work any more, but I know that my phone does have the offline voice recognition module already downloaded (a mere 22 MB for english).. is there some flag we have to change in the ANE to permit offline recognition? It would be far superior for an app not to incur data charges to do recognition…

  41. Thanks for pointing out the issue Edward. I haven’t really had the chance to get to this. Yet to get my hands on a 4.1 device 🙂
    Missing out on an important usecase though. Will try to get this sorted.

  42. Dosn’t seem to work on Galaxy SII (android-version 2.3.4) / Desire HD (android-version 2.3.5).
    The air app crashes in 99% of all applications tests. Sometimes directly after the app starts .. sometimes when the speech recognition popup appears (sorry! .. the app .. quit unexpectedly ..).

    this is how my applicationname-app.xml file looks:
    ..

    <![CDATA[

    ]]>

    ..

    i’m using flash cs6

  43. i’ve deleted the “<" at the beginning .. hope it will be posted now

    ![CDATA[
    uses-permission android:name=”android.permission.INTERNET”/>
    application>
    activity android:name=”com.inoel.ane.speech.extensions.SpeechHandler”>
    /activity>
    /application>
    /manifest>]]>

  44. @73C, verified to be working on Desire HD (android-version 2.3.5). Please check the comments above on problems with the App descriptor or the usage. I’m pretty sure the problem is with incorrect usage.

  45. yes your right. my fault! everything was ok with the ane-related stuff. the problem was somewhere else ( the app quits on deactivation ^^ ). so great work and thanks for that extension! and sorry for spaming with unrelated stuff but it was late that day 😉
    btw. works on android 4.0.4 too ( Galaxy SII )

  46. @73C Good to know you got it working. And just got excited knowing it works on IceCream too. I hadn’t tested that. Thanks!

  47. Hi, I am trying to use your life saver extension, but I am unable to run it on Flash CS6, it publish ok, and the issupport flag its enabled, but the event is never triggered. Any suggestion?

  48. Sumeet Basak

    Can we have speech recognition without opening the default speech recognition UI?

  49. I doubt that’s possible Sumeet.. Bypassing native speech recognition UI is not possible in the case of native apps as well!

  50. This is seriously awesome. Got it working on the first try, and it couldn’t be any easier to use.

  51. AnaFB

    Hello, thank you so much for your work. I’m using Flash Pro CS6 and I can’t understand why this doesn’t work for me.

    So when I publish my flash I have this error “1046: Type was not found or was not a compile-time constant: Speech.”

    I added de .ane file
    http://i41.tinypic.com/yorw0.jpg
    and this is my .xml file
    http://pastebin.com/kS7VSUeA

    I also prove the .swc tip you post above. Same result 🙁

    Hope you can help me

  52. AnaFB

    Ok, finally it’s working but I don’t know how to get the string.

    Box.text=e.data; doesn’t work in Flas CS6

    Any Idea?
    Thanks

  53. Kassim

    Is this works without internet? I mean offline speech recognition

  54. @Kassim, No. It needs internet. It has a potential of working without internet on the JellyBean, but I haven’t tried that!

  55. Sanjeev

    Hi Noel,

    Thank you so much for sharing this ANE. I am new in Flash builder and I am trying to implement your’s ANE in my android APP. It is working fine but can you guide me that how can I control it to stop playing or how can I control it by saying few commands like “Play”, “Stop”, “Next”, “Back”. Right now it is going back automatically. I want it to control on my commands not automatically. Please help and let me know if you can…

    Thanks & Regards,
    Sanjeev

  56. @Sanjeev
    Having your app react to commands is just a matter of reading the textual representation of the voice recognized, and then performing appropriate actions, right? 🙂

  57. Hi, i’m new to Flash builder and would like is there a possible to make the voice recognition able to detect mandarin instead of english?.
    And do u know any ways to compare 2 speeches to see whether they are similar? 1 of the voice will be pre-recorded. The other voice will be the user saying out the word or sentence where the app is suppose to detect it.

    The Sample is very fun to play ^v^

  58. ครูนรินทร์ Thailand

    How can I hide, move or resize Prompt “Speak now” on the Native Speech Recognition UI

  59. Hello Immanuel,

    Thanks so much for this fantastic ANE. I just wanted to know weather you have tested this on KitKat. I am experiencing some trouble as the listen function doesnt seem to work on this version. Thanks again for the ANE and the help.

    Regards,

    Gautham

  60. @gru_narin @ครูนรินทร์ Thailand
    The Speak Now prompt cannot be hidden. If you have a way to hide the prompt with native Android APIs, go ahead, download the ANE source, and modify it as per your needs. See the previous comment for ANE source location.

  61. Hi Immanuel,

    It turned out the problem was not with the android 4.4 but with my xml file. Despite making the changes you have said, my flash cs6 just reverts back to what it was before. I have checked the manually change permissions options despite that flash modifies the xml file and removes the four lines of code.

    Below is my XML file could you pls tell me if you notice something wrong

    <!–
    Usage:

    To localize the description, use the following format for the description element.

    English App description goes here
    French App description goes here
    Japanese App description goes here

    To localize the name, use the following format for the name element.

    English App name goes here
    French App name goes here
    Japanese App name goes here

    –>

    micCheck
    1.0.0

    micCheck

    micCheck

    micCheck.swf
    standard
    false
    true
    false
    portrait
    cpu
    false

    false
    false

    <![CDATA[

    ]]>

    com.inoel.ane.speech

  62. Euriza Ann

    I already used the above codes but if I debugging it there’s an error. UNDEFINED SPEECH

  63. @Gautham I’m sorry Gautham. I’m unable to see any issues in the snippet you posted.

    @Euriza Ann. Do you have any more info on the error you are facing?

    @Gerry The code must work everywhere, including JellyBean, and releases beyond that. What I mean’t, was that the device must have capability to recognize and parse voice without the help of internet. The code is actually agnostic to you. So if the voice search on the Google search widget / Google Now works, this code must work as well!

  64. Dear Noel,
    your extension is very interesting and I would like to test.
    I work in Flash CC, i have included the .ane and i have tried to export the .apk but it generates errors. I past your simple code:
    var speech:Speech = new Speech(“Please speak something!”);
    trace(“Speech Supported: ” + speech.isSupported);
    speech.listen();

    errors:
    Scene 1, Layer ‘Layer 1’, Frame 1, Line 1, Column 12 1046: Type was not found or was not a compile-time constant: Speech.
    Scene 1, Layer ‘Layer 1’, Frame 1, Line 1, Column 25 1180: Call to a possibly undefined method Speech.

    you can help me?
    thanks, Antonio

  65. Antonio

    @immanuel, I have try and try but not work.

    I have create a new Android project for see if work, and in the first frame:


    var speech:speech; new Speech(“Please speak something !”);
    trace(“Speech Supported: ” + speech.isSupported);

    I have add correctly the ane file into Actionscript settings and compiled the XML file.

    When i compile it return an error: Constructor functions must be instance methods.

    I understand that it is simple, but i’m a designer and not a programmer 😉

    what i’m wrong?

    tks

  66. hi, new user here… this is my first comment.. First of all I thank you for providing this ANE. I used Adobe Flash Professional CC and when I compile, i get this error: “Included native extension XXXXXX included a SWF version 14 which is greater than the version of the root swf of this application : 11” .. I have never seen this error before therefore I have no idea what it is saying.. How do I solve this problem?

  67. @Immanuel i an using your ANE and my code is:-
    speech = new Speech(“Hello !”);
    trace(“speech is recognize : “+speech.isSupported);
    speech.listen();
    speech.addEventListener(speechToTextEvent.VOICE_RECOGNIZED,onRecognized);
    speech.addEventListener(speechToTextEvent.VOICE_NOT_RECOGNIZED,onNotRecognized);

    function onNotRecognized(event:speechToTextEvent):void
    {
    trace(” no recognition”);
    }
    function onRecognized(event:speechToTextEvent):void
    {
    trace(” recognized”);
    trace(“Now on recognitiondata is ! “+event.data);
    }
    but event speechToTextEvent.VOICE_RECOGNIZED doesn’t fire and above code only trace
    speech is recognize : true

  68. I have the same problem with AnaFB,because I also used Flash Pro CS6.
    How to resolve the problem.
    Who can help me?

  69. I have a couple questions. Is this also compatible with Flash Pro CC (2014) or just Flash Builder? Also in the near future will there be a AIR for iOS release?

  70. Immanuel, thanks for the great and simple Speech ANE! Looks like you wrote this a couple years ago, and I am surprised that it took me this long to finally implement it into my apps. Again, thanks for your hard work, I am very appreciative.

Leave a Reply

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>