Merge branch 'master' of https://github.com/Microsoft/AI

solutions/Virtual-Assistant/src/csharp/testharnesses/assistant-SpeechDevicesSDK/license.md

@@ -0,0 +1,77 @@
+# Speech Devices SDK license agreement
+
+## MICROSOFT SOFTWARE LICENSE TERMS<br>SPEECH DEVICES SOFTWARE DEVELOPMENT KIT (SDK) 
+
+> IF YOU LIVE IN (OR ARE A BUSINESS WITH YOUR PRINCIPAL PLACE OF BUSINESS IN) THE UNITED STATES, PLEASE READ THE "BINDING ARBITRATION AND CLASS ACTION WAIVER" SECTION BELOW. IT AFFECTS HOW DISPUTES ARE RESOLVED.
+
+These license terms are an agreement between you and Microsoft Corporation (or one of its affiliates). They apply to the software named above and any Microsoft services or software updates (except to the extent such services or updates are accompanied by new or additional terms, in which case those different terms apply prospectively and do not alter your or Microsoft's rights relating to pre-updated software or services). **IF YOU COMPLY WITH THESE LICENSE TERMS, YOU HAVE THE RIGHTS BELOW.**
+
+**1\. INSTALLATION AND USE RIGHTS.**
+
+a)	General. You may install and use any number of copies of the software on your devices, solely to evaluate and test it for your internal business purposes. You may not use the software in a live operating environment unless Microsoft permits you to do so under another agreement.
+
+b)	Included Microsoft Applications. The software may include other Microsoft applications. These license terms apply to those included applications, if any, unless other license terms are provided with the other Microsoft applications.
+
+c)	Third Party Components. The software may include third party components with separate legal notices or governed by other agreements, as may be described in the Third Party Notices file(s) accompanying the software.
+
+d)	Package Managers. The software may include package managers, like NuGet, that give you the option to download other Microsoft and third-party software packages to use with your application. Those packages are under their own licenses, and not this agreement. Microsoft does not distribute, license or provide any warranties for any of the third-party packages.
+
+e)	If you are a direct competitor, and you access or use the software for purposes of competitive benchmarking, analysis, or intelligence gathering, you waive as against Microsoft, its subsidiaries, and its affiliated companies (including prospectively) any competitive use, access, and benchmarking test restrictions in the terms governing your software to the extent your terms of use are, or purport to be, more restrictive than Microsoft's terms. If you do not waive any such purported restrictions in the terms governing your software, you are not allowed to access or use this software, and will not do so.
+
+**2\. SCOPE OF LICENSE. The software is licensed, not sold. Microsoft reserves all other rights. Unless applicable law gives you more rights despite this limitation, you will not (and have no right to):**
+
+a)	work around any technical limitations in the software that only allow you to use it in certain ways;
+
+b)	reverse engineer, decompile or disassemble the software;
+
+c)	remove, minimize, block, or modify any notices of Microsoft or its suppliers in the software;
+
+d)	use the software in any way that is against the law or to create or propagate malware; or
+
+e)	share, publish, or lend the software (except for any distributable code, and then subject to the applicable terms above), provide the software as a stand-alone hosted solution for others to use, or transfer the software or this agreement to any third party.
+
+f)	You must use the software only in combination with Microsoft Azure Cognitive Services, Speech service
+
+**3\. DATA COLLECTION.** This software may interact with other Microsoft products that collect data that is transmitted to Microsoft. To learn more about how Microsoft processes personal data we collect, please see the Microsoft Privacy Statement at http://go.microsoft.com/fwlink/?LinkId=248681.
+
+**4\. PRE-RELEASE SOFTWARE.** The software is a pre-release version. It may not operate correctly. It may be different from the commercially released version.
+
+**5\. FEEDBACK.** If you give feedback about the software to Microsoft, you give to Microsoft, without charge, the right to use, share and commercialize your feedback in any way and for any purpose. You will not give feedback that is subject to a license that requires Microsoft to license its software or documentation to third parties because Microsoft includes your feedback in them. These rights survive this agreement.
+
+**6\. PERIOD.** This agreement is effective on your acceptance and may be terminated by you or Microsoft at any time. 
+
+**7\. TERMINATION.** Without prejudice to any other rights, Microsoft may terminate this agreement if you fail to comply with any of its terms or conditions. In such event, you must destroy all copies of the software and all of its component parts.
+
+**8\.EXPORT RESTRICTIONS.** You must comply with all domestic and international export laws and regulations that apply to the software, which include restrictions on destinations, end users, and end use. For further information on export restrictions, visit (aka.ms/exporting).
+
+**9\. SUPPORT SERVICES.** Microsoft is not obligated under this agreement to provide any support services for the software. Any support provided is "as is", "with all faults", and without warranty of any kind.
+
+**10\. UPDATES.** The software may periodically check for updates, and download and install them for you. You may obtain updates only from Microsoft or authorized sources. Microsoft may need to update your system to provide you with updates. You agree to receive these automatic updates without any additional notice. Updates may not include or support all existing software features, services, or peripheral devices.
+
+**11\. BINDING ARBITRATION AND CLASS ACTION WAIVER.** This Section applies if you live in (or, if a business, your principal place of business is in) the United States.  If you and Microsoft have a dispute, you and Microsoft agree to try for 60 days to resolve it informally. If you and Microsoft can't, you and Microsoft agree to binding individual arbitration before the American Arbitration Association under the Federal Arbitration Act ("FAA"), and not to sue in court in front of a judge or jury. Instead, a neutral arbitrator will decide. Class action lawsuits, class-wide arbitrations, private attorney-general actions, and any other proceeding where someone acts in a representative capacity are not allowed; nor is combining individual proceedings without the consent of all parties. The complete Arbitration Agreement contains more terms and is at aka.ms/arb-agreement-1. You and Microsoft agree to these terms.
+
+**12\. ENTIRE AGREEMENT.** This agreement, and any other terms Microsoft may provide for supplements, updates, or third-party applications, is the entire agreement for the software.
+
+**13\. APPLICABLE LAW AND PLACE TO RESOLVE DISPUTES.** If you acquired the software in the United States or Canada, the laws of the state or province where you live (or, if a business, where your principal place of business is located) govern the interpretation of this agreement, claims for its breach, and all other claims (including consumer protection, unfair competition, and tort claims), regardless of conflict of laws principles, except that the FAA governs everything related to arbitration. If you acquired the software in any other country, its laws apply, except that the FAA governs everything related to arbitration. If U.S. federal jurisdiction exists, you and Microsoft consent to exclusive jurisdiction and venue in the federal court in King County, Washington for all disputes heard in court (excluding arbitration). If not, you and Microsoft consent to exclusive jurisdiction and venue in the Superior Court of King County, Washington for all disputes heard in court (excluding arbitration).
+
+**14\. CONSUMER RIGHTS; REGIONAL VARIATIONS.** This agreement describes certain legal rights. You may have other rights, including consumer rights, under the laws of your state, province, or country. Separate and apart from your relationship with Microsoft, you may also have rights with respect to the party from which you acquired the software. This agreement does not change those other rights if the laws of your state, province, or country do not permit it to do so. For example, if you acquired the software in one of the below regions, or mandatory country law applies, then the following provisions apply to you:
+
+a)	Australia. You have statutory guarantees under the Australian Consumer Law and nothing in this agreement is intended to affect those rights.
+
+b)	Canada. If you acquired this software in Canada, you may stop receiving updates by turning off the automatic update feature, disconnecting your device from the Internet (if and when you re-connect to the Internet, however, the software will resume checking for and installing updates), or uninstalling the software. The product documentation, if any, may also specify how to turn off updates for your specific device or software.
+
+c)	Germany and Austria.
+
+> (i)	Warranty. The properly licensed software will perform substantially as described in any Microsoft materials that accompany the software. However, Microsoft gives no contractual guarantee in relation to the licensed software.
+
+> (ii)	Limitation of Liability. In case of intentional conduct, gross negligence, claims based on the Product Liability Act, as well as, in case of death or personal or physical injury, Microsoft is liable according to the statutory law.
+
+> Subject to the foregoing clause (ii), Microsoft will only be liable for slight negligence if Microsoft is in breach of such material contractual obligations, the fulfillment of which facilitate the due performance of this agreement, the breach of which would endanger the purpose of this agreement and the compliance with which a party may constantly trust in (so-called "cardinal obligations"). In other cases of slight negligence, Microsoft will not be liable for slight negligence.
+
+**15\. DISCLAIMER OF WARRANTY. THE SOFTWARE IS LICENSED "AS IS." YOU BEAR THE RISK OF USING IT. MICROSOFT GIVES NO EXPRESS WARRANTIES, GUARANTEES, OR CONDITIONS. TO THE EXTENT PERMITTED UNDER APPLICABLE LAWS, MICROSOFT EXCLUDES ALL IMPLIED WARRANTIES, INCLUDING MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT.**
+
+**16\. LIMITATION ON AND EXCLUSION OF DAMAGES. IF YOU HAVE ANY BASIS FOR RECOVERING DAMAGES DESPITE THE PRECEDING DISCLAIMER OF WARRANTY, YOU CAN RECOVER FROM MICROSOFT AND ITS SUPPLIERS ONLY DIRECT DAMAGES UP TO U.S. $5.00. YOU CANNOT RECOVER ANY OTHER DAMAGES, INCLUDING CONSEQUENTIAL, LOST PROFITS, SPECIAL, INDIRECT OR INCIDENTAL DAMAGES.**
+
+**This limitation applies to (a) anything related to the software, services, content (including code) on third party Internet sites, or third-party applications; and (b) claims for breach of contract, warranty, guarantee, or condition; strict liability, negligence, or other tort; or any other claim; in each case to the extent permitted by applicable law.**
+
+**It also applies even if Microsoft knew or should have known about the possibility of the damages. The above limitation or exclusion may not apply to you because your state, province, or country may not allow the exclusion or limitation of incidental, consequential, or other damages.**

solutions/Virtual-Assistant/src/csharp/testharnesses/assistant-SpeechDevicesSDK/readme.md

@@ -0,0 +1,82 @@
+# Speech Devices SDK Integration with a Virtual Assistant
+
+## Overview
+
+This sample demonstrates how to integrate the [Speech Devices SDK](https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/speech-devices-sdk-qsg) with a bot hosted on the [Microsoft Bot Framework Direct Line](https://docs.microsoft.com/en-us/azure/bot-service/bot-builder-howto-direct-line?view=azure-bot-service-4.0&tabs=cscreatebot%2Ccsclientapp%2Ccsrunclient). This is built using an implementation of the [Virtual Assistant](https://github.com/Microsoft/AI/tree/master/solutions/Virtual-Assistant
+) and sends related events to the bot on conversation startup.
+
+* The client uses the Speech Devices SDK to demonstrate speech to text (both keyword & continuous recognition listening modes) and text to speech.
+* The client is connected via websocket to a Bot Framework Direct Line bot, and will parse bot responses into TTS.
+* The conversation is rendered similar to the web chat experience, new messages will be displayed after the speech synthesizer finishes playing the corresponding activity.
+* User can configured common settings in the application to configure the client to work with their own bot.
+* If a user is participating in the Neural TTS Preview, they can provide their keys in configuration and toggle it under settings.
+
+## Setup
+
+For basic setup of your DDK and running an Android application, visit the documentation for the [Speech Devices SDK](https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/speech-devices-sdk-qsg) with a bot hosted on the [Microsoft Bot Framework Direct Line](https://docs.microsoft.com/en-us/azure/bot-service/bot-builder-howto-direct-line?view=azure-bot-service-4.0&tabs=cscreatebot%2Ccsclientapp%2Ccsrunclient). This is built using an implementation of the [Virtual Assistant](https://github.com/Microsoft/AI/tree/master/solutions/Virtual-Assistant
+). This will explain what software to install, how to deploy an Android application on your DDK device, and install a custom wake word. You can deploy our sample application in the same method.
+
+The settings screen of the sample application allows user to make quick configuration changes without needing to dive into the code. The default values for these can be set at `/res/values/configuration.xml`.
+You will also need to provide connection settings to the Speech Service and can find details to get a key at [Speech Service documentation](https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/get-started). 
+If these settings are missing, the application will raise an alert and not start.
+
+| Key       | Description  |
+| ------------- |:-------------:| 
+| Default_directline_secret | The Direct Line secret for your hosted bot. |
+| Default_from_user_id | ID attached to the user activities sent to bot. |
+| Default_from_bot_id | ID of your hosted bot, this is used to filter activities as they are passed through the web socket.|
+| Default_locale | Bot's locale to determine responses *(you can update your device's language settings for the application to show a fully localized environment)* |
+| Default_inputhint | The input hint that determines what recognizer mode is enabled upon conversation startup. If the conversation starts with a greeting from the bot, it is recommended to leave this as ignoringInput and let the recognizer handle this as activities are parsed. <br> *Valid values: ignoringInput, acceptingInput, expectingInput* |
+| Default_latitude | Mock latitude that can be sent in a location event to the bot. |
+| Default_longitude | Mock longitude that can be sent in a location event to the bot. |
+| Default_devicegeometry | Device microphone configuration. <br> *Valid values: Circular6+1, Linear4* |
+| Default_selectedgeometry | Software microphone configuration. <br> *Valid values: Circular6+1, Circular3+1, Linear4, Linear2* |
+| Default_gender | Gender of your bot's TTS voice. <br> *Valid values: male, female* |
+| Default_neural | Boolean to determine if Neural TTS is enabled for your demo. <br> *Valid values: true, false* |
+
+### Using the Speech Recognizer
+
+The application will switch between different recognition modes based on the input hints returned by the bot.
+
+| Input Hint | Speech Recognizer | Description | Sample Utterance | 
+| ------------- |:-------------:| -----:| -----:|
+| Ignoring input      | No recognizer enabled | The bot is not expecting any response from the user and the microphone is closed. This may be used when a bot has multiple activities to send concurrently. | n/a |
+| Accepting input      | Keyword recognizer | The bot is passively awaiting a response from the user, it will open the microphone but not react unless the configured keyword has been uttered. | *"Assistant, send an email to Mark"* |
+| Expecting input      | Continuous recognizer | The bot is awaiting a response from the user and will leave the microphone open. This is useful for multi-turn dialogs where a user will have a natural conversation with their bot.| *"Send an email to Mark"* |
+
+For more details on how to use input hints, visit [Add input hints to your messages](https://docs.microsoft.com/en-us/azure/bot-service/bot-builder-howto-add-input-hints?view=azure-bot-service-4.0&tabs=cs).
+
+### Customizing your wake word
+You can use a provided wake word (Assistant, Computer or Machine) or create your own, either way you will need to configure your Roobo to work with that wake word.
+- Create your own custom wake word following the instructions at [Create a custom wake word using Speech service](https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/speech-devices-sdk-create-kws)
+- To install the necessary files on your Roobo device, follow Step 5 of [Get started with the Speech Devices SDK](https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/speech-devices-sdk-qsg) to install your custom word.
+- Update the Keyword variable in Configuration.java with your wake word.
+
+## Virtual Assistant
+
+### Localization
+
+Under **Settings**, update the locale to connect to a multi-language bot. This sets the locale for the Direct Line conversation & the Text to Speech voice. The client is configured for English, Chinese, French, Italian, Spanish, & German. 
+Visit [Language and region support for Speech Service API](https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/language-support) for other supported languages.
+
+### Event Architecture
+When a new conversation is started, the client sends multiple events to the Virtual Assistant on startup.
+1. The `IPA.TimeZone` event with the Android device's time zone.
+2. The `IPA.Location` event using mocked coordinates in `configuration.xml`. If you make any changes to the coordinates mid-conversation (under **Settings**), the client will send an updated event when the conversation is resumed.
+
+# Contributing
+
+This project welcomes contributions and suggestions.  Most contributions require you to agree to a
+Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
+the rights to use your contribution. For details, visit https://cla.microsoft.com.
+
+When you submit a pull request, a CLA-bot will automatically determine whether you need to provide
+a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions
+provided by the bot. You will only need to do this once across all repos using our CLA.
+
+This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
+For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
+contact [[email protected]](mailto:[email protected]) with any additional questions or comments.
+
+## License
+Copyright (c) Microsoft Corporation. All rights reserved.

solutions/Virtual-Assistant/src/csharp/testharnesses/assistant-SpeechDevicesSDK/virtualassistant/.gitignore

@@ -0,0 +1,8 @@
+*.iml
+.gradle
+.idea
+/local.properties
+.DS_Store
+**/build
+/captures
+.externalNativeBuild