Exploring Windows Mixed Reality, switching between 2D / 3D and embedding Web Views

In these days, I’m exploring the options for switching between Unity 3D and XAML 2D views integrating the access to UWP APIs for content hosted on Web Views.

This scenario could be particularly useful if an app needs to reuse existing code, perhaps available in a website, with the requirement to access the Windows Runtime when executed in Windows Mixed Reality devices and activated from a Unity 3D scene.

Creating the Unity project

To start, I created a new Unity scene and imported the HoloToolkit package downloaded from here.

I applied the Scene and Project settings from the HoloToolkit|Configure menu and added the following prefabs available from the imported package:

  • HoloLens Camera
  • InputManager
  • Cursor

Then, I added a simple cube which, when gazed and air-tapped, permits to trigger the view switching.

I defined a new script TapBehaviour to capture the event and call the 2D view:

The interesting part here is the one using the AppViewManager for switching views: after some research, I decided to use this class from GitHub as a starting point for handling the logic via CoreApplicationView.

I created a new script in Unity (AppViewManager.cs) and added the source code from the GitHub repository.

In this way, it is possible to retrieve the list of registered views or a specific one using for instance

And then simply transition to the new page by calling the Switch() or SwitchAsync() methods.

Creating and registering the 2D views

The 2D view named ContentPage needed to be defined and registered in the UWP project generated by Unity using the File|Build Settings|Build command after setting the UWP Build type to XAML to be able to define additional views in my project

Then, I opened the project in Visual Studio and added a new page called ContentPage.xaml containing a WebView and a button to switch back to Unity 3D

And the code makes use again of the AppViewManager previously imported in Unity:

We are now making use of two pages MainPage and ContentPage, respectively hosting the 3D and 2D content. Before using them, I needed to register both in the App.cs class right after initialising the Unity player:

Accessing UWP APIs from the hosted web page

The Web View is actually showing content from this web page hosted in my personal web-site which tries to access the UWP APIs if hosted with elevated permissions:

To enable Windows Runtime access, I modified the Package.appxmanifest and added the following under the Application section:

After this step, I was able to launch the app using an HoloLens device or emulator:

And then switch to the 2D view hosting the web page accessing UWP APIs on HoloLens by tapping the cube:

And return to the Unity 3D view using the “Click me” button.

As usual, the source code is available for download on GitHub.

Experiments with HoloLens, Bot Framework and LUIS: adding text to speech

Previously I blogged about creating a Mixed Reality 2D app integrating with a Bot using LUIS via the Direct Line channel available in the Bot Framework.

I decided to add more interactivity to the app by also enabling text to speech for the messages received by the Bot: this required the addition of a new MediaElement for the Speech synthesiser to the main XAML page:

Then I initialized a new SpeechSynthesizer at the creation of the page:

And added a new Speech() method using the media element:

When a new response is received from the Bot, the new Speech() method is called:

And then the recognition for a new phrase is started again via the MediaEnded event to simulate a conversation between the user and the Bot:

As usual, the source code is available for download on GitHub.

Unity: “Failed to initialize unity graphics” error

Today, I launched Unity for some HoloLens and Mixed Reality dev and received the following error:

After some research, I verified the latest graphic drivers were available in my machine and ran a dxdiag to check that all the DirectX features were enabled:

Then, I searched for command line arguments on the official documentation and modified the launch link as below:

And this last step allowed me to run again Unity in my machine:

Microsoft Bot Framework: using a LuisDialog for processing intents

In the previous post, I blogged about integrating a Holographic 2D app with Bot Framework and LUIS.

I also spent some time going through these great samples for some presentations and found a very nice implementation for handling the Bot messages when LUIS intents are recognised.

Instead of using the exposed LUIS endpoint and parsing the returned JSON, the framework already provides a specific LuisDialog<> type which can be used for handling the various intents, in order to make the code cleaner and more extensible.

I’ve then modified the HoloLensBotDemo sample and added a new RootLuisDialog:

And this is all the code now needed for the Bot.

The updated source code is available on GitHub.

Happy coding!

 

Experiments with HoloLens, Bot Framework, LUIS and Speech Recognition

Recently I had the opportunity to use a HoloLens device for some personal training and building some simple demos.

One of the scenarios that I find very intriguing is the possibility of integrating Mixed Reality and Artificial Intelligence (AI) in order to create immersive experiences for the user.

I decided to perform an experiment by integrating a Bot, Language Understanding Intelligent Services (LUIS), Speech Recognition and Mixed Reality via a Holographic 2D app.

The idea was to create a sort of “digital assistant” of myself that can be contacted using Mixed Reality: the first implementation contains only basic interactions (answering questions like “What are your favourite technologies” or “What’s your name”) but these could be easily be expanded in the future with features like time management (via the Graph APIs) or tracking projects status, etc.

Creating the LUIS application

To start, I created a new LUIS application in the portal with a list of intents that needed to be handled:

In the future, this could be further extended with extra capabilities.

After defining the intents and utterances, I trained and published my LUIS app to Azure and copied the key and URL for usage in my Bot:

Creating the Bot

I proceeded with the creation of the Bot using Microsoft Bot framework downloading the Visual Studio template and creating a new project:

The Bot template already defined a dialog named RootDialog so I extended the generated project with the classes required for parsing the JSON from the LUIS endpoint:

And then processed the various LUIS intents in RootDialog (another option is the usage of the LuisDialog and LuisModel classes as explained here):

Then, I tested the implementation using the Bot Framework Emulator:

And created a new Bot definition in the framework portal.

After that, I published it to Azure with an updated Web.config with the generated Microsoft App ID and password:

Since the final goal was the communication with an UWP HoloLens application, I enabled the Diret Line channel:

Creating the Holographic 2D app

Windows 10 UWP apps are executed on the HoloLens device as Holographic 2D apps that can be pinned in the environment.

I created a new project using the default Visual Studio Template:

And then added some simple text controls in XAML to receive the input and display the response from the Bot:

I decided to use the SpeechRecognizer APIs for receiving the input via voice (another option could be the usage of Cognitive Services):

The SendToBot() method makes use of the Direct Line APIs which permit communication with the Bot using the channel previously defined:

And then I got the app running on HoloLens and interfacing with a Bot using LUIS for language understanding and Speech recognition:

The source code of the project is available on GitHub here.

Happy coding!

Microsoft Bot Framework: showing a welcome message at the start of a new conversation

Recently I’ve worked on some projects related to Bot Framework and enjoyed the functionalities which permit to automate actions in response to user interactions.

It is important to provide the user with a great experience: one “nice touch” can be achieved by providing a welcome message at the beginning of a new conversation.

The first solution I tried was triggering the welcome message in the ConversationUpdate activity:

When I ran this, the message was presented twice in the Bot Framework Emulator:

After some investigation, I discovered that the ConversationUpdate activity is triggered both when the connection to the Bot is established and when a new user joins the conversation.

As explained on GitHub, the correct way to handle this case is by showing the welcome message only when a new user is added:

Using this approach the welcome message is displayed properly:

Happy coding!

Validate input in Xamarin.Forms using INotifyDataErrorInfo, custom Behaviors, Effects and Prism

This post is also available in the Premier Developer blog.

I have recently been investigating the support available in Xamarin.Forms for validation and, in particular, researched the possibility of using INotifyDataErrorInfo to complement the traditional approach of using Behaviors.

In simple scenarios, it’s possible to perform validation by simply attaching a Behavior to the required view, as shown by the following sample code:

In this case, we are validating the input and modifying the UI if the number of characters entered is less than 5.

What about more articulated scenarios when multiple business rules are required to be checked for the input values?

In these cases, we could take advantage of other types available to make our code more structured and extensible:

  • INotifyDataErrorInfo: available in Xamarin.Forms. When implemented it permits specifying custom validation supporting multiple errors per property, cross-property errors and entity-level errors;
  • DataAnnotations decorate the data models using attributes which specify validation conditions to be applied to the specific field;
  • Forms Behaviors specify the specific UI to be applied to the specific validation scenarios, integrating with INotifyDataErrorInfo and DataAnnotations.

To start exploring this approach, I created a new Xamarin.Forms Prism solution using the Prism Template Pack which generated the following project structure:

Then, I added the following new model to be validated using DataAnnotations and INotifyDataErrorInfo:

The model uses attributes declared in the SystemComponentModel.DataAnnotations namespace which can be referenced in the solution modifying the Portable Class Library profile of the UsingValidation common project:

Quick tip: to be able to change the PCL profile I had to remove all the NuGet packages used by the common project, remove the Windows Phone 8 profile and then add back all the removed NuGet packages to the UsingValidation PCL.

To use the capability offered by INotifyDataErrorInfo, the model needs to implements 3 members defined in the interface:

  • GetErrors() returns an IEnumerable sequence of strings containing the error messages triggered by validation;
  • the HasErrors property returns a boolean value indicating if there are validation errors;
  • ErrorsChanged event can be triggered to Notify if the validation errors have been updated.

This interface is quite flexible and is designed to be customised depending on the different scenarios needed: I took as a starting point this implementation available on GitHub and modified it accordingly: I decided to separate the implementation of INotifyDataErrorInfo in a different base class called ValidationBase which contains the following code using a Dictionary<string, List<string>> needed for storing the generated validation errors:

The validation is performed by this method which evaluates the DataAnnotations decorating the model using the Validator available in the System.ComponentModel.DataAnnotations namespace and then stores the error messages in the dictionary:

At this stage, I needed a solution for linking the UI to the model, and modifying the visuals depending on the presence or not of validation errors.

The ViewModel for the sample page, contains only a property storing an instance of the item defined in the model:

Then, the corresponding XAML contains two Entry views used for input and a ListView used for showing the validation errors:

The sample page uses a Behavior called EntryValidationBehavior which take care of changing the colour of the Entry background views in the case validation errors are present:

The UI is also fine-tuned using a Xamarin.Forms effect applied only to the UWP platform, in order to change the colour of the Entry border when validation errors occur:

And this is the result when the application is executed on Android and UWP:

Conclusions

Xamarin.Forms provides a rich set of features for implementing validation: the usage of INotifyDataErrorInfo, Data Annotations, Behaviors and Effects permit the handling of complex scenarios including multiple conditions per field, cross-property validation, entity-level validation and custom UI depending on the platform.

The source code of the sample demo is available for download on GitHub here.

Happy coding!

Useful links for exam 70-357: Developing Mobile Apps

Last week I received confirmation that I passed the exam Developing Mobile Apps which I took in beta last July while I was preparing for UWP – App Architecture and UX/UI & UWP – App Data, Services, and Coding Patterns.

Since this last exam completes my certification journey for MCSD / MCSA: App Builder – Universal Windows Platform, I think it’s useful to share some resources that I found valuable for preparation.

First of all, I’ve been asked a couple of times what’s the procedure to participate in beta exams (these are the ones that are not yet in final form): I found that the best way to be notified when a new one is available is by following the Born to Learn website (I got my free voucher for 70-357 after I read this blog post).

The exam objectives are listed below and are explained here in the official exam description: I found useful going through every single topic in order to cover as many cases as possible.

  • Develop a XAML page layout for an adaptive UI (10–15%)
  • Implement page navigation and lifecycle events (10–15%)
  • Implement data access and data binding (20–25%)
  • Implement feature detection for adaptive coding (10–15%)
  • Manage user input and custom user interactions (10–15%)
  • Manage authentication and identity management (10–15%)
  • Implement notifications, background tasks, and reusable components (20–25%)

Some tips are also available in this Microsoft Ignite session by Mark Rosenberg.

Another resource that I found valuable is the official preparation test: the questions are different from the ones available in the certification exam, but the coverage of the measured skills is properly done and I found it precious for highlighting the topics I needed to review.

I took the exam using the proctored modality from home and worked quite well since I didn’t have to take time off from work and also had much more choices in term of time-slot availability: in the future, I will definitely look forward to using it for other exams.

Other links that I used for preparation are available in my previous post here.

Happy coding!

I’ve just achieved the MCSD App Builder certification

In the previous days, I’ve received confirmation that I achieved the new certification

MCSD App Builder-logo-Blk

Microsoft Certified Solutions Developer: App Builder  it validates that you have the skills needed to build modern mobile and/or web applications and services.

The exams required are fully explained on the certification website: interestingly, the new MCSA (Microsoft Certified Solutions Associate) path is now a mandatory step for earning the MCSD and can be chosen between Web Applications and Universal Windows Platform.

valuable post in the “Born to Learn” site describes the changes and the transition options available from the existing MCSD.

The new certifications will not have an expiry date, and there will be the possibility to re-earn them by passing every year an additional exam taken from a list of electives.

 

Using Prism modularization in Xamarin.Forms

Recently, Prism for Xamarin.Forms 6.2.0 has been released with many notable improvements including a new bootstrapping process, AutoWireViewModel behaviour changes, deep-linking support,  modularity and Prism template pack enhancements (full release notes available here).

Today, I fired up Visual Studio to have a play with this new version and decided to try the Xamarin.Forms support for Prism Modules: this is a very useful feature which allows to separate clearly the various parts of the application and improve quality, extensibility and readability of the code.

After downloading the new template pack, I created a new solution selecting New Project => Templates => Visual C# => Prism => Forms => Prism Unity App:

PrismTemplatePack

The new wizard is very useful and permits to select the platforms to target in the project: I selected Android, iOS and UWP and the project was generated targeting the three platforms with a shared PCL. NuGet packages were already updated to the latest version so no need for further actions.

While exploring the new project structure and the new modularization stuff, I decided to create a new Xamarin.Forms portable class library containing a module with a single View/ViewModel (SamplePage SamplePageViewModel) visualised when a user interacts with a button on the home page.

The new module required the definition of the following class implementing the Prism IModule interface:

To keep the logic separated from the rest of the app, I decided to register the navigation type for SamplePage inside the Initialize() method of the module which will be triggered when the module loads.

I also applied Xamarin.Forms XAML compilation to the module to improve performance, which is always great to have 😉

It’s worth noting that in this new Prism release the default value for the attached property ViewModelLocator.AutowireViewModel is set to true, so we can omit it and the framework will automatically associate SampleViewModel as the BindingContext for the view:

I then explored the new breaking changes for the bootstrapping process: the application class now needs to inherit from the PrismApplication class and two new virtual methods OnInitialized() and RegisterTypes() permit respectively to specify the implementation for navigating to the home page and registering the known types / ViewModel’s for navigation:

The third overridden method, ConfigureModuleCatalog(), informs the app to initialize the catalog with the module we created previously and set the initialization mode to OnDemand which means the module is not activated when the application starts, but it must load explicitly. This feature is particularly useful in cases in which some functionalities of the app must initialise after some other requirements like, for instance, successful authentication or applications extended via modules.

The sample view was in place, so I proceeded with the implementation of the HomePage: I wrapped the existing one in a NavigationPage to allow the correct back stack and then created two commands for initializing the module and navigating to the SamplePage defined previously:

and the corresponding ViewModel:

The module is initialized by injecting the Prism ModuleManager and then calling the LoadModule() method:

The navigation to the page defined in the module is performed by:

The property IsSampleModuleRegistered permitted to control the CanExecute() for the button commands using this nice fluent syntax using ObservesProperty(()=>….) available in Prism:

Here we go: I found the Prism implementation in this new version very stable and working well with Xamarin.Forms. The modularization capabilities are useful to write clean, maintainable and extensible apps.

The source code is available for download as part of the official Prism samples on GitHub.

Looking forward to exploring all the other capabilities available in Prism and Xamarin.Forms. Happy coding!