Experiments with HoloLens, Mixed Reality Toolkit and two-handed manipulations

I’ve always been a big fan of manipulations, as in the past I worked on some multi-touch XAML Behaviors implementing rotate, translate and scale on 2D objects.

As I progress with my learning about HoloLens and Windows Mixed Reality, I had on my to-do list the task of exploring how to recreate this scenario in the 3D Mixed Reality context. Finally, during the weekend, I started some research while preparing a demo for some speaking engagements I’ll have over the next weeks.

As usual, the Mixed Reality Toolkit is a fantastic help for speeding up development, so I started my investigation analysing the GitHub repo and found that the dev branch now contains a new readme illustrating all the steps required to use a new Unity script TwoHandManipulatable.cs which enables rotate, translate and scale to 3D objects in Unity using two hands tap gestures with HoloLens and the Motion Controllers with the Immersive headsets.

I decided to give this a try using a model imported from the Remix 3D repository.

Importing the model from Remix 3D

I fired up Paint 3D and selected More models to explore the ones available online in Remix 3D: this is a great container of assets you can use in Mixed Reality apps. I chose to explore a model of the Mars Rover, so I selected and opened it:

Then I exported it as a 3D FBX object to be able to import in Unity and saved it on my local machine as MarsRover.fbx.

Creating the Unity project

I started a new Unity Project, and copied the folder Assets\MixedRealityToolkit from the dev branch of the MRT GitHub repository.

After applying the Mixed Reality Project and Scene settings from the corresponding menu, I was ready to import the Mars Rover model and associate the manipulation script.

I selected Assets->Import New Asset, searched for the previously saved mode MarsRover.fbx and adjusted the scale to X:60, Y:60, Z:60 to have it correctly visualised in my scene. Then, I inserted a new Capsule Collider for enabling interaction with the object:

Adding the TwoHandManipulatable script

After selecting the imported model, I actioned the Add Component button from the inspector tab and searched for the Two Hand Manipulatable script in the toolkit and then added it to the asset together with a BoundingBoxBasic for showing the boundaries when applying manipulations.

And set the manipulation mode to Move Rotate Scale.

The scene was completed, so I only had to test it with the device: I selected the File->Build Settings, generated the package and deployed it to HoloLens, and I got the Mars Rover ready to be explored with two-handed manipulations. Amazing! 

The source code is available on my GitHub playground here.

I will speak at the DDD Wales 2018 conference

I’m very excited to announce that I will be speaking about HoloLens, Windows Mixed Reality and Cognitive Services at the DDD (Developer! Developer! Developer!) Wales 2018 conference in Swansea on 24th March.

I’m looking forward to meeting all the community! See you there? 🙂

Analysing visual content using HoloLens, Computer Vision APIs, Unity and the Windows Mixed Reality Toolkit

In these days, I’m exploring the combination of HoloLens/Windows Mixed Reality and the capabilities offered by Cognitive Services to analyse and extract information from images captured via the device camera and processed using the Computer Vision APIs and the intelligent cloud.
In this article, we’ll explore the steps I followed for creating a Unity application running on HoloLens and communicating with the Microsoft AI platform.

Registering for Computer Vision APIs

The first step was to navigate to the Azure portal https://portal.azure.com and create a new Computer Vision API resource:

I noted down the Keys and Endpoint and started investigating how to approach the code for capturing images on HoloLens and sending them to the intelligent cloud for processing.

Before creating the Unity experience, I decided to start with a simple UWP app for analysing images.

Writing the UWP test app and the shared library

There are already some samples available for Cognitive Services APIs, so I decided to reuse some code available and described in this article here supplemented by some camera capture UI in UWP.

I created a new Universal Windows app and library (CognitiveServicesVisionLibrary) to provide, respectively, a test UI and some reusable code that could be referenced later by the HoloLens experience.

The Computer Vision APIs can be accessed via the package Microsoft.ProjectOxford.Vision available on NuGet so I added a reference to both projects:

The test UI contains an image and two buttons: one for selecting a file using a FileOpenPicker and another for capturing a new image using the CameraCaptureUI. I decided to wrap these two actions in an InteractionsHelper class:

I then worked on the shared library creating a helper class for processing the image using the Vision APIs available in Microsoft.ProjectOxford.Vision and parsing the result.

Tip: after creating the VisionServiceClient, I received an unauthorised error when specifying only the key: the error disappeared by also specifying the endpoint URL available in the Azure portal.

I then launched the test UI, and the image was successfully analysed, and the results returned from the Computer Vision APIs, in this case identifying a building and several other tags like outdoor, city, park: great!

I also added a Speech Synthesizer playing the general description returned by the Cognitive Services call:

I then moved to HoloLens and started creating the interface using Unity, the Mixed Reality Toolkit and UWP.

Creating the Unity HoloLens experience

First of all, I created a new Unity project using Unity 2017.2.1p4 and then added a new folder named Scenes and saved the active scene as CognitiveServicesVision Scene.

I downloaded the corresponding version of the Mixed Reality Toolkit from the releases section of the GitHub project https://github.com/Microsoft/MixedRealityToolkit-Unity/releases and imported the toolkit package HoloToolkit-Unity-2017.2.1.1.unitypackage using the menu Assets->Import Package->Custom package.

Then, I applied the Mixed Reality Project settings using the corresponding item in the toolkit menu:

And selected the Scene Settings adding the Camera, Input Manager and Default Cursor prefabs:

And finally set the UWP capabilities as I needed access to the camera for retrieving the image, the microphone for speech recognition and internet client for communicating with Cognitive Services:

I was then ready to add the logic to retrieve the image from the camera, save it to the HoloLens device and then call the Computer Vision APIs.

Creating the Unity Script

The CameraCaptureUI UWP API is not available in HoloLens, so I had to research a way to capture an image from Unity, save it to the device and then convert it to a StorageFile ready to be used by the CognitiveServicesVisionLibrary implemented as part of the previous project.

First of all, I enabled the Experimental (.NET 4.6 Equivalent) Scripting Runtime version in the Unity player for using features like async/await. Then, I enabled the PicturesLibrary capability in the Publishing Settings to save the captured image to the device.

Then, I created a Scripts folder and added a new PhotoManager.cs script taking as a starting point the implementation available in this GitHub project.

The script can be attached to a TextMesh component visualising the status:

Initialising the PhotoCapture API available in Unity https://docs.unity3d.com/Manual/windowsholographic-photocapture.html

Saving the photo to the pictures library folder and then passing it to the library created in the previous section:

The code references the CognitiveServicesVisionLibrary UWP library created previously: to use it from Unity, I created a new Plugins folder in my project and ensured that the Build output of the Visual Studio library project was copied to this folder:

And then set the import settings in Unity for the custom library:

And for the NuGet library too:

Nearly there! Let’s see now how I enabled Speech recognition and Tagalong/Billboard using the Mixed Reality Toolkit.

Enabling Speech

I decided to implement a very minimal UI for this project, using the speech capabilities available in HoloLens for all the interactions.

In this way, a user can just simply say the work Describe to trigger the image acquisition and the processing using the Computer Vision API, and then naturally listening to the results.

In the Unity project, I selected the InputManager object:

And added a new Speech Input Handler Component to it:

Then, I mapped the keyword Describe with the TakePhoto() method available in the PhotoManager.cs script already attached to the TextMesh that I previously named as Status Text Object.

The last step required to enable Text to Speech for receiving the output: I simply added a Text to Speech component to my TextMesh:

And enabled the speech in the script using StartSpeaking():

I also added other two components available in the Mixed Reality Toolkit: Tagalong and Billboard to have the status text following me and not anchored to a specific location:

I was then able to generate the final package using Unity specifying the starting scene:

And then I deployed the solution to the HoloLens device and started extracting and categorising visual data using HoloLens, Camera, Speech and the Cognitive Services Computer Vision APIs.

Conclusions

The combination of Mixed Reality and Cognitive Services opens a new world of experiences combining the capabilities of HoloLens and all the power of the intelligent cloud. In this article, I’ve analysed the Computer Vision APIs, but a similar approach could be applied to augment Windows Mixed Reality apps and enrich them with the AI platform https://www.microsoft.com/en-gb/ai.

The source code for this article is available on GitHub: https://github.com/davidezordan/CognitiveServicesSamples

 

Experiments with HoloLens, Bot Framework, LUIS and Speech Recognition

Recently I had the opportunity to use a HoloLens device for some personal training and building some simple demos.

One of the scenarios that I find very intriguing is the possibility of integrating Mixed Reality and Artificial Intelligence (AI) in order to create immersive experiences for the user.

I decided to perform an experiment by integrating a Bot, Language Understanding Intelligent Services (LUIS), Speech Recognition and Mixed Reality via a Holographic 2D app.

The idea was to create a sort of “digital assistant” of myself that can be contacted using Mixed Reality: the first implementation contains only basic interactions (answering questions like “What are your favourite technologies” or “What’s your name”) but these could be easily be expanded in the future with features like time management (via the Graph APIs) or tracking projects status, etc.

Creating the LUIS application

To start, I created a new LUIS application in the portal with a list of intents that needed to be handled:

In the future, this could be further extended with extra capabilities.

After defining the intents and utterances, I trained and published my LUIS app to Azure and copied the key and URL for usage in my Bot:

Creating the Bot

I proceeded with the creation of the Bot using Microsoft Bot framework downloading the Visual Studio template and creating a new project:

The Bot template already defined a dialog named RootDialog so I extended the generated project with the classes required for parsing the JSON from the LUIS endpoint:

And then processed the various LUIS intents in RootDialog (another option is the usage of the LuisDialog and LuisModel classes as explained here):

Then, I tested the implementation using the Bot Framework Emulator:

And created a new Bot definition in the framework portal.

After that, I published it to Azure with an updated Web.config with the generated Microsoft App ID and password:

Since the final goal was the communication with an UWP HoloLens application, I enabled the Diret Line channel:

Creating the Holographic 2D app

Windows 10 UWP apps are executed on the HoloLens device as Holographic 2D apps that can be pinned in the environment.

I created a new project using the default Visual Studio Template:

And then added some simple text controls in XAML to receive the input and display the response from the Bot:

I decided to use the SpeechRecognizer APIs for receiving the input via voice (another option could be the usage of Cognitive Services):

The SendToBot() method makes use of the Direct Line APIs which permit communication with the Bot using the channel previously defined:

And then I got the app running on HoloLens and interfacing with a Bot using LUIS for language understanding and Speech recognition:

The source code of the project is available on GitHub here.

Happy coding!

Useful links for exam 70-357: Developing Mobile Apps

Last week I received confirmation that I passed the exam Developing Mobile Apps which I took in beta last July while I was preparing for UWP – App Architecture and UX/UI & UWP – App Data, Services, and Coding Patterns.

Since this last exam completes my certification journey for MCSD / MCSA: App Builder – Universal Windows Platform, I think it’s useful to share some resources that I found valuable for preparation.

First of all, I’ve been asked a couple of times what’s the procedure to participate in beta exams (these are the ones that are not yet in final form): I found that the best way to be notified when a new one is available is by following the Born to Learn website (I got my free voucher for 70-357 after I read this blog post).

The exam objectives are listed below and are explained here in the official exam description: I found useful going through every single topic in order to cover as many cases as possible.

  • Develop a XAML page layout for an adaptive UI (10–15%)
  • Implement page navigation and lifecycle events (10–15%)
  • Implement data access and data binding (20–25%)
  • Implement feature detection for adaptive coding (10–15%)
  • Manage user input and custom user interactions (10–15%)
  • Manage authentication and identity management (10–15%)
  • Implement notifications, background tasks, and reusable components (20–25%)

Some tips are also available in this Microsoft Ignite session by Mark Rosenberg.

Another resource that I found valuable is the official preparation test: the questions are different from the ones available in the certification exam, but the coverage of the measured skills is properly done and I found it precious for highlighting the topics I needed to review.

I took the exam using the proctored modality from home and worked quite well since I didn’t have to take time off from work and also had much more choices in term of time-slot availability: in the future, I will definitely look forward to using it for other exams.

Other links that I used for preparation are available in my previous post here.

Happy coding!

I’ve just achieved the MCSD App Builder certification

In the previous days, I’ve received confirmation that I achieved the new certification

MCSD App Builder-logo-Blk

Microsoft Certified Solutions Developer: App Builder  it validates that you have the skills needed to build modern mobile and/or web applications and services.

The exams required are fully explained on the certification website: interestingly, the new MCSA (Microsoft Certified Solutions Associate) path is now a mandatory step for earning the MCSD and can be chosen between Web Applications and Universal Windows Platform.

valuable post in the “Born to Learn” site describes the changes and the transition options available from the existing MCSD.

The new certifications will not have an expiry date, and there will be the possibility to re-earn them by passing every year an additional exam taken from a list of electives.

 

Using Prism modularization in Xamarin.Forms

Recently, Prism for Xamarin.Forms 6.2.0 has been released with many notable improvements including a new bootstrapping process, AutoWireViewModel behaviour changes, deep-linking support,  modularity and Prism template pack enhancements (full release notes available here).

Today, I fired up Visual Studio to have a play with this new version and decided to try the Xamarin.Forms support for Prism Modules: this is a very useful feature which allows to separate clearly the various parts of the application and improve quality, extensibility and readability of the code.

After downloading the new template pack, I created a new solution selecting New Project => Templates => Visual C# => Prism => Forms => Prism Unity App:

PrismTemplatePack

The new wizard is very useful and permits to select the platforms to target in the project: I selected Android, iOS and UWP and the project was generated targeting the three platforms with a shared PCL. NuGet packages were already updated to the latest version so no need for further actions.

While exploring the new project structure and the new modularization stuff, I decided to create a new Xamarin.Forms portable class library containing a module with a single View/ViewModel (SamplePage SamplePageViewModel) visualised when a user interacts with a button on the home page.

The new module required the definition of the following class implementing the Prism IModule interface:

To keep the logic separated from the rest of the app, I decided to register the navigation type for SamplePage inside the Initialize() method of the module which will be triggered when the module loads.

I also applied Xamarin.Forms XAML compilation to the module to improve performance, which is always great to have 😉

It’s worth noting that in this new Prism release the default value for the attached property ViewModelLocator.AutowireViewModel is set to true, so we can omit it and the framework will automatically associate SampleViewModel as the BindingContext for the view:

I then explored the new breaking changes for the bootstrapping process: the application class now needs to inherit from the PrismApplication class and two new virtual methods OnInitialized() and RegisterTypes() permit respectively to specify the implementation for navigating to the home page and registering the known types / ViewModel’s for navigation:

The third overridden method, ConfigureModuleCatalog(), informs the app to initialize the catalog with the module we created previously and set the initialization mode to OnDemand which means the module is not activated when the application starts, but it must load explicitly. This feature is particularly useful in cases in which some functionalities of the app must initialise after some other requirements like, for instance, successful authentication or applications extended via modules.

The sample view was in place, so I proceeded with the implementation of the HomePage: I wrapped the existing one in a NavigationPage to allow the correct back stack and then created two commands for initializing the module and navigating to the SamplePage defined previously:

and the corresponding ViewModel:

The module is initialized by injecting the Prism ModuleManager and then calling the LoadModule() method:

The navigation to the page defined in the module is performed by:

The property IsSampleModuleRegistered permitted to control the CanExecute() for the button commands using this nice fluent syntax using ObservesProperty(()=>….) available in Prism:

Here we go: I found the Prism implementation in this new version very stable and working well with Xamarin.Forms. The modularization capabilities are useful to write clean, maintainable and extensible apps.

The source code is available for download as part of the official Prism samples on GitHub.

Looking forward to exploring all the other capabilities available in Prism and Xamarin.Forms. Happy coding!

I’m now a Microsoft Certified Solutions Developer – Universal Windows Platform

I’ve just completed the exams required to earn the MCSD: Universal Windows Platform Solutions Developer certification

MCSD Universal Windows Platform

As described in the official certification website, with this certification you

Demonstrate your expertise at planning the designing and implementing Universal Windows Platform apps that offer a compelling user experience, leverage other services and devices, and use best coding practices to enhance maintainability.

I decided to start my MCSD journey last June while studying for the Xamarin Mobile Developer Certification: I discovered that preparing for the exams is an excellent way to learn new skills and “stretch your muscles” for ensuring you know a lot of details related to the technologies you work on and become a better developer.

The current requirements include the following exams:

As described in the “Born to Learn” website, the requirements will change on March 31, 2017, and exams 354 and 355 will be substituted by

Some links I found useful for preparation:

 

Xamarin.Forms: sharing resources in XAML using Styles

Previously, I’ve blogged about creating a custom Behavior in Xamarin.Forms. The sample published on GitHub initially used the default template available in Visual Studio which instantiates the Application singleton from C# using the following App class:

This is a great approach, however very often it is useful to define common resources in XAML and reuse them across the application using Styles.

To achieve this, It’s possible to transform the App class in a partial one and add a new App.xaml source file containing a resource dictionary to be shared:

In this code, I’ve defined a new resource with a specific Padding to be applied only on iOS via the OnPlatform class, and an explicit style named PageStyle targeting ContentPage types.

The modified code behind is now taking care of parsing/loading the XAML (via the InitializeComponent() call) and then creates an instance of SamplePage:

The page can now consume the new style specifying Style=”{StaticResource PageStyle}” (dynamic resources are also supported):

This is a basic example of using styles in Xamarin.Forms: a complete coverage of the various options is available on the Xamarin official documentation.

The source code is available for download here.

Enabling PlayReady support on Xbox One UWP Developer Preview

In the previous days I was trying the recently released Xbox One UWP Developer preview with these samples involving Smooth Streaming / PlayReady and noticed that playback was not working properly on the console with the MediaElement always returning the error MEDIA_ERR_SRC_NOT_SUPPORTED – 0x80070005.

As highlighted in this forum post, the following capability must be added to the app manifest in order to enable playback of PlayReady DRM protected content on Xbox One UWP developer preview:

I’ve updated the Smooth Streaming / PlayReady samples and they are now available for download here on GitHub.