Exploring Windows Mixed Reality, switching between 2D / 3D and embedding Web Views

In these days, I’m exploring the options for switching between Unity 3D and XAML 2D views integrating the access to UWP APIs for content hosted on Web Views.

This scenario could be particularly useful if an app needs to reuse existing code, perhaps available in a website, with the requirement to access the Windows Runtime when executed in Windows Mixed Reality devices and activated from a Unity 3D scene.

Creating the Unity project

To start, I created a new Unity scene and imported the HoloToolkit package downloaded from here.

I applied the Scene and Project settings from the HoloToolkit|Configure menu and added the following prefabs available from the imported package:

  • HoloLens Camera
  • InputManager
  • Cursor

Then, I added a simple cube which, when gazed and air-tapped, permits to trigger the view switching.

I defined a new script TapBehaviour to capture the event and call the 2D view:

The interesting part here is the one using the AppViewManager for switching views: after some research, I decided to use this class from GitHub as a starting point for handling the logic via CoreApplicationView.

I created a new script in Unity (AppViewManager.cs) and added the source code from the GitHub repository.

In this way, it is possible to retrieve the list of registered views or a specific one using for instance

And then simply transition to the new page by calling the Switch() or SwitchAsync() methods.

Creating and registering the 2D views

The 2D view named ContentPage needed to be defined and registered in the UWP project generated by Unity using the File|Build Settings|Build command after setting the UWP Build type to XAML to be able to define additional views in my project

Then, I opened the project in Visual Studio and added a new page called ContentPage.xaml containing a WebView and a button to switch back to Unity 3D

And the code makes use again of the AppViewManager previously imported in Unity:

We are now making use of two pages MainPage and ContentPage, respectively hosting the 3D and 2D content. Before using them, I needed to register both in the App.cs class right after initialising the Unity player:

Accessing UWP APIs from the hosted web page

The Web View is actually showing content from this web page hosted in my personal web-site which tries to access the UWP APIs if hosted with elevated permissions:

To enable Windows Runtime access, I modified the Package.appxmanifest and added the following under the Application section:

After this step, I was able to launch the app using an HoloLens device or emulator:

And then switch to the 2D view hosting the web page accessing UWP APIs on HoloLens by tapping the cube:

And return to the Unity 3D view using the “Click me” button.

As usual, the source code is available for download on GitHub.

Experiments with HoloLens, Bot Framework and LUIS: adding text to speech

Previously I blogged about creating a Mixed Reality 2D app integrating with a Bot using LUIS via the Direct Line channel available in the Bot Framework.

I decided to add more interactivity to the app by also enabling text to speech for the messages received by the Bot: this required the addition of a new MediaElement for the Speech synthesiser to the main XAML page:

Then I initialized a new SpeechSynthesizer at the creation of the page:

And added a new Speech() method using the media element:

When a new response is received from the Bot, the new Speech() method is called:

And then the recognition for a new phrase is started again via the MediaEnded event to simulate a conversation between the user and the Bot:

As usual, the source code is available for download on GitHub.

Unity: “Failed to initialize unity graphics” error

Today, I launched Unity for some HoloLens and Mixed Reality dev and received the following error:

After some research, I verified the latest graphic drivers were available in my machine and ran a dxdiag to check that all the DirectX features were enabled:

Then, I searched for command line arguments on the official documentation and modified the launch link as below:

And this last step allowed me to run again Unity in my machine: