A HoloLens 2 device (or you can use the emulator) Unity Hub with Unity 2019.3.X installed and the Universal Windows Platform (UWP). Again you can tick this module (UWP) during the installation process) Unity Mixed reality Tool Kit (MRTK) version 2.3.0 — Download from here. Let’s get started. ARToolKit is written in native C/C code, and an additional wrapper is needed for development on Unity, which is recommended for HoloLens. (If you are not good at Direct3D, and Windows App) Luckily, after spending some time on this, I successfully get some pieces of code written, and actually make ARToolKit runs on HoloLens.

• Unity Hololens App
• Hololens Toolkit Unity 2018
• Hololens Unity Toolkit Download
• Deploy From Unity To Hololens
• Hololens 2 Unity Tutorial
• Hololens 2 Tutorials

A button gives the user a way to trigger an immediate action. It is one of the most foundational components in mixed reality. MRTK provides various types of button prefabs.

Button prefabs in MRTK

Examples of the button prefabs under MRTK/SDK/Features/UX/Interactable/Prefabs folder

Unity UI Image/Graphic based buttons

• UnityUIInteractableButton.prefab
• PressableButtonUnityUI.prefab
• PressableButtonUnityUICircular.prefab
• PressableButtonHoloLens2UnityUI.prefab

Collider based buttons

The Button (Assets/MRTK/SDK/Features/UX/Interactable/Prefabs/Button.prefab) is based on the Interactable concept to provide easy UI controls for buttons or other types of interactive surfaces. The baseline button supports all available input methods, including articulated hand input for the near interactions as well as gaze + air-tap for the far interactions. You can also use voice command to trigger the button.

PressableButtonHoloLens2 (Assets/MRTK/SDK/Features/UX/Interactable/Prefabs/PressableButtonHoloLens2.prefab) is HoloLens 2's shell style button that supports the precise movement of the button for the direct hand tracking input. It combines Interactable script with PressableButton script.

For HoloLens 2, it is recommended to use buttons with an opaque backplate. Transparent buttons are not recommended because of these usability and stability issues:

• Icon and text are difficult to read with the physical environment
• It is hard to understand when the event triggers
• Holograms that are displayed through a transparent plane can be unstable with HoloLens 2's Depth LSR stabilization

How to use pressable buttons

Unity UI based buttons

Create a Canvas in your scene (GameObject -> UI -> Canvas). In the Inspector panel for your Canvas:

• Click 'Convert to MRTK Canvas'
• Click 'Add NearInteractionTouchableUnityUI'
• Set the Rect Transform component's X, Y, and Z scale to 0.001

Then, drag PressableButtonUnityUI (Assets/MRTK/SDK/Features/UX/Interactable/Prefabs/PressableButtonUnityUI.prefab), PressableButtonUnityUICircular (Assets/MRTK/SDK/Features/UX/Interactable/Prefabs/PressableButtonUnityUICircular.prefab), or PressableButtonHoloLens2UnityUI (Assets/MRTK/SDK/Features/UX/Interactable/Prefabs/PressableButtonHoloLens2UnityUI.prefab) onto the Canvas.

Collider based buttons

Unity Hololens App

Simply drag PressableButtonHoloLens2 (Assets/MRTK/SDK/Features/UX/Interactable/Prefabs/PressableButtonHoloLens2.prefab) or PressableButtonHoloLens2Unplated (Assets/MRTK/SDK/Features/UX/Interactable/Prefabs/PressableButtonHoloLens2Unplated.prefab) into the scene. These button prefabs are already configured to have audio-visual feedback for the various types of inputs, including articulated hand input and gaze.

The events exposed in the prefab itself as well as the Interactable component can be used to trigger additional actions. The pressable buttons in the HandInteractionExample scene use Interactable's OnClick event to trigger a change in the color of a cube. This event gets triggered for different types of input methods such as gaze, air-tap, hand-ray, as well as physical button presses through the pressable button script.

You can configure when the pressable button fires the OnClick event via the PhysicalPressEventRouter on the button. For example, you can set OnClick to fire when the button is first pressed, as opposed to being pressed and released, by setting Interactable On Click to Event On Press.

To leverage specific articulated hand input state information, you can use pressable buttons events - Touch Begin, Touch End, Button Pressed, Button Released. These events will not fire in response to air-tap, hand-ray, or eye inputs, however. To support both near and far interactions, it is recommended to use Interactable's OnClick event.

Interaction states

In the idle state, the button's front plate is not visible. As a finger approaches or a cursor from gaze input targets the surface, the front plate's glowing border becomes visible. There is additional highlighting of the fingertip position on the front plate surface. When pushed with a finger, the front plate moves with the fingertip. When the fingertip touches the surface of the front plate, it shows a subtle pulse effect to give visual feedback of the touch point.

In HoloLens 2 shell-style button, there are many visual cues and affordances to increase the user's confidence on interaction.

The subtle pulse effect is triggered by the pressable button, which looks for ProximityLight(s) that live on the currently interacting pointer. If any proximity lights are found, the ProximityLight.Pulse method is called, which automatically animates shader parameters to display a pulse.

Inspector properties

Box ColliderBox Collider for the button's front plate.

Pressable ButtonThe logic for the button movement with hand press interaction.

Physical Press Event RouterThis script sends events from hand press interaction to Interactable.

InteractableInteractable handles various types of interaction states and events. HoloLens gaze, gesture, and voice input and immersive headset motion controller input are directly handled by this script.

Audio SourceUnity audio source for the audio feedback clips.

NearInteractionTouchable.csRequired to make any object touchable with articulated hand input.

Prefab layout

The ButtonContent object contains front plate, text label and icon. The FrontPlate responds to the proximity of the index fingertip using the Button_Box shader. It shows glowing borders, proximity light, and a pulse effect on touch. The text label is made with TextMesh Pro. SeeItSayItLabel's visibility is controlled by Interactable's theme.

How to change the icon and text

MRTK buttons use a ButtonConfigHelper component to assist you in changing the button's icon, text and label. (Note that some fields may be absent if elements are not present on the selected button.)

Creating and Modifying Icon Sets

An Icon Set is a shared set of icon assets used by the ButtonConfigHelper component. Three icon styles are supported.

• Quad icons are rendered on a quad using a MeshRenderer. This is the default icon style.
• Sprite icons are rendered using a SpriteRenderer. This is useful if you prefer to import your icons as a sprite sheet, or if you want your icon assets to be shared with Unity UI components. To use this style you will need to install the Sprite Editor package (Windows -> Package Manager -> 2D Sprite)
• Char icons are rendered using a TextMeshPro component. This is useful if you prefer to use an icon font. To use the HoloLens icon font you will need to create a TextMeshPro font asset.

To change which style your button uses, expand the Icons dropdown in the ButtonConfigHelper and select from the Icon Style dropdown.

You can create a new button icon set with the asset menu: Create > Mixed Reality Toolkit > Icon Set. To add quad and sprite icons, simply drag them into their respective arrays. To add Char icons, you must first create and assign a font asset.

In MRTK 2.4 and beyond, we recommend custom icon textures be moved into an IconSet.To upgrade the assets on all buttons in a project to the new recommended format, use the ButtonConfigHelperMigrationHandler.(Mixed Reality Toolkit -> Utilities -> Migration Window -> Migration Handler Selection -> Microsoft.MixedReality.Toolkit.Utilities.ButtonConfigHelperMigrationHandler)

Importing the Microsoft.MixedRealityToolkit.Unity.Tools package required to upgrade the buttons.

If an icon is not found in the default icon set during migration, a custom icon set will be created in MixedRealityToolkit.Generated/CustomIconSets. A dialog will indicate that this has taken place.

Creating a HoloLens Icon Font Asset

First, import the icon font into Unity. On Windows machines you can find the default HoloLens font in Windows/Fonts/holomdl2.ttf. Copy and paste this file into your Assets folder.

Next, open the TextMeshPro Font Asset Creator via Window > TextMeshPro > Font Asset Creator. Here are the recommended settings for generating a HoloLens font atlas. To include all icons, paste the following Unicode range into the Character Sequence field:

Once the font asset is generated, save it to your project and assign it to your Icon Set's Char Icon Font field. The Available Icons dropdown will now be populated. To make an icon available for use by a button, click it. It will be added to the Selected Icons dropdown and will now show up in the ButtonConfigHelper. You can optionally give the icon a tag. This enables setting the icon at runtime.

To use your Icon Set select a button, expand the Icons dropdown in the ButtonConfigHelper and assign it to the Icon Set field.

How to change the size of a button

HoloLens 2's shell-style button's size is 32x32mm. To customize the dimension, change the size of these objects in the button prefab:

1. FrontPlate
2. Quad under BackPlate
3. Box Collider on the root

Then, click Fix Bounds button in the NearInteractionTouchable script which is in the root of the button.

Update the size of the FrontPlate

Update the size of the Quad

Update the size of the Box Collider

Click 'Fix Bounds'

Voice command ('see-it, say-it')

Speech Input HandlerThe Interactable script in Pressable Button already implements IMixedRealitySpeechHandler. A voice command keyword can be set here.

Speech Input ProfileAdditionally, you need to register the voice command keyword in the global Speech Commands Profile.

See-it, Say-it labelThe pressable button prefab has a placeholder TextMesh Pro label under the SeeItSayItLabel object. You can use this label to communicate the voice command keyword for the button to the user.

How to make a button from scratch

You can find the examples of these buttons in the PressableButtonExample scene.

1. Creating a pressable button with cube (near interaction only)

1. Create a Unity Cube (GameObject > 3D Object > Cube)
2. Add PressableButton.cs script
3. Add NearInteractionTouchable.cs script

Hololens Toolkit Unity 2018

In the PressableButton's Inspector panel, assign the cube object to the Moving Button Visuals.

When you select the cube, you will see multiple colored layers on the object. This visualizes the distance values under Press Settings. Using the handles, you can configure when to start press (move the object) and when to trigger event.

When you press the button, it will move and generate proper events exposed in the PressableButton.cs script such as TouchBegin(), TouchEnd(), ButtonPressed(), ButtonReleased().

2. Adding visual feedback to the basic cube button

MRTK Standard Shader provides various features that makes it easy to add visual feedback. Create a material and select shader Mixed Reality Toolkit/Standard. Or you can use or duplicate one of the existing materials under /SDK/StandardAssets/Materials/ that uses MRTK Standard Shader.

Check Hover Light and Proximity Light under Fluent Options. This enables visual feedback for both near hand(Proximity Light) and far pointer(Hover Light) interactions.

3. Adding audio feedback to the basic cube button

Since PressableButton.cs script exposes events such as TouchBegin(), TouchEnd(), ButtonPressed(), ButtonReleased(), we can easily assign audio feedback. Simply add Unity's Audio Source to the cube object then assign audio clips by selecting AudioSource.PlayOneShot(). You can use MRTK_Select_Main and MRTK_Select_Secondary audio clips under /SDK/StandardAssets/Audio/ folder.

4. Adding visual states and handle far interaction events

Interactable is a script that makes it easy to create a visual state for the various types of input interactions. It also handles far interaction events. Add Interactable.cs and drag and drop the cube object onto the Target field under Profiles. Then, create a new Theme with a type ScaleOffsetColorTheme. Under this theme, you can specify the color of the object for the specific interaction states, such as Focus and Pressed. You can also control Scale and Offset, as well. Check Easing and set duration to make the visual transition smooth.

You will see the object respond to both far (hand ray or gaze cursor) and near(hand) interactions.

Custom button examples

Hololens Unity Toolkit Download

In the HandInteractionExample scene, see the piano and round button examples which are both using PressableButton.

Each piano key has a PressableButton and a NearInteractionTouchable script assigned. It is important to verify that the Local Forward direction of NearInteractionTouchable is correct. It is represented by a white arrow in the editor. Make sure the arrow points away from the button's front face:

See also

After following some on-line video’s and snooping through some other’s people code I felt I had to start bottom-up to get at least a feeling for how you setup a HoloLens app – even though I don’t have a HoloLens. Call it my natural curiosity. So I followed my fellow MVP Morten Nielsen’s blog – highly recommended – as he starts with the real basics. Very good if you hardly have a clue what you are doing ;)

In his third post, I ran into a snag. According to his second post, I had unzipped the HoloLens toolkit into the assets folder as instructed, and started to add occlusion to my project. Morten explains in detail what this is – it makes holograms disappear behind physical objects like a wall when they are ‘in front’ of the holograms (which, in reality, they never can be as they are projected on a screen not 2 inch from your eyes, but that is the magic of HoloLens).

So I added a rectangular box as a Hologram, having it stick out of a wall like halfway, so I was supposed only to see the front part. That I did, but I did also see something else:

Yikes – that is my box all right, but where did that horrible magenta color come from? To this moment, I still don’t know, but I found out how to fix it.

First of all, it really helps if, in addition to making the settings to your project that Morten describes in his first post, you tick at least “Unity C# project” as well in Build Settings:

I do the “Development Build” as well, although I don’t really know if this is neccesary).

This makes all the scripts end up in your Visual Studio solution as well, and what is more, you can debug them. Thus I learned that the script “SpatialMappingRenderer.cs” (in folder AssetsHoloToolkitSpatialMappingScriptsSpatialMappingComponent) looks for materials “Occlusion” and “WireFrame” that are clearly not there.

If you debug this script (you can now thanks to the setting above) you will see both OcclusionMaterial and RenderingMaterial end up as null. The materials are in HoloToolkit/SpatialMapping/Materials but changing “HoloToolkit” to “HoloToolkit/SpatialMapping/Materials” does not have any effect.

Deploy From Unity To Hololens

So I went back to the Unity editor, selected the Spatial Mapping Renderer script again, changed the dropdown “Render mode” from “Occlusion” to “Material”, and that made the property “Render Material” pop up. I found the Occlusion Material on top of it, and it was accepted.

If I now debug the code, the Occlusion material is still null, but RenderingMaterial is set, and lo and behold:

Now I am seeing what I expect to see – a box sticking out of a wall.

Hololens 2 Unity Tutorial

Disclaimer – I am just happily stumbling along trough Unity3D and HoloLens, not hampered by much knowledge of either. I found a problem (that I maybe caused myself) and I fixed it. I hope it helps someone. If I messed up, maybe someone can enlighten me. Unlike some other people, I am not afraid to look a n00b because in this case, that’s just what I am.

Hololens 2 Tutorials

The project itself can be found in a good old-fashioned ZIP file as I don’t want to clutter up my GitHub account with every little thing I try at this stage. I am still in the stage of ‘programming by changing things and observing their effects’, so I hope you will humor me. And maybe this will help someone, who knows.