Awesome

360° Video Heatmaps

360° video is a new medium for storytelling and expression, but as such it also presents significant creative challenges. The rules are unwritten, the vocabulary still being discovered. When someone watches a 360° video in VR, they can look anywhere, possibly missing key parts of the story or even your million-dollar special effects extravaganza. So how do you guide the viewer’s gaze? How do you let the viewer know when they can explore the scene and when they should focus? Determining the answer to such questions — in other words, writing the rules and film-making vocabulary for the 360° video medium — will require experimentation, trying different techniques to see what works and what doesn’t.

Unity Analytics 360° video heatmaps can help you discover the answers by showing you where your viewers look and how they react. When you add the heatmap sender component to your video player, Unity Analytics samples the gaze direction of your viewers and sends that data to the Unity Analytics service. You can then view the collected heatmap data overlaid on the video itself in the Unity Editor.

Heatmaps indicate where viewers are looking using color gradients. The more people who look at the same area of the video the hotter that part of the heatmap overlay appears:

Note that the 360° Video heatmaps use statistical sampling appropriate for a large audience. It is not intended for tracking the gaze of a single viewer in precise detail.

Setup

1. Download and import the 360° Video Heatmaps SDK.

2. Enable Unity Analytics for the project

3. Add the Heatmap Sender component to collect heatmap analytics.

4. Create a playback scene for viewing heatmap data.

Requirements

• Unity 5.6+

• Unity Video Player component

• Equirectangular 360° and 180° video (monoscopic and stereoscopic)

• Video rendered using the included Skybox/Panoramic-Heatmap shader (for heatmap display)

• Compute shader support (for heatmap display)

If you are displaying video on a computer monitor rather than a VR headset, you must supply controls so that viewers can aim the scene camera. Otherwise, all the heatmap data will be focused on the same spot of the video and won’t provide any useful data for analysis.

Enable Video Heatmap Analytics

Enable analytics for 360° Video Heatmaps on the Unity Analytics Dashboard:

1. Navigate to your Analytics dashboard:

   1. In the Unity Editor, open the Services window (menu: Window > Services).

   2. Click Analytics on the Services window to go to the Analytics section.

   3. Click Go to Dashboard.

      

2. On the dashboard, click the Configure tab.

3. Under Feature Settings, click the Activate Heatmaps button.

   

Collect Heatmap Analytics Data

To collect heatmap Analytics data for a 360° video, add the HeatmapSender component to the scene Camera.

1. Complete the Setup steps to enable Analytics and install the 360° Video Heatmaps SDK.

2. Select the Camera in your Unity scene.

3. Click Add Component in the camera Inspector window.

4. Type Heatmap Sender in the search field.

   

5. Click the Heatmap Sender component in the list to add it to the Camera object.

   

Note that the Heatmap Sender component finds the Video Player object automatically when the scene loads. However, if you have more than one Video Player in a scene, then you must drag the correct Video Player’s object to the Heatmap Sender’s Video Player property.

If you have already created a scene for playing 360° videos using the Unity Video Player component and the panoramic skybox shader, then you are done. Otherwise, set up the Video Player render texture and the Skybox material.

Set up the Video Player

Note: for more information on the Video Player settings, see Panoramic 2D and 3D Video Shader in Unity, the following is just a quick guide.

To set up the Video Player to render into a texture:

1. On the Video Player component, set the Render Mode to Render Texture.

2. Create a render texture asset appropriate for the video:

   1. Create a Render Texture Asset (menu: Assets > Create > Render Texture).

   2. Select the new Render Texture asset to view its properties in the Inspector.

   3. Set the size to match your video.

   4. Set Depth Buffer to No depth buffer.

      

3. Drag the render texture to the Video Player’s Target Texture field.

   

Set up the Skybox

Note: for more information on the Video Player settings, see Panoramic 2D and 3D Video Shader in Unity, the following is just a quick guide.

To set up the skybox to display the video texture:

1. Create a material for the skybox:

   1. Create a new Material Asset (menu: Assets > Create > Material).

   2. Select the new Material Asset to view its properties in the Inspector.

   3. Set the Shader to Skybox/Panoramic-Heatmap. Note: you can use the standard Skybox/Panoramic shader in the video player application that you distribute to viewers, but to see the heatmap data, you must use the heatmap version.

   4. Drag the render texture used for the Video Player into the Spherical (HDR) texture field.

   5. Set the other shader fields as appropriate for your video clip. (You do not need to set the Spherical Heatmap (HDR) and Gradient Lookup fields, these are set automatically.)

      

2. Open the Lighting window (menu: Window > Lighting > Settings).

3. In the Environment section, drag the new material asset to the Skybox Material field.

   

Setting the sample rate

When you activate the heatmaps feature on the Unity Analytics dashboard, the system automatically creates a Remote Settings key named heatmaps_sample_rate with a value of 10%. This setting means that a viewer has a 10% chance of being included in the test audience. If you have a large audience, a 10% sample rate should be sufficient for most purposes. You can use different sample rates for debug builds of your video player application and release builds.

To set the sample rate:

1. Navigate to your Unity Analytics Dashboard for the project.
2. Click the Remote Settings tab.
3. To set the sample rate for release applications, choose the Release configuration. Otherwise, choose the Development configuration.
4. Locate the heatmaps_sample_rate key in the list of settings.
5. Click the pencil-shaped edit button to the right of the screen.
6. Enter the desired sample rate.
7. Click Save.
8. Click Sync.

Repeat this process to change the sample rate in the other configuration, if desired.

Note that events are always sent when you run your video player scene from the Unity Editor.

Testing the Analytics setup

Use the Validator at the bottom of the Analytics Services window to check that the heatmap analytics events are sent when you view a video in the Unity Editor:

1. In the Unity Editor, open the Services window (menu: Window > Services).

2. Click Analytics on the Services window to go to the Analytics section. Note the Validator section at the bottom of the Analytics window.

3. Press Play in the Editor to start the video.

4. Every half second, you should see a custom event named Heatmap.PlayerLook appear in the window:

   

Viewing 360° video heatmaps

You can view heatmaps superimposed on your 360° videos using the same video player that you used to collect the analytics data — as long as you use the heatmap version of the Skybox/Panorama shader. You can also make a separate heatmap viewer scene if that makes more sense in your project.

The general process for analyzing how viewers watch your video is:

1. Collect data. It can take a few hours for the Unity Analytics Service to process any collected data.

2. Download the processed data using the Raw Data window.

3. View the heatmaps superimposed over the video using the Heatmapper window.

Creating a player scene

To create a scene to display your heatmap data, follow the instructions above to create a Video Player component that renders into a skybox texture. (You do not need a Heatmap Sender component for this scene.)

1. Set up the Video Player

2. Set up the skybox

When setting up the material used for the skybox, you must use the Skybox/Panorama-Heatmap shader.

Download the Collected Data

Use the Raw Data window (menu: Window > Unity Analytics > 360 Video Heatmaps > Raw Data) to download the data collected when viewers use your video player application.

To download your heatmap data:

1. Open the Raw Data window.

2. Check that the UPID and API Key have the correct values for the current project. (Click the Project Config button for quick access to the page listing these values.)

3. Set the Start and End Dates as desired, using the format YYYY-MM-DD.

4. Click Create Job. The new job is added to the Job List. Job processing can take a few minutes.

5. Click Get Jobs after a few minutes. When the job is ready to download, a Download button appears for the job.

6. Click the Download button to get the data. The data is downloaded to the location specified in the Output path field and is ready to display.

View the Heatmap Data

Once the data is collected and you have downloaded it locally, open the Heatmapper window (menu: Window > Unity Analytics > 360 Video Heatmaps > Heatmapper).

First, make sure that the input Path is the same as the Output path listed on the Raw Data window. (If the two paths are different, you must change the Raw Data Output path and redownload the data.)

Next, drag the scene Video Player component to the Video Player field and the skybox material used to render the video to the Video Material field. If you have more than one video clip in the project, set the one to watch in the Video Clip field.

Depending on how much data you have collected, you might want to increase the Radius and Decay Rate settings. These settings make individual points more visible. For information about the other settings, refer to the Heatmapper window topic.

To watch the video with superimposed heatmaps in the Unity Game or Scene views, press the Play button on the Heatmapper window. You can also scrub the video using the Frame slider.

Note that heatmaps are not superimposed over the video when the Editor is in Play mode, only in Edit mode.

The Flattened Heatmap View provides a full-frame, equirectangular view of the video so that you can see the entire frame and heatmaps without having to rotate the camera.

Component and Asset Reference

The 360° Video Heatmaps SDK includes the following components and assets for collecting and viewing 360° video heatmap data. (Other components, scripts, and assets in the SDK package should be considered internal.)

• Heatmap Sender component

• Raw Data window

• Heatmapper window

• Flattened Heatmap View

• Skybox/Panoramic-Heatmap shader

Heatmap Sender component

Attach the HeatmapSender component to the scene camera. This component sends heatmap events to the Analytics Service and reports where the camera is looking (using the camera’s forward direction).

Raw Data window

The Raw Data window provides access to the heatmap data collected by the Unity Analytics Service. Open the Raw Data window using the Unity Editor menu: Window > Unity Analytics > 360 Video__ Heatmaps__ > Raw Data.

Before you can display heatmap data, you must create a job and download the data to your local computer using the Raw Data window. See the Download the Collected Data topic.

Heatmapper window

The Heatmapper window controls the display of the heatmap data. Open the Heatmapper window using the Editor menu: Window > Unity Analytics > 360 Video Heatmaps > Heatmapper.

For information on using the Heatmapper window to view your heatmap data, see the View the Heatmap Data topic.

Flattened Heatmap View

The Flattened Heatmap View displays a "flattened" equirectangular projection of the video with the heatmap overlay. This window enables you to see an entire video frame at once. Open the Flattened Heatmap view using the Editor menu: Window > Unity Analytics > 360 Video Heatmaps > Flattened Heatmap View.

Use the Heatmapper window to control the content and appearance of the Flattened Heatmap View.

Skybox/Panorama-Heatmap shader

The Panorama-Heatmap shader is a version of the Panoramic 2D and 3D Video shader modified to display the heatmap. This shader is required to view heatmaps, but you can use either version in the player that you distribute to viewers.