Implementing 360 Video in Unity for Gear VR and Cardboard

This entry will describe our efforts to get 360 video working in Unity for apps running on Gear VR and Cardboard (both iOS and android). This is more of a work in progress than a full guide, but I hope it helps. Feel free to leave questions or suggestions!

The overall process is:

  1. Start with google cardboard camera or oculus camera demo scene.
  2. Add a sphere with an equirectangular UV mapping and inward facing normals around the camera.
  3. Purchase a plugin to play a movie on that sphere’s texture. (Note: if you just want to run on Gear VR, you can adapt their movie example to play on the inside of a sphere. I’m not sure how to extract their movie playing code to a cardboard android app, though).
  4. Use mp4s or ogg vorbis files that are compatible with the platform. Some resolutions and frame rates don’t work for me.

Movie & Video Plugins for Unity and Android / iOS / Gear VR.

The most success so far has been achieved using the Easy Movie Texture plugin for Unity, currently $45. It works on android (utilizes Android MediaPlayer) and iOS and the developer gives great support. It even supports streaming and playing from the SD card. Note that it doesn’t work in the Editor, so you have to build to see the results.

The best part of this plugin is that it comes with a demo scene including a sphere to play back equirectangular videos. Getting a suitable sphere is a little tricky, since the default sphere in Unity has normals facing outwards rather than in, and doesn’t have the right UV mapping.

I also tried prime31’s iOS video plugin, LiveTexture, which was $75. This one was nice because videos would play inside the Editor, but I had to provide my own sphere and it doesn’t work on android. AND, sounds don’t play. The plugin provides a way to sync an audio track along with the video, but this seems likely to get out of sync. I had trouble with larger format videos being very choppy. A 2048×1024 was unwatchable on iPhone 5S, which is my lower end target for playing back 360 video with cardboard. Anything below 1000px of vertical resolution starts too look poor under a magnified 360 view.

Another one that’s also $75, but free to try, is Mobile Movie Texture. This one requires ogg vorbis files, and also doesn’t play video at the same time. It does support android and iOS.

One other plugin I tried plays back a sequence of images: Universal Video Texture ($20). This also doesn’t sync audio, but has the potential for extreme hi-def playback at the cost of extreme app size–not ideal for a mobile app. I believe a lot of the compression in an mp4 or ogg vorbis file is due to the compression algorithm knowing that a certain pixel doesn’t change from one frame to the next.

A final plugin that was recently pulled from the store by the developer (too much work to maintain maybe?) was Video Texture Pro 2. Here’s an interesting forum post by its author about video support in Unity.

I just came across a forum post on reddit talking about SPlugins Movie Texture for Mobile ($30) that may also work on android and iOS with sound, but haven’t tried it yet.

360 Equirectangular Sphere for VR Movie Playback in Unity

If you’d like a sphere to use with LiveTexture or one of the image sequence players, check out the roundscreen.obj (has holes at poles) here or an inside facing normals sphere with 32×16 vertices I made in blender. I’m not sure how the number of vertices impacts the resulting quality vs performance. Lower poly spheres seem to look very bad near the poles. Here are the instructions I followed to create the UV wrapped sphere if you’d like to try out much higher vertex counts. If you find a sphere with the normals facing out rather than in, you can run this reverse normals script on the object.

(With LiveTexture, I also had some strange wrapping issues with non-powers-of-2 video resolutions (ie, not 1024,2048, etc). I got around those by commenting out the line including “updateMaterialUVScaleForTexture” in the plugin code.)

In addition to reversing the normals, you may also need to mirror the UV map–ie, it might playback flipped. The Easy Movie Texture plugin flips the UV y coordinates (on iPhone only?) by iterating through all the UVs and reversing the y:

vec2UVs[i] = new Vector2(vec2UVs[i].x, 1.0f -vec2UVs[i].y);

Video Resolution, File Location, Frame Rate and Other Considerations for VR on iOS and Android

I was able to play 4096×2048 mp4s on Android, however files with a 50 fps frame rate stalled repeatedly on a Galaxy Note 4, whereas 25 fps files played perfectly (Using Easy Movie Texture plugin). Reducing the bitrate in half or the resolution had no effect on playability, only the frame rate mattered. I’m still trying to wrap my head around what frame rate I see given that the Unity app is running at one frame rate and the video at another.

One tip for experimenting with different files is to play off the SD card on android. This way you don’t have to import the videos into Unity. Placing an mp4 into the Streaming Assets folder in a Unity project causes unity to try to import it to a ogg vorbis file, which can take hours for videos over a few minutes, and is completely unnecessary since the plugin plays the original mp4. You can also make the extension a bogus name like .mp42 to prevent unity from converting the video. This worked for me on Android, but not iOS.

To play a file from the SD card, simply give the Str File Name variable in Easy Movie Texture the following full path:

file:///storage/extSdCard/myvideo.mp4

Note that the path to the sdcard may be different on different phones. The above is only for the Galaxy Note 4. You can find out your path with the following command:

adb shell 'echo ${SECONDARY_STORAGE%%:*}'

Note on iOS, be sure to run one of the patches depending on your version of Unity inside the EasyMovieTexture folder: Unity5_Patch_IOS or Unity463_Patch_IOS.

Another big restriction on iOS, at least on an iPhone 5s is that the max file size is 1920×1080. So for 2×1 equirectangular movies, this is 1920×960, almost half the size of the android. I believe the iPhone 6 can play larger files, but have not tested yet.

Open Questions

  • What is the limit of playback fps and resolution on both iOS and Android? I know Gear VR supports 4096×2048 at 60fps, but is this possible from within Unity?

 

 

Looking to create custom, native VR Apps without having to learn any of the above or touch a single line of code? Our VR Forge system might be just what you need to get going quickly and profitably. Take a look to see if it’s right for you!