How Warner Music Created the CARYS AR Avatar for Fans
Warner Music wanted to create an engaging content experience for fans of the musician CARYS for her song “Princesses Don’t Cry” of viral TikTok fame. They wanted to capture her signature dance, which was popularized with the #PrincessDanceChallenge hashtag.
The experience needed to be:
- Easy to share on social media to reach her fans and followers
- Easy to use since many users may never have used AR before
- Global since she has fans on every continent
Together with JAXX, they developed an all-new AR concept which was released to audiences on social media, inviting them to create videos with the CARYS avatar. (Click here for CARYS AR experience)
JAXX is a Toronto-based content marketing studio. They served as the creative lead for the project, specializing in creating augmented reality content.
SpinVFX is a visual effects studio providing visual effects for the film and television industry. Their award-winning team is behind the VFX of major Hollywood projects such as John Wick, Game of Thrones, and Stranger Things.
About this Article
In this article, we will explain how you can make an AR avatar yourself, using standard software for professional results.
BrioVR is a creative-friendly platform and real-time rendering medium that empowers creators to focus on end results. With no apps or coding required, the experience is frictionless for both users and creators.
Step 1: 3D Scanning
To scan CARYS, we used SpinVFX’s 3D scanning rig, complete with 144 DSLR cameras. The rig is mobile, and is typically used for on-set scanning for film and TV shoots, such as The Umbrella Academy.
For this project, CARYS was captured in a single 3D scan. Additionally, the SpinVFX scanning team used the scanning rig to capture every pose of the dance for reference. Changing facial expressions are part of the dance too, so close up photos of her face were also taken.
TIP: Although you may have a great 3D scan or model, reference material is always important to have. Try to capture as much as possible.
How to 3D Scan (Photogrammetry)
Scanning is possible without a Hollywood-quality 3D scanning rig. Lots of free scanning software is available online, or even through mobile apps like Display.Land. You can also create a 3D scan from basic, 2D photos using a phone or camera. Use a high-resolution camera if possible since the greater number of pixels that you have, the more detail you’ll get in the scan.
Choose a subject with a lot of texture or variation (i.e. in their clothing). However, avoid reflective or shiny materials (i.e. jewellry)
Shoot your photos/scan in a bright space with even lighting and no hard shadows. However, be sure that the lighting is not over-exposed.
Lock the camera exposure so that the brightness between each image is maintained
Ensure a 40% overlap between photos, with a minimum of 60 pictures
Model should stand in a ‘T’ or ‘A’ pose (if you’re shooting an object, use thin objects like toothpicks to prop up overlapping appendages).
Begin by shooting from further away with the whole subject in frame
Complete a 360 at eye level, lower angle, and then elevated angle, then move closer and complete 360 of the face or other important details.
These photos will be aligned in the scanning software to recreate a 3D mesh.
For processing the photos, you can use software such as Agisoft Photoscan, Capture Reality, Meshlab, and Meshroom from AliceVision (Free).
Step 2: Cleaning Up Your 3D Model
After your mesh has been scanned you will end up with a model composed of several million polygons. However, this will make it too heavy to run on mobile devices.
To ensure that your avatar will run smoothly in augmented reality, you will need to ‘clean up’ and remodel your scan so that it is both friendly for animation, and will perform well in AR.
Max poly-count for mobile Augmented Reality: 100K
Best performance poly-count for mobile Augmented Reality: 40K
You should aim for a max final-scene size of 10MB for the best performance on phones.
To clean up the scan, select a tool to trace new polygons that fit your scan. Maya, Mudbox, 3D-Coat, Zbrush, Blender, Meshlab, Meshmixer, etc. all have good tools for this.
SpinVFX used Maya, and the retopology tools within it, to get a clean result of CARYS.
Zbrush, Mudbox or 3DCoat can also be used to fill holes, smooth out noise and create a new topology in the 3D scan. Be sure to have enough edge loops around the joints of the character so that they can deform properly when you get to the animation stage.
When the mesh is ready, add UVs and fill the texture space so that fewer pixels are wasted in the final texture. UV editing tools can be found in many DCC software packages. Maya, 3D Max, Rizom UV, Blender, Modo, Zbrush, and 3DCoat all have great UV tools.
Step 3: Texture
The advantage of photogrammetry is that once the photos have been aligned in 3D space, you can import your clean, UV’d model back into that space, and reapply the photos for a beautiful look in no time.
For CARYS, we imported the clean model back into RealityCapture software and projected the photos back into the model.
After your textures are projected in the scanning software you can use tools like Substance Painter, Mudbox or Mari to clean up any errors in the projection. The final textures will later be imported into BrioVR.
Max number of textures: < 8
Max texture size: 2048×2048
Max number of materials: 8-10
Be sure that for one mesh you use one material, no materials in face sets. If you need to create the shading for different materials within one mesh, just create this within your PBR textures and that should get you the same results.
TIP: if you would like to include transparency in AR, you should add this into the diffuse or base colour texture as a .png file.
Step 4: Motion Capture for Animation
While the CARYS model was being prepared, she was invited to the SpinVFX studio to perform the dance in their strap-based motion capture suit. In order to accurately recreate her, her movements needed to be captured very precisely. The motion would later be imported into BrioVR for playback.
This is a camera-less system, but witness cameras were set up to record her from two different angles while she performed several ‘takes’. This reference material was critical to ensuring accuracy in not only the major joints, but the smaller appendages too (i.e. fingers, etc.)
With the model and Motion capture complete, a rig was created with a simplified set of controls. Careful attention was spent to balance the deformation of the joints so that movement was anatomically correct while playing back on the AR device.
You must have 1 connected skeleton hierarchy in your scene
Max number of joints in the skeleton: 254
When you create your skeleton, be sure that no more than 4 joints have any influence on a single vertex.
Max frame range: <400 fps
The more frames in the animation the larger the file size will be resulting in longer load times and poorer performance.
When the CARYS rig was ready, there was an opportunity to perfect the motion capture data in the animation stage in Maya. Adding subtle finger, head and shoulder movements, made her appear more realistic and life-like.
Animation in AR plays on a loop, so ensure that your animation completes the loop smoothly. For CARYS, we added a pose at the end to ensure that we could loop her back to the beginning in a single, cohesive motion, rather than ‘snapping’ her back to the starting position.
The final model, skeleton and animation were then exported in an .FBX file format for BrioVR.
Step 5: Preparing your AR Avatar
The .FBX file is then imported into BrioVR. Within BrioVR you can add other objects, add lights and materials, and set up your custom branding in the publish settings.
BrioVR is cloud-based, and automatically converts .OBJ and/or .FBX to GLTF or USDZ as part of the publishing process.
Voilà! You will be provided a unique URL that runs equally on iOS and Android. That link can be shared anywhere online that links can be distributed, including all social media platforms. It can also be embedded as an i-frame on websites.
Although AR is for mobile and tablet devices only, desktop users can view the scene in 3D. A unique QR code will appear for them to scan to launch the AR experience.
Augmented Reality lets you view digital assets and models in the context of the real world. BrioVR makes it easy for you to distribute your work in 3D, VR and AR.
Sign up to BRIO here: www.briovr.com