Sculpting in Virtual Reality

One of the exciting things about Virtual Reality is the new avenues it opens up for interaction, not only in games but in applications and tools. My biggest wishlist item since Oculus was announced was a 3D modeling app, and while there’s still a lot of stuff that needs to be figured out, Oculus Medium has delivered a very compelling peek of the future. Let me talk about my process and what I’ve learned.

First let me say that I’m not a professional modeler. I’ve taken a class in Blender, and some online courses in Zbrush, but I am by trade a UX Designer. I did study traditional fine art in school so I know traditional sculpting skills but due to the heavily technical nature of 3D art/design (as opposed to 2D stuff which is much more immediate) there’s a lot I don’t know. Oculus Medium is fantastic because of the way that it emulates real life sculpting but it’s also not as simple as just switching from Zbrush or Mudbox to Oculus.

Oculus Medium is voxel based. This essentially means that it uses 3D pixels. Picture+ Element= Pixel, Volume+Element=Voxel. A pixel is a grid, and a voxel is a grid of grids, making a cube. A lot like the game minecraft. Voxels are like raster graphics. Traditional 3D rendering and 3D software uses vertexes, which are basically like 3D vectors. If you’re familiar with the difference between Adobe Illustrator/Flash, and Adobe Photoshop, then Oculus Medium is more like 3D Photoshop.

Voxels are not generally used for rendering 3D objects because they take a whole lot of memory. If you want a more technical explanation of why, I highly recommend this video by Craig Perko where he describes the reasons in depth:

Voxels have a lot of advantages over traditional vertex modeling. Voxel sculpting feels more intuitive to me and I found myself really enjoying my time in VR sculpting. This sculpture took two days of work, which is a bit long, but most of that was spent figuring out the interface and how to best approach the work.

After doing the skeleton warrior I decided to fish around for a new challenge. A friend on facebook suggested I try to model Beyoncé’s costume she wore at the Grammys. I’m not a Bey fan, but her Grammy outfit looked cool and I had recently read a beautiful poem by Warsan Shire, London Poet Laureate, and contributor to Beyonce’s newest album. Her poem The Unbearable Weight of Staying is worth a meditation.

The Beyoncé sculpt offered a range of challenges that I thought would be interesting to try inside Oculus Medium such as human likeness (humans have spent millions of years refining our facial recognition wetware and as a result we’re very very good at it), the varying levels of detail, and the gossamer like dress that she wore.

The intuitive nature of voxel modeling is very straightforward. You click the button and a ball appears in the air trailing behind your hand’s movement. You set it to erase and you can selectively carve away what you don’t want there. It’s essentially as simple as that. You can even create custom brushes, called “stamps” that function similar to Photoshop. Photoshop brushes are a repeating pattern of pixels taken from a thumbnail. Stamps don’t have the sophisticated extra features of Photoshop brushes (YET!) but they work on the same principle. You drag a ball around, and you get a big tube in its diameter. You drag a disk around and you get a flat thin stroke that you can lay down like a layer of clay onto a sculpture.

The nature of voxel sculpting can be contrasted to a similar program by Google, called Tilt Brush that uses vertexes. It can be used by a skilled artist to make beautiful piece of art that look like paintings in the air (check out my favorite Tilt Brush Artist Elizabeth Edward’s work below), but the vertex nature of it also limits it in certain ways. It means that when you want to delete a brush stroke, you must delete the entire stroke, and can not erase only a part of it. Each stroke is it’s own vertex group, similar to the way that each vector brush stroke in Illustrator gets it’s own sublayer. You can selectively erase parts of vectors in Illustrator, so I imagine the same would be eventually true of Tilt Brush or Quill (another vertex based 3D art program)but as of now, you are committed to the entire stroke, or you erase it entirely. Even if this were fixed, the vertexes are 2d flat shapes strokes with shaders/textures applied in 3D space and have no depth/mass, so you can only really make a sketchy painterly piece, which has limited use in other applications like games (though obviously a clever person could probably turn it to their uses if they wanted).

Voxels have mass, because they’re 3D cubes which then have a smoothing algorithm applied to them to give them a consistent surface rather than the blocky look of Minecraft. If you want to use your work to a more traditional vertex based animation or game engine application, you have to convert it it to an OBJ using the export feature. This brings up the second difficulty of voxel based sculpting. Vertex models don’t take up as much ram as Voxels because they only describe the points that make up the mesh surface of the model. Voxels hold not only the surface but also every single internal 3D “pixel”. If you have a giant flat space, in a vertex model that giant flat space can be made up of basically 4 points, while the same flat space has to be made up of hundreds of voxels, just like a 900px by 900px raster image is going to be larger than a 90px by 90px image even if they’re basically the same thing.

Vertexes are capable of variable levels of resolution in the same mesh. Areas of large undifferentiated geometry can be represented by fewer points, and then at areas of high detail you can have a bunch of of vertices. So if you want to have a highly detailed head attached to a big simple body, then you have to waste a lot of extra RAM holding that simple body at the same resolution as the highly detailed face. So to get around this with the Beyoncé sculpt, I separated her face and body into two parts, and used her jewelry to disguise this. Each of these elements were on a separate layer and had different levels of resolution that were then piece together. Her face is something like 6 levels of resolution up from her body. So even while the head is represented in space as a smaller object in Oculus Medium it actually includes 6 times more voxels. Think of it kind of like having a high resolution smart object played inside of a lower resolution image in Photoshop, and maybe that can give you an idea.

Even with this division of resolution I still ran into a lot of RAM problems. Voxels aren’t commonly used for a reason, and even 6 times more resolution felt restrictive at times on the face. It doesn’t help that I’m also pretty out of practice on my figurative work. This is just a hardware problem though and eventually will be less of a problem which is why I believe that the voxel rendering choice by Oculus was the right one to make.

The fact that voxels are defined at every point in space evenly like this, taking up so much memory, is a issue but also a great boon too. Since every voxel is taken into account equally, you can easily hollow out objects, or sculpt objects that are unattached from the main figure without having to create a separate vertex group etc. You can see this with the Beyoncé sculpt and the way I did her hair. Near the ends of her hair you see unattached pieces of hair. This is because of the way I created a custom hair brush and feathered the hair form. Similar to the kind of technique that 2d concept artists use to quickly generate hair, grass, or tree leaves, but applied in 3D space, I modeled a tiny part of her hair, and then rapidly repeated it’s form to create the overall shape and texture of her hair. The unattached pieces of hair at the end were unintentional but I liked the painterly effect and kept it. Once you have more memory and resolution at hand (and a greater range of options to tweak like in Photoshop brushes) I can see this system becoming extremely extremely powerful. There’s a whole dimension that has yet to be explored in 3D modeling with this functionality.

Another challenge I mentioned was making the gossamer-like dress. Originally I made that by take a leaf shaped stamp and creating a threadbare dress in the shape of the dress, but this proved challenging. Exported alone, the dress was 1.5 gigs and had something in excess of a million polys. That’s a lot! (Typical video game models have something in the low thousands if not hundreds). It was a complete mess to work with. I tried cleaning it up with Mesh Lab (which I highly recommend: but to no avail. My computer just couldn’t take it. So I ended up remodeling the dress in Blender and setting it to a transparent material.

Which brings me to this key aspect of sculpting in voxels. You have to treat what you are making as a solid object. It’s like throwing out a lump of rough earthenware opaque clay, and doing delicate things can be difficult because you can’t rely on texture/materials to give the ephemeral appearance and trying to translate that lacy appearance literally is extremely memory intensive.

Even doing delicate features like the face, which is completely solid, is a bit of a challenge. Sculpting with vertex programs allows for a certain amount of surface control (because you’re essentially just dealing with the surface, there is no “inner space” like with voxels”. If your handle trembles in Oculus Medium you can end up with a big divot in the cheek or lip where you don’t want it. I believe there are some UI solutions that could be used to get around this difficulty (aaaay, drop me a line Oculus!) but it’s still a challenge to take into account.

In conclusion, I think Oculus Medium is a super promising start to VR Voxel sculpting and I recommend it to any Oculus or Vive users (I actually own a Vive and just used ReVive to run it no problems). I’d recommend it primarily as a method of roughing out shapes and composition, and then cleaning up in a more traditional vertex program like Blender, Mudbox, etc. (or Zbrush though that’s technically also non-vertex that’s a whole another story to explain). I am by no means a professional modeler, and there are tons of more advanced features that will need to be worked out for game assset optimization (and probably an entirely different program for VR box modeling), I’m sure, but I’m excited to keep learning and figuring out my workflow with the program.

Also published on Medium.