In the previous post, I explored how to improve the results of generating 3D objects with zero-1-to-3, a technique that allows us to control the shape and appearance of the objects. For almost any real-life usage of the result of generation, we need to convert it to a mesh with texture, as that’s the most popular way to represent 3D objects (for now). They are the most widely supported and easiest to edit and manipulate medium for such data. So, in this post, I generate a mesh and texture from the result and check available parameters.
This time, I used 4800 steps to train the model, which should result in slightly better quality than before. The loss stopped decreasing after that point, so more steps would not improve the quality significantly. You can see the running average of the loss for a 10-step window in the figure below.
To convert a NeRF into a mesh, we need to use a process called surface extraction. Surface extraction is a technique that finds the boundary between occupied and empty cells in the voxel grid, and creates a mesh that follows that boundary. Threestudio supports two methods for surface extraction: Marching Tetrahedra and Marching Cubes. Both of them extract a polygonal mesh of an isosurface from a 3-D volume. They both work by dividing the volume into cells and finding the intersections of the isosurface with the cell edges. Isosurface is a surface defined by a certain value in the space of a field – like lines of the contour map.
- Marching Cubes – argument:
system.geometry.isosurface_method=mc-cpu– It uses the original twelve edges of the cube to find the intersections with the isosurface. It has more configurations (256) than Marching Tetrahedra (16), increasing the lookup tables’ size. The other downside of this method is that some ambiguities can cause holes in the mesh. In threestudio implementation of this method, there are no predefined resolutions, with VRAM being the only limiting factor. I was able to generate a mesh with a resolution of 300, which created a way denser mesh, than the default Marching Tetrahedra method.
- Marching Tetrahedra – default, argument:
system.geometry.isosurface_method=mt– It splits each cube into six irregular tetrahedra by cutting diagonally through each of the three pairs of opposing faces. It computes up to nineteen edge intersections per cube, whereas Marching Cubes only requires twelve. It has fewer configurations (16) than Marching Cubes (256), which reduces the size of the lookup tables. Furthermore, it does not suffer from ambiguities that can cause holes in the mesh, because each tetrahedron has a unique way to separate the positive and negative vertices. It provides a slightly better sampling resolution than Marching Cubes, because of the additional intersections. Threestudio implements only 3, predefined, resolutions for this method: 32, 64, and 128 (default). The downside of this method is that the tessellation of a cube with tetrahedra requires a choice regarding the orientation of the tetrahedra, which may produce artificial „bumps” in the isosurface because of interpolation along the face diagonals. That effect was visible in the generated examples.
UV unwrapping is done XAtlas library. After the mesh UVs generation, colors are sampled in export_obj_with_mtl method. CV2 inpainting is used for filling UV island padding.
Generated meshes
I started with default values:
Some glitches are clearly visible. Firstly: bumps on smooth surfaces, as mentioned earlier, can be noticed. Change to Marching cube algorithm should fix that:
There are white parts of the model: I think, that those could be fixed by moving isosurface „inside” the object by increasing the system.geometry.isosurface_threshold value. But, just to check how it will look like, I tried lower value first:
The result matched expectations: There are a lot more white, blocky elements. I tried a few more values to find a sweet spot:
The value of 75 seems to produce the least amount of white artifacts without removing too much of the volume from the objects. Doing that results in artifacts visible on the lest two meshes from the right.
With the Marching Cubes algorithm, I was able to generate a mesh with a resolution of 300. The bump artifacts are gone, mesh is way smoother, but also way denser:
The mesh is way too dense for any real-life use cases. It should be refined, with remeshing and decimation.