Really cool idea, I wonder why nobody tried this earlier.
However, the main problem I have with this approach is the voxels. They model the geometry and only the geometry. The far more important aspect of topology is left out. Thus, the results suffer the same problems like 3D scanning / photogrammetry does: It is practically unusable as it can not even be textured, yet alone animated or used in fabrication (except for Lego-models I guess). So point 5 of the future work is the biggest one in my opinion.
Voxels are directly analogous to 2d pixels, which means they benefit from the myriad of CNN based techniques developed over the last few years (GAN in particular)
3d manifolds are sparse, and more analogous to 2d vector graphics. There are approaches for dealing with this type of data (eg. spectral graph NN) but they don't work as well for 3d topology as CNNs do for dense pixel data, as far as I know.
In the near term, it might be better to explore approaches that use voxels, then generate the topology heuristically.
Electronic Arts used 3D scanning / photogrammetry in the new Star Wars Battlefront II.
You're right - the topology is a huge problem - but we're starting to see specialized tools that can take minimal input from the artist and automate the topology creation workflow.
Here's the GDC talk where the E.A devs go over how they converted the scans to game usable 3D models.
https://youtu.be/U_WaqCBp9zo
Well they have at least for certain parts. I'd guess based on what they use, the whole workflow here was really only possible in the last couple of years.
I'll update this comment another time but back in 2016 my company needed something like this and there was already research using GANs to generate objects from basic parameter inputs. The text portion wasn't there.
I believe that research came out of Stanford actually.
Ya using voxels was definitely a big drawback to my approach here but was more intended to just see what's possible.
I think a better approach would be something similar to the StructureNet paper I mention in the post and use graph based models to actually attempt to capture the topology. But they did it with super explicitly defined part trees as training data the hard part would be finding a way to do that in a unsupervised manner so you could actually make use of the massive amount of unlabelled 3D models available.
However, the main problem I have with this approach is the voxels. They model the geometry and only the geometry. The far more important aspect of topology is left out. Thus, the results suffer the same problems like 3D scanning / photogrammetry does: It is practically unusable as it can not even be textured, yet alone animated or used in fabrication (except for Lego-models I guess). So point 5 of the future work is the biggest one in my opinion.