3D graphics today are no longer bound by the limitations of screens and devices. Instead, they transport us into immersive digital worlds that feel as real as the one outside your window. This future is not science fiction; it’s the present, and it’s being powered by a fascinating concept called Gaussian Splatting. In the pages that follow, we’ll delve into its inner workings, demystify its magic, and explore how it’s leaving its indelible mark on everything from 3d graphics, game development to visual storytelling.
Balancing Realism and Efficiency : Photogrammetry & NeRF
In recent years, the world of 3D graphics has made significant strides in pushing the boundaries of realism. One prominent technique, Photogrammetry, involves capturing numerous photos of an object and then reconstructing them into a polygonal mesh. However, it has a notable drawback: when dealing with objects that have shiny or featureless surfaces, accurately mapping them can be difficult as there are limited surface features available for matching between photos. The process also depends on favorable lighting conditions which can lead to occasional issues with blurry or poorly exposed scenes.
Photo Credit : Professor Jeff Price, Associate Professor, Game Design and Virtual Reality, The University of Texas at Dallas
Another such innovation is Neural Radiance Fields (NeRF), which utilizes neural networks to replicate intricate lighting dynamics, resulting in highly detailed and realistic visuals. However, NeRF comes with its own set of challenges, particularly in its ability to handle dynamic scenes. As the demand for more lifelike 3D graphics grows, it becomes evident that a new approach is necessary to harmonize the balance between visual fidelity and computational efficiency.
Photogrammetry vs NeRF
Photo Credit: Jamie Portolese, GISP, Senior Research Data Scientist, CBRE.
“NeRF comes with its set of challenges, especially when real-time rendering is a prerequisite for dynamic applications like gaming”, says Kaveh Kamali, Senior Research Engineer, Unity.
Gaussian Splatting: The Future of 3d Graphics
Enter Gaussian Splatting, a revolutionary technique poised to address the challenges of real-time rendering in 3D graphics. While Neural Radiance Fields (NeRF) had raised the bar for rendering quality, Gaussian Splatting represents a novel approach, departing from the conventional use of polygons and triangles to depict objects. Instead, it employs “blurry” mathematical volumes known as Gaussians, which are derived from sparse points acquired during camera calibration. Through an efficient training process, these Gaussians are refined, adapted, and manipulated to provide a compelling blend of quality and speed, making Gaussian Splatting an exciting prospect in the world of 3D graphics. For an easy breakdown, just hit play on Dylan Ebert’s 2-minute video above.
“Gaussian splatting comes with a handful of advantages over NeRFs. They are generally easier and faster to compute, smaller files and more accessible to store and since they have been researched for around 20 years and don’t require machine learning, the final results are much more predictable.”, says Michael Tanzillo, Head of 3D Technical Artists, Adobe, Substance 3D.
Creative Masterpieces : Crafting 3D Realities with Gaussian Splatting
Now that we’ve got a handle on Gaussian Splatting, it’s time for the really cool stuff! Let’s dive headfirst into a world of mind-blowing innovations, all brought to life with the magic of Gaussian Splatting. Get ready to have some serious fun while we explore the wacky, wild, and wonderful side of 3D art!
Bad Decisions Studio co-founded by Farhad Shababi and Faraz Shababi — brought these 3D Gaussian Splatting tests into Unreal Engine 5 to unlock new creative possibilities. Utilizing an available plugin from the Unreal Engine Marketplace, they examined quality, performance, cropping tools, and even set up cameras to craft mesmerizing cinematic videos. Their YouTube channel is an enthralling hub of such 3d creations, showcasing an array of content, from transforming iconic Hollywood film scenes into 3D to exploring 3DGS within VR.
Another intriguing application comes from Chris Young, who skillfully integrated a Gsplat into Unreal Engine for Fortnite by employing Niagra Particles; you can see the Fortnite avatar navigating within a realistic 3D environment in the video below.
Kishimisu, a Parisian Machine Learning enthusiast, developed a cutting-edge 3D Gaussian Splatting renderer tailored specifically for WebGL, enabling an astonishing 1.3 million splats in real-time.
Arata Fukoe, a japanese video director, achieved an incredible level of realism by combining 34 GoPros with 3D Gaussian Splatting within Unity with a meticulous orchestration of camera work within the Unity framework.
Gaussian Splatting 4D: 3D Graphics Meet the Fourth Dimension
4D Gaussian Splatting, Dylan Ebert
Now that you’ve learned about Gaussian Splatting, let’s venture into Gaussian Splatting 2 aka Gaussian Splatting 4D. This next step is all about adding a dynamic twist using something called deformation fields. Think of it as turning a collection of still images into an action-packed video. These fields are a super-efficient, speedy neural network similar to the technology that powers AI. It might sound a bit complex, but the future with 4D Gaussian Splatting seems incredibly exciting. We’re just scratching the surface, and it’s all open source, which means everyone can jump in and explore the endless possibilities!
Laan Labs, a product development and consulting firm based in New York, created something interesting by merging 3d and AR. Watch how an incredible dynamic display of moving 3D splats come to life within an AR environment.
Augmented Reality 3D Gaussian Splatting in Real Time on iPad
While the early findings are quite noteworthy, the 3D motion sequences recorded by the research team require a substantial 20 minutes of filming for mere seconds of actual footage. This duration is insufficient for covering the extensive time demands of a complete feature film, exemplifying a substantial gap.
“For studios looking to capture an actor’s few motions and re-use them, it’s a great start. And for video game designers, XR/VR designers, it’s hard to imagine that this technique will not be useful.”, Carl Franzen, Editorial Leader, VentureBeat.
Gaussian Splatting has ignited a revolution in 3D (or even 4D) graphics, but its impact stretches far beyond pixels and polygons. In film production and gaming, Gaussian Splatting’s ability to deliver efficiency and unparalleled detail can set a new benchmark. It will empower creators to craft worlds with precision. Even in Augmented and Virtual Reality, it will redefine the boundaries ensuring that every frame tells a story that’s more vivid and immersive than ever before. It’s not just the next big thing; it’s a transformative tool that will continue to inspire and reshape our digital world.
Edited by Rufah Kaka.
Follow us as we’ll be sharing the latest updates from the tech savvy world and insights on our journey as we continue to revolutionize and fabricate immersive realities! Socials below. Until next time!
