Friday, 18 September 2020

How to Upscale Video to 4K, 8K, and Beyond

For the past nine months, I’ve been working on what I’ve named the Deep Space Nine Upscale Project (DS9UP). The goal of DS9UP is to create a new, much-improved version of the show by applying modern processing techniques to the original DVD source before using AI-based software to create a higher-resolution version of the show. It’s had me thinking about upscaling and upscalers in general. Upscaling isn’t a feature we talk about much, but how your TV handles it (or, alternately, how you use the capability on a PC) can have a significant impact on how you experience content.

The word “upscale” generically means “to improve the value or quality of something.” In the video and PC space, it’s almost always a reference to increasing the effective resolution of a piece of content. There is typically an understood difference between upscaling and native resolution. If you upscale a 1080p video into 4K, it means you are taking a 1080p input and using a combination of hardware and software to create a larger image. This upscaled image will not be identical to a native 4K signal, but it should offer a better picture than what was previously available on your 720p or 1080p television.

Keyword: “should.” Video scalar quality in TVs can vary widely between different product families. In some cases, you might be better off using a GPU to drive a picture than relying on the TV’s native rescaling capability, while other TVs have excellent upscalers. Manufacturers rarely disclose their upscaling hardware choices, but higher-end TVs should have improved upscaling capabilities. If you have a UHD Blu-ray player paired with an older or lower-quality 1080p or 4K TV, you might even get better results by running all video signals through the Blu-ray player rather than the television. Generally speaking, a good TV upscaler is considered to be as good or better than a GPU.

How the Scalar Sausage Gets Made

The most basic function of a video scaler is to take whatever image it receives — 480i, 720p, 1080p — and stretch it across the entire screen. Without this functionality, a 1080p signal would take up just a fraction of a 4K television’s display. This simple resizing is typically done by taking each individual 1080p pixel and creating four pixels out of it (remember, 4K is four times the pixels of 1080p).

But many TVs and Blu-ray players do more than just perform a simple 1:4 mapping. They also use video processing techniques to extrapolate what details ought to be present in the scene. How well this works depends on the type of content being upscaled.

Image by DisplayNinja

In the image above, you can see how the upscaled 4K is much more nuanced than the simple 1:4 mapping in the second grid from the left. If you’re having trouble seeing the difference between the 1080p upscale and the native 4K, look at the left-side blocks in the very first row and the right-side blocks in the very last row. The native 4K image resolves into distinctly different colors than the 1080p image in R1C2 (That’s 1st row, 2nd column), R1C3, and R8C8. As the amount of available horsepower in televisions has improved, the quality and sophistication of their integrated upscalers have grown as well. Some modern TVs have sophisticated sharpening algorithms to reverse the blur created by upscaling and interpolation algorithms good enough to almost match the precision of a native 4K signal.

How does all this look with real content? A 2017 Rtings article can help answer that question. The first image below is from a 4K set displaying 1080p in upscaled 4K, while the second is native 1080p.

Image by Rtings

If you have trouble seeing a difference between the two images, open both of them in a new tab and focus your eyes near the center of the image. See the house with a brown roof near the center of the image, with three windows facing approximately south-southwest and a fourth pointed east? (All directional cues based on north being “up”, not the direction of the sunlight). Look at that specific spot in both images, and the roofs in the buildings immediately adjacent. The difference should jump out at you. In this case, even using TVs that date back to 2015, the 1080p upscale to 4K is better than the 1080p image.

Image by Rtings

If you have an older TV or a budget 4K model, there’s one obvious method of improving your TV’s upscaling: Buy a better television. Unfortunately, it’s impossible to predict how well this will work without knowing exactly what you own now and what you plan to purchase to replace it. The older your current TV, the better the chances that a new set will deliver upgrades in all respects, but many of those improvements may have nothing to do with the way your upscaler handles <4K content.

If you aren’t happy with your current TV, can’t replace it at the moment, and happen to own a high-end Blu-ray or UHD player, you can also try running content through its upscaler rather than relying on the television to handle it. In some cases, a top-end UHD Blu-ray player may deliver a better experience than an entry-level 4K TV from a few years back. If you’re still using a DVD player to feed a picture to a mediocre 1080p or 4K panel when/if you play DVDs, and you can swap over to a high-end Blu-ray / UHD Blu-ray player instead, I’d try it. It may or may not help, but it definitely won’t hurt. What you’re trying to do here is route the signal through the upscaler that’ll give it the best quality kick.

Still need a higher-quality picture? You’re in luck.

Real-Time AI Processing

I haven’t tested the most recent Nvidia Shield myself, but there’s a demo you can actually play with on Nvidia.com to apply the effect the TV offers. Here’s a screenshot of the effect. I’ve positioned the slider over the lizard’s eye because it’s the easiest place to see the upscaler’s impact:

Upscale-Comparison

Left: Regular upscale
Right: Nvidia Shield

Still not clear? Here’s an enlarged version of the same screenshot.

The image on the left is a traditional upscaler, the image on the right is Nvidia’s Shield when asked to scale up 720p or 1080p content to 4K (content below 720p is not supported for upscaling, at least not yet). The AI component of the upscaler obviously improves the overall image quality. In my experience with applications like TVEAI, this is a fair representation of the improvements that can be achieved.

Third-party reviews of the Shield agree. Slashgear writes that when it works, the effect is “fairly astonishing.” So far as I’m aware, the Shield is currently the only set-top or 2D box offering this kind of functionality.

Preprocessing Video

This step is technically optional, because there’s no reason you can’t take a DVD source file and feed it directly through Topaz or a similar product. Often, however, this is not an ideal way to extract maximum quality from your source. It is possible to achieve meaningful improvements in image quality with traditional video editing tools. The first image below is from Deep Space Nine episode 1×03, “Past Prologue.”

Here’s the same image after being processed but before upscaling. I’ve applied denoising filters and a sharpening filter, respectively. There’s a careful balance to be struck with some of these techniques, between blending away features and drawing out detail.

For another example, here’s a pair of shots from Stargate: SG-1.

No pre-processing, no upscaling.

Processed, but not upscaled. Again, there’s a modest-but-detectable sharpness improvement. I haven’t fine-tuned my work on SG-1 the way I have for DS9, so I could likely improve this more with a bit of practice.

The primary purpose of this specific article is to cover upscaling software, but there’s a modest set of improvements you can achieve without using it at all. The workloads don’t take long to run, either. We may dive into that in the future.

Video Upscaling via Third-Party Software

Finally, there’s the option to use a third-party upscaler, like Topaz Video Enhance AI. I’ve made extensive use of TVEAI as part of the Deep Space Nine Upscale Project, and can confirm that the application is capable of yielding stunning results. One major limitation of TVEAI, however, is that it currently only supports Intel and Nvidia platforms for GPU-accelerated processing. I’m testing alpha versions of the software with AMD GPU support already, however, so the feature is coming.

Thanks to AI-based upscaling, historic sci-fi canards like the “Enhance” function are now a reality. It’s a truism in video encoding that no application on Earth can put data back where it never existed, and that’s still true today. The reason we can now “enhance” images to improve their clarity is that AI-based applications are capable of analyzing a video field and estimating what detail would exist if the image were in higher quality already.

One of the typical ways to train a video-enhancing application like TVEAI is to provide the neural net with the same image or video in both high and low quality. The neural net is then tasked with finding a way to make the low-quality source look as much like the high-quality source as possible. Instead of trying to teach a computer what lines and curves look like by providing painstaking examples, we’ve developed the ability to make the computer do the work of teaching itself. That’s why “Enhance” has gone from a complete joke to a plausible reality in a matter of a few years.

Applications like Topaz Video Enhance AI can cost several hundred dollars and they don’t run in real-time — the RTX 2080 appears capable of 90-110 frames per minute when upscaling 640×480 video to 2560×1920. The result of using these applications, however, is a vastly better picture than you’ll see from any other source.

DVD source on the left, upscale on the right.

I suspect we’ll see AI video processing and upscaling become more important over time as Intel, Nvidia, and AMD introduce their next-generation graphics processors. There’s an awful lot of old content orphaned on bad-quality source, and building specialized AI processors to fix it will likely consume some silicon for all of the concerned parties over the next few years.

Here’s the upscaled version of the same screenshots I posted above. First, Dr. Bashir:

There’s a color shift here — I can fix it, but hadn’t done so before taking this screenshot. Upscaled using Gaia-HQ.

In this case, the upscaler has removed the glare of the windshield that’s in the previous DVD shots. This doesn’t particularly bother me — the Goa’uld are shown to have the ability to create forcefields over naked space and I don’t miss the window, but it’s absolutely different than what’s on the DVD and I’d never argue otherwise. It’s possible that a different model would preserve this detail more accurately.

Finally, here’s a version of the credits I’ve uploaded since my “What No Fan Has Seen Before” article last week. This is new footage — I finally squashed the ripple.

Upscalers are amazing and only getting better. That’s true no matter how you consume content or what technology you use to do it. Depending on the shows you like and how much time you want to sink into the project, there are tremendous improvements to be had… or you can just wait a few years, and buy a better TV. You can read more about DS9UP at the links below.

Now Read:



Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)