It’s been several months since I wrote an update on my ongoing efforts to restore Deep Space Nine. I took a break from the project through much of June due to a move and an associated injury but jumped back into it in July and have been at work steadily since then. The majority have my time has been focused on understanding how shifting the episode into various alternative frame rates would impact motion smoothness and image quality.
In the past, I’ve written updates when I hit specific milestones I’d set for myself or discovered something I thought was interesting. This is more of a progress report. So, to begin: A bit of recap: I’m a lifelong Deep Space Nine fan who started this project in January and has pursued it since. I’ve been learning about video processing and encoding from scratch as I’ve worked, and according to everyone I’ve talked to, I didn’t pick a beginner-level project.
Deep Space Nine is a VFR (Variable Frame Rate) show, which means the DVD alternates between playing back at 23.976 fps and 29.97 fps at various points within the episode. This is a common format for late-1990s science fiction. Shows encoded in this fashion include Babylon 5, Stargate SG-1, Buffy the Vampire Slayer, Star Trek: The Next Generation (DVD-only), and Star Trek: Voyager.
The episode of Deep Space Nine I chose to treat as my test vehicle, “Sacrifice of Angels,” is about 14 percent 29.97fps footage and ~86 percent 23.976fps footage. The problem is, applications like AviSynth cannot edit VFR video and must convert it to CFR (Constant Frame Rate). Applications like DaVinci Studio Resolve can technically “handle” VFR files, in that it will ingest them properly, but the resulting output periodically pauses in a way I couldn’t find a clean solution for. For now, unless I figure that out, processing the show requires that it be converted to CFR as an initial step.
If you encode a VFR show at 23.976 CFR, the 29.97fps content will be cut to 23.976fps and the playback may not be perfectly smooth. In some cases, you won’t see any stutter because there’s not enough motion on screen for the frame decimation to be visible. There’s a several-minute block of 29.97fps content in “Sacrifice of Angels” when Dukat, Dumar, Weyoun, and the female Changeling are all talking at Ops. While there are a few telltale signs, you only really see it when Dukat walks around the table — and this is after both postprocessing and upscale.
The reason that Dukat’s hand and body are blurred as he moves is that, if you go frame-by-frame, what he’s doing looks like this:
I boosted the brightness a bit here, to make the shadow easier to see. Most of the frame looks normal, but you can see where Dukat’s hand is going to be in the next frame. The error is visible but small and confined to one part of the screen.
The fact that a lot of TNG-era Star Trek is conversation makes the frame rate shifts that much easier to deal with, but it’s still noticeable as heck when it happens. My goal has been to find an automated method of processing DS9 that would typically produce better motion during 29.97fps content sections. I spent the last few months playing with various methods of converting the show’s frame rate to see what the options would look like.
Source Sensitivity
The transmutative property of mathematics states that when you multiply two numbers together, it doesn’t matter what order you write the numbers in. 1 * 2 * 3 * 4 = 24. So does 4 * 3 * 2 * 1. Video processing is not transmutative. The order in which you apply filters changes what the final output will be. Video processing workflows need to be duplicated exactly in order to guarantee accurate results, up to and potentially including using the exact same application and filter versions.
There are a few reasons I’ve been exploring the outcomes for Handbrake and MakeMKV as opposed to using DVD Decrypter to create a VOB copy of the DVD data in 59.94 interlaced format.
First and foremost, I’ve yet to figure out how to get the video output quality to look anywhere near as good as what I’ve achieved with HB/MMKV without creating scripts for each episode. In point of fact, I haven’t completely figured out the episode scripting, either. This is what I get for taking my last programming course circa “I Want It That Way.”
While we’re being honest, I want it this way, but we don’t get everything we want, do we ViacomCBS? Image from the “What We Left Behind” DS9 documentary, showing what a remastered Defiant would look like.
My own best. Spoiler: The professional one is better.
Second, Handbrake offered some really simple options to batch up and test a huge range of file encode presets. In mid-July, I ripped “Sacrifice of Angels” more than 250 separate times in Handbrake in order to examine the impact of various quality control settings, H.264 flags, frame rates, and deinterlacing options. Third, I finally figured out how to hand StaxRip a set of flags that would synchronize the audio/video playback of a VFR MakeMKV file, and I wanted to experiment with it. Finally, part of learning something is figuring out what not to do. I make a lot of mistakes and I make some of them on purpose, just to see how various ideas change the final output.
I have spent a great deal of time during the last two months playing with various methods of changing frame rates. AviSynth has a number of filters for changing the frame rate and different source filters yield subtly different outputs. I’ve experimented with various methods of interpolating up to 119.88fps before trimming back down — either to a compromise frame rate like 59.97 or back to 23.976. I’ve done a lot of testing combining a pass through Davinci Studio Resolve through further processing with AviSynth, or before AviSynth, or after. I’ve experimented with various H.264 quality levels and specific presets to look at the impact these would have on the areas of troublesome motion in the show. To be honest, I worked out a strategy for what I wanted to encode and allowed the encoding to race ahead of my actual evaluations. I’m still evaluating what I’ve created. If any of these methods had yielded a single clear winner, I’d have said so, but I’ve certainly seen some intriguing differences among the data. I’ve even played with some of the AI-based methods of interpolation to see how they’d compare.
Separately from this, I’ve experimented with deinterlacing based on 59.97 VOB files. Even with script help from some of the community at Doom 9, I haven’t found a single, broad, fire-and-forget solution that gave me as clear an image quality as what I’ve gotten from MakeMKV and Handbrake. Part of the reason I chose to stick with these sources when evaluating motion is that I knew I’d already achieved something reasonably close to what I’d consider final quality. I wanted to hold that set of variables constant and experiment with the methods I’d already worked with, especially when I had trouble achieving the same image quality. Still hoping to find one, but that’s why I chose to focus my time where I did.
The Pros and Cons of 119.88fps
One way of solving the 23.976fps and 29.97fps playback problem is to shift content up to 119.88fps. The problem with 119.88 — well, one of them, because there’s not just a problem — is that you’ve definitionally quintupled your workload. If it takes 15 wall-clock hours of mixed CPU and GPU processing time to upscale an episode of 23.976 DS9, it’ll take ~75 hours for 119.88.
That’s not great. And to add insult to injury, you need a 120Hz display to watch the output without dropping half the frames.
I’m still messing around with 119.88, because so far I’ve gotten the best overall results in those troublesome patches at this frame rate, but it’s hard to imagine attempting to do the show this way. Ampere would have to be more than 2x faster than the GTX 1080 Ti to make the GPU processing times anything near reasonable.
Alternatively, one can attempt a frame rate between 23.976 and 119.88, and I’ve been doing some experimenting there as well. These frame rates all require either the film or video portion of the material to shift playback speed by a non-integer multiplier, which means there’s always some degree of detectable something. What varies is just what that something is, and how often it pops up. I’ve also tested the outcomes if you upscale the video first, then process it. The end results are pretty good, but the clock time penalty for processing 2560×1920 clips versus 720×480 clips is larger than the resolution increase alone would suggest.
Where This Is Going
My plan is to assemble a set of options that make some reasonable tradeoffs as far as motion smoothness versus processing time versus frame rate, with at least two and possibly three targets. I’ve also been experimenting with masking and antialiasing lately, including using a version of an episode with fewer aliasing problems as an external antialiasing guide for a version of the same episode optimized for smoother motion. And it works!
…ish.
One of the things I’ve learned is that when searching for a best-fit line that will safely adjust a television show, you may be very lucky to find a single method that works for an episode. Asking for a method that globally works well for 176 episodes is asking a lot.
Most of the time, what you get is…ish, and some things are a heck of a lot ‘ishier’ than others. The external clip concept is interesting, but after playing around with it for a little while I’ve got my doubts about whether it can work. There are scenes that it transforms as perfectly as I could ask for — and scenes that, uh, don’t.
This is what “Don’t” looks like.
I suppose a pertinent question to y’all would be: How much of what doesn’t work are you curious to see in the first place? I haven’t posted or talked much about failed experiments to date, and the reason this story doesn’t have more video is that I’m not sure what people would find interesting in the first place. It doesn’t seem all that interesting to just talk about what doesn’t work. If you find this sort of work-in-progress more interesting, or if you’d find it more interesting if I gave you more to look at, say so.
Now Read:
- Deep Space Nine Upscale Project Season Finale: What We’ve Brought Ahead
- Deep Space Nine Upscale Project (DS9UP): Technical Goals and FAQ
- Upscaling Star Trek: Deep Space Nine Using Topaz Video Enhance AI
No comments:
Post a Comment