The following unhinged rant article is the sole opinion of its author. ExtremeTech does not discriminate against variable frame rate media.
In my previous two Star Trek: Deep Space Nine articles, I’ve discussed my efforts to upscale the content, provided some video and image samples, and discussed the process of slowly attempting to remaster the show using commercially available software. I have continued to make real progress on the show in terms of overall image quality, to the point that I’m turning my attention to fixing some other problems I haven’t been sure how to deal with.
Specifically: The frame rate.
In Which My Barely Video-Literate and Underslept Self Attempts to Explain the Mechanics of Hell
Star Trek: The Next Generation, Deep Space Nine, and Voyager weren’t recorded in anything as sensible as a steady 23.976fps. When these shows aired in the 1980s and 1990s, they were constructed from at least two different types of content: 23.976fps progressive content and 29.97fps interlaced content. Episodes of these shows do not run at one frame rate; they run at different frame rates depending on what’s happening on-screen. A show that runs at one frame rate, start to finish, uses Constant Frame Rate encoding (CFR). Deep Space Nine, TNG, and VOY all use VFR. Stargate SG-1 and Babylon 5 do the same thing. Most of the best nerd content of the late 1990s and early 2000s is locked behind media that go for this kind of manipulation.
One of the things I haven’t talked about, but have been privately confused by, is why so many of the filters I deployed in an attempt to fix DS9’s frame rate weren’t producing the results I wanted. When I reported in my last article that DS9 used 3:2 pulldown, in which three progressive frames of data are followed by two interlaced frames, I was right — and I was wrong.
The Deep Space Nine episode “Sacrifice of Angels” absolutely contains video footage that was mastered with 3:2 pulldown applied. It is not made of this footage. It contains interlaced frames that were there when the videos were transferred to DVD in the first place. It isn’t made of this footage, either. No, Deep Space Nine, or at least “Sacrifice of Angels,” is mostly 23.976fps (which looks fine), but with enough 29.97fps footage spliced into it to make it jarring.
Part of the problem is that interlacing and 3:2 pulldown are a lot more visible on modern LCDs than they were on older CRTs. Issues like this have cropped up in gaming as well. If you look back at reviews of these DVDs, they were lauded for their quality when the show came out in the early 2000s — and probably reviewed on CRT monitors. This, at least, is what a long-suffering video editor friend told me when I pestered him to answer my questions for three months straight a few hours.
Part of the problem is that TVEAI does a really good job enhancing half-field interlaced frames when they appear on-screen. They’re far more visible when upscaled than at native resolution.
This is what a telecined or interlaced frame looks like. You can clearly see that it’s just half a field of visual information, not the full frame.
Here’s another issue: A lot of video editing applications, including AviSynth and TVEAI itself, are none too fond of VFR content.
When I feed a MakeMKV-derived source file directly into Topaz Video Enhance AI, it detects more than 80,000 frames in an episode that ought to contain around 65,000 frames. TVEAI exposes no encoding options to the end-user, and the default output from content scaled in this mode runs at 29.97fps. If you were the kind of kid who enjoyed playing 33 1/2 RPM records at 45 RPM, you’ll be very excited by this result. If you weren’t, Captain Benjamin “Alvin” Sisko is not an improvement on the original version. (Assuming, at least, that I was nuts enough to pitch-shift the audio to match it).
If you don’t want to run content at 29.97fps, there’s another option — re-encode it at 23.976fps. Convert the interlaced frames, lower the frame rate, and you’ll end up with imperfect, highly noticeable jerks and jumps during fast-paced space combat scenes — which are exactly the ones we are trying to preserve. This is known as judder. Judder sucks.
Topaz Video Enhance AI is not the only application that gets confused by VFR content. I’ve been using StaxRip as a front-end for AviSynth+ and have been mystified as to why the program constantly detects a frame rate of 24.66fps when loaded with MakeMKV source. It turns out that 24.66fps is the average frame rate over the entire episode if you average together the relatively small number of 29.97fps scenes with the much larger number of 23.976fps scenes.
So. The goal is to duplicate the smoothness of the 29.97fps content in the stream without judder. I’ve actually made some real progress towards that goal — enough to say I feel like I’m nearing the end of the first stage of this process. (Color grading is going to be the second stage, oh joy of my heart.)
Where I’m running into trouble, however, is figuring out to remux the audio. I haven’t yet figured out how to synchronize audio to video playback at new frame rates, and the judder management isn’t perfect yet. This (near) final video is from several days ago. I’ve actually improved further on this since I ran the encode. And since if you’re reading this, you’re probably wanting to see the content, so I’ve got an interim version of the credits I’m happy to share. You’ll have to ignore the misaligned audio and you need to set YouTube to 4K for best image quality, even if you don’t have a 4K monitor.
Let’s compare this new video with the video I created back in February. There are a few things I want to specifically call out. Both videos were upscaled using Topaz Video Enhance AI, but Topaz has updated the application in the intervening period. This does not seem to have had a large impact on image quality, but may have tweaked it in some ways. The top video is April, the bottom video is from February. Run both of them in 4K mode or as close to it as you can get.
Speed: The old video runs at 49.32fps, while the new one is encoded at 24.66fps. I’m still grappling with StaxRip / AviSynth+ to get this problem out of the video, but 24.66fps is close enough to see what it’s going to look like, where 49.32fps looks artificially smooth to eyeballs used to slower video refresh.
Blur: The February video is over-sharp when it passes through the comet, while this April video is too blurry. I’ve already cleaned this up substantially since I uploaded the video a couple of days ago, so no worries on that score.
Aliasing: The February edition of DS9 is heavily aliased in certain places, like when the runabouts approach the logo in the first part of the credits. The new version shows far less aliasing. With that said, there are certain shots, like the Defiant’s final approach into the wormhole, where the ship is badly aliased no matter what I do. Haven’t figured a solution for that yet.
Shimmer: The February video makes it look as if DS9 (the station, not the show) is crawling with ants in certain places. The reason the April video looks a bit blurred is that my efforts to lock that problem down were a little too enthusiastically applied. Watch the station pans to see the difference here.
Judder: My new, enthusiastic, and least-wanted best friend has been substantially improved, but you can see some clear instances of judder in the April video, including when the runabouts pass in front of the camera and the Defiant passes the outer docking pylon. I’ve already got an encode that improves sharpness and the Defiant’s passage while yanking the frame rate down to 23.976fps rather than this dorky 24.66 faux-hybrid thing.
Overall, you can see the rate of progression from February to April in these two videos. February represents the beginning of my work, when I was just running an application through TVEAI to upscale. The April video represents the application of additional filters.
What Comes Next?
I figure it’s time I lay out a formal roadmap for where I’m going with this thing.
Video Quality: After three months of working on DS9 and literally hundreds of video encodes, I am hopeful that I’m nearing the end of the road, as far as video work. Judder is the last major problem to solve, to the extent that judder can be “solved” in the first place. I have attempted some rather … whacky experimentation to solve this problem. I launched 17 different encodes of the MakeMKV source file last night. The first 12 had serious names that reflected the settings I picked for them. #13 – #17 were named “F***It.mkv.” I’ll go downstairs later on and check the results before repairing to AviSynth documentation to attempt more testing.
I’d like to know more about antialiasing scripts than I do (most of them reference anime, not video content) and I need to run some tests to confirm that DAA is actually still getting me a visual improvement. I’d love to find a better AA solution to clean up a few niggling issues.
Color Reprocessing: I have decided to separate out any work I do on DS9’s color from the work I’m doing on the video. I haven’t dropped this aspect of the project, but I’m not picking it back up until I have the visual processing nailed down. Not everyone is going to want color tweaking in the first place, after all. Users should note that Topaz Video Enhance AI introduces some color shifting when it upscales. In the initial version of the application, this applied a subtle lightening effect in HQ-CG mode, as shown below:
It’s subtle, but you can see that the image on the right is slightly lighter than the one on the left.
Topaz recently updated the application to a new version that now appears to have a different impact on color, but I wanted to show the original. I’m keeping both the original and newer version of the application handy to cross-compare video output.
Audio: HELP WANTED: INQUIRE WITHIN.
More seriously, I expect to nail down the audio issues, too. I just can’t remember how I solved the problem last time.
Tools Used: Right now, MakeMKV, StaxRip 2.0.8 with AviSynth+, and Topaz Video Enhance AI itself. The front-end is a useful way to code AviSynth scripts. The goal is to create this project with as much free software as possible. Still evaluating a run through Handbrake, if only to try syncing the audio/video that way.
Once I’ve hammered the workflow into shape, I’m going to publish a step-by-step tutorial on how to do this on ExtremeTech. Honestly, it doesn’t take that much in terms of filters — well, not yet. I haven’t shared my specific settings yet because I’m still trying to figure out which variables and filters are the right ones to use. I don’t want to publish pieces of a guide that then become separated from the final product.
Finally: A great many people have reached out to me following the publication of my second article with encouragement, well wishes, and technical assistance. A longer list of thank-yous will accompany the final version of my article, once I’ve checked in with people and figured out how they want to be acknowledged. The help I have received from some of you has been vital to pulling off this project. Anyone with ideas on how to improve image quality or solve the judder problem is welcome to get in touch.
Once Deep Space Nine is complete, I’m moving on to Buffy the Vampire Slayer, another beloved show with an execrable 16:9 “conversion” that very few people seem to like. I’ll be using the 4:3 version of the show to see what kind of improvements I can bring to the Scooby Gang. We’ll also check in with the Blu-ray version of Firefly to see how Topaz Video Enhance AI handles upscaling already-HD content.
Now Read:
- Deep Space Nine Upscale Project Update: ‘Sacrifice of Angels’
- Upscaling Star Trek: Deep Space Nine Using Topaz Video Enhance AI
- Star Trek: Voyager Gets 4K Upscale Remaster via AI
No comments:
Post a Comment