It’s been nine months since my last Deep Space Nine Upscale Project update, where I am using various AI-based tools to remaster Star Trek: Deep Space Nine. The methods and scripts I am publishing today fix motion problems and visual errors that the original Orinoco and Rio Grande encode methods did not resolve. I’m retiring the 60fps Orinoco variant for now. There no longer seems to be a need for it.
I’ve got new video clips here, but we’ll be looking at a lot of screenshots today as well. Many of the errors I want to talk about are 1-5 frames long, and it doesn’t make sense to build a tiny video clip for every single one when a couple of comparison frames will do.
This project relies upon two applications: AviSynth and Topaz Video Enhance AI. AviSynth is a free video editor you can operate from the command line or through various front ends. Topaz Video Enhance AI is a commercial product that I’ve written about multiple times in the past. TVEAI has been updated several times since my last story. It’s now much faster and supports more models than it did previously.
Kasidy Yates. Defiant Encode with high sharpening filters. Right-click on images and choose “Open in New Tab” for best comparison.
The model recommendations I make today are different than the model recommendations I gave in September because the relative quality of the models has changed. In the past, the “Artemis” model family handled DS9 very poorly. But as of TVEAI 2.2.0, Artemis High Quality is now the best overall choice for upscaling Star Trek: Deep Space Nine. This could change again in the future. I’ll be keeping an eye on it. In many cases, you’ll want to open multiple images in their own windows and swap between the tabs for best comparison.
Here’s the latest version of the DS9 credits, pre-processed and upscaled with the method I’ll report today:
And here, for reference, are the credits I showed off in September. There are motion errors in the September encode that do not exist in the work I’m discussing today. This clip is specifically from the season 6 episode “Sacrifice of Angels.”
Watch the station pan immediately after the blue flash. There are ripples in the video above that the newer method avoids. You can also see the Defiant flying into the wormhole in the new credits; the September encode loses this bit of detail.
The goal of the Deep Space Nine Upscale Project (DS9UP) has always been to provide a straightforward method anyone could follow to improve the quality of Star Trek: Deep Space Nine, provided you own the original DVD set. Last September, I promised to publish the full tutorial on how to perform this task. I ultimately held off on doing so because I wasn’t satisfied with either of the two methods I suggested, code-named Rio Grande (23.976fps) and Orinoco (60fps). I’m far happier with the final product now and more willing to recommend it across the board. I’ve committed to writing that tutorial in the near future. At present, this is a two-step process, with initial processing in AviSynth and upscaling in Topaz Video Enhance AI.
In September, I went with Rio Grande and Orinoco as titles for specific processing methods. Today, I’m introducing another. This time, the quality is high enough to deserve a more appropriate code name: Defiant.
Note: Italicized references to Defiant refer to the in-universe USS Defiant. References to “Defiant” without italics are references to a specific pre-processing and upscaling method. References to Rio Grande and Orinoco refer to now-deprecated pre-processing and upscaling methods.
Defiant: Fewer Flaws, More Teeth
Defiant improves on Rio Grande in a number of ways. It handles the 23.976 –> 29.97 transitions more gracefully than its predecessor and fixes motion in places where Rio Grande broke it. It also corrects output errors that existed in both Orinoco and Rio Grande. Here’s one example:
Rio Grande encode, AHQ via TVEAI 2.2.0.
When upscaled using Rio Grande, the back of the USS Defiant warps when fired upon. The Defiant encode model corrects this problem, improves image quality, and outputs in 2560×1920 as opposed to 2618×1920. Not stretching the image improves overall clarity. These Rio Grande images have been resized to the same aspect ratio to prevent that blur from interfering in comparison shots.
Fixed frame. AHQ via TVEAI 2.2.0, Sharpness=1.0.
Here’s another example. In the Rio Grande-derived image below, there are some cross-connected lines on the front of the saucer section.
Defiant repairs these issues:
Weaker starfields can be slightly boosted with additional noise. TVEAI’s Gaia-CG 1.5.3 handles these scenes better, but it’s not so readily available.
Although the second shot is better than the first, it’s also a stronger sharpening filter. You have to be careful with how much you sharpen DS9. A certain amount improves the show, but too much will create artifacts.
Here’s an example of a frame where Artemis High Quality + Defiant doesn’t yield the best possible image quality. First, here’s Artemis High Quality:
Artemis High Quality.
Next, here’s the same frame upscaled using Topaz’s Strong DeHalo model:
Artemis Strong DeHalo, after sharpening the video within an inch of its life.
Kira looks much better in this frame. It’s not the kind of technique you can count on, however.
Not pictured: Quality.
Defiant is not flawless. There are some blended frames that pop up here and there that I’d like to get rid of. But like its titular namesake, what Defiant brings to the table more than compensates for its weak spots. And given the number of CGI shots in the source DVDs that begin on badly interlaced frames as opposed to progressive ones, the blending I’m talking about is the better solution.
The Strengths and Weaknesses of a ‘Best Fit’ Solution
There is no single script that will treat every episode of Deep Space Nine in an idealized way and Defiant is no exception. The first problem is that the show’s DVD quality is inconsistent. Seasons 1 and 2 are in very poor condition. Season 3 is better, but still not great. Seasons 4-7 improve further with each successive season, but the show’s lighting, color scheme, and image quality still vary a great deal. Many of the special effect shots in Seasons 1-3 are also in bad shape.
Here’s another clip from “Sacrifice of Angels,” upscaled with the latest techniques. A comparison clip from earlier in the project is available here.
There are a few ways you can adjust Defiant’s parameters to compensate for these differences. I’ll discuss them below. I intend to continue poking around to see if I can find further methods of improving the earlier seasons.
I want to clear up a misconception I’ve had a hand in perpetuating. In the past, I theorized that the radical difference in quality between Seasons 1-2 and Seasons 3-7 was due to an equipment change on the Paramount soundstage. This was not the case. The early seasons look bad because the DVD transfer was done poorly. Unfortunately, this limits the amount of improvement I can offer for these episodes.
But just because there’s no single script that will treat every episode ideally doesn’t mean we can’t process the show in a manner that consistently improves it. The processing method I’ve developed for Deep Space Nine is designed to consistently improve the show with a minimum of introduced artifacts or other problems.
Here’s a clip from “In the Pale Moonlight.” The late seasons of DS9 are more amenable to high levels of sharpening than the earlier seasons.
The strength of a best-fit solution like Defiant is that it simplifies the project. The trade-off is that better quality is still achievable if one is willing to process the entire show frame by frame.
Be Aware of Baked-in Source Errors
There are a fair number of errors baked into the original Deep Space Nine DVDs. Early in “Sacrifice of Angels,” during the “Previously, on Star Trek: Deep Space Nine” sequence, there’s a clip of the Federation fleet moving into position. Right at the beginning, the USS Defiant jumps forward for no reason. It looks like the kind you’d see if AviSynth had mistakenly deleted a frame.
That’s not what happens. The error is in the original NTSC (but not PAL) DVD stream. The only way to fix it while holding 23.976fps is to manually create an interpolated frame and then insert it. This is the last shot of the clip before a fade to black, which means there’s a painless opportunity to insert an interpolated frame and then remove a black frame just a few seconds later in order to keep the frame count synchronized with the audio. Since there’s no dialog, a single frame insertion followed by a deletion is not noticed by the audience.
Here’s another example. The image below is from the original DVD with no processing of any kind. The only thing I’ve done is increase the size of the frame up to 2560×1920:
Original frame from DVD source, upscaled to 2560×1920. No other changes.
That black line in the middle of the phaser beam exists in both the original source and in the upscaled variant. This error does exist in the PAL version as well. Here’s the NTSC upscaled frame.
Defiant encode, Artemis High Quality, TVEAI 2.2.0
Here’s yet another example. This shot is from “Way of the Warrior,” the opening episode of the fourth season:
Check the red box. This error goes on for multiple seconds, always in the same place.
I have no idea what that is. I haven’t seen the PAL DVD, but I imagine it is baked in there, too. I can’t fix this at all. The error is huge and it runs on for dozens of frames, moving as the camera moves. There are techniques that can blend frames to fix errors, but they can’t fix this.
I’m not promising that Defiant can’t or won’t introduce errors in some spots, but always check the baseline DVD before you assume the flaw came from the encode. In places where Defiant does introduce flaws, let me know about them. I’ll see if I can find a solution.
Improve Your Upscale Quality With This One Weird Trick
It is still possible to slightly improve Topaz VEAI quality by injecting additional grain and noise, but the technique is not nearly as useful as it once was. I pioneered this idea with Topaz 1.5.3 and Gaia-CG back in September, but Topaz 2.2.0 models do not respond to it as strongly as older models did.
Applying additional noise and grain will still sometimes draw a bit of additional detail out around the eyes and mouth of a character. It can strengthen background starfields slightly, as shown below. Look directly to the left of the USS Defiant‘s nose and you’ll see several stars are brighter in one frame compared with the other.
Defiant encode, Sacrifice of Angels, more grain injected. Look just to the left of the ship’s nose.
Same scene, same encode, but with less grain. While the difference is small, injecting additional noise + grain can give you a small uplift, even in TVEAI 2.2.0 when using Artemis HQ.
Injecting a bit of noise and grain after QTGMC is also helpful because the denoising effect it applies is strong enough that the output can look a bit over-smoothed if you don’t.
The Same Encode Can Yield Very Different Results
Different versions of Topaz have yielded very different results when upscaling the same clip from Deep Space Nine. Here’s the same image, from the same encode. I upscaled it in Gaia-CG using Topaz 1.5.3 on top and Artemis HQ in Topaz 2.2.0 on bottom:
The first image is sharper than the second, even though both are based on the exact same input file. This pays off in some parts of the model. Some lines are much less blurry and hull details are sharper. Unfortunately, there are also places where Gaia-CG has over-sharped the image.
I tuned this iteration of the Defiant encode model to look sharper with Artemis High Quality. It turns out that what sharpens Artemis High Quality can over-sharpens 1.5.3’s version of Gaia-CG. In my opinion, 1.5.3 still produces better battle scene output than what Topaz 2.2.0 generates today, but you have to tune the AviSynth run more conservatively.
The stated goal of the DS9 Upscale Project is to create a simple project a person could follow, and mixing and swapping upscale models on a scene-by-scene basis cuts against that goal. That’s why I’m recommending Artemis HQ overall. I can beat Artemis HQ’s image quality in many cases by using noised, tuned input with Gaia-CG, but I can’t offer the same kind of one-stop solution that AHQ does.
Here’s an example of what noised QTGMC can do from “In the Pale Moonlight” versus Artemis HQ:
Artemis High Quality, TVEAI 2.2.0.
This is solid output. I’ve increased the brightness slightly to make the vertical line error near the dress in-back look more noticeable. But I can beat it if I use a carefully tuned version of Gaia-CG from TVEAI 1.5.3.
Gaia-CG, TVEAI 1.5.3.
Image quality here is very similar, but the color blending in the background is better using the older version of Gaia-CG. By injecting additional noise, I avoid the over-smoothing that makes people dislike this rendering method.
But this is a tiny level of detail repair in exchange for using an older, slower, and more persnickety version of the application. If you’re willing to work frame by frame or scene by scene, it’s a useful tool in your back pocket. If you aren’t, Artemis High Quality and TVEAI 2.2.0 are the best options.
I recommend keeping previous versions of Topaz archived in case you want to test them later.
How to Create Defiant and What to Vary When You Do
As before, I’m going to provide the AviSynth script I’ve used. This time, the tutorial explaining how to use this software if you don’t already know will be forthcoming.
To render out Defiant, use the following script. Note: Our back end either insists on injecting smart quotes into code samples, making it impossible to copy and paste them, or refuses to display the code word-wrapped. So here’s my compromise. I’m going to show the text here, so you can read the entire string. Do not copy from the block of text immediately below. Instead, copy the second block of text — the one you can’t entirely read. Don’t worry about the fact that you can’t see the entire code sample. Start at the top and copy the entire block, plus a little bit extra of the paragraph below it. Paste the entire block into Notepad, remove the extra bits at the end, and you’ll have the entire script in usable format.
TFM(pp=5, mode=2, micmatching=3).TDecimate(hybrid=1)
QTGMC(ShutterBlur=3,ShutterAngleSrc=180,ShutterAngleOut=180,SBlurLimit=8,Preset=”Very Slow”, TR2=3, InputType=1,SourceMatch=3, MatchEnhance=1.0, MatchPreset=”Very Slow”, TR0=2, TR1=2, MatchPreset2=”Very Slow”, sharpness=0.6, SMode=2, Rep0=22, Rep1=22, Rep2=22, RepChroma=True, Sbb=3, SubPel=4, NoiseProcess=1, ChromaNoise=True, GrainRestore=0.5, NoiseRestore=0.25,DenoiseMC=True, NoiseTR=2)
pSharpen(strength=50, threshold=90, ss_x=4.0, ss_y=4.0)
MAA2(mask=1, chroma=true,ss=4, aa=128, aac=128, threads=8, show=0)
LSFmod(defaults=”slow”, strength=50, Smode=5, Smethod=3, kernel=11, preblur=”ON”, secure=true, Szrp=16, Spwr=4, SdmpLo=4, SdmpHi=48, Lmode=4, overshoot=1, undershoot=1, Overshoot2=1, Undershoot2=1, soft=-2, soothe=true, keep=20, edgemode=0, edgemaskHQ=true, ss_x=4.0, ss_y=4.0, dest_x=704, dest_y=480, show=false, screenW=1280, screenH=1024)
Copy from below:
TFM(pp=5, mode=2, micmatching=3).TDecimate(hybrid=1)
QTGMC(ShutterBlur=3,ShutterAngleSrc=180,ShutterAngleOut=180,SBlurLimit=8,Preset="Very Slow", TR2=3, InputType=1,SourceMatch=3, MatchEnhance=1.0, MatchPreset="Very Slow", TR0=2, TR1=2, MatchPreset2="Very Slow", sharpness=0.6, SMode=2, Rep0=22, Rep1=22, Rep2=22, RepChroma=True, Sbb=3, SubPel=4, NoiseProcess=1, ChromaNoise=True, GrainRestore=0.5, NoiseRestore=0.25,DenoiseMC=True, NoiseTR=2)
pSharpen(strength=50, threshold=90, ss_x=4.0, ss_y=4.0)
MAA2(mask=1, chroma=true,ss=4, aa=128, aac=128, threads=8, show=0)
LSFmod(defaults="slow", strength=50, Smode=5, Smethod=3, kernel=11, preblur="ON", secure=true, Szrp=16, Spwr=4, SdmpLo=4, SdmpHi=48, Lmode=4, overshoot=1, undershoot=1, Overshoot2=1, Undershoot2=1, soft=-2, soothe=true, keep=20, edgemode=0, edgemaskHQ=true, ss_x=4.0, ss_y=4.0, dest_x=704, dest_y=480, show=false, screenW=1280, screenH=1024)
If you feel the final output is too sharp, change “Sharpness=0.5” to “Sharpness=0.2” and see if you like the output better. If you think the output isn’t sharp enough, nudge the value upwards. There are scenes of DS9 that benefit from a sharpness as high as 1.0 without erroring, but this much pre-sharpening typically creates haloing on-screen. The default 0.5 recommendation is deliberately conservative.
You can also adjust the sharpness and threshold levels in the other parts of the script if you feel like experimenting. Generally speaking, earlier seasons of DS9 are less willing to tolerate being sharpened than the later ones are.
I recommend using low levels of grain and noise restore (0.25 – 0.5), then adding some additional grain with TVEAI. When you run Topaz VEAI, use Artemis High Quality and inject grain at 2.5 – 2.7, 1.00 sizing.
If you want softer output than AHQ, try Artemis Medium Quality. Gaia-CG does not yield very good results in DS9’s CGI scenes at the moment, unless you hunt down Topaz 1.5.3. Gaia-HQ’s quality is much worse in DS9 than it used to be, so I don’t recommend this method at all.
All of these steps will be covered in more detail in the future. A few more quick tips:
Topaz’ JPG output as of 2.2.0 will result in markedly worse final upscale quality and the media encoder does not always handle audio data well. You are best off using Topaz to output PNGs and then combining the PNGs with a separate audio tack via FFmpeg.
Once you’ve run DGIndex to create a D2V file, right-click on the file, choose “Properties”, and set it to “Read Only.” StaxRip occasionally decides to rewrite D2V files and change their frame rates without prompting. This is obviously a bug, but it’s a consistent issue with certain specific episodes.
You do not need to encode at lossless rates to preserve image quality, apart from ordering Topaz to save to PNG, not JPG. I output my AviSynth encode at CRF 6, “Very Slow” preset. After Topaz finishes its PNG output, I re-encode my files using FFmpeg at CRF 16.
Do Not Try This on Pirated Versions of Deep Space Nine
Warning: My methods require the original DVDs. The DVD needs to be processed using DVD Decrypter and run through DGIndex to create a D2V file.
The Defiant script above will not work on files encoded with Handbrake. It will not work if you used MakeMKV, for reasons I’ll explain. If you apply the scripts I’ve written to files that were not handled in the manner I describe, the motion in your final output will be broken. Image quality will also be lower than if you started with the original DVDs.
I have not created a method of improving pirated or previously ripped versions of DS9 and I do not intend to do so.
If you care enough about DS9 to tackle this project, don’t shoot yourself in the foot. Start from the best source available. The time it takes to rip and pre-process the video is much shorter than the time it takes to upscale 11.9 million frames. I created this project as a way to honor the special effects artists, makeup artists, actors, and other crew that made the show what it is. Buying the DVDs is also the best way to do that. Do not use my work as a justification or excuse for pirating Deep Space Nine.
Why Starting With MKV Files Won’t Work
When MakeMKV is used to encode DS9, the application creates a variable rate encode. Most of any given episode runs at 23.976fps, but there are occasional sections that step up to 29.97fps. AviSynth, unfortunately, does not support variable-rate editing. When you drop your MakeMKV source into an AviSynth front-end like StaxRip or MeGUI, the application will force a 24.66fps frame rate. This de-synchronizes audio playback in a way that can’t be fixed with a static timing shift.
There is no simple way around this problem. It is possible to force audio/video synchronization by using the “ChangeFPS(24000/1001)” command instead of calling TFM and TDecimate, but this will break motion in most of the 29.97fps sections. I didn’t consider this a viable option and I don’t think you should, either.
The DS9 Upscale Project’s Requirements Are Formidable… For Now
I’m not going to sugarcoat it. If you want to upscale the show the way I’m suggesting, you’ll need 300-400GB of storage per episode for your PNG files and double that for two-parters (unless you break them up). You can crunch episodes down to 3GB with H.264 and as little as a third of that with H.265, but the PNG storage requirements are formidable. While you can use an HDD, I consider an SSD a requirement. A powerful CPU isn’t a necessity, but it sure makes things easier.
Depending on the GPU you have, upscaling a single episode can take between 8-16 hours. If you output to PNG, you always have the option to pause the upscale when you want to use your GPU for something else. Keep in mind that frequent reboots are a fact of life if you’re upscaling on the regular. You can also save time by starting with Season 3 or 4 if you want the greatest quality uplift for your time. Seasons 1 and 2 improve, but the degree of improvement is much smaller.
I know this project is on the outer edge of what could be considered plausible for even a dedicated fan. That’s alright. First of all, you don’t have to commit to upscaling the entire show to upscale a few of your favorite episodes, and you don’t have to buy the entire boxed set if you only want to tackle a single season.
But second — and more importantly — the difficulty of upscaling DS9 to acceptable image quality is dropping like a rock. Performance has already roughly doubled on same-generation hardware between 1.01 and 2.2.0. There will be an RTX 4000 and an RTX 5000. At some point, it’s going to become possible to get the kind of improvements I’m delivering in real-time.
That doesn’t mean my ongoing work will be obsolete. It probably means I’ll just have moved on to more advanced techniques. One great rule of thumb in 3D gaming and video both: If you don’t have to worry about running the output in real-time, you can pour a lot more effort into making it look nice. Deep Space Nine needs all the help it can get. Topaz Video AI, thankfully, continues to improve. It’s much faster and more stable now than it was nine months ago.
Anyone who believes they have ideas for how to improve Defiant’s output is welcome to contact me. Thank you to those who already have. This project has generated more email over a longer period of time than any other set of articles I’ve ever written. I believe the Defiant quality model has reached a point to make it worth starting my own encode of the show. Those of you who have followed this project, I hope you feel the same.
May your stem bolts always self-seal,
Hruska out.
- Google Deploys AI to Build Better AI Hardware Accelerators
- Google, Seagate AI Identifies Problem Hard Drives Before They Fail
- New Intel AI Turns GTA V Into a Photorealistic Game
No comments:
Post a Comment