Friday, 9 July 2021

How to Upscale Star Trek: Deep Space Nine With Topaz Video Enhance AI

I’ve written a number of articles about Deep Space Nine and my efforts to improve and upscale it over the past 16 months, but this is the long-promised tutorial — the explanation of how to do what I’ve done, in more detail than I’ve shared it before. If you’ve been waiting for the step-by-step explanation, this is what you’ve been waiting for.

 

Why I Don’t Use Terms Like HD, UHD, or 4K

The reason I don’t call this version of DS9 “DS9 4K” or even “DS9 HD” is because this isn’t HD-quality footage and nothing I’ve learned in the past 16 months can change that. Upscaling can only improve detail that already exists unless you use an approach that seeks to generate new detail — and I haven’t. Native DVD frames of Deep Space Nine are 720×480. That’s 37.5 percent as many pixels as a frame of 720p (1280×720) and 4.1 percent as many pixels as a frame of 4K content. Anyone who promises you a “4K” version of Deep Space Nine has smuggled film out of the studio and acquired some serious VFX chops or is talking up a resolution quality they can’t actually deliver.

Additionally, Deep Space Nine’s seasons are of highly variable quality. Seasons 1 & 2 are much worse than the rest of the show. While these can be improved, they still begin from a much lower baseline and benefit less.

Some of Deep Space Nine’s early special effects, especially external station shots and the credits, are in very poor condition on the DVD. The shot of Jake Sisko fishing on the holodeck in Emissary was done with a green screen whose detail transferred very poorly. I can only assume that whoever created the Season 1 & 2 DVDs really hated DS9’s pre-Dominion storyline because that holodeck background made it to disc with all the clarity of a mid-70s Playboy shoot.

I am willing to tolerate these acknowledged flaws in the DS9 Upscale Project because nobody I know of has had better luck repairing them. Improved color grading would undoubtedly help, but Seasons 1 & 2 look more like VHS tapes than DVD. Let me show you what I mean. I’m going to use an online tool to create easy comparison shots — unfortunately, I can’t embed its output in-page. “Resized” images are from the original DVD, blown up to 2560×1920 to compare against upscaled shots. Resized shots have been put through AviSynth to extract progressive frames but have not been processed with QTGMC.

Upscaled station vs. resized station: This shot shows how marginal some of the improvements are, due to problems with the underlying footage. There’s not enough detail to upscale in the first place.

Upscaled runabout vs. resized runabout: The upscale does nothing to harm the image, but it doesn’t exactly help much, either. Again, this is the result of the base footage, not the upscaler. The improvement is even smaller here.

Season 1 – 3 Credits versus Season 6 credits (resized): This shot isn’t between an upscaled version and a resized image. This shot shows two resized shots from the credits — one from Season 2 and one from Season 6, and it illustrates just how much the Deep Space Nine DVD quality improved over time. The resized Season 6 credits look better than the upscaled Season 3 credits in the shot below.

Season 1-3 Credits versus Season 6 credits (Upscaled): This is a comparison between the upscaled version of the same shot. The quality jump is not small.

But while the DS9 Upscale Project does not represent what Deep Space Nine would look like in native 720p, 1080p, or 4K, it represents the highest overall image quality I have achieved in 16 months of work. It is far ahead of any real-time method of improvement that I’m currently aware of. Achieving this level of quality requires the original DVDs and the same results are unlikely to be achieved if other sources are used. As I’ve said previously, my work is not an excuse or justification for pirating Deep Space Nine.

Because this is a tutorial, we’ll spend less time discussing the overall project and more time on the nuts and bolts of how you can perform this work yourself. From the beginning, I’ve promised that once I figured out a reliable enough method of performing this task to feel as if I could recommend it, I would do so.

This guide explains how to create the highest-quality version of Deep Space Nine I currently know how to create without requiring any frame-by-frame or scene-by-scene editing. This is a one-stop “best fit” line, but it’s a good best-fit line if you’ve got the horsepower for it. Because not everyone does, I’ll be publishing additional tutorials with advice on how to create a still-improved version of the show without requiring the same amount of GPU and CPU horsepower in the future.

The one piece of paid software this job relies on is Topaz Video Enhance AI. When I’ve compared TVEAI in the past to solutions like DVDFab, TVEAI continues to come out well ahead.

Be advised that TVEAI is still a work in progress. I would not recommend it as a paid product if I did not believe it capable, but capable and perfect are not the same thing. The product is young and the field is rapidly advancing.

What You’ll Need (Hardware)

  • At least a quad-core CPU of reasonably current make and model
  • The Deep Space Nine boxed DVD set
  • A DVD-ROM drive
  • ~350GB of storage for intermediate processing per episode (Explained in more detail below)
  • A reasonably new GPU
  • ~8GB of RAM

The hardware requirements are vague because they represent more of a soft floor than a hard cutoff. A Ryzen 7 4800U mobile processor is capable of running the AviSynth script I’ll recommend in 3 hours, 45 minutes, and upscales one frame of video every 1.08 seconds or so on its integrated GPU. An RTX 2080 in Topaz VEAI 2.3.0 runs more like 0.4 seconds per frame. That’s roughly 19.5 hours of upscaling per episode for the 4800U and 7-8 hours for an RTX 2080.

There are several reasons why I recommend outputting your video as a series of PNG files rather than video. First, it allows you to stop and restart the encode. If you can afford to leave your system encoding overnight but need it during the day, you can use it to do this work in the off-hours, without needing to worry about losing a video file that was only half complete.

The second reason we use PNG files is that Topaz’s JPG output sometimes has visual errors that meaningfully affect final video quality. Storing an entire episode of DS9 in PNG files can require up to 280GB of storage and double that if you want to do a two-parter in one go. An SSD is not required, but I strongly recommend one.

Finally, TVEAI sometimes fails to preserve audio properly, requiring you to patch it in manually regardless.

350GB is actually a bit more storage than you should need for just the PNGs for an episode. 350GB should be enough for the VOB files, PNG files, AviSynth .h264 or .h265 intermediate files and the final output without requiring you to delete anything partway through the process to free space.

This tutorial will not work properly if you attempt it using MKV files or pirated files. The process is tuned to work exactly as written.

What You’ll Need (Software)

The only paid application on this list is Topaz Video Enhance AI, which is available for a free trial.

DVD Decrypter

Install DVD Decrypter and choose the region that corresponds to the disc set you’ve purchased.

Navigate to the “Mode” menu and click on “IFO.” This should give you a screen like the below:

Pick whichever episode you want to work on. DVD Decrypter will create a new directory for each DVD, but you’ll need to manually name the subdirectories if you want them to reflect the title of the episode in each. PGC 1 is the first episode on the disc, PGC 2 is the second, etc. You can choose to handle the entire DVD in one contiguous file, but the next step is somewhat simpler if you separate files here.

It typically takes 7-10 minutes for DVD Decrypter to do its work. Once you’ve finished with DVD Decrypter, you can close it. Closing the app sometimes leads it to throw an error (at least, it does on my own machine). You can kill it from Task Manager without harm if this occurs.

Output: VOB files, copied to your hard drive in a directory and a subdirectory organization of your choice.

DGIndex

DVD Decrypter created a file on your local hard drive at “C:\DS9S1D1” or “DS9S4D3” depending on which disc you started with. Navigate to the episode you want to work with and you’ll find a pair of files with names like “VTS_02_1.vob.” Launch DGIndex and navigate to “Open”, under “File.” Choose both VOB files from the appropriate directory and load them in order:

When finished, hit “Ok,” re-open the “File” menu, and choose “Save Project.” DGIndex will scan through your video file and complete its work. The end result of this will be a .D2V file.

Note: StaxRip sometimes likes to edit D2V files. In instances where a Deep Space Nine episode has a very high percentage of film (upwards of 95 percent), StaxRip will decide that the proper frame rate for your input video is supposed to be 23.976fps, not 29.97fps. As a result, running the script I recommend against an episode like “Emissary” will produce ~19fps video.

To avoid this problem, right-click on the D2V file you just created and choose “Properties.” Select “Read-only” and hit “OK.” StaxRip will yell at you about this problem when you load the file, but it won’t fail. Once you’ve set your project file to Read-only, we’re done with DGIndex.

Output: One .D2V file, to be kept in the same location as the .VOB files from the previous step.

StaxRip

Once you’ve created and locked your D2V file, launch StaxRip and load your D2V file by dragging and dropping it into the application. I’m working with the DS9 episode “Indiscretion” below:

There’s a lot going on here, but you don’t have to care about most of it. First, double-click on the “Mpeg2Source” window. This will open a sub-window where you can enter script parameters.

Unfortunately, I still don’t have a way to print the script I recommend easily. You can copy it below. Don’t worry about the fact that it isn’t completely visible; it will copy correctly if you grab a bit of text from the paragraph below, then delete it.

TFM(pp=5, mode=2, micmatching=3).TDecimate(hybrid=1)
QTGMC(ShutterBlur=3,ShutterAngleSrc=180,ShutterAngleOut=180,SBlurLimit=8,Preset="Very Slow", TR2=3, InputType=1,SourceMatch=3, MatchEnhance=1.0, MatchPreset="Very Slow", TR0=2, TR1=2, MatchPreset2="Very Slow", sharpness=0.6, SMode=2, Rep0=22, Rep1=22, Rep2=22, RepChroma=True, Sbb=3, SubPel=4, NoiseProcess=1, ChromaNoise=True, GrainRestore=0.5, NoiseRestore=0.25,DenoiseMC=True, NoiseTR=2)
pSharpen(strength=50, threshold=90, ss_x=4.0, ss_y=4.0)
MAA2(mask=1, chroma=true,ss=4, aa=128, aac=128, threads=8, show=0)
LSFmod(defaults="slow", strength=50, Smode=5, Smethod=3, kernel=11, preblur="ON", secure=true, Szrp=16, Spwr=4, SdmpLo=4, SdmpHi=48, Lmode=4, overshoot=1, undershoot=1, Overshoot2=1, Undershoot2=1, soft=-2, soothe=true, keep=20, edgemode=0, edgemaskHQ=true, ss_x=4.0, ss_y=4.0, dest_x=704, dest_y=480, show=false, screenW=1280, screenH=1024)

One way to improve your performance is to add the line “Prefetch(x)” to the bottom of the script, where “x” stands for the maximum number of threads you want to run. Sane values are 2-8. Lower values will leave you more CPU performance for other things. If you intend to use your PC while encoding, set this value to ~50 percent of your cores or less. Be aware that values larger than 8 are generally more likely to cause instability than to improve performance. If I wanted to encode two files simultaneously while also using a 16-core PC, I’d launch both StaxRip runs with a value of Prefetch(4) and leave myself eight threads for other tasks.

To control your output formula, look to the right-hand side of the image. Click on the “x265” text, and a menu will open, offering you various other codecs and output options.

If you want to adjust those options to fine-tune them, clock on the “Options” button. This will open an additional sub-menu:

x264 options. I recommend “Very Slow” and a quality level of “6”, though the latter is a bit arbitrary.

If you want to change your final container options, click on the blue text labeled “MKV.” This will give you an array of choices. Personally, MKV files work fine for me. StaxRip detects both audio tracks and preserves both through the encode process, so you don’t need to change these settings unless you specifically want to encode into different formats. Click “Next” and then hit “OK” to start the process.

Pro Tip: If you are experimenting with script settings, don’t render an entire episode to do it. You can stop StaxRip at any time by hitting the “Abort” button. Once the application returns to the main page, hit “Next” again. The application will tell you that a Video and Audio encoding already exists and ask if you want to re-use the video you already rendered or re-encode it. Select “Re-use.”

This will create an MKV file of whatever partial episode you rendered out for test purposes. You can plug this into Topaz VEAI and check the output. Just remember to set the final rendering frame manually. Otherwise, you’ll render a ton of blank, black frames. They might upscale beautifully, but you’ll never be able to tell.

Output: One MKV file, assuming you are fine with StaxRip’s default options. If not, one MP4, MOV, or another output file of your choice and configuration. DS9 files encoded with these presets should be between 3GB and 5GB depending on how much noise you inject. Injecting noise will make your final output larger. This is your pre-processed, pre-upscale version of Deep Space Nine.

Topaz Video Enhance AI 2.3.0

Once you’ve started TVEAI, navigate to the directory where your MKV output from the previous step is saved. Open the file. You can use the { and } options to select the section of the video you wish to render. If you wish to upscale the entire file, you do not need to bother with these settings. Switch the “AI Model Selector” section at the top of the right-hand side of the application to “All” from “Suggested.” Choose “Artemis High-Quality v11” and “400%” below that.

Scroll down and choose your grain settings. I recommend between 2.4 – 2.7. Above 3.0 gets pretty heavy. I recommend that you output to PNG files. TVEAI can output video files in ProRes at 4:2:2, but the final output files are quite large for video files and you’ll wind up compressing them again or taking them apart in FFmpeg and rebuilding them with less obnoxious size requirements anyway. Set TVEAI for PNG output and you can stop the upscaler at will, use your GPU for other things, and then fire it up again when convenient.

Output: A series of PNG files, typically created in a subdirectory of wherever your MKV file is stashed.

FFmpeg

FFmpeg is a command-line utility you can use to take your PNG files and wrap them back up into a video file. Once you’ve downloaded FFmpeg, you have two options. You can add it to your Windows PATH (good idea, best security practice) or you can do what I did, and dump the executables directly into your C:\Windows\System32 folder. There are all sorts of good reasons not to do this related to security and best practices and not encouraging end-users to muck around in folders they ought not to touch, so I’ll leave it up to you which method you prefer.

It’s not as bad as it might look, I promise.

Once TVEAI is finished with its work, visit the original subdirectory where you dropped the VOB files and from which you presumably ran StaxRip. You should find some audio files with the extension *.ac3. Feel free to rename them to something more plausible, like “Indiscretion-2ch.ac3” and “Indiscretion-5ch.ac3.” The file with a larger Kbps number is your 5.1 audio, the smaller number is your 2-channel audio. You can pick which audio you want to encode as your primary channel depending on your preference on this topic.

Copy both of these files into the same directory where your PNG files are stored. An upscaled episode of DS9 contains ~65,143 PNG files.

Open a command-line prompt by hitting the Windows key and typing CMD.

Open the subdirectory where your PNG files and audio files are stored. Click in the address bar and hit Ctrl-C to copy the address.

Return to your open command prompt. Type “cd” and hit the space bar once, and then paste the address you just copied. If you are working off a different drive (D instead of C), you will need to type “d:\” and hit enter before beginning this sequence of commands. If you use default names for your files and subdirectories, your final output will be in a directory named something like: “C:\DS9S6D2\Sacrifice_4.00x_2560x1920_ahq-11_png”

Once you’ve navigated to the appropriate directory, modify the following command to fit your own requirements. This command can be copied directly off the web page because there are no smart quotes involved (smart quote formatting is what wrecks AviSynth scripts in our backend). In the commands below, replace “Indiscretion-2ch.ac3” or “Indiscretion.mkv” with the file names of your own episodes and their associated audio tracks.

ffmpeg -r 23.97602 -f image2 -s 2560×1920 -i %06d.png -i Indiscretion-2ch.ac3 -vcodec libx264 -crf 16 -pix_fmt yuv420p -acodec copy Indiscretion.mkv

The name of the .ac3 file should be replaced with the name of the audio file you want to use. If you want to keep both audio tracks, we split the process up into steps. There’s undoubtedly a way to do this in a single ingestion, but I haven’t quite nailed the syntax for that yet. Start with:

ffmpeg -r 23.97602 -f image2 -s 2560×1920 -i %06d.png -vcodec libx264 -crf 16 -pix_fmt yuv420p -acodec copy Indiscretion.mkv

This will compile your frames into an MKV file, but you won’t have any audio. When the video file has finished compiling itself, type:

ffmpeg -i Indiscretion.mkv -i Indiscretion-2ch.ac3 -i Indiscretion-5ch.ac3 -map 0 -map 1 -map 2 IndiscretionCombined.mkv

This will compile your final file and output with multiple audio files. If you are upscaling at 200 percent instead of 400 percent, make the appropriate resolution changes to the “-s 2560×1920” variable. If you want to use H.265 instead of H.264, replace “libx264” with “libx265.” Lower or higher compression factors can be set by adjusting the crf number.

Bonus Round: Subtitles

If you want subtitles, you can have them. There’s an additional step here involving a program called CCExtractor and an application called MKVToolnix.

To use CCExtractor, run the application and drag both of your VOB files into it, as shown below:

Navigate to the far right-hand side of the tab list. You can’t actually see the “Execute” tab you’re looking for by default and there’s no way to widen the application enough to include it by default. Ah well.

Now you have a .SRT file and can install MKVToolNix. MKVToolNix will finish this entire procedure up for us.

Drag your completed MKV file sans subtitles and your subtitles file into the application. MKVToolNix will ask you what kind of file you want to create. Choose to add a new source file to the current multiplex settings, as shown below.

Choose these options and select your final file name. “Start Multiplexing.” Rejoice! You have finished an episode while retaining both audio tracks and subtitles.

How to Adjust StaxRip Image Quality

The only way to get exactly the output I’ve shown throughout this process is to follow my exact steps. But that doesn’t mean there isn’t room to experiment. If you feel as though the final output is inappropriately sharpened (or not sharp enough), adjust the Sharpness setting in the QTGMC script. You can also experiment with removing the “psharpen” and “MSAA” calls altogether. I would strip LSFMod out last — there’s resize code embedded in it that outputs in 704×480, resulting in a proper final file size of 2560×1920 as opposed to the over-stretched 2618×1920.

If you feel as though your final output is too plastic-looking, there are several ways to adjust for this. Decrease sharpness in QTGMC downwards to 0.2 and change the “strength” variable of LSFMod downwards. You can also try injecting more grain and noise in the final output. If you choose to adjust NoiseRestore above 0.5, remove the “ChromaNoise=True” line from the QTGMC script. If you don’t, the output will turn slightly greenish. The effect doesn’t get very noticeable until above 0.75, but there’s no reason to leave it there in the first place.

Increasing noise output in QTGMC may slightly improve overall image quality in Artemis-HQ, but the effect is not as large as it was in previous versions of Gaia-CG. It can still improve background starfields, however.

No Longer Left Behind

This article will be updated as my understanding of best practices continues to expand. Readers have requested that I take a look at Voyager, to see how best to adapt Deep Space Nine’s processing methods to that TV show. In the future, I’ll be swapping the Defiant for the Delta Quadrant. I’ve tossed a few screenshots into a slideshow below, for those curious to see how Voyager holds up under the same processing methods, though I’ve already experimented with the best way to tweak them. VOY was mastered with higher-quality footage than most of DS9 and the difference is obvious when you put them side by side. Consider this a preview:

 

Am I “done” with Deep Space Nine?

Yes — and no.

I’m still considering color grading, but DaVinci Studio Resolve can’t edit or create MKV files, and the approach laid out above is based on them. Regrading color is also a new field for me and I can’t make any promises about how quickly I’ll be able to tackle it. Color grading wasn’t contemplated in the original project as I created it, and I do not wish to delay this tutorial even longer while I learn.

In the beginning, I set out to create a method of upscaling Deep Space Nine that would do the show justice compared to Netflix or even what you could buy on DVD. I have no doubt that the quality of the show will improve as tools such as Topaz Video Enhance AI improve. Nevertheless, I have reached a point where I feel comfortable recommending this process to people.

One day, footage of the quality I’ve shown throughout this project will be available in real-time. By the time it is, I’ll have built something better through the use of non-real-time rendering. I spent 20 years waiting for ViacomCBS to tackle this project and I don’t intend to take my eye off it until we either have a native 4K version of the show build directly by ViacomCBS or third-party AI tools have advanced to the point to render such a version redundant.

But the goal of the Deep Space Nine Upscale Project was never to wait until 2030 when the perfect tools become available. I created this project once it became clear that AI tools and free utilities like AviSynth could be used to create a version of Deep Space Nine that would be sufficiently better than baseline to be worth the effort of creating in the first place. This tutorial was always the story I wanted to write. Now, I’ve written it.

Different people will reach different conclusions on whether the show deserves this kind of effort. That’s okay. Every time I publish one of these articles, someone different pops out of the woodwork to say how the Netflix version or the baseline DVDs are fine. I’m glad people feel that way. As someone who loves Deep Space Nine, I want people to enjoy the show. If you’re happy with the version on TV without spending a few hundred (or thousand) hours improving it, I’m glad for you.

I wasn’t.

Now, nobody else has to be, either.

If you think you can beat my quality, do it. Let me know where the quality improvements come from and I’ll include them here and credit you.

I still hope ViacomCBS remasters Deep Space Nine. I don’t think it will surprise anyone at this point to say I’d love to be part of such an effort. I’ve been asked how I’d feel if a remaster was announced next week and my honest answer would be “thrilled.” I believe if such a thing happens, it’ll be because my own project and others like it collectively illustrated to Paramount that there was an appetite for a better version of the show. I’m also willing to bet it’d sell better than TNG did at its Blu-ray debut if Paramount either 1). Committed to doing both VOY and DS9, 2). Released both shows on disc and on ParamountPlus, and 3). Didn’t try to sell it for $80 a season.

If you love Deep Space Nine enough to even contemplate this project, you’re the person I had in mind when I pitched this story to my EiC, Jamie Lendino. It is not linear. The things we love rarely are. My most heartfelt thank-you to the multiple individuals who worked on Deep Space Nine’s VFX team and have contacted me throughout this process to offer insight and support.

Other stories and assignments may take me to the depths of the Delta Quadrant or bury me in CPUs but I will stand with you again, here, in this place where I belong.

Hruska out.

Now Read:



Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)