In Depth: Upscaling Video
If you ever need to work with old video footage that’s in the standard format, here are several options and tools to make the job smoother and get better results.
Table of contents
Introduction
When I started my career in video post-production, “standard definition” was, in fact, the standard. Now there are so many resolutions. I originally wrote this article about 7 years ago, when a cinematographer friend of mine asked me to upscale footage for his reel from the1990s to HD. In that article I focusing upscaling 720×480 (480i) into 1920×1080 (1080p) footage.
Video standards have changed in just a few short years. 4K Ultra High Definition (4K UHD) is now a standard feature in over 60% of the televisions sold in North America, and that number is increasing rapidly over the last few months as people can’t go out for entertainment during the pandemic. For example, if you’re watching an old standard-definition television show on a 4K UHD screen, it will look a bit stretched and blurry.
But, upscaling is not just for old footage. Some people are shooting at 1080p and upscaling their renders to 4K for YouTube. Why would they do that?
- They may not have a camera that is capable of shooting 4K.
- Editing 4K footage can be slow.
- 4K footage takes up a ton of space on your drives!
Luckily, upscaling technology has seen major improvements over the past few years, especially in the area of Artificial Intelligence (AI). We’ll dive into AI and I’ll also discuss upscaling video in Adobe products, Resolve, with AI, and using a few plug-ins with upscaling technology.
Please note: This article is not a review of products, but more of a guide to the upscaling options available.
What is Upscaling?
Upscaling means stretching a smaller video to fit a larger screen. The goal is to increase the size of the video while maintaining quality. Other terms like upsizing and uprezzing are also commonly used and refer to the same task. Upscaling can be done by just scaling the footage, where you will often get stretched and blurry results. It can be also done with software and plugins that sharpen and clean up blur, jagginess, color artifacting, and noise. This article covers some of the software available.
Back in the day, when I worked at Postworks, I was always making graphics for Avid Media Composer or Symphony and I was always converting square pixels to rectangular D1/DV. I had a system down. If you’ve dug up older footage, there’s a good chance that it is set to D1/DV, so if you are looking at it without Pixel Aspect Ratio (PAR) Correction turned on, your footage will likely look squished. If PAR Correction is ON, you may not realize it’s squished so just be aware of this issue.
Pixel Aspect Ratio resources
- Pixel Aspect Ratio, Part 1 – Fitting rectangular pixels into square holes (pdf) by Chris & Trish Meyer, CyberMotion
- Pixel Aspect Ratio, Part 2 – Keeping your work in perspective &-; as well as in the right aspect ratio (pdf) – by Chris & Trish Meyer, CyberMotion
- Exporting Video for YouTube: Pixel Aspect Ratio Basics
- Pixel Aspect Ratio Calculator – There are a bunch of these on the web so if you don’t like this one, just Google: Pixel Aspect Ratio Calculator
Artificial Intelligence (AI) in Upscaling
There have been major leaps in AI technology over the past few years, and it’s pretty exciting.
These videos below are from 1911 and 1896. They are blown up to 4K, 60fps. The enlargement was done with Gigapixel AI from Topaz Labs. The algorithm for adding frames is called Depth-Aware Video Frame Interpolation (DAIN), powered by neural networks.
A Trip Through New York City in 1911
“Arrival of a Train at La Ciotat” (The Lumiere Brothers, 1896)
Deep Space 9 Upscale Project
There’s a project out there called Deep Space Nine Upscale Project (DS9UP) to upscale Star Trek: Deep Space 9 from the DVD source to 4K. Joel Hruska writes about his experiences using Topaz Video Enhance AI, DaVinci Resolve, and AviSynth to uprez the show. The project is pretty interesting because he talks about his challenges and what he’s learned during the project. Take a look at the comparisons below.
The NVIDIA SHIELD TV brings real-time AI upscaling right to the television.
Upscaling in your NLE/Host
Most host applications and NLEs now have built-in tools to make upscaled footage look better. We’ll cover a few of the more popular tools below.
Upscaling with Adobe Tools
Adobe has a feature called Detail-preserving upscale. Below, tutorials for After Effects, Premiere Pro, and Photoshop.
How to Upscale Video (with the best results) in After Effects
In this livestream from Adobe, Jason Levine shows the preferred method for upscaling video content in After Effects. To elaborate, he uses an effect called Detail Preserving Upscale. It’s similar to ‘preserve details’ in Photoshop. In fact, using this process instead of simply scaling the layer in a comp, gives better-looking results. In addition, there’s more control over detail and noise.
UPSCALING VIDEO: How to upscale 1080p to 4k | NO BLUR, PIXELATION, NOISE (2020)
Themika Thennakoon explains how to upscale using Adobe Media Encoder 2020.
Upscaling in Premiere Pro
How to Upscale Video to 4K or Downscale to 1080P Easily – This tutorial by Farrah Penn explains it pretty clearly.
Upscaling in DaVinci Resolve
Upscale 1080p Footage to 4k with Davinci Resolve Superscale!!
Jay Lippman has become my go-to guy for Resolve tutorials. In this video, he uses DaVinci Resolve’s Superscale.
Turn 1080p Footage Into 4K & Beyond! DaVinci Resolve SUPERSCALE Tutorial
Tim Crowe gives this quick tutorial on Superscale.
Upscaling with Plug-ins
The major benefit of upscaling with plug-ins is the ease of use. There are several plugins out there that can upscale your footage.
Reaching Hearing Impaired Viewers
15% of American adults, aged 18 and over, have trouble hearing. In addition, one in eight people in the United States (13% or 30 million) aged 12 years or older has hearing loss in both ears, based on standard hearing examinations.[1]
Reach people who watch with no sound
A staggering 85% of Facebook videos are viewed without sound![6] Viewers are often watching in a quiet environment, where they don’t want to disturb others, for example, in bed or in the office. Others are in a noisy environment where they can’t hear the sound.
Improve Grasp of Content
It is helpful for people with certain learning disabilities to read captions. Non-native English speakers often watch videos with sound and English subtitles because it helps them to absorb the content better, especially if the actors have thick accents.
Attract foreign-language speakers
Not everyone speaks English. English currently has 1.132 million speakers. Mandarin Chinese has 1.117 million speakers.[2] For many, it’s easier to read than it is to listen. In addition, many people use subtitles for language learning.
Increase viewer engagement
Facebook for Business shared that “Internal tests show that captioned video ads increase video view time by an average of 12%, so we’re soon releasing a new tool that makes adding captions to video ads easier than ever.”[3]
Improve SEO on search results
YouTube will suggest sections of your video based on Google searches when subtitles are used. This can help with bringing viewers. Based on your captions, Google can list video content, complete with timestamps, in Google searches.
Upscaling with Boris Continuum Complete BCC UpRez
BCC UpRez is included in both Boris Continuum Complete and Boris Continuum Image Restoration Unit. The following hosts are supported:
- Adobe After Effects
- Adobe Premiere Pro
- Avid Media Composer
- Avid DS
- Apple Final Cut Pro 7
- Sony Vegas Pro
- Boris RED 5
Boris Continuum Complete UpRez does a decent job with upscaling video and keeping video looking sharp with its Magic Sharp and Magic Smooth settings. Instead of applying the filter to the footage, make a comp sized Solid layer and set the Source Layer, which should be below the Solid in the Timeline. Set the source Pixel Aspect Ratio (in this case, D1/DV NTSC (0.91). I chose a custom scale to match the other samples and set Quality to Magic Sharp. Of the upscale plug-ins for After Effects, my clear choice is Boris Continuum BCC UpRez. It does the nicest job keeping footage crisp and downplaying artifacting.
Here’s the interface for BCC UpRez
Here are the Frame Size options.
More on BCC
- Buy Boris Continuum BCC Image Restoration unit which contains BCC Uprez. $299 at Toolfarm
- Buy Boris Continuum Complete for After Effects and Premiere Pro
- For other hosts, click here
- Download a free 14 day trial of Boris Continuum Complete – not watermarked!
Upscaling with Red Giant Magic Bullet Instant HD
With Red Giant Magic Bullet Instant HD, drag the footage into a comp that is the destination size, in this case, HDTV 1080p. The effect is applied directly to the footage and there is a long list of Output Size preset to choose from.
Here are the Output Settings options.
Instant HD has parameters for adjusting Filter Type, Sharpness, Quality, and Antialiasing. I’ve had mixed results with Instant HD. Sometimes the Instant HD uprezzed footage looks a lot better and other times no better than the After Effects scaled footage. I used the Output Size preset “Fill 1920×1080 (2133×1422) DV NTSC”.
Red Giant Instant HD ReSizer was acquired from Digital Anarchy and instead of scrapping it or working into Instant HD, it is bundled as a separate plug-in with Instant HD, giving the user another option.
ReSizer also comes with a few presets but has options to manually size the video. The user can set the Aspect Ratio, method of Filling the Frame, Cropping options, and Field Order. There are of course Quality settings as well. Under Advanced the Smoothness settings will remove blur artifacting. With a low setting it works well but with higher values will blur footage. Sharpness will obviously sharpen the footage.
Instant HD and ReSizer, support the following hosts:
- Adobe After Effects CS6, CS5.5, CS5, CS4, CS3
- Adobe Premiere Pro CS6, CS5.5, CS5, CS4, CS3*
- Apple Final Cut Pro 7, 6, 5.1.4
*Instant HD for Adobe Premiere Pro CS4 and CS3 runs on Windows OS only.
Noise Industries FxFactory Pro Upscale
With Noise Industries FxFactory Pro Upscale, the effect is applied directly to the source footage. Instead of it upscaling the footage to the size of the composition, it allows the user to choose a portion of the video and scale it to the size of the video frame only. This works well if your footage is the same size as the final output but closer shots are needed. Since it doesn’t really serve the same purpose as the others, it was not included in the comparison. With Upscale, the Height and Width should be equal or the video will be stretched. Also if Edge Smoothing is not set to low, the video will artifact.
FxFactory Upscale asks the user to set the region to upscale. This region will fill the original video frame.
Upscaling Video in Photoshop
I really hadn’t thought of upscaling video in Photoshop, but the Adobe help file on Scale or flip a layer for After Effects had a great tip. I had this article almost finished and decided to add this Photoshop section. Here’s the tip:
Adobe Photoshop provides fine control over resampling methods used for scaling of images. For fine control of resampling, you can export frames to Photoshop to change the image size and then import the frames back into After Effects.
Though it’s not very well suited for movies, the content-aware scaling feature in Photoshop is very useful for extending and scaling still images. This feature can be useful when repurposing images for wide-screen formats that were created for standard-definition formats.
I had to try it! It was not so straight forward. This is a quick rundown of what I had to deal with. I ended up creating a tutorial, below that.
Pixel Aspect Ratio – The footage was NTSC D1/DV (0.91 aspect ratio) and the footage had to be upscaled to NTSC DV 1920×1080 (square pixels).
Actions and Scripts
The footage was interlaced and Photoshop brings in each frame as a layer when using the video import function. Because Photoshop brings in each frame as a layer, to effect each layer, each layer must be selected one by one and the filter applied &-; one by one. Tedious to say the least. I created Actions for deinterlacing the video and another for sharpening and removing noise. I created scripts that pointed to these Actions. I saved the scripts in the Scripts folder for Applications > After Effects > Presets > Scripts and then restarted After Effects so that the scripts would show up under the File menu.
Script for deinterlacing
The Set of Actions is named VideoActions and the Action I created is named remfields. This script was applied before the footage was scaled. You never want to scale interlaced footage.
for(var a = 0; a < doc.layers.length; a++) {
doc.activeLayer = doc.layers[a];
app.doAction(“remfields”, “VideoActions”);
}
Script for Unsharp Mask and Remove Noise
I did this with a single action and this was applied after the footage was upscaled to it’s final size.
var doc = activeDocument;
for(var a = 0; a < doc.layers.length; a++) {
doc.activeLayer = doc.layers[a];
app.doAction(“UnsharpMask”, “VideoActions”);
}
Issues you may run into with older footage
Older SD formats don’t have the quality of more modern larger formats, as many old formats are captured from tape. A lot of older footage has also been compressed into legacy video formats or use old codecs, which can create problems. You may not even have access to the source footage anymore. Older video is likely to be interlaced, which may just be another step to deal with but not affect the footage.
Interlaced video
No matter which method you use to upscale your footage if you are using interlaced footage, make sure that you removing interlacing first!!
This is what your fields will look like if you don’t remove interlacing before scaling. It’s not pretty.
The information below is an excerpt from my book Plug-in to After Effects: Third Party Plug-in Mastery (Out of Print) and is pulled from Chapter 15, Reference and Workflow.
Interlaced video is comprised of two separate fields that make a single frame of video. These fields are alternating lines of the image that create each frame, or each field makes up half the resolution of the frame. Fields were introduced to smooth out motion and reduce flicker that was introduced by the slow rate of the broadcast signal because early televisions didn’t handle content with a lot of movement too well. The frame rate in NTSC television is 30 frames per second (29.97, actually), making the field rate 60 interlaced fields per second (technically 59.94).
What do the p and i stand for?
If you see the term 1080i or 1080p, it means HDTV that is interlaced (with interlaced fields) or progressive (no interlaced fields). The i stands for interlaced and p is for progressive.
In After Effects, the process of removing interlacing is known as separate fields. Newer cameras and software will embed information into the video and After Effects will automatically separate fields for you. Most of the time, After Effects built-in Interpolation Rules do a good job of auto-detecting fields by referencing at the frame size and sometimes codec.
To separate fields manually in After Effects, select your footage in the Project panel. Click the Interpret Footage button at the bottom left of the Project panel or go to File > Interpret Footage > Main. Under Fields and Pulldown, select Lower Field First in the Separate pop-up if you’re using for NTSC D1. To improve the look of the video, check Preserve Edges, which by default is unchecked.
RE:Vision Effects FieldsKit’s Effect Controls set at the defaults.
RE:Vision Effects FieldsKit provides many more field separation and pulldown options than After Effects built-in effects. FieldsKit comes with three plug-ins: Deinterlacer, Reinterlacer and Pulldown. Using proprietary field reconstruction and adaptive motion techniques, FieldsKit Deinterlacer can construct nicer looking frames and better edges by using surrounding pixel information. FieldsKit Deinterlacer used in combination with RE:Vision Effects Twixtor, can mimicking the look of 24fps film with video footage.
Red Giant Magic Bullet Frames Effect Controls panel.
Another way to give your interlaced footage the look of 24 fps progressive film is Red Giant Magic Bullet Frames. Magic Bullet Frames is a fast de-interlacing filter that uses a motion-adaptive algorithm to smooth video. This 24p conversion includes a feature to make sure that clips stay the same duration when they are converted from 29.97i frames to 24 frames during the pull-down process, so no clip trimming is required. Magic Bullet Frames includes six plug-ins: Broadcast Spec, Deartifacter, Frames, Frames Plus, Letterboxer and Opticals. The Deartifacter plug-in can remove compression from DV footage.
More Information
- To better understand working with fields and 3:2 pulldown, Chris Meyer, co-author of the Creating Motion Graphics with After Effects books, has online training modules on Lynda.com titled Understanding Fields and Interlacing in After Effects and Working with 3:2 Pulldown in After Effects. Non-subscribers can get a free 7-day pass by going to www.lynda.com/go/chrisandtrish.
- RE:Vision Effects FieldsKit for After Effects Compatible – MSRP: $89.95, Toolfarm Price: $85.45. Save 5% at Toolfarm Everyday!
- RE:Vision Effects FieldsKit for Final Cut Pro & Express – – MSRP: $89.95, Toolfarm Price: $85.45. Save 5% at Toolfarm Everyday!
- Red Giant Magic Bullet Frames – MSRP: $199, Toolfarm Price: $179.10. Save 10% at Toolfarm Every Day!
Pixel Aspect Ratio
Back in the day, when I worked at Postworks, I was always making graphics for Avid Media Composer or Symphony and I was always converting square pixels to rectangular D1/DV. I had a system down. If you’ve dug up older footage, there’s a good chance that it is set to D1/DV, so if you are looking at it without Pixel Aspect Ratio (PAR) Correction turned on, your footage will likely look squished. If PAR Correction is ON, you may not realize it’s squished so just be aware of this issue.
Pixel Aspect Ratio resources
- Pixel Aspect Ratio, Part 1 – Fitting rectangular pixels into square holes (pdf) by Chris & Trish Meyer, CyberMotion
- Pixel Aspect Ratio, Part 2 – Keeping your work in perspective &-; as well as in the right aspect ratio (pdf) – by Chris & Trish Meyer, CyberMotion
- Exporting Video for YouTube: Pixel Aspect Ratio Basics
- Pixel Aspect Ratio Calculator – There are a bunch of these on the web so if you don’t like this one, just Google: Pixel Aspect Ratio Calculator
Conclusion
Upscaling video is certainly possible but you won’t always end up with stellar results. The more you are upscaling, the worse your results will be so don’t upscale if you don’t have to! You do have options. If one doesn’t give you the results you want, try another. Good luck!
More Info on Upscaling
- Video resize methods comparison – The info here is a bit old but maybe worth a look. It’s pretty technical.
Note: This article was originally published in 2013 and has been updated to add current software.
Posted by Michele Yamazaki
