In Depth: Upscaling or Upsizing Video

Introduction

When I started my career in video post-production, standard definition was the standard. Now there are so many resolutions. A friend of mine who is a cinematographer has some really great spots that he worked on in the 1990s and would like them to be upscaled HD for YouTube and Vimeo. This article is for you, Peter!

I’m focusing on turning 720×480 (480i) into 1920×1080 (1080p) footage in this article. I’ll discuss scaling the video in After Effects, Photoshop, and using a plug-in with upscaling technology.

Upscaling video is a pretty easy task however making it look good is a big problem. Older SD formats don’t have the quality of more modern larger formats, as many are captured from tape. A lot of older footage has also been compressed into legacy video formats or use old codecs, which can create problems. You may not even have access to the source footage anymore. Old video is likely to be interlaced, which may just be another step to deal with but not affect the footage.

After going through all of these options available for upscaling, I can ultimately surmise that there is really no method that will magically turn your SD footage to amazing looking HD. Not a single one looked great. This is not a review of products, but more of a guide to the options available I’ll focus on how to minimize problems and tools for the job. Here are the main problems you’ll likely encounter after upscaling from SD to HD, such as:

• Softening video
• Bad color artifacts
• Jaggy video edges, especially after sharpening
• Noise

You will have much better results converting 720p to 1080p with any method. I chose to do a larger upscale amount because this is what Peter was asking about.

Interlaced Video

No matter which method you use to upscale your footage, make sure that you removing interlacing first!!

This is what your fields will look like if you don’t remove interlacing before scaling. It’s not pretty.

The information below is an excerpt from my book Plug-in to After Effects: Third Party Plug-in Mastery, and is pulled from Chapter 15, Reference and Workflow.

Interlaced video is comprised of two separate fields that make a single frame of video. These fields are alternating lines of image that create each frame, or each field makes up half the resolution of the frame. Fields were introduced to smooth out motion and reduce flicker that was introduced by the slow rate of the broadcast signal, because early televisions didn’t handle content with a lot of movement too well. The frame rate in NTSC television is 30 frames per second (29.97, actually), making the field rate 60 interlaced fields per second (technically 59.94).

In After Effects, the process of removing interlacing is known as separating fields. Newer cameras and software will embed information into the video and After Effects will automatically separate fields for you. Most of the time, After Effects built-in Interpolation Rules do a good job of auto detecting fields by referencing at the frame size and sometimes codec.

To separate fields manually in After Effects, select your footage in the Project panel. Click the Interpret Footage button at the bottom left of the Project panel or go to File > Interpret Footage > Main. Under Fields and Pulldown, select Lower Field First in the Separate pop-up if you’re using for NTSC D1. To improve the look of the video, check Preserve Edges, which by default is unchecked.

RE:Vision Effects FieldsKit’s Effect Controls set at the defaults.

RE:Vision Effects FieldsKit provides many more field separation and pulldown options than After Effects built-in effects. FieldsKit comes with three plug-ins: Deinterlacer, Reinterlacer and Pulldown. Using proprietary field reconstruction and adaptive motion techniques, FieldsKit Deinterlacer can construct nicer looking frames and better edges by using surrounding pixel information. FieldsKit Deinterlacer used in combination with RE:Vision Effects Twixtor, can mimicking the look of 24fps film with video footage.

Red Giant Magic Bullet Frames Effect Controls panel.

Another way to give your interlaced footage the look of 24 fps progressive film is Red Giant Magic Bullet Frames. Magic Bullet Frames is a fast de-interlacing filter that uses a motion-adaptive algorithm to smooth video. This 24p conversion includes a feature to make sure that clips stay the same duration when they are converted from 29.97i frames to 24 frames during the pull-down process, so no clip trimming is required. Magic Bullet Frames includes six plug-ins: Broadcast Spec, Deartifacter, Frames, Frames Plus, Letterboxer and Opticals. The Deartifacter plug-in can remove compression from DV footage.

More Information

• To better understand working with fields and 3:2 pulldown, Chris Meyer, co-author of the Creating Motion Graphics with After Effects books, has online training modules on Lynda.com titled Understanding Fields and Interlacing in After Effects and Working with 3:2 Pulldown in After Effects. Non-subscribers can get a free 7-day pass by going to www.lynda.com/go/chrisandtrish.
• RE:Vision Effects FieldsKit for After Effects Compatible – MSRP: $89.95, Toolfarm Price: $85.45. Save 5% at Toolfarm Everyday!
• RE:Vision Effects FieldsKit for Final Cut Pro & Express – – MSRP: $89.95, Toolfarm Price: $85.45. Save 5% at Toolfarm Everyday!
• Red Giant Magic Bullet Frames – MSRP: $199, Toolfarm Price: $179.10. Save 10% at Toolfarm Every Day!

Pixel Aspect Ratio

Back in the day, when I worked at Postworks, I was always making graphics for Avid Media Composer or Symphony and I was always converting square pixels to rectangular D1/DV. I had a system down. If you’ve dug up older footage, there’s a good chance that it is set to D1/DV, so if you are looking at it without Pixel Aspect Ratio (PAR) Correction turned on, your footage will likely look squished. If PAR Correction is ON, you may not realize it’s squished so just be aware of this issue.

Pixel Aspect Ratio resources

• Pixel Aspect Ratio, Part 1 – Fitting rectangular pixels into square holes (pdf) by Chris & Trish Meyer, CyberMotion
• Pixel Aspect Ratio, Part 2 – Keeping your work in perspective &-; as well as in the right aspect ratio (pdf) – by Chris & Trish Meyer, CyberMotion
• Exporting Video for YouTube: Pixel Aspect Ratio Basics
• Pixel Aspect Ratio Calculator – There are a bunch of these on the web so if you don’t like this one, just Google: Pixel Aspect Ratio Calculator

Examples and Video Comparison

Here are a couple of examples of upscaling video in After Effects by using scale, Boris Continuum Complete BCC UpRez and Red Giant Magic Bullet Instant HD, and even some Photoshop.

Upscaling in Adobe Premiere Pro

I read online that Premiere Pro’s algorithms are better for upscaling video than After Effects, but After Effects CC (v12.0) adds bicubic sampling, which improves upscaling in After Effects.

The upscaling process goes something like this in Premiere Pro:

Create a new Sequence at the HD setting.

In the Project panel, right click the SD footage to bring up the flyout menu. Choose Modify > Interpret footage. This is where you will deinterlace the footage.

Under Pixel Aspect Ratio, choose Conform to: Square Pixels (1.0)

Bring in your SD footage and place it into the sequence. You will get a Clip Mismatch Warning. Click the Keep existing settings button.

Scale the footage. I scaled mine to 267.0 which filled the frame. I also applied unsharp mask because the footage was getting soft.

Here it is before and after:

Here is a tutorial I found online that has tinny audio but the tutorial covers everything about upscaling in Premiere Pro.

Upscaling in Adobe After Effects

This is about as simple as it gets.

1. First, take care of any interlaced video.
2. Create a new comp that is the final size that you need your video. For the sake of this article, everything will be NTSC HDTV 1920×1080.
3. Drag your SD footage into an HD comp in After Effects.
4. Select the footage layer in the Timeline and go to Layer > Transform > Fit to Comp, Fit to Comp Height or Fit to Comp Width. This will depend on your settings and if your video was 720×480 D1/DV or 640×480.
5. Reposition video as needed. You may need to keyframe the Anchor Point to keep your elements centered.

Fit to Comp Width in After Effects.

Todd Kopriva of Adobe sent me a note saying “Also worth noting is that After Effects CC (12.0) adds bicubic sampling, which makes upscaling in After Effects much better.”

More info

• Scale or flip a layer from After Effects / Layer properties in Adobe Help
• Buy Adobe After Effects CS6
• Adobe Creative Cloud for Teams & Businesses – Price Drop! First Year Pricing at $39.99 a Month Through August 31, 2013

Upscaling with Plug-ins

The major benefit of upscaling with plug-ins is the ease of use. Both plug-ins in this section, Boris Continuum Complete UpRez and Red Giant Magic Bullet Instant HD, are so easy to use and very fast. With both, there is no need to worry about Pixel Aspect Ratio. This function is built in to both.

Upscaling with Boris Continuum Complete BCC UpRez

BCC UpRez is included in both Boris Continuum Complete and Boris Continuum Image Restoration Unit. The following hosts are supported:

• Adobe After Effects
• Adobe Premiere Pro
• Avid Media Composer
• Avid DS
• Apple Final Cut Pro 7
• Sony Vegas Pro
• Boris RED 5

Boris Continuum Complete UpRez does a decent job with upscaling video and keeping video looking sharp with its Magic Sharp and Magic Smooth settings. Instead of applying the filter to the footage, make a comp sized Solid layer and set the Source Layer, which should be the below the Solid in the Timeline. Set the source Pixel Aspect Ratio (in this case, D1/DV NTSC (0.91). I chose a custom scale to match the other samples and set Quality to Magic Sharp.

Of the upscale plug-ins for After Effects, my clear choice is Boris Continuum BCC UpRez. It does the nicest job keeping footage crisp and downplaying artifacting.

Here’s the interface for BCC UpRez

Here are the Frame Size options.

More on BCC

• Buy Boris Continuum BCC Image Restoration unit which contains BCC Uprez. $299 at Toolfarm
• Buy Boris Continuum Complete for After Effects and Premiere Pro
• For other hosts, click here
• Download a free 14 day trial of Boris Continuum Complete – not watermarked!

Upscaling with Red Giant Magic Bullet Instant HD

With Red Giant Magic Bullet Instant HD, drag the footage into a comp that is the destination size, in this case, HDTV 1080p. The effect is applied directly to the footage and there is a long list of Output Size preset to choose from.

Here are the Output Settings options.

Instant HD has parameters for adjusting Filter Type, Sharpness, Quality and Antialiasing. I’ve had mixed results with Instant HD. Sometimes the Instant HD uprezzed footage looks a lot better and other times no better than the After Effects scaled footage. I used the Output Size preset “Fill 1920×1080 (2133×1422) DV NTSC”.

Red Giant Instant HD ReSizer was acquired from Digital Anarchy and instead of scrapping it or working into Instant HD, it is bundled as a separate plug-in with Instant HD, giving the user another option.

ReSizer also comes with a few presets but has options to manually size the video. The user can set the Aspect Ratio, method of Filling the Frame, Cropping options and Field Order. There are of course Quality settings as well. Under Advanced the Smoothness settings will remove blur artifacting. With a low setting it works well but with higher values will blur footage. Sharpness will obviously sharpen the footage.

Instant HD and ReSizer, support the following hosts:

• Adobe After Effects CS6, CS5.5, CS5, CS4, CS3
• Adobe Premiere Pro CS6, CS5.5, CS5, CS4, CS3*
• Apple Final Cut Pro 7, 6, 5.1.4

*Instant HD for Adobe Premiere Pro CS4 and CS3 runs on Windows OS only.

More on Red Giant Magic Bullet Instant HD

• Buy Red Giant Magic Bullet Instant HD Only $89.10 (MSRP: $99)! Save 10% Everyday at Toolfarm.
• Buy Red Giant Magic Bullet Suite – $719.10 (MSRP: $799.00). Save 10% Everyday at Toolfarm.
• Download a demo of Magic Bullet Suite

Noise Industries FxFactory Pro Upscale

With Noise Industries FxFactory Pro Upscale, the effect is applied directly to the source footage. Instead of it upscaling the footage to the size of the composition, it allows the user to choose a portion of the video and scale it to the size of video frame only. This works well if your footage is the same size as the final output but closer shots are needed. Since it doesn’t’t really serve the same purpose as the others, it was not included in the comparison. With Upscale, the Height and Width should be equal or the video will be stretched. Also if Edge Smoothing is not set low, the video will artifact.

FxFactory Upscale asks the user to set the region to upscale. This region will fill the original video frame.

Upscaling Video in Photoshop

I really hadn’t thought of upscaling video in Photoshop, but the Adobe help file on Scale or flip a layer for After Effects had a great tip. I had this article almost finished and decided to add this Photoshop section. Here’s the tip:

Adobe Photoshop provides fine control over resampling methods used for scaling of images. For fine control of resampling, you can export frames to Photoshop to change the image size and then import the frames back into After Effects.

Though it’s not very well suited for movies, the content-aware scaling feature in Photoshop is very useful for extending and scaling still images. This feature can be useful when repurposing images for wide-screen formats that were created for standard-definition formats.

I had to try it! It was not so straight forward. This is a quick rundown of what I had to deal with. I ended up creating a tutorial, below that.

Pixel Aspect Ratio – The footage was NTSC D1/DV (0.91 aspect ratio) and the footage had to be upscaled to NTSC DV 1920×1080 (square pixels).

Actions and Scripts

The footage was interlaced and Photoshop brings in each frame as a layer when using the video import function. Because Photoshop brings in each frame as a layer, to effect each layer, each layer must be selected one by one and the filter applied &-; one by one. Tedious to say the least. I created Actions for deinterlacing the video and another for sharpening and removing noise. I created scripts that pointed to these Actions. I saved the scripts in the Scripts folder for Applications > After Effects > Presets > Scripts and then restarted After Effects so that the scripts would show up under the File menu.

Script for deinterlacing

The Set of Actions is named VideoActions and the Action I created is named remfields. This script was applied before the footage was scaled. You never want to scale interlaced footage.

Script for Unsharp Mask and Remove Noise

I did this with a single action and this was applied after the footage was upscaled to it’s final size.

Tutorial: How to Upscale Video in Photoshop

When I de-interlaced and upscaled entirely in Photoshop, the footage did seem to be slightly sharper. I may be delusional. The process was a pain in the butt and I would rather have a slightly softer image than mess with doing it in Photoshop. I’d recommend this process for short clips but not with longer pieces of footage. This process is also good if there’s something you need to do in Photoshop and can’t do in After Effects or Premiere. The quality is not THAT much better for all of the hassle it entails. You would not need the scripts if you don’t need to de-interlace or apply effects to each layer, so that would speed up the process.

Synthetik Studio Artist

Studio Artist 4 is a digital art, image and video processing software, that provides graphic synthesizer editing, intelligent visual processing and smart automatic drawing. Being an After Effects artist, Studio Artist is VERY different from what I’m used to.

Users can use the DeInterlace image operation effect to just grab the odd or even fields.

Because this is not software I am used to, I contacted John Dalton of Synthetik with some questions. He responded very quickly, “We don’t provide any specific aspect ratio correction controls for the situation you mention. For most generic paint animation effects this really isn’t an issue. You can use our interactive warp effects to do aspect ratio adjustment.

He included a link to “an interview with David Kaplan, who was the director for Year of the Fish, which is a feature length Studio Artist generated animated film that won an award at Sundance. discusses in the interview how he worked with Studio Artist to adjust the aspect ratio of standard-def 4:3 video to HD in 16:9. David also used Supersizer for the film, and discusses that in the interview as well.”

More info

• Buy Synthetik Studio Artist 
• Studio Artist tutorial on Supersizer and Resizing the canvas.

Legacy Products

A couple that are no longer available but could be useful if you own them:

• Topaz Enhance
• Deemon SuperResolution

Conclusion

Upscaling video is certainly possible but you won’t always end up with stellar results. The more you are upscaling, the worse your results will be so don’t upscale if you don’t have to! You do have options. If one doesn’t give you the results you want, try another. Good luck!

More Info on Upscaling

• Video resize methods comparison – The info here is a bit old but may be worth a look. It’s pretty technical.