In the days of CRT monitors, TV shows used to be broadcast in interlaced mode, which is unsupported by modern flat-panel displays. All online streaming services and video stores provide progressive video, so they must deinterlace the data first. This article compares the deinterlacing strategies of Apple iTunes, Netflix, Microsoft Zune, Amazon VoD and Hulu by comparing their respective encodings of a Futurama episode.
If you have dealt with video formats before, you probably know about interlacing, a 1930s trick to achieve both high spatial and temporal resolution at half the (analog) data rate: In NTSC countries, there are 60 fields per second (PAL: 50), and every field is half the vertical resolution of a full frame. When film footage at 24 frames per second has to be played at 30 fps (NTSC), every frame has to be shown 1.25 times – in other words, every fourth frame has to be shown twice. This introduces jerky motion (judder), but it can be improved by using the 60 Hz temporal resolution: Frame A gets shown for 2 fields, frame B for 3 fields, frame C for 2 fields, and so on. This way, every source frame gets shown for 2.5 fields, i.e. 1.25 frames – this method is called a telecine 2:3 pulldown.
A lot of TV material is produced at 24 fps and telecined, for several reasons: Standard movie cameras can be used instead of TV cameras, 24 fps can be converted to 25 fps PAL more easily than 30 fps NTSC, and for cartoons, this means that only 24 (or 12) frames have to be drawn for every second.
Unfortunately, interlacing only works with ancient CRT TVs – modern LCD screens can only show progressive video. And while DVDs are specified to encode interlaced video, more modern formats like MPEG-4/H.264 and VC-1 usually carry progressive data. So when playing DVDs, the DVD player or the TV have to deal with the interlacing problem, and in case of modern file formats, it’s the job of the converter/encoder.
The naive way of converting an interlaced source to progressive is to combine every two fields into a frame. This works great if the original source material was 30 fps progressive (which is rare for NTSC but common for PAL), but for telecined video, since two out of every six frames are combined from two different fields, this leads to ugly combing effects.
If the source material was 24 fps, an inverse telecine can be done, recovering the original 24 frames per second. Unfortunately, it is not always this easy, since interlaced video may switch between methods, and sometimes use different methods at the same time, e.g. overlaying 30 fps interlaced captions on top of a 24 fps telecined picture, or compositing two telecined streams with a different phase. “Star Trek: The Next Generation” is a famous offender in this category – just single-step through the title…
In the following paragraphs, let us look at an episode of Futurama and how the deinterlacing was done by the different providers of the show. Futurama was produced in 24 fps and telecined. Some of the editing seems to have been done on the resulting interlaced video, so the telecine pattern is not 100% consistent.
NTSC DVD
The NTSC DVD is basically just an MPEG-2-compressed version of the US CCIR 601 broadcast master. It encodes 720×480 anamorphic pixels (which can be displayed as 640×480 or 720×540) and has all the original interlacing intact. This is a frame at 640×480 and properly inverse telecined:
Hulu
Hulu (480p version) took the original image without doing any cropping on the sides. You can clearly see this picture is only half the vertical resolution, meaning one of the fields got discarded. It seems this was Hulu’s deinterlacing strategy, since throughout the complete video, everything is half the vertical resolution, whether there is motion or not. This also keeps the video at 30 fps, and effectively shows every fourth frame twice, introducing stronger judder.
iTunes
iTunes crops the picture to get rid of the black pixels in the overscan area and scales it to 640×480. They run a full-blown 60 Hz deinterlace filter on the video. Such a filter is meant to take a live television signal as an input, with a temporal resolution of 60 Hz. While this looks fine on frames with no or little motion, vertical resolution is halved as soon as there is motion. Basically, it is the wrong filter. Like Hulu, iTunes preserves the 30 fps, introducing a stronger judder. (The video encoding is H.264 at 1500 kbit/sec.)
Netflix
Netflix seems to do the same as iTunes – maybe they even got the data from iTunes? The image is cropped and scaled to 640×480, they run a deinterlace-filter and retain the 30 fps, leading to halved resolution when there is motion, and stronger judder.
Amazon Video on Demand
Amazon Video on Demand with its horribly inconvenient Unbox Player (Windows only, requires 1 GB of extra downloads and two reboots) did a better job. Like Netflix and iTunes, they cropped the picture and scaled it to 640×480, but they actually did a real inverse telecine. In some segments (like the end credits), the algorithm failed because of inconsistencies of the original telecine, so it reverted to half the vertical resolution. And like the others, Amazon also encodes at 30 fps, i.e. judder. (The video encoding is VC-1 at 2600 kbit/sec.)
Zune
Microsoft’s Zune Store provides a cropped video at 640×480 at the original 24 fps and with a bitrate of 1500 kbit/sec (VC-1). Looking through it frame by frame reveals that they used a brilliant detelecine/deinterlace algorithm. On the DVD, the panning at the beginning of the “Robot Hell” song is very tricky: It breaks the standard telecine pattern (PPPIIPPPII becomes PPPIPPPI), it seems every fifth frame was removed.
The pan consists of a pattern of three progressive frames, and then one interlaced frame, which is composed of the previous frame and the current frame. Consequently, every fourth frame has half its resolution wasted by the repeated lines of the previous frame, i.e. every fourth frame only exists at half resolution in the DVD master material.
Hulu discards half the vertical resolution for every frame anyway, and the deinterlacing algorithms of iTunes and Netflix discard half the resolution whenever there is motion. The Amazon algorithm does a good job when the telecine pattern is correct, but in this case, it gets confused and encodes all frames of the pan in half resolution. The Zune algorithm does a brilliant job here: The progressive frames stay at full resolution, and it extracts the half-resolution picture out of every fourth frame:
This is the fourth picture at full size – you can see half the vertical resolution is missing (it was never there in the first place!), but the algorithm did a very good interpolation job:
The Zune video is almost perfect. It recombines all fields correctly and recovers all single fields, scaling them up so that it’s hardly visible there is information missing. If you ignore the 720 vs. 640 horizontal pixels, the resulting 24 fps video contains all information of the DVD version, but with all interlacing removed, and with zero judder. Too bad it’s not H.264, but DRMed and only plays on Windows (XP+), Zune and Windows Phone 7.
Summary
| Provider | Cropping | Resolution | Deinterlacing | fps | Encoder | Bitrate (kbits/sec) |
| NTSC DVD | no | 720×480 | none | 30 | MPEG-2 | 6500 |
| Hulu | no | 640×480 | discard | 30 | H.264? | ? |
| iTunes | yes | 640×480 | 30 Hz deinterlace | 30 | H.264 | 1500 |
| Netflix | yes | 640×480 | 30 Hz deinterlace | 30 | H.264/VC-1 | ? |
| Amazon VoD | yes | 640×480 | detelecine+decomb | 30 | VC-1 | 2600 |
| Zune | yes | 640×480 | fuzzy detelecine | 24 | VC-1 | 1500 |
Note: H.264 and VC-1 compress significantly better than MPEG-2; a rule of thumb is to divide the MPEG-2 bitrate by 2.3 to get a comparable H.264/VC-1 bitrate. So the Amazon bitrate is fine and the video is about the same quality (sharp picture, no compression artefacts) as the DVD, but the iTunes and Zune versions are not (artifacts can be seen on single frames).
It is scary how little effort seems to be going into video conversion/encoding at major players like iTunes, Netflix and Hulu. Amazon did a kind of okay job converting the source material properly, and only Microsoft did an excellent job. The NTSC DVDs still give you the maximum quality – but of course, if you watch them on an LCD, the burden of deinterlacing is on your side. Handbrake with “detelecine” (for the bulk of it) and “decomb” (for exceptions) turned on, and with a target framerate of “same as source” will generate a rather good MP4 video similar to Amazon’s, but without the judder.
Are there any stores I missed? Can someone check the PAL DVD as well as digital PAL and NTSC broadcasts? What is the magical detelecine/deinterlace program Microsoft uses?
Interlacing is still present in high end projectors e.g. Doremi.
NTSC is 29.97fps not 30fps
I see lots of temporal artifacts on UK TV from NTSC -> PAL conversion. I use the opening credits of Will & Grace as my canonical example, interestingly that have changed (for the better) as the show got more series.
Final Cut’s conversion introduces temporal artifacts.
I have been working on an FFT fps transcoder for a while but nothing to show as of yet.
I’ve spent quite a bit of time in India and if you watch Star TV satellite broadcasts on a CRT you will see lots of footage with reversed fields.
Get LCD panels out of the edit suites!!!
http://en.wikipedia.org/wiki/STAR_TV_%28Asia%29
I don’t have anything useful to contribute, just wanted to say this is a very interesting & well-written post.
It’s worth saying that more recent episodes from itunes and similar are done right. I think this was just early teething problems.
@maht: NTSC isn’t even 29.97 fps, but 30/1.001 fps 🙂
@Anon: “just early teething problems”? No. The original FOX seasons all have interlacing issues, and the movies as well as the Comedy Central episodes were produced in 24 fps HD, so the source material for iTunes was already 24 fps.
Wow! Microsoft did something right for once! Great article BTW.
Always good to preface these articles with, “You might not want to know this.” Ever since I’ve studied video fidelity I curse it when my entertainment turns into QA.
Too bad open source video isnt main stream. It be nice to see XviD covered.
@Directory Guy… read my name
Thanks for the explanation and screen shots, that explains a lot of artifacts I have always been wondering about. I also wonder what is keeping these companies from encoding properly, is it simply too CPU intensive?
have any of you guys ever thought about the fact that all of these companies have to deliver these assests across the entire country and need to account for the entire spectrum of different bandwidths?
Also, consider the sources of their content. Would it be correct to assume that the studios and vendors selling the content wouldn’t give the highest quality sources so that they can minimize the loss in profits from their dvd and bluray sales?
also, why would you use animated content, designed for broadcast television, for your test? I recommend doing your comparison with some live action content that all of these companies deliver in HD. As well written as your article was, I felt it was somewhat biased and a bit lazy. Use multiple examples of different types of content please.
I give kudos to the author of the article for taking the time to compare the quality differences between the different online video providers. However, I must agree with the previous post that this is not a true apples to apples comparison and let me explain why.
Because the content is the same does NOT mean that each of the providers received the same source file. Depending on what division of the studio the deals are made with (Home Video, DVD, BluRay, etc), this will be a determining factor in the source file that the provider receives. Was it tape based? Was it a digital file? HD? SD? etc. For example, Microsoft could have gotten access to an HD tape master like D5 or HDCam which could have been run through a piece of hardware like Teranex to create a 23.976p HD digital source file. On the flip side, other vendors might have only gotten access to an SD version because they had rights for the content earlier? If SD is the case then one provider might have received an interlaced source and another provider could have received a progressive source file.
There are a lot of assumptions you are making when you say that “…little effort seems to be going into video conversion/encoding at major players like iTunes, Netflix and Hulu.”
Also, I find it strange that you’re complaining about “motion judder” at 29.97 fps progressive with animation content. Particularly with the title “Futurama”. This particular series (along with other popular animated series such as “Family Guy” and “The Simpson”) are animated at 12.0 fps (sometimes even 8.0 fps), then converted to 24.0 fps, then telecined to 29.97 interlaced. Duplicate frames are inherent in these types of animations so don’t expect 60i motion.
Another reason for judder may be the setup of your TV as well. A lot of recent flat panel TV’s have a pseudo-120Hz feature which, when enabled, can cause motion judder on normal content. If this feature is enabled, try disabling this feature. All the TV is doing is interpolating extra frames/lines to create the effect of smoother motion. Essentially it’s “making up data” that isn’t really there and injecting it into the picture. The algorithm isn’t always perfect so some content may look pleasing with this feature enabled while other content may display obvious glitches. For example, if you have the feature enabled, try watching a scene with a very fast moving object that pans the scene. You’ll most likely see glitches and motion judder here.
As stated earlier, I enjoyed reading the article and the discussion this brings regarding quality. But try comparing a title where all providers give access to HD 23.976 content. I suggest using a title like “Heroes” since this displays true motion and all the content providers have HD versions of this series.
@anon:
1. Bitrate has nothing to do with the issues discussed, which is properly dealing with interlaced source material.
2. Sources of content: True, iTunes, Neflix and Hulu could have already gotten badly deinterlaced material from their sources. In this case, of couse the source is to blame, but also iTunes, Netflix and Hulu for selling low quality goods. In this case, it’s not like higher quality video was unavailable – Microsoft got it.
3. Animated vs. live action: I would love to do the same analysis with live action video (but not anything recent and in HD, because it would not be interlaced) if someone can suggest a good show to test this on. Sitcoms should either be 24p originally or maybe 60i. Shows with lots of editing and special effects are especially interesting – too bad “Star Trek: The Next Generation” is not available for download on any of the platforms.
@Steven Kang:
1. I think I am indeed comparing apples to apples here – if you look at the perspective of a US customer. I want to watch Futurama, and I can either get a Netflix subscription (about 9 dollars/month, soon 8), rent the show from iTunes (about a dollar), buy it from iTunes, Amazon or Zune (about 2 dollars), or watch clips for free (with ads) on Hulu. Given the choice, I’m interested in the highest quality.
2. Different source material: I agree that it is very well possible that the different providers had different sources. In this very example, the Zune version of the Futurama pilot shows a “TV PG” logo during the first few seconds, while the others (including the DVD) don’t – Microsoft might indeed have had a different source than the others. But in this specific case, Microsoft certainly did not have a source that was superior to the DVD, since I showed that their file doesn’t have the missing fields in the “Robot Hell” pan – just like the DVD version. They either deinterlaced a source with quality equivalent to the DVD, or they got a deinterlaced version of it already. They definitely did not get the 24p original master. And still, even if some providers had access to lower-quality sources (e.g. already badly deinterlaced): In this case, the people who did the encoding are not to blame, but still the provider as a whole is, because they were not able to cut the right deals to deliver the same quality I can get from other providers.
3. “little effort seems to be going into video conversion/encoding”: You are right, let me rephrase: “little effort seems to be going into either getting high quality sources or video conversion/encoding”, or “little effort seems to be going into producing high-quality progressive video for the customer”
4. Judder in Futurama: True, a lot of the animation is done at 12 fps, i.e. the artists rarely draw more than 12 frames for every second of the show. But it is not true that Futurama is produced at 12 fps and then converted to 24 fps: Some scenes indeed have 24 drawings per second (e.g. Pilot: Fry riding the bike at 0:35), and all computer animations (the space ship, e.g. at 20:09), all moving objects and all pans are 24 fps – and it’s these that count: steadily moving objects and pans! The Futurama title is the best example here, it’s consistently 24 fps.
5. TV Judder: I was doing my tests on an LCD monitor. The flight towards “Madison Cube Garden” at 0:32 in “Hell Is Other Robots” looks jerky on iTunes and Netflix, but it looks great in the Zune version. I understand that TVs have their own refresh rate and might display frames twice or cause tearing, but even then the video from the provider should be at the optimal frame rate for the source material, and it should be up to the decoder/display device to do the necessary conversion (telecine, switch refresh modes, duplicate frames, maybe interpolate frames…), shouldn’t it?
6. modern 24fps HD content: The point of my article was to compare how the different providers coped with a telecined TV show. If I compare a modern TV show that is produced in 720p24 and has no overscan area, all I do is compare their respective bitrates and the quality of the H.264/VC-1 encoders they use, which is somewhat less exciting. 😉
“special and temporal resolution”??? Do you perhaps mean **spatial** and temporal resolution?
@GetItRight:
Hah, good one. Thanks for pointing it out. Fixed.
Very cool article! It’s rare to see someone giving this close a look to video processing. And it reveals the critical truth about compression. Good codecs can keep good sources looking good, but it’s bad preprocessing where things REALLY get ugly.
I know the guys who handle the ingest workflow for Zune, and they in fact work really hard on preprocessing content. And 480i animation sources like these are about as painful as it gets.
If you want to see how things go with REALLY challenging sources, check out anything from the first season of Family Guy.
@Steven Kang
Deinterlacing (30i->30p) instead of IVTCing (30i->24p) a telecined video will introduce judder to practically anything. Unfortunately good IVTC systems (as good as TFM, or at least pullup) seem to be rare commercially, or maybe professionals don’t know what to do with them. Even when they’re animation companies uploading to iTunes.
I know that MS does not work from retail DVDs for their content so the odd cadence seen on the consumer side may not apply to the master they get to encode from.
Additionally the department that handles their encoding (internally refered to as “vidlab”) has some of the best compressionists you can find.
Contrast this to netflix which takes region 1 dvds sourced from PAL masters and just rips them to their service making them unwatchable.
Good article. A couple of points:
– Your rule of thumb comparing MPEG-2 to H.264/VC-1 doesn’t really work because of the huge variation in compression efficiency/quality of encoder implementations for any one format, particularly H.264. For example, x264 offers considerably better compression efficiency than many (most/all) H.264 encoder implementations.
– H.264 and VC-1 both have interlacing compression modes. H.264 actually has two – PAFF and MBAFF. But you’re absolutely right, editing interlaced/telecined material is a massive sin. Don’t do it! 🙂
HandBrake’s methods are very good and somewhat derived from some AviSynth filters with some special sauce added on top. I used this and some other references for researching my own attempt at cracking this problem for GStreamer. I will be blogging about this when it is done.
Have fun! 🙂
Years ago, I wrote a plugin for “transcode” that reverses telecine, and documentated how telecine works – for those people that would like to find out, it is here: http://users.softlab.ece.ntua.gr/~ttsiod/ivtc.html
It would interesting to learn what algorithm MS used – when I wrote the transcode plugin (8 years ago), Donald Graft’s code was the one I based my work on, and it was the best. I believe, however, that it would have failed on the problem spot you mention (the PPPIPPI one)…
@ADFan: Most or all of the VOD services work from mezzanine files like Cineform AVI files or High Profile MPEG-2. There’s some use of ripped DVDs, but thats obviously sub-standard.
However, lots of content is only archived as 480i, so the mezzanine file will have all the weird field stuff seen n the DVD. It just won’t have compression artifacts.
@Robert Swain: Yes, VC-1 and H.264 both have native interlaced modes. However all the VOD services today are encoding progressive AFAIK. Windows 7 has a pretty good real-time Bob mode (so 30i gets played at 60p), which is good.
For animation, 24p is enough. But for high-motion content like sports or reality, better to do a bob deinterlace to 480p60 instead of throwing away half the temporal information and sticking to 480p30. This requires really good (and slow) deinterlacers to do well, though.
@astrange: Correct, however if you receive a source that is 30p there’s not much to do.
@ADFan: There are rights issues with a lot of content, particularly content that originates from PAL regions. Some studios which will go un-named provide NTSC Tape masters which were converted from PAL Tape masters so there’s no way to go back to the original state in these cases.
When video is provided to companies such as hulu, it is often a digital file encoded by the content owner/provider specifically for hulu and to hulu’s source specification. So the original source master is not being provided to hulu, but rather a high resolution encode. That file is then re-encoded by hulu. The problems could arise in either of those encodes.
We would need to examine multiple sources from each provider to determine a pattern before laying the blame on the provider.
I have wrestled with the interlacing issue with my personal dvd rips and decided to tell my video card to output 1080i and rip everything in it’s de-interlaced glory. I couldn’t stand seeing those artifacts after having done all of the rips that I did, however I did notice that hulu and netflix content suffered from the same problem. I guess the people just aren’t bothered or perhaps I’m just more observant than most. I made this blog post to document my struggles….. http://danshirah.homedns.org/personal/?p=786. Now the concept of PAYING for FLAWED MATERIAL? That really kills me. It does help me understand why high definition cable is still at 1080i……
Haha, good choose about Futurama show 🙂
I got in a big debate with Hulu over this a few weeks ago. Along with their new platform launch with improper subtitle size I brought attention to a few glaring issues. It looks like they finally listened and shows are being deinterlace pre-stream. They probably listened to the fact that it takes half the bandwidth to broadcast progressive rather than interlaced material.
I’ve been trying for years to correctly encode my Futurama DVDs, with limited success. Today, I got a free trial for Hulu and was surprised to see how great the quality is. I think, if you have a chance, it would be great to see what you think about their process now. It’s what I’m trying to achieve with my own collection.
Nevermind. I was able to watch more episodes, and while there aren’t combing artifacts, there are still the issues you described in your article.
I’m pretty sure I can help resolve all the issues… 2/3 pulldown, mixed cadence, Deinterlacing, cross-format conversions, including Uprising or Ai upscaling, using interpixel expansion. Everything thing should be resolved by 24p. I was part of that dev in the late 90’s… I love restoring and preserving the past for the future! Nothing like a good-looking progressive frame 😉 Who should I speak with? Best, Bradford