Enterprise Technology By Phil Edholm

I have occasionally been asked in this blog to step outside of the narrow field of networking and computing to comment on other technical issues of the day. So, in the interests of being a little provocative, I decided to a do an analysis on a topic a ways outside of my everyday world. If you are not interested in a mental thought process in a parallel field, skip this post please.......

I found this article about digital media storage to be interesting. It details a number of questions around both the practice and cost of digital archiving and questions in a number of ways. I got to thinking about the comments made and thought it was an interesting topic to post about.

COMPUTERWORLD WorldDigital crisis: Motion pictures may fade to black
Dian Schaffhauser
"February 08, 2008 (Computerworld) Current storage technologies may have a reputation for being plentiful and cheap, but not necessarily in Hollywood, where a recent study warns that the annual cost of archiving a digital film is 11 times that of storing celluloid film.
According to "The Digital Dilemma," a report recently released by the Academy of Motion Picture Arts and Sciences, digital film storage costs $12,510 per year, compared with $1,059 for celluloid. More dramatically, source materials -- those outtakes and audio recordings that often make up bonus content for special edition products -- cost 429 times as much to store, a whopping $208,500 per year for digital materials vs. $486 for film.
The report's authors state the data explosion could turn into digital movie extinction, unless the studios push the development of storage standards and data management practices that will guarantee long-term access of their content."

NOTE: the article is fairly long, so I will leave it to the reader to follow the link above for more details. It will open in a new window to look at my comments versus the article quotes.

While I am not in any way an expert in film, my first reaction was that the claim that storing a digital film would cost not 11x as much but actually $212,000 per year versus a mere $1,500 for film was a really suspicious claim. Even the 11x seemed high (until you realize that digital is storing 3-5x more information as discussed below). Essentially, the claim is that to store something on digital tape costs over two orders of magnitude (141 times actually) more than on film. I also am not qualified to comment on the artistic value of making a film in either digital or film media, my only interest is in the discussion of archival quality around resolution, color density, and archival size and longevity. I will leave the discussion of the artistic merits to others who are more embedded into the process. The following are some thoughts on the article and trying to noodle on what I read and researched about the issue..............

GUT REACTIONS

My first reaction is that this is one of those tales by a bunch of folks that have a vested interest, either personal or business, in the concept of film stock and the analysis and preservation of a "relatively" fragile chemical media. I believe the same genre of people were decrying the automobile as being far inferior to the horse for transportation, both in reliability and in cost and a myriad of other factors....

At the top level, there seem to be many inconsistencies here:

First, the size of the retained record in minutes of material seems to be 2-5x larger with digital production (not archiving). This is presented as an issue, but seems more of a new paradigm in the artistic/production processes that has been introduced with digital media. With digital they shoot more because it is easier...…why not throw out the extraneous stuff…oh, because it MAY have value…but it never existed in the film version, so therefore film is, by definition, a lesser media as it is not capable of capturing the entire "event" of the making of a film due to cost……if there is no value to the 5x increase in base material, a simple process of rejection is apparent. On the other hand, if there is value to this material then we should retain it and only use digital. Also, all of the "intermediate" screening versions are kept in post production in digital, while they are not in celluloid….wow, so we have a record of the steps the team took from their first cut to the last…either it is worth saving or just throw it away...…the sum of all of these comments is that the increase in material per film is due to the digital film-making process, not the archiving.

Tape access is not random???...…is accessing spools of celluloid somehow become random???????....or easier???? This is discussed in more detail below as I believe this may be a significant part of the cost differential quoted.

What they do not emphasize is that celluloid deteriorates with age…there is a gradual reduction in the capability of the particles to effectively image over time as acids, oxygen, and other reagents react with the film stock. This deterioration is not absolute, it is a linear process as the atomic structure of the film deteriorates through contact with the atmosphere. The argument in the industry is that a combination of low temperatures and low humidity will essentially preserve current color film stock for very long periods. If you assume that even with low temp/humidity film deteriorates over time, then the next archival print will be a result of that deterioration (assuming it is not produced digitally). This appears to be primarily in color resolution. On the other hand, with digital media, using a RAID type of mechanism that enables the recovery of lost data and BERs, the archival version can be transferred across storage systems indefinitely with no loss in quality. It is not clear whether the lifespan of magnetic media is shorter/longer/same as that of film stock, as this would drive when an archival copying action is required.

MATHEMATICAL ANALYSIS (Some quick thoughts)

One of the questions raised is whether the digital storage is sufficient and then what the size of the stored files are. I tried to do a little analysis of this to validate (and contrast) the numbers in the report. Also there were comments on quality.
Anyway, the following is a quick analysis of the claims, both validating and (potentially) contesting…..

Per Brad Templeton http://pic.templetons.com/brad/photo/pixels.html , a frame of 35 MM film has 20M “good” pixels. This is for still camera stock, but let’s assume the same is true for each frame of film stock. The current digital projection standard is 4 Kpixels across, which yields 2250 pixels high in the 16x9 format (this is contrasted to the 1920x1080 of HDTV). The digital projection standard yields 8.8 Mpixels, however, that is multiplied by 3 for the 3 color pixels, so the total is about 26 Mpixels, slightly higher than the “good” pixels in 35MM film. Per Templeton, he argues that a 5300x4000 resolution is necessary to equal top quality film, but this is for a 5x4 still image, so the 26 Mpixels is appropriate for full density with the 16x9 image size. If we assume that this is used, then digitizing a 35 MM print will take about 26 Mpixels per frame with 16 bits of color density and 24 frames per second. This yields 1.27Gbytes per second. Assuming the numbers are correct, that for every minute of final film, there are 60 minutes to be archived and that the film is 2 hours long, the total storage is: 547 TBytes (they actually have 208 Tbytes). The 2 PBytes is not really reasonable for most features which run significantly under 2 hours, though I would think that would be appropriate for a film like “The Burning of the Red Lotus Temple” (Chinese, 1928), which ran 27 hours!!!!, though I would be surprised if there are many extras to archive….."Fanny and Alexander" (1982) comes in at 5 hours, which yields 1.5 PBytes if it actually had the extras. for purposes of the rest of the analysis, I will use the 208 Tbytes quoted in the article.

Now, HP has a tape storage system with a max capacity of 5673 TBytes….so that is only about 27 movies with automated access at these rates…..this means an automatic storage system is probably out as the benefits of quick access do not justify the cost. However, such a system would be useful to actually create or bring back the archival information. The tapes could be removed for long term storage. So, just like film stock, the tapes will have to be manually archived and retrieved.

So the question then is would it be the same storing digital tapes as film? If the film reel has 22 minutes of film on a 12 inch reel, then the 2 hour movie with 20x increase (per the article) would need 133 film reels (see the 129 cans in the article - I will use that number) to store all of the materials with a single copy. If we assume the same film shot as above on digital equivalence, with 800 GByte tape cartridges, and a 208 Tbyte total, it would require about 120 tapes...this would be reduced to 60 tapes with 2:1 compression. I am not sure on the cost points, but if we assume the cost of a tape is the same as a reel, the 50% smaller number of 800Gbyte tapes should generate a decrease in cost. The other issue is ease of recovery, though this should not be a huge issue, you could use one of the above machines to put the data on and off the tapes. Also, with the compressed tape, each tape has twice as much information, reducing the mount time to find appropriate information.

All of this tells me that this is a very complicated issue and is not really a networking issue at all….it is a storage density and size issue and the “inefficiency “ of digital (note that if digital had the same 20 minute to 1 as film versus the 60 to 1 used on the above analysis, the actual number of tapes would be equal to the number of film reels. So the key difference may be two assumptions; that the digital system is automatic (resulting in much higher costs), and the “extra filming/editing overhead” of digital. If the “director” was to remove the extraneous taped material and reduce to the 20 to 1 ratio, and the storage was really archival (which is actually what the reels are for heavens sake), then I think they would be roughly equivalent. The real issue in this document is the assumption of .$.16 per minute archived, or the $500/year/Tbyte. This is a claim they base on the San Diego Supercomputer center. I wonder what type of storage they are quoting, I bet it is some form of automated archiving system and includes active tape management and automated retrieval.

To pursue this further, it would be interesting to talk more with someone like IBM, HP or Fujitsu who makes storage systems to analyze the cost per reel versus tape. Also, I assume that there are some mechanisms for error correction/RAID striping across tapes for data reliability (this would eliminate the requirement for three (3x) distinct copies, potentially driving 2.4x using 4in5 striping and then stored as 1.2x in 2 locations).

Finally, I wonder about optical storage. Sony is doing a 5.25 inch optical system with 40 GBytes per disk and has plans to go to 120 Gbytes per disk. Obviously, even at 120GB per disk, a movie archive of 208 Tbytes is a lot of disks...(866 with 2:1 compression). However, if this technology were applied to a large size disk (think LaserDisk - 12") and if area is proportional to capacity, the 12 inch disk would have 7 times the 120 Gbytes of the Blu-ray disk and if used 2 sided would therefore have 1.68 Tbytes of capacity per disk. If we assume 2:1 compression, then an entire digital movie of 208 Tbytes could be held in about 61 disks. With 4in5 stripping across the disks for error protection, 75 or so disks could hold a entire movie, and should have virtually zero degradation over time. As each disk would be about .25 inches in a secure holder versus the about 1.75 inches for the film cartridge, the total archival volume is reduced by 86% per unit. This means that a film archive would take about 1 square foot for the 12" size and have 129x1.75" of linear shelf space or about 18 feet of shelf space. In a 4 foot wide 7 foot tall cabinet with 6 12" shelves (24 cubic feet of 12" deep storage), only one movie could be accommodated. In the same cabinet using optical 12" disks, 16 movies could be accommodated. In cubic feet, the film storage is 18 cubic feet, while the optical disk, even accounting for the 25% add for reliability, would take up about 1.5 cubic feet (.25x60=15inches/12=1.5 cubic feet).

As an aside, tape storage can be in steel cases that hold 60 tapes in a 20"x5"x20.75" form factor. Each 60 tape box is about 1.2 cubic feet, therefore the 60 tapes for the movie described above would require a single case or about 1.2 cubic feet or about 10% the physical storage of film or essentially equivalent to the optical above. I am not sure of the archival value of these boxes or the longevity of the tapes, that would need comment.

In any case, after spending a little time thinking through the parameters. I really find the assertions that it would cost 147 times as much to store movies in digital format to probably be a wild exaggeration.....

Anyway, interesting mental exercise……good for keeping the brain aware, and now I have a new topic to discuss……