Importance of formats...

Philippe — Sun, 24 Jun 2007 16:42:00 +0700

I wrote this document in February 2007, as a coursework during the last year of my French engineering school. It's entitled "OK, Houston, we've had a problem here.", in reference of the problem that concerned the Apollo 13 mission, and the problem of the NASA tapes, nowadays. The aim was to write about 3000 words about an Internet related topic. I chose the formats because, as a regular reader of the formats-ouverts.org blog, I believe in their importance in this field, and I wanted to share these information with my classmates. I got a lot of information on this blog, but not only (cf my sources at the end).

Introduction

Any information needs a protocol, a format, to be read and understood. If you read this text, you can understand the English language. If you make a phone call, your phone can transform voice in electric signals. The Nü Shu writing system is a Chinese system derived from Mandarin only used by women. It is also almost completely unknown, because its last proficient user died in 2004. Because it remained secret until 1995, and because it was customary to burn writings during the author's funeral, there are a few Nü Shu manuscripts left to help us understand this system. This kind of issue is possible in the computer field. If your favorite software's company decides to stop its development, or, worse, closes, all your documents created with this software are lost.

These examples illustrate the importance of the formats. Formats are important because they are everywhere. They are important because they are the key to the access to information. An open or standard format is necessary when you seek interoperability, data conservation, or security. A closed (often proprietary) or non-standard format can lead to the problems we just saw, loss of information, control of the user or balkanization of a specific field. That's why their choice should not be neglected. In France, since June 2004, a law defines what is an open format, and the government, providing the general repository of interoperability, tries to promote the use of open formats in its administration.

Unfortunately, people are unaware of the problems that can occur because of a wrong format. Several organizations (World Wide Web Consortium, etc) and Web sites (http://formats-ouverts.org, etc) try to show that formats are important, but companies with their economical goals are engaged in a war of formats.

Importance of formats

Definition

Let's define what a format is. Wikipedia gives this definition on its Web site (translated from French) [WIK02] :

“A format is the way (eventually normalized), in the computing field, to represent data with binary numbers. A format can be a protocol, a file format, a convention. It can be proprietary, open, standardized. Standard formats are used when interoperability between different software is aimed.”

A format can also define the type of connector (e.g. USB), the character encoding (e.g. ASCII), etc. Formats are important because they are everywhere. They were everywhere before the development of computers, and were already creating problems. The most common format is probably the language we use to communicate. If you can read this document, it's because you understand English. But how can we read a text if we can't understand it's language ? And what if nobody can understand it ? That's the problem with the Phaistos disc [WIK03], which, unlike the Rosetta stone, contains writings in only one language. As we have nothing to compare the disc with, the language can't be understood.

The source of many problems

This kind of problem also exists in the computing field. It is impossible to open a document if we don't have a software that knows how to decode it. This is just the source of another issue. In order to open a certain file, we need a specific software, sometimes in a specific version. To run this software, we need a specific operating system, which runs on a specific type of computer. We have here a chain of formats, that can control and limit the user in its choices and its freedom.

The domination of Microsoft and Intel in the computing field is based on this. Lots of files are exchanged in formats that are readable by softwares that exist on Microsoft's operating systems only. Such systems can be installed on Intel (x86) computers only. In order to read this file, you need to respect all these constraints. The computing field is not limited to file formats. For example, the Skype software, which allows to make phone calls over the Internet, uses its own protocol to communicate, hence it is impossible to make it communicate with other softwares¹. What if real phone calls couldn't be made between two different telephone operators, or telephone brands ?

An important choice

Let's now consider a document provided by a government to its citizens. Any citizen should have access to this document, regardless of the format in which it is written. This means, being able to read it from any type of computer, with any operating system and any software. This is interoperability. This also means being able to read it in 10 or 20 years. This is data conservation. The French radioactive waste management centers have to store their data during 300 years. To do that, they have chosen a special paper and a special ink. Digital formats weren't a solution because of the problems we just talked about. They would have required the conservation of all the elements of the chain : the software, the operating system and the computer. We will see that governments and public administrations are considering these issues and are publishing recommendations, and sometimes laws, to favor the use of open formats.

Choosing the right format

Five questions

Now that we know that formats are important, we need to know how to choose them. We will see that choosing the wrong format can lead to loss of information, or control and limitation of the user. On its website dedicated to open formats [FOO04], Thierry Stoehr explains that we should ask ourselves 5 questions before choosing a format :

Are the file format, protocol or programming language open ?
Are the metadata (author, title, date, etc) open ?
Is the license (terms of use, etc) soft concerning the user ?
Are the hardware (computer, etc) and the physical support (cdrom, etc) durable ?
Is the document's follow-up accessible ?

These questions can ensure that your information will be accessible in time and from any platform.

Governments can help

Since information provided by governmental services must be accessible by everyone, several governments have voted laws that make the use of Open Format Document (ODF) mandatory for office documents. ODF is an OASIS standard approved by the ISO as an international standard. In France, since June 2004, a law defines what an open format is [LOI00] :

“Is considered an open standard any communication, interconnection or exchange protocol, and any substitutable data format which technical specifications are public and free of access or implementation restrictions.”

France, Norway, Belgium, Denmark and the state of Massachusetts also recommend open formats for their applications and documents [LMI00] ,[FOO02], [FOO03]. Administrations seem to follow these recommendations, the French ministry of agriculture, the national police, and the ministry of finances have chosen the Open Office software to edit their documents. ODF being the default file format for this application, this ensure that any other office software can implement it in order to read ODF documents. Other administrations or cities (Amsterdam, Munich) in the world are also interested in reducing costs, ensuring data conservation and interoperability by choosing open formats and free softwares.

Some people even go further with the definition of an open standard. The European Interoperability Framework for pan-European e-Government Services gives this definition [EIF00] :

“The following are the minimal characteristics that a specification and its attendant documents must have in order to be considered an open standard :
The standard is adopted and will be maintained by a not-for-proﬁt organization, and its ongoing development occurs on the basis of an open decision-making procedure available to all interested parties (consensus or majority decision etc.).

The standard has been published and the standard specification document is available either freely or at a nominal charge. It must be permissible to all to copy, distribute and use it for no fee or at a nominal fee.

The intellectual property - i.e. patents possibly present - of (parts of) the standard is made irrevocably available on a royalty-free basis.”

Compatibility against interoperability

Another difficulty when choosing a format is that compatibility and interoperability mean the same for a lot of people. Compatibility is when a software understand a format (or protocol) that is not its default one, e.g. Apple Pages 2 is compatible with the Microsoft Word (.doc) format. Interoperability is when several softwares understand each other because they use the same open format (or protocol), e.g. WengoPhone and BeWip are interoperable because they both use the SIP protocol.

Compatibility between formats
Compatibility with a de-facto standard
Interoperability

Illustrations by Camille Moulin

Now that we've established that we should use open standards to store our documents or to communicate, let's see a comparison of open standards and closed standards. A detailed list can be found on Wikipedia [WIK00], although it only concerns file formats :

Functionality	Closed standard	Open standard
Formatted text	Microsoft Word (.doc) Apple Pages (.pages)	Open Document Text (.odt)
Spreadsheet	Microsoft Excel (.xls)	Open Document Spreadsheet (.ods)
Image	Tagged Image File Format (.tiff)	Bitmap (.bmp)
Lossless audio	WAVEform (.wav)	Free Lossless Audio Codec (.flac)
Instant Messaging	Microsoft Messenger	Jabber
Voice over IP	Skype	Session Initiation Protocol (SIP)

Table 1: List of closed and open standards

Several organizations contribute to create standards. The most known are :

The Internet Engineering Task Force (IETF) which publish Request For Comments (RFC), for example the SIP RFC : http://www.ietf.org/rfc/rfc2543.txt
The Institute of Electrical and Electronics Engineers (IEEE) which publish norms, for example the Ethernet IEEE 802.3 norm.
The International Organization for Standardization (ISO) which standardizes norms, such as ISO 15836, the Dublin Core metadata element set.
The World Wide Web Consortium (W3C) which publish recommendations to promote compatibility between World Wide Web technologies, such as XHTML.

A detailed list of organizations can be found on Wikipedia [WIK01].

It's risky

We've all heard about the several delays in the Airbus A380 production. These delays are caused by the complex organization of the several factories in France and Germany. Whereas German factory of Hamburg uses paper, French factories use a specific software to manage their productions. This software was finally adopted by Hamburg factory, but needs a long training to be efficiently used. Meanwhile, the project is more and more late. Second, when French and German both use a software to conceive electrical plans, they are not the same and, of course, not compatible. So, choosing a wrong software, therefore the wrong format, is not without risks.

Incurred risks with closed formats

User controlled and limited

There are several risks of choosing a closed format. The most important one is probably the fact that a user, when his documents are encoded with a closed format, is controlled and limited in its freedom. Anyone should be able to change of operating system or word processing software freely. On the other hand, anyone should be able to read a document sent to him without having to purchase the specific software. Roberto Di Cosmo, researcher in computer science, explains [COS00] Microsoft's strategy to force its users to purchase regularly its softwares. How do they manage to control the user this much ? They make impossible for a former version of a software, let's say Word, to read correctly a document created with the latter version. For example, it's hard, even impossible, to open a Word 2003 document with the Word XP software. If this situation comes up, you will be forced to purchase the latest version of Word. So you're controlled because you are forced to buy the latest version of your software(s) regularly, and you're limited because if you want to change of software, you have to find another software compatible with your documents. Lots of softwares are compatible with the Microsoft's file formats so the limitation is not that strong, much less are with the Apple Mail.app's format (.emlx) [PIL00].

Balkanization

Another problem caused by closed formats is the balkanization of the markets. In this case, balkanization means the subdivision of a market by several vendors, each one promoting its product, base on a closed format. The best example is the music market, with its particularities : on-line stores, Digital Rights Management (DRM) and mobile players. This market is led by Apple with its iTunes platform, and Sony with its Connect platform. The problem comes from the formats of the music sold on these platforms : closed, unique for each vendor, and incompatible with each other. Furthermore, music bought on iTunes can't be played on a Sony player, and vice-versa. This tends to split the market and to loose the consumer. This problem already happened a few years ago when the Web was divided in two : one for Microsoft and one for Netscape. It was solved because they both made an effort and followed the standard recommendations published by the W3C. Hence, the Web is now a place where almost every website is visible from almost any browser, on any platform. Not all websites from all browsers because some websites remain accessible from certain browsers only, like Connect which requires Internet Explorer 5.5 and more.

Hackers to the rescue

Closed formats are also the reason why some people crack softwares. The famous DVD Jon, which cracked the protection on DVDs (Content Scrambling System, CSS). Any player capable of playing a DVD must implement this protection system. Unfortunately, Linux users couldn't play their DVDs because no software implement the CSS protection. Using this crack, and of course creating it, makes the user an outlaw, because of the Digital Millennium Copyright Act (DMCA) in the United States, and its equivalent EUCD in Europe (DADVSI law in France). DRM, which are partly in cause in the balkanization of the digital music market, are also the reason why some developers create cracks such as DRM Dumpster, or FaireUse4WM. This is not the only case where users must brake the law to be able to use something they legally bought. Reverse engineering is a method that consists in analyzing how something works (a software, a codec, a protocol, etc) to recreate it. The French law authorizes this under certain circumstances, but when the system is protected by a patent, it becomes illegal. The Skype protocol was recently cracked by a Chinese company [PAG00], after a reverse engineering operation. Since the protocol was not patented, yet kept private, this is not illegal. Nevertheless, when it comes to personal information, I think performing such operations is justified in order to know what is going on inside.

Conclusion

We've seen that formats, if they are not considered as an important parameter in your choices, can lead to bad and sometimes irremediable problems. We explained why such problems can happen, whereas consumers are expecting an easier way to work and to entertain using computers and digital contents. Some firms take advantage of these expectations to make profit, and to abuse the consumer.

The best way to prevent himself from such issues is to get information about the importance of the choice of a computer, its operating system, the softwares and the documents you use. Information that can be provided by international organizations, Web sites, and even by local associations.

The message that tries to give this work is not to necessarily use open formats and standards, but to consider them when you seek interoperability between different softwares, long conservation of your data, etc. This is just easier to achieve with open formats than with closed. Furthermore, any document needs to be stored, and the support is also very important. Meanwhile, the war of formats continues.

Bibliography / Webgraphy

WIK02: Wikipedia, Format de données, 2007-01-31

WIK03: Wikipedia, Phaistos Disc, 2007-01-30

FOO04: Formats Ouverts, Pour les formats ouverts,

LOI00: Legifrance, LOI n° 2004-575 du 21 juin 2004 pour la confiance dans l'économie numérique, 2004-06-22

LMI00: Le Monde Informatique, France: un rapport gouvernemental soutient les standards ouverts, 2006-10-04

FOO02: Formats Ouverts, Une nouvelle prise de position importante du Massachussets, 2005-09-02

FOO03: Formats Ouverts, Le Danemark déclare qu'il va utiliser le format ouvert ODF, 2006-05-26

EIF00: European Interoperability Framework, EIF For Pan-European eGovernment Services, 2004-11-01

WIK00: Wikipedia, Alternatives ouvertes aux formats fermés, 2007-01-29

WIK01: Wikipedia, Standards organization, 2007-01-08

COS00: Roberto Di Cosmo, Le Hold Up Planétaire, 2006-08-31

PIL00: Mark Pilgrim, Juggling oranges, 2006-06-16

PAG00: Charlie Paglee, Skype Protocol Has Been Cracked, 2006-07-13

Table of figures

Illustration 1: The chain of formats

Illustration 2: Five questions

Illustration 3: Compatibility between formats

Illustration 4: Compatibility with a de-facto standard

Illustration 5: Interoperability

Table 1: List of closed and open standards

Illustration 6: Balkanisation of the music market

Screenshot 1: Fake Skype (left) in action

riendutout.org - Univers libre