A good deal of the simplicity of Relax NG comes from a careful definition of its scope and a focus on the validation of the structure of XML documents instead of the values placed within its nodes, however, even without using external libraries which will the subject of our next chapter ("Chapter 8: Datatype libraries"), Relax NG includes a simple and efficient support for values, enumerations, lists and whitespace processing which we will study in this chapter.

!!Values

The elementary pattern on which enumerations are built is "value". The syntax and semantics of the "value" pattern is straightforward: the pattern will only be matched if the value found in the instance document matches the value specified in the "value" pattern. Out of the scope of an enumeration which we will see in the next section, "value" patterns can be used to check fixed values such as version identifiers of XML vocabularies. If we wanted a highly specialized vocabulary to describe the book with the ISBN number "0836217462" and only this one, we could replace the "text" pattern by a "value" pattern and write:

    <element name="isbn">
     <value>0836217462</value>
    </element>

or, with the compact syntax:

   element isbn {"0836217462"}

and the schema will validate a book with a ISBN number equal to "0836217462" and refuse any other ISBN number.

!!Co-occurrence constraints

A more common usage of "value" patterns, still out of the scope of an enumeration, is to define so-called "co-occurrence constraints" where the value of a node (often an attribute) changes the content model of another node (often an element). In our library, we have both the "author" and "character" elements have a close semantic. We may want to group them as two variations over a "person" element differentiated by a "type" attribute and write:

   <person id="CMS" type="author">
    <name>Charles M Schulz</name>
    <born>1922-11-26</born>
    <died>2000-02-12</died>
   </person>

and

   <person id="PP" type="character">
    <name>Peppermint Patty</name>
    <born>1966-08-22</born>
    <qualification>bold, brash and tomboyish</qualification>
   </person>

However, in this kind of schemas, it is often required to validate that the content models are different according to the value of the "type" attribute and this can be done using "value" patterns. If we still want that all the authors precede the characters, we can just update the definitions of the elements describing authors and characters and keep them in sequence in the definition of the book element:

   <element name="book">
    <attribute name="id"/>
    <attribute name="available"/>
    <element name="isbn">
     <text/>
    </element>
    <element name="title">
     <attribute name="xml:lang"/>
     <text/>
    </element>
    <zeroOrMore>
     <element name="person">
      <attribute name="type">
       <value>author</value>
      </attribute>
      <attribute name="id"/>
      <element name="name">
       <text/>
      </element>
      <element name="born">
       <text/>
      </element>
      <optional>
       <element name="died">
        <text/>
       </element>
      </optional>
     </element>
    </zeroOrMore>
    <zeroOrMore>
     <element name="person">
      <attribute name="type">
       <value>character</value>
      </attribute>
      <attribute name="id"/>
      <element name="name">
       <text/>
      </element>
      <element name="born">
       <text/>
      </element>
      <element name="qualification">
       <text/>
      </element>
     </element>
    </zeroOrMore>
   </element>

Or, using the compact syntax:

  element book {
   attribute id {text},
   attribute available {text},
   element isbn {text},
   element title {attribute xml:lang {text}, text},
   element person {
    attribute id {text},
    attribute type {"author"},
    element name {text},
    element born {text},
    element died {text}?}*,
   element person {
    attribute id {text},
    attribute type {"character"},
    element name {text},
    element born {text},
    element qualification {text}}*
  }

We will see the main restrictions of W3C XML Schema which are non restrictions for Relax NG in "Chapter 16: Determinism and Datatype Assignment", but we need to mention here that this schema and schemas expressing co-occurrence constraints in general cannot be expressed with W3C XML Schema since they are not "deterministic". To be totally accurate, I have shown in my book "XML Schema" that some of them can be expressed in W3C XML Schema using "xs:key" as a tricky hack, but this doesn't work for the general case and this isn't something easy to implement in a schema translator.

However, if we are now using a single "person" element, this is probably to develop the interoperability between these elements and we may prefer allow to mix the definitions of characters and authors and express this part of the schema as zero or more "person" elements having two possible definitions such as:

   <element name="book">
    <attribute name="id"/>
    <attribute name="available"/>
    <element name="isbn">
     <text/>
    </element>
    <element name="title">
     <attribute name="xml:lang"/>
     <text/>
    </element>
    <zeroOrMore>
     <element name="person">
      <choice>
       <group>
        <attribute name="type">
         <value>author</value>
        </attribute>
        <attribute name="id"/>
        <element name="name">
         <text/>
        </element>
        <element name="born">
         <text/>
        </element>
        <optional>
         <element name="died">
          <text/>
         </element>
        </optional>
       </group>
       <group>
        <attribute name="type">
         <value>character</value>
        </attribute>
        <attribute name="id"/>
        <element name="name">
         <text/>
        </element>
        <element name="born">
         <text/>
        </element>
        <element name="qualification">
         <text/>
        </element>
       </group>
      </choice>
     </element>
    </zeroOrMore>
   </element>

Or:

   element person {
    (
     attribute id {text},
     attribute type {"author"},
     element name {text},
     element born {text},
     element died {text}?
    ) | (
     attribute id {text},
     attribute type {"character"},
     element name {text},
     element born {text},
     element qualification {text}
    )}

Now that we have brought closer the definitions of the two contents for the "person" element, we see that an attribute and the two first sub-elements are common and can be "factorized". The definition of the "person" element can thus be simplified as:

     <element name="person">
      <attribute name="id"/>
      <element name="name">
       <text/>
      </element>
      <element name="born">
       <text/>
      </element>
      <choice>
       <group>
        <attribute name="type">
         <value>author</value>
        </attribute>
        <optional>
         <element name="died">
          <text/>
         </element>
        </optional>
       </group>
       <group>
        <attribute name="type">
         <value>character</value>
        </attribute>
        <element name="qualification">
         <text/>
        </element>
       </group>
      </choice>
     </element>

or:

   element person {
    attribute id {text},
    element name {text},
    element born {text},
    ((
     attribute type {"author"},
     element died {text}?
    ) | (
     attribute type {"character"},
     element qualification {text}
    ))
   }


In the compact syntax first, note that we have had to use a double parenthesis to express our choice. This is because the operators used at any level must be homogeneous (you can't mix ",", "|" and "&" at a same level since it would be ambiguous). The other thing to note is that if we have been able to "factorize" the "id" attribute and the name and born elements and keep the "type" attribute and its two possible values in the choice, this is because we have been able to group the elements which are different with the attribute used to do the distinction between content models and this has been possible not only because the example had been carefully prepared for this but also because of the semantic of implicit "interleave" given to the "attribute" patterns which lets us locate the attribute either in or out of the choice. Finally, we should note that this factorization is just a syntactical variation and that even when such simplification is impossible the co-occurrence constraint may still be expressed even though it would have been more verbose.

!!Enumerations

We have in hand all the elements needed to define enumerations: what is an enumeration if not a choice between several values? Enumerations are this written in Relax NG through choices of values. A good candidate for an enumeration in our library is the "available" attribute which could be defined as:

    <attribute name="available">
     <choice>
      <value>available</value>
      <value>checked out</value>
      <value>on hold</value>
     </choice>
    </attribute>

or:

   attribute available {"available"|"checked out"|"on hold"}

As expected, this definition will validate values such as "available", "checked out" and "on hold", but also and this might be more unexpected , values such as " available ", "checked out     " or even " on     hold " with multiple spaces, tabs or CRs between "on" and "hold" and we will see the reason of this behavior -and how to change it if needed- in the next section.

!!Whitespaces and native datatypes

We use to say that Relax NG doesn't natively support datatypes and, strictly speaking, this assertion is not totally accurate: Relax NG does include a native type system but this type library is extremely weak and consists of only two datatypes (token and string) which only differ by the whitespace processing applied before the validation. The whole Relax NG datatype system can be seen as the ability to add validating transformations to text nodes. These transformations transform text nodes into canonical formats (ie a formats where all the different formats for a same value are converted into a single normalized or "canonical" format) and eventually detect format errors. The two native datatypes do not detect format errors (their formats are broad enough to allow any value) but still transform text nodes in their canonical forms and this makes a difference for enumerations.

Enumerations are thus the first place, before next chapter where we will see more complete datatype libraries, where we will see datatypes at work and this is simply done by adding a "type" attribute in "value" patterns.

Up to now, we haven't specified any datatype when we've written "value" elements and by default, Relax NG consider that they have the default type "token" of the built-in library. The transformation performed on text values per this datatype is a full whitespaces normalization similar to the one which is performed by the XPath "normalize-space()" function: all the sequences of one or more whitespaces -ie characters #x20 (space) #x9 (tab), #xA (linefeed) and #xD (carriage return)- are replaced by a single space and the leading and trailing space is then trimmed.

If we come back on previous examples, writing:

    <attribute name="available">
     <choice>
      <value>available</value>
      <value>checked out</value>
      <value>on hold</value>
     </choice>
    </attribute>

or:

   attribute available {"available"|"checked out"|"on hold"}

has been using the default type value (token) and was equivalent to the following:

 <attribute name="available">
  <choice>
   <value type="token">available</value>
   <value type="token">checked out</value>
   <value type="token">on hold</value>
  </choice>
 </attribute>

or:

 attribute available {token "available"|token "checked out"|token "on hold"}

When the "token" datatype is used, the string normalization is applied on the value defined in the schema and to the value found in the instance document and the comparison is done on the result of the normalization, which explains why we have seen that "on hold" was matching "  on   hold   " with spaces or tabulations added before, between and after the words.

To suppress this normalization, we can use the second builtin datatype ("string") which doesn't perform any transformation on the values and write:

 <attribute name="available">
  <choice>
   <value type="string">available</value>
   <value type="string">checked out</value>
   <value type="string">on hold</value>
  </choice>
 </attribute>

or:

 attribute available {string "available"|string "checked out"|string "on hold"}

With this new definition, the value of our attribute must match exactly the value specified in the schema, ie the strings "available", "checked out" and "on hold".

Note that the native "token" and "string" datatypes have the same definition than the W3C XML Schema "token" and "string" datatypes with the difference than, as we will see in the next chapter "Chapter 8: Datatype Libraries", the additional restrictions which can be applied through "param" attributes to W3C XML Schema datatypes are not available with the native datatypes.

Also not that the name of the "token" datatype, borrowed to W3C XML Schema is highly confusing. In the general meaning in our IT jargon, a token is a piece of string between two delimiters, i.e. what we would call a "word" in plain English. The "token" datatype is not a word, otherwise "on" and "hold" would be valid tokens but "on hold" wouldn't be one. The "token" datatype is more a "tokenized" datatype, in the sense that it's a string made ready to be easily cut into token because non significant whitespaces have been removed. This confusion is dangerous since it's leading many people into using the "string" datatype when what they really need is "token" (we will see in a next section that using the "string" datatype should be reserved to very specific cases).

!!Beware of string datatypes in attributes

The fact that no whitespace normalization is performed when the builtin "string" datatype is used may lead to some surprises when we define attributes since the XML parsers must remove the line feeds and carriage returns which they find there and since the value is defined as a text node in the Relax NG which will not be submitted to the same treatment.

This behavior may be confusing in "both ways" and our previous schema defining that the attribute must match "on hold" will always match an attribute where the space between "on" and "hold" will have been replaced by a line feed such as in:

 <book id="b0836217462" available="on
 hold">

This is rather normal since any XML parser must send an attribute's value "on hold" in this case and no schema language will change this. The reverse is also possible and if we had written our schema as:

 <attribute name="available">
  <choice>
   <value type="string">available</value>
   <value type="string">checked out</value>
   <value type="string">on
 hold</value>
  </choice>
 </attribute>

The compact syntax doesn't allow new lines within quotes and to translate this into the compact syntax, we need to introduce a couple of new features to show two different possibilities to include new lines in values.

The first one is borrowed from Python and if instead of using simple (') or double (") quotes, you use three simple (''') or three double (""") quotes, you can include pretty much everything in your values including new lines:

 attribute available {string "available"|string "checked out"|string """on
 hold"""}

or:

 attribute available {string "available"|string "checked out"|string '''on
 hold'''}

The second way is through escaping the new line character using the syntax "\x{A}" (where "A" is the Unicode value of new line in hexadecimal):

 attribute available {string "available"|string "on hold"|string "on\x{A}hold"}

This pattern defines that the attribute could contain a value with a line feed, ie something which can only happen in XML if the newline in the attribute is explicitly specified through is numeric value, such as in:

 <book id="b0836217462" available="who&#x0A;knows?">

!!Rule of thumb about string datatypes

That's amazing when you think about all the complex applications that have been made possible by SGML and XML, but something apparently as simple as whitespace processing has  consistently been a nightmare for users and programmers.

The "string" datatype will expose you to all the issues related to whitespace handling: a wide class of users and applications will make their best to either remove or add whitespaces in your documents and your document will become invalid.

The "token" datatype will keep you much less exposed to this nightmare and that's the reason why Relax NG has chosen it as its default datatype but that's true for W3C XML Schema datatype too: you shouldn't use the "string" datatype unless you have a good reason to do so.

Examples of good reasons to use the "string" datatype include all the variety of program listings in which you want to preserve whitespaces.


!!Using different types in each value

In our previous schema, we have have been obliged to define the type for each "value" pattern:

 attribute available {string "available"|string "checked out"|string "on hold"}

This is both verbose and very powerful and there is no restriction forbidding to use different datatypes in the different alternatives of an enumeration. While this would be confusing with our two builtin types to accept "available" as a "token" and "checked out" as a "string", this will become something more useful in the next chapter when we will have more datatypes at hand and could specify for instance that we want to accept either "true" as a boolean or "on hold" as a token.

!!Exclusions

What if, instead of giving a list of values which are allowed, I want to give a list of values which are forbidden? The "except" pattern has been designed for this purpose, and to exclude the value "0836217462" from the possible ISBN numbers, we would write:

    <element name="isbn">
     <data type="token">
      <except>
       <value>0836217462</value>
      </except>
     </data>
    </element>

Or, using the compact syntax:

   element isbn {token - "0836217462"}

Although this looks simple, we must note that the type can be defined at two different levels here: it must be defined in the "data" pattern and may be defined in the "value" pattern and these two definitions have a different meaning. The type attached to the "data" pattern defines a validation performed on the text node and the type attached to the "value" pattern defines how the value should be interpreted and which whitespace processing should be performed.

In our example, both types are "token" and values such as " 0836217462 " would be excluded as well as "0836217462". The two datatypes could also be different such as in:

    <attribute name="available">
     <data type="token">
      <except>
       <choice>
        <value type="string">available</value>
        <value type="string">checked out</value>
        <value type="string">on hold</value>
       </choice>
      </except>
     </data>
    </attribute>

Or, using the compact syntax:

 attribute available {token -(string "available"|string "checked out"|string "on hold")}

And in this case, the first control would be done on the datatype "token" and the comparison would use the datatype "string".

!!Lists

Relax NG supports the description of text nodes which are lists of whitespace separated values through the "list" pattern which is pretty non typical in that it can be seen as the only pattern which transforms the structure of the document at validation time by splitting text values into lists of tokens ("token" being used here in the meaning of words, not in the meaning of the "token" datatype). The benefit of doing so is that within a "list" pattern, all the pattern constraining data values may be used combined with the compositors which will let us constrain the combination of these tokens.

Funny enough, if we use a "list" pattern without defining a cardinality, we may not get what we expect. An attribute defined as:

    <attribute name="see-also">
     <list>
      <data type="token"/>
     </list>
    </attribute>

or, using the compact syntax:

  attribute see-also {list {token}}

Would not match a list of tokens (such as see-also="0345442695 0449220230 0449214044 0061075647 0061075612") but a list of exactly one token (such as see-also="0345442695"). This is because the "list" pattern splits the text value into a list of token and this list is then evaluated against the patterns which are included within the "list" pattern. If we want a list of any number of "tokens", we must thus use a "zeroOrMore" pattern to express it:

    <attribute name="see-also">
     <list>
      <zeroOrMore>
       <data type="token"/>
      </zeroOrMore>
     </list>
    </attribute>

Or, using the compact syntax:

 attribute see-also {list {token*}}

This would consider the "see-also" attribute as a list of tokens and wouldn't add any constraint (this will of course be different when we will have more datatypes) but we could have used other compositors in the "list" pattern exactly as we have used them in other contexts. To express that we want a list with between 1 to 4 tokens, we could say:

    <attribute name="see-also">
     <list>
       <data type="token"/>
       <optional>
        <data type="token"/>
       </optional>
       <optional>
        <data type="token"/>
       </optional>
       <optional>
        <data type="token"/>
       </optional>
     </list>
    </attribute>

Or, using the compact syntax:

  attribute see-also {list {token, token?, token?, token?}}

That's verbose but we have already seen that we had no other possibilities to define number of occurrences with RelaxNG...

We could also constraint the values of these tokens, for instance through an enumeration:


    <attribute name="see-also">
     <list>
      <oneOrMore>
       <choice>
        <value>0836217462</value>
        <value>0345442695</value>
        <value>0449220230</value>
        <value>0449214044</value>
        <value>0061075647</value>
       </choice>
      </oneOrMore>
     </list>
    </attribute>

Or:

   attribute see-also {list {("0836217462"|"0345442695"|"0449220230"|"0449214044"|"0061075647")+}}

A last point to note is that this mechanism gives us the ability to define different constraints for the different members of a list. To illustrate this feature, we need to take another example. Let's say that we want to give the dimension of a book by giving its three dimensions and a unit, such as:

  <book id="b0836217462" available="true" dimensions="0.38 8.99 8.50 inches">

In this case, we can define the "dimensions" attribute as:

    <attribute name="dimensions">
     <list>
      <data type="token"/>
      <data type="token"/>
      <data type="token"/>
      <choice>
       <value>inches</value>
       <value>cm</value>
       <value>mm</value>
      </choice>
     </list>
    </attribute>

Or:

   attribute dimensions {list {token, token, token, ("inches"|"cm"|"mm")}}


!!Data versus text

In the previous chapter (Chapter 6: More patterns) we have spend a lot of time to give a detailed description of the "text" pattern and of its behavior within "interleave" patterns. There is another pattern to describe and attach datatypes to text nodes and even though this pattern will become useful with the introduction of the datatype libraries in next chapter, we can describe its core features right now to be exhaustive on the subject of text nodes.

This pattern is the "data" pattern. The data pattern accepts a "type" attribute (like we have seen for the "value" pattern) and checks that the value is valid per this type. Since our two builtin types accept any value, the "data" pattern with builtin types is almost equivalent to a "text" pattern... Almost only because the "data" pattern does not mean like the "text" pattern "zero or more text nodes" but "one text node" and also because the "data" pattern has been designed to represent data and that it is forbidden in mixed content models since the authors of the Relax NG specification have considered that this is a bad practice.

This restriction apply to all the patterns matching a single text node (ie, "data", "value" and "list") which can never be associated with patterns sibling matching elements (ie elements which could have add the same parent element in the same instance document). In practice, this means that we won't be able to use a "data" pattern describe content models such as:

 <price><currency>USD</currency>20</price>

or

 <price>20<currency>Euro</currency></price>

These content models have been considered bad practice by the authors of the Relax NG specification who advise to reformulate them as:

 <price>
  <amount>20</amount>
  <currency>USD</currency>
 </price>

or

 <price currency="USD">20</price>

This is the second time (the first one being when we've seen that there is no "unordered non interleaved" pattern in the previous chapter) that we see Relax NG giving a priority to good practices over the ability to describe all the combinations possible per the XML recommendation. This second case is actually increasing the complexity of the implementations of Relax NG processors which must check that "data" patterns are not included directly or not within mixed content models while the support of data in mixed content models would have been implied by the general algorithms without any additional complexity. The only benefit for Relax NG processors is that they can skip whitespaces occurring between two elements since they cannot match a "data" element which is forbidden between two elements but this benefit seems really minimal compared to the possibilities which are lost by this restriction.

This restriction appears to come from a strict distinction between data oriented and document oriented applications of XML, mixed contents having been considered to belong to document oriented applications which shouldn't need datatypes and datatypes to belong to data oriented applications which shouldn't need mixed contents!
