reduce-historical/r38/packages/mathml/testing.tex

\chapter{Testing}

In order to confirm that the task of interfacing MathML and OpenMath has
been achieved, we must test the program in a variety of situations. We
will be ensuring that all guidelines described in Chapter \ref{analysis}
are properly adhered to. The testing will also prove the efficiency of
the overall design.

The most important feature to look for is that the translation process does not alter or modify the expressions being translated. Additionally, it
is essential that all results produced by the translator are compliant to the OpenMath and MathML standards. We will focus on ensuring semantic
value is preserved as well as making sure that CDs appear correctly next to symbols, MathML attributes are accurate, and that OpenMath symbols not
handled are properly dealt with. Finally we will test MathML outputs with the widespread MathML renderer: IBM TechExplorer.

We will aim to test varios specific aspects in more detail. We will describe the testing method and the results obtained.

The testing should verify whether the program accomplishes its task of translating OpenMath to MathML.

\section{Translation}

In order to verify whether the translation process is attained, we have
created a test suite comprising over 170 MathML and OpenMath examples,
more than 80 of each. Most of these examples come from the standard
specifications and others have been designed to test specific aspects
of the translator. Many of these examples were extensively used
throughout the implementation phase.

By running these examples through the translator, it was possible to
carefully check if the output produced for each example corresponded to
the expected result. This careful analysis, example by example,
has proven that in most cases, semantic value was preserved and a
proper translation was carried out.  However, it is important to
concentrate on the difficulties arising from translating OpenMath and
MathML.

\subsection{Content Dictionaries}

\paragraph{Aim:} Given that symbols from different CDs may have the
same name albeit different meanings, we must test the ability of the
translator to relate symbols to their CDs and recognize the difference
in meaning different CDs convey.

\paragraph{Testing method:} In the test suite of 170 examples, there
are a set of expressions testing the ability of the translator to
relate symbols to their CDs. These examples contain expressions with
symbols from different CDs but with the same name, to test the
translator's faculty to properly handle them. There are examples
where correct symbols appear next to wrong CDs and vice versa. The
translator should recognize only the valid CDs.

\paragraph{Results:} All examples were correctly treated. The
translator was capable of recognizing valid symbols by consulting the tables, translating them
accordingly. The results produced by these examples were very satisfactory.

\paragraph{Aim:} To test whether OpenMath expressions generated have
symbols appearing next to the correct CDs.

\paragraph{Testing Method:} Running all the examples contained in {\tt 
examples.mml} will generate a large number of OpenMath expressions.
By looking at them carefully we will see if symbols are correctly
related to their CDs.

\paragraph{Results:} All examples analysed were correct. The generated
symbols appeared next to the appropriate CDs. However, whenever a
MathML element mapped to a symbol which belonged to more than one CD
(like {\it inverse} for instance), there was no rule determining which
CD to place. Nevertheless, all translation were correct.

\subsection{MathML Attributes}

\paragraph{Aim 1:} In many cases attribute values modify the semantic
value of MathML expressions and must be taken into account when
translating. We would like to determine if the translator detects
attribute values and takes them into account when translating.

\paragraph{Aim 2:} Various OpenMath symbols map to MathML elements with
specific attribute values. Does the translator detect these symbols in order to 
generate MathML elements with their correct attribute values?

\paragraph{Testing Method:} We must gather all examples containing
elements whose attribute values convey semantic meaning. We translate
these elements from MathML to OpenMath and see if the semantic meaning
has been preserved. We then translate back to MathML and the translator
will have recognized these OpenMath elements and reproduced the
original MathML element with the correct attribute value.

\paragraph{Results:} All results were satisfactory. Attribute values were
recognized, correctly interpreted and semantic meaning was preserved in
all cases.

\subsection{Extensibility}

\paragraph{Aim:} To test how well the translator copes with OpenMath
symbols having no equivalent MathML element. We want to determine if they
are properly translated and preserve their structure and meaning.
\paragraph{Testing Method:} There are various types of situations where
extensibility mechanisms must be employed. These are:

\begin{itemize}

\item Both the CD and symbol are not recognized

\item The symbol is not recognized

\item The CD is not recognized

\end{itemize}

In these three cases, the translator must employ MathML \verb|<semantic>|
tags to preserve semantic meaning. We will tes expressions
based on these three cases.

\paragraph{Results:} The translator automatically detected the unknown
symbols and enveloped them inside \verb|<semantic>| tags. The translator
worked well in cases where the OpenMath symbol translated was constructed
by {\it application}.  An example of this is the {\tt rank} operator seen
in figure \ref{rank}. In other cases (such as {\it binding}) the results
were poor and in many cases incorrect.

\section{Standard Compliance}

For this translator to produce usable results it must imperatively conform to MathML and OpenMath standards. This implies that all expressions
produced must be lexically and syntactically correct according to the specifications.  We must examine that the translator can parse and generate
valid expressions.

\subsection{Parsing of Expressions}

\paragraph{Aim:} Valid MathML and OpenMath expressions must be parsed
without difficulty by the translator. Lexical and syntax errors must be
detected and attribute values must be extracted for use in translating.

\paragraph{Testing Method:} The translator should correctly parse a large
amount of valid MathML and OpenMath examples taken from the standards.
These examples are contained in the 170 examples mentioned earlier and are considered to be correct. The
World Wide Web Consortium also offers a test suite for testing
applications for MathML compliance. This test suite is a series of valid
MathML expressions, which if parsed prove that an application is MathML
compliant. Unfortunately the URL link to this test suite was broken
throughout the duration of this project and has not been able to be used.

We will also introduce incorrect expressions to see if the translator
declares them as erroneous.

\paragraph{Results:} The translator performs well. All supported
operators are correctly parsed. Syntax is validated in both standards, thus distinguishing amongst correct and incorrect expressions.
Unsupported elements as described in section ref{nosupport} cause the
translator to abort. The translator is MathML and OpenMath compliant.

\subsection{Generation of Expressions}

\paragraph{Aim:} To determine if expressions produced by the translator
are MathML and OpenMath compliant.

\paragraph{Testing Method 1:} In order to determine whether the OpenMath
expressions are compliant, we shall introduce them back into the
translator. If the translator correctly parses them then they are
compliant.  Additionally, we will check the expressions individually to
ensure they are correct.

\paragraph{Testing Method 2:} To determine if MathML expressions produced
are MathML compliant, we shall translate them back to OpenMath. If the
translator reads through them correctly then we can conclude the
expressions are compliant.  Furthermore, we will try and render the
generated MathML expressions using IBM's TechExplorer. If TechExplorer
renders all MathML expressions generated then we have more reason's to
confirm that the MathML output is MathML compliant. 

\paragraph{Results:} The testing procedures all produced satisfying results. The translator's output can be parsed by itself validating the
expressions. Furthermore, IBM's TechExplorer successfully parsed and rendered a large number
of generated MathML1.0 expressions. \\

The results produced throughout these tests were mostly accurate, and when not, the translator was corrected. We recomend the user to run all the
examples in {\tt examples.om} and {\tt examples.mml} to get a better idea of the translation efficiency. It is possible that mistakes have pased
unnoticed. Reassuringly, the design is robust enough and most bugs should be quick to eliminate.

\section{Interface Limitations}

Testing of the interface has also revealed its limitations. Various
aspects of the translation process have not been properly solved. It is
important that we enumerate the areas where the translator performs
poorly.

In section \ref{boundvars} we mentioned that some OpenMath operators
encoded their bound variables within lambda expressions. We also said
that this was not compulsory. The translator however only deals with
expressions where the bound variable is within a lambda expression. Other
cases cause the translator to abort promptly.

The analysis in section \ref{scope} discussed the importance of defining
each operator's scope. The OpenMath/MathML translator gets confused when
scopes are ambiguous and aborts.

MathML element {\tt partialdiff} is not translated properly in most
cases. Only when the variables of differentiation have an order of
derivation equal to one. In all other cases the translator produces
incorrect results or aborts.

The translator will reject MathML expressions containing operators
defined within \verb|<semantic>| tags. This is clearly something which
will have to be implemented in the future, since the \verb|<semantic>|
tags may contain OpenMath code. 

The translator is capable of distinguishing incorrect expressions from valid ones. Unfortunately there was not enough time to implement a
constructive set of error messages. These error messages should have been able to quickly inform the user why a translation might have been
aborted, or what problems there are with the input impeding tranlation.


Finally, there is an aspect of REDUCE which limits the interface.
Contrary to XML, which is case sensitive as is stated in the XML
standard~\cite{xml}, REDUCE is case insensitive. Consequently when
translating an expression with variables or function names using
capital letters, REDUCE will produce only small letters.  This may in
some occasions create confusion for the user or even distort the
semantics.

\section{Conclusion}

Although 170 examples might not be enough to test the translator in all situations, they did demonstrate that the translator coped well with the
difficulties of OpenMath/MathML translation. Because these examples are a representative selection of most OpenMath and MathML operators and
situations, the satisfying results confirm that the task of accurate translating has been achieved.