The TOIA - Technologies for Online Interoperable Assessment - project has just completed its beta testing stage and so we thought we would give it a spin. First a caveat, I'm not someone who claims to be an expert in CAA so my evaluation is that of a real tyro … but I guess that's why I was asked to give my opinion. The perspective of a more experienced CAAist will be provided by a colleague in a later Auricle article. The release date for the project is July 2004. The TOIA Project is an eagerly awaited project and is funded by JISC as part of the Exchange for Learning (X4L) Programme. With the big player in the CAA market, Questionmark Perception dominating, there are great hopes for TOIA, not least because it is free to higher and further education.
My first contact with the TOIA software was as an instructor and the opening menu of items was presented. Where next? The TOIA Web site provides a Quick Start Guide for new Users. As with guides of this type you do get the chance to walk through the process of creating and publishing an assessment. However this does not really provide a novice instructor with sufficient skills to be able to create and manage online assessments.
At this point my next port of call would normally be the full documentation for the software but none appears to be available, at least not from the TOIA Web site. At this stage of a major software project the lack of substantial documentation does concern me. Producing such material is a non-trivial exercise. I do hope that the software is not released before all the documentation is to hand.
Having done the 'walk through 'a number of questions were raised in my mind. One was 'What is the difference between Manage Assessments and Manage Assignments?' which are options under the Assessments menu item. I accessed the Help menu but this does not have assignments as a help topic which is very surprising. Perhaps I was missing the obvious, but I think not, as a colleague also had the same difficulty. I found the answer to the difference in a question type but my point in raising this is not to pick a particular issue, but as a user of the software I need answers and the lack of documentation and unhelpful Help does not provide what I need.
There's many things I like about TOIA. The student view of the TOIA interface is clean and uncluttered. When a student logs on to TOIA he/she is presented with an assessment schedule made available by the instructor. The taking of the test is also very straightforward. Results are available to the student, but only if the assessment author has set this at the time the assessment was created.
Authoring questions is relatively straight forward. The version that has been available for beta testing has nine question types namely Multiple choice, Multiple response, Pull-down list, Fill in the blanks, Matrix, Essay, Macromedia flash, Java applet, File Upload. Two notable exceptions are Hot Spot and Drag and Drop. These and other question types can be purchased but the TOIA Web site does not show what the costs are likely to be. I did initially have a problem rendering questions using MathML because I was using the Sun Java engine. Disabling this engine cured the problem. Without listing the details there are other niggles which I am hopeful will be sorted out when the designers have had a chance to see the feedback from the beta testing.
My examination of TOIA has been an overview and it does seem to offer many of the features that are desirable in CAA software. However there seem to me to be a number of issues that need to be addressed to make it user friendly enough for most academics. It does not have a sufficiently large bank of question types to satisfy the needs of some authors although others can be purchased.
I feel that at present, as a novice CAAist, I need to develop more confidence and skills before using TOIA as my CAA software. Someone, more experienced in CAA software and services, however, may take a different view.
One final point. As mentioned earlier, this project is funded by the JISC Exchange for Learning (X4L) Programme until the end of July 2005. Does this mean that the TOIA software is free in its present form only until that date? This will be an important issue for an institution to consider before committing to implementing this CAA solution.
