My colleague, Graham Blacker's, recent article on the beta release of the new TOIA system hopefully gave readers an insight into how a person unfamiliar with Computer Aided Assessment might view the system. We threw him in at the deep end really:) Now it's my turn so read on. I'm the CAA person in our team and am currently co-ordinating a small project looking into the use of Questionmark and so it was good to have the opportunity to review TOIA.
As with Graham's previous article, the caveat with the following is that these comments apply to the beta version of the system. We have, along with others, passed our feedback along to the TOIA project team, and look forward to version 1.0 in July. Also, as Derek Morrison's comment to Graham's article stated the TOIA team deserve plaudits for the open way they make available the current iterations of the key project deliverable.
Many in the CAA community have been looking forward to the release of TOIA and I for one was interested in seeing if it was showing promise as a future alternative to the likes of Questionmark and WebMCQ.
Let's start - with a discussion about money.
TOIA is 'free' to all UK HEIs and FE colleges. That is, the basic version is free. The premium version, with whistles and bells we haven't yet seen (more questions, extended reporting, LDAP authentication and support), will apparently cost £15,000. Not an insignificant amount of money.
It's possible to argue, however, that this compares favourably with the sums involved in, say, a site-wide licence for Questionmark. Nevertheless, I think that there is a perception amongst some academic colleagues, that TOIA will be entirely free in all its forms.
TOIA's FAQ states that the:
“TOIA project is currently funded by JISC until October 2005 and we will be providing support until then. UK FE/HE users will be able to use the system indefinitely. If demand is sufficient we will endeavour to continue the support service and issue future upgrades. TOIA has been built in conjunction with Freedom2Learn Ltd which is likely to further develop the system in years to come.”
I'll leave readers to draw their own conclusion about what this means for the long term funding of TOIA.
But what about the system itself?
Graham has made some comments, so I'm not going to repeat those here but I'll give a brief overview and highlight what I think are some issues with the system in its present guise.
The system separates the creation of questions and assessments in a similar way to Questionmark.
Instead of a hierarchial structure of topics and subtopics built into the questions there is something called the 'Content Structure' which holds the hierarchial strucuture instead. Questions are then associated with a topic or subtopic within that content structure. Unfortunately, it appeared that it was not possible to do the same with assessments which means it's not that really that different to Questionmark. The documentation also infers that you can attribute learning objects within this structure which would be useful but the documentation didn't make it clear how I would go about that.
It's the acutal creation of questions and assessments which is more interesting; it's all web based. Great, I thought, at last something that will be accessible to those academics that are not windows based.
Uh, no. The web based system employs a WYSIWYG editor (HTML and MathML) which is fantastic. The downside is that this functionality is only available in Internet Explorer, because the editor functionality takes advantage of some of the Active X extensions that are available through IE. While it supports IE 5.0 and above, this does mean that a section of the HE community (such a Computer Science/Maths) may miss out.
TOIA’s FAQ states that the reason they choose this route is that IE is used by 95% of the online community. Well, that’s true at present but it’s looking like the next version of IE will be the last, and this still doesn’t take into account all those who choose (or may be told) to use Mozilla/Firefox as their browser. While I can fully appreciate the technical reasons for going down the Windows/IE route, I hope the TOIA team strive to develop cross platform or multi-platform solutions because I think this is an under developed area.
While I found it relatively easy, as a Questionmark user, to create questions and assessments, there were areas that tripped me up and I could see a novice having more problems, as Graham found out.
The adminstration and reporting are similar to Questionmark, in that you can batch upload students and run reports on their assessments. The manual assignment of students to groups could soon become a pain - but I think this is an area that both TOIA and Questionmark could work on. The reporting looks comparable to Questionmark but being caught up with other areas of the system didn't give me the opportunity to look into this area as closely as I would have liked and I'm interested to see what extended reports the Premium version will offer.
Overall, I think TOIA has great potential. In my humble opinion, it still has some usability issues to sort out with the interface, and I have to agree with Graham, it really needs to produce some serious in-depth documentation, but if it listens to it's users and gets these problems ironed out then it's going to give the likes of Questionmark some stiff competition and, for the UK HE and FE sector as consumers, that can only be a good thing.
