The EIS development has involved five staff members up to now, but the ministry says it plans to start outsourcing the development soon.

Developing the EIS began back in 2009. At that time, the national examination and qualification center (Eksami- ja kvalifikatsioonikeskus) concluded a procurement won by Fujitsu Estonia, at price tag of €211,548.

Henry Kattago, the deputy secretary general at the Ministry of Education and Research, told ERR he could not comment on the specifics of the situation at the time, as he did not work for the ministry then, while the public procurement system was different from the present-day one.

Kattago said: "Nowadays, relevant criteria are indeed applied in the execution of development procurement rounds like these, and there are various conditions set in respect of development teams, the systems themselves, development capabilities, timeline, and so on."

At present, EIS development work is not outsourced. According to Kattago, the system has been developed significantly over the years, but essentially it is the same system that it was initially intended to be.

The EIS development team is located within the technology management department at the Ministry of Education and Research.

As to why the Ministry handles the EIS development internally itself, and does not outsource it, Kattago said that since 2021, the IT systems the ministry manages have been consolidated under the purview of its technology management department.

"Unfortunately, EIS is the last such system being developed internally, and we have actually planned to switch to an external development partner this year," he said. "These decisions were made at the end of last year or the beginning of this year."

Thus, the development of EIS will be outsourced in the future. Until now, Kattago mentioned, the EIS team has consisted of a developer, a system administrator, two user support experts, and a product manager, with the product manager bearing the primary responsibility for the development and management of each technology product.

Kattago could not specify exactly how much the government has spent on EIS over the years, but in recent years, the annual expenditure has been about €300,000, a figure that has remained fairly consistent over the last five years.'

Procedural requirements have been developed, Kattago said, to ensure operability. This means that every IT product being developed or commissioned must be fully tested before being used.

He said: "This has been standard practice, standard procedure, but in this case, for some reason, it was omitted."

We now need to understand precisely why this happened, though initially, it seems likely due to human error, Kattago went on. "Whether someone overlooked something or made an error, but in any case, the processes are structured in a way that quality control takes place with every product before it goes live."

Kattago conceded that the EIS, given what happened on Saturday, when the joint tests for Tallinn and Harju County state high schools failed due to EIS glitches, had not been optimized for testing on such a scale.

This should have been identified during prior testing, he went on.

The EIS was initially created for administering national exams at basic schools (Põhikool, or junior high) rather than high school, and as a daily aid for teachers for conducting more day-to-day tests, pop quizzes etc.

Kattago said: "Overall, this objective has remained the same, then later, level exams, vocational exams, entrance exams, and so on came in as add-ons, meaning the demands on the system have multiplied over time."

How long it will take to thoroughly investigate the causes of the recent EIS outages is as yet unclear.

"It is clear today that an independent external expert-involved investigation process has to be initiated, in an attempt and a desire to clarify all the circumstances, review all the management chains, and decision-making processes, to identify how all these events were possible. Once the causes are identified, then we can likely move forward more effectively in the future," Kattago went on.

Kattago reiterated that essentially, the recent EIS malfunctions were due to human error.

Professor: Nothing fundamentally wrong with the EIS source code

Tanel Tammet, a professor at Tallinn University of Technology (TalTech), said that EIS source code, which is public, leaves a generally good impression, at first glance.

"I wouldn't say that the code looked either bad or good but in a peculiar way. It looked quite normal, fine and generally rational," the professor continued.

"Certainly, there are some aspects where it might get a bit complex, since, as always, people tend to go too easily down too complicated a path, but it is not hopelessly complex."

At the same time, that the system did not cope with the generated load came as no surprise to Tammet.

He said: "It's a little bit surprising that a mess like this took place; that this system, as it exists now, was actually launched into a live exam system at a critical moment. That does surprise me, and it shouldn't have happened."

Tammet said he believes there are three aspects at play in the saga.

The first of these is organizational.

He said: "Judging by the comments by the Ministry of Education people, who were probably aware of the issues, I got the impression that this system was not initially intended for such mass-scale exam administration; that it was designed for much smaller workloads, and for much more specific tasks. It simply wasn't designed with what it was later used for in mind."

Main problem was a lack of testing

Thus, Tammet suggested, people in the ministry looking for a solution might have seen that such a system already existed and therefore could be used. This in itself does not mean that things should go wrong, but the next point is pure load testing.

"If it was put in a role where suddenly a lot of students started entering their exams and if you fail to enter once, then you try frantically again, likely the load grew avalanche-like very easily," he said.

Tammet noted that it's quite difficult to systematically test when practically at the same time, a lot of requests come from outside, from different machines. Proper testing with high load is a complicated procedure and, in his opinion, the main problem that arose was that it was omitted.

Consequently, Tammet said he deduces that the people who made the decision to use the system did not have competence in fundamental IT, which in turn means the fault does not lie with the original developers.

Thirdly, Tammet pointed out that the system uses many tools which while they may be convenient for a developer, are not very efficient. The system it can be assumed, could work well on an ordinary computer or server and without the need for a large cluster or the parallel operation of many machines. These loads are not large, but the system would need to be optimized for a larger load such as that employed in the state high school entrance exams, so as to eliminate inefficient constructions, he said.

A certain amount of passing the buck, even unintentionally, will have happened too, he added.

"This is quite a common chain of events, whereby you have a system consisting of a group of people who communicate with each other, and mistakes happen. Someone assumes that someone else knows or is delving into an issue, but this someone doesn't in fact do so or know, then they in turn hope someone else does instead. To summarize, I think what we should actually grasp more is that there needs to be greater IT competence at government agencies; this is currently a bit lacking," Tammet concluded.

In northeastern Estonia, state high schools have said they will revert to paper-and-pen exams on April 20, to avoid a repeat of the crash that affected the exams in Tallinn last Saturday.

--

Follow ERR News on Facebook and Twitter and never miss an update!