There is no generic answer to questions such as "Which MT engine produces the best quality output?" or "What is the expected productivity increase from implementing MT?".

Much depends on the project context: language pair, domain, customisation efforts, source content but also the expectations that have been set. Running a pilot project can help companies calculate ROI and make informed decisions on which technology to deploy and how to customise in the most efficient way.

CrossLang has developed an MT evaluation methodology based on industry standards and supported by proprietary tools. MT pilot programs have been carried out for many companies which helped them choose the best solution for their particular needs – to this date no two programs have been the same as every company has a different set of parameters and objectives.

Quality

Quality evaluations are primarily based on human assessments with multiple unbiased informants as well as generated with statistical methods. The quality scores can be complemented by an error categorisation that provides more insight in the options for improving the quality of the output. The result is an objective analysis or transparent comparison of the participating engines, languages and customisation levels.

Productivity

The CrossLang post-editing environment measures the time spent by human resources correcting the MT output in order to compare the resulting throughput with the traditional, 100% manual translation approach. This allows companies to predict productivity increases and eventually cost savings.

Usability

Our usability programs aim at measuring whether the translated version of the content serves its purpose. In the case of online support content this is ultimately in solving the user's problem. It is possible that the user is not looking for perfect linguistic quality but just needs to understand the information. A usability evaluation measures the intelligibility of the translation by means of a reading comprehension test.