In this paper we present and discuss the results of the evaluation of an Intelligent Computer Assisted Language Learning (ICALL) system that operates over the Web. In particular, we aimed at evaluating the system along three dimensions: a) the effect of the intelligent features of the system on the learning outcome of students, b) the system's ability to provide individualized support to students that leads to more effective use of the system and c) the general usability and friendliness of the ICALL. To achieve this, we conducted an empirical study, where we compared the intelligent system with a non-intelligent version of it. The results of the study revealed that the students of the Web-based ICALL had gained more knowledge of the domain and had been able to interact with the system more effectively as compared to the students that had used the non-intelligent version of the system. However, the students of the intelligent version of the system found it more difficult and they needed more time to get acquainted with the system in comparison to the students of the non-intelligent system.