A Survey of Automated Programming Hint Generation: The HINTS Framework

Automated tutoring systems offer the flexibility and scalability necessary to facilitate the provision of high-quality and universally accessible programming education. To realise the potential of these systems, recent work has proposed a diverse range of techniques for automatically generating feedback in the form of hints to assist students with programming exercises. This article integrates these apparently disparate approaches into a coherent whole. Specifically, it emphasises that all hint techniques can be understood as a series of simpler components with similar properties. Using this insight, it presents a simple framework for describing such techniques, the Hint Iteration by Narrow-down and Transformation Steps framework, and surveys recent work in the context of this framework. Findings from this survey include that (1) hint techniques share similar properties, which can be used to visualise them together, (2) the individual steps of hint techniques should be considered when designing and evaluating hint systems, (3) more work is required to develop and improve evaluation methods, and (4) interesting relationships, such as the link between automated hints and data-driven evaluation, should be further investigated. Ultimately, this article aims to facilitate the development, extension, and comparison of automated programming hint techniques to maximise their educational potential.

Using rapid evaluation methods to assess service delivery changes: Lessons learned for evaluation practice during the COVID-19 pandemic

The COVID-19 pandemic required large-scale service delivery changes for government, and provided the opportunity for evaluators to step up and support decision makers to understand the impact of these changes. Rapid evaluation methods (REM) provide a pragmatic approach for generating timely information for evidence-based policy and decision-making. Grounded in developmental and utilisation-focused evaluation theory, REM incorporates a team-based, mixed methods design, executed over a 6–8-week period. Customised rubrics were used to rigorously assess effectiveness and scalability of practice changes to inform COVID-19 response planning. REM is an alternative approach to full-scale evaluation models frequently implemented to assess policies and programs. Adapted use of REM suggests that meaningful insights can be gained through use of smaller scale evaluations. This article shares lessons learned from a novel rapid evaluation method applied in the context of the COVID-19 pandemic. The rapid evaluation approach was implemented to provide real-time insights and evaluative conclusions for 15 program and practice adaptations across Victorian health and human service settings. The article shares insights about the practical applicability of balancing rigour and timeliness when implementing a rapid evaluation, and strengths and limitations of working within a fast-paced evaluation framework. Findings can inform evaluative practice in resource and time-limited settings.

Evaluation of Socially-Aware Robot Navigation

As mobile robots are increasingly introduced into our daily lives, it grows ever more imperative that these robots navigate with and among people in a safe and socially acceptable manner, particularly in shared spaces. While research on enabling socially-aware robot navigation has expanded over the years, there are no agreed-upon evaluation protocols or benchmarks to allow for the systematic development and evaluation of socially-aware navigation. As an effort to aid more productive development and progress comparisons, in this paper we review the evaluation methods, scenarios, datasets, and metrics commonly used in previous socially-aware navigation research, discuss the limitations of existing evaluation protocols, and highlight research opportunities for advancing socially-aware robot navigation.

Smart Urban Mobility System Evaluation Model Adaptation to Vilnius, Montreal and Weimar Cities

To date, there is no developed and validated way to assess urban smartness. When evaluating smart city mobility systems, different authors distinguish different indicators. After analysing the evaluation indicators of the transport system presented in the scientific articles, the most relevant and influential indicators were selected. This article develops a hierarchical evaluation model for evaluating a smart city transportation system. The indicators are divided into five groups called “factors”. Several indicators are assigned to each of the listed groups. A hybrid multi-criteria decision-making (MCDM) method was used to calculate the significance of the selected indicators and to compare urban mobility systems. The applied multi-criteria evaluation methods were simple additive weighting (SAW), complex proportional assessment (COPRAS), and technique for order preference by similiarity to ideal solution (TOPSIS). The significance of factors and indicators was determined by expert evaluation methods: the analytic hierarchy process (AHP), direct, when experts evaluate the criteria as a percentage (sum of evaluations of all criteria 100%) and ranking (prioritisation). The evaluation and comparison of mobility systems were performed in two stages: when the multi-criteria evaluation is performed according to the indicators of each factor separately and when performing a comprehensive assessment of the smart mobility system according to the integrated significance of the indicators. A leading city is identified and ranked according to the smartness level. The aim of this article is to create a hierarchical evaluation model of the smart mobility systems, to compare the smartness level of Vilnius, Montreal, and Weimar mobility systems, and to create a ranking.

Study on the extension evaluation of smart logistics distribution of fresh agricultural products based on the perspective of customers

The rapid development of the Internet has accelerated the expansion of e-commerce sacle of fresh agricultural products. The actual audience of smart logistics distribution of fresh agricultural products is customers, and customers enjoy the process and results of distribution services. However, the current research mainly selects indicators from the aspects of enterprise performance, cost and technical level based on the perspective of managers and technicians, which make it difficult to truly reflect customers’ feelings in the evaluation results. At the same time, the evaluation methods mainly focus on the comprehensive evaluation method and fuzzy evaluation method. These evaluation methods are greatly affected by subjective factors in the evaluation grade distribution, and the assignment is often relatively complete and inaccurate. To solve these problems, this paper constructs the evaluation index system of intelligent logistics distribution of fresh agricultural products from the perspective of customers, so that the selection of indicators is more in line with the real wishes of customers. And we use the extension function to construct the correlation function for multi-level extension evaluation to ensure the accuracy of the evaluation results. Taking X logistics enterprise as an example, this paper verifies the scientificity of the evaluation index system of intelligent logistics distribution of fresh agricultural products through empirical research, which has reference significance for further improving the intelligent logistics distribution of fresh agricultural products.

Research Progress of Bonding Agents and Their Performance Evaluation Methods

Bonding agents are an important type of additive that are used to increase the interfacial interaction in propellants. A suitable bonding agent can prevent the dewetting between the oxidant and binder, and thus effectively improve the mechanical properties of the propellant. In the current paper, the bonding mechanisms and research progress of different types of bonding agents such as alcohol amine bonding agents, borate ester bonding agents, aziridine bonding agents, hydantoin bonding agents, neutral polymer bonding agents, and so on, are reviewed and discussed. The evaluation methods of their bonding performances including molecular dynamic simulation, contact angle method, in situ loading SEM, characterization analysis, and mechanical analysis are summarized to provide design ideas and reference for future studies.