CSF: Formative Feedback in Autograding

2021 ◽  
Vol 21 (3) ◽  
pp. 1-30
Author(s):  
Georgiana Haldeman ◽  
Monica Babeş-Vroman ◽  
Andrew Tjang ◽  
Thu D. Nguyen
Keyword(s):  
Formative Feedback ◽  
Test Suite ◽  
Knowledge Map ◽  
Test Cases ◽  
Not Given ◽  
Average Accuracy ◽  
Comprehensive Test ◽  
Functional Correctness ◽  
Logical Errors ◽  
Future Work

Autograding systems are being increasingly deployed to meet the challenges of teaching programming at scale. Studies show that formative feedback can greatly help novices learn programming. This work extends an autograder, enabling it to provide formative feedback on programming assignment submissions. Our methodology starts with the design of a knowledge map, which is the set of concepts and skills that are necessary to complete an assignment, followed by the design of the assignment and that of a comprehensive test suite for identifying logical errors in the submitted code. Test cases are used to test the student submissions and learn classes of common errors. For each assignment, we train a classifier that automatically categorizes errors in a submission based on the outcome of the test suite. The instructor maps the errors to corresponding concepts and skills and writes hints to help students find their misconceptions and mistakes. We apply this methodology to two assignments in our Introduction to Computer Science course and find that the automatic error categorization has a 90% average accuracy. We report and compare data from two semesters, one semester when hints are given for the two assignments and one when hints are not given. Results show that the percentage of students who successfully complete the assignments after an initial erroneous submission is three times greater when hints are given compared to when hints are not given. However, on average, even when hints are provided, almost half of the students fail to correct their code so that it passes all the test cases. The initial implementation of the framework focuses on the functional correctness of the programs as reflected by the outcome of the test cases. In our future work, we will explore other kinds of feedback and approaches to automatically generate feedback to better serve the educational needs of the students.

scholarly journals Test Case Reduction Using Ant Colony Optimization for Object Oriented Program

2015 ◽  
Vol 5 (6) ◽  
pp. 1424
Author(s):  
Sudhir Kumar Mohapatra ◽  
Srinivas Prasad
Keyword(s):  
Ant Colony Optimization ◽  
System Development ◽  
Object Oriented ◽  
Ant Colony ◽  
Test Suite ◽  
Test Case ◽  
Test Cases ◽  
Reduction Techniques ◽  
Maintenance Stage ◽  
The Cost

Software testing is one in all the vital stages of system development. In software development, developers continually depend upon testing to reveal bugs. Within the maintenance stage test suite size grow due to integration of new functionalities. Addition of latest technique force to make new test case which increase the cost of test suite. In regression testing new test case could also be added to the test suite throughout the entire testing process. These additions of test cases produce risk of presence of redundant test cases. Because of limitation of time and resource, reduction techniques should be accustomed determine and take away. Analysis shows that a set of the test case in a suit should satisfy all the test objectives that is named as representative set. Redundant test case increase the execution price of the test suite, in spite of NP-completeness of the problem there are few sensible reduction techniques are available. During this paper the previous GA primarily based technique proposed is improved to search out cost optimum representative set using ant colony optimization.

Comparison of Multi-Objective Evolutionary Algorithms to Prioritize Regression Test Cases

2021 ◽  
Vol 9 (12) ◽  
pp. 1445-1454
Keyword(s):  
Fault Detection ◽  
Evolutionary Algorithms ◽  
Fitness Function ◽  
Optimal Solution ◽  
Test Suite ◽  
Test Case ◽  
Test Cases ◽  
Nsga Ii ◽  
Multi Objective ◽  
Regression Test

Regression testing is one of the most critical testing activities among software product verification activities. Nevertheless, resources and time constraints could inhibit the execution of a full regression test suite, hence leaving us in confusion on what test cases to run to preserve the high quality of software products. Different techniques can be applied to prioritize test cases in resource-constrained environments, such as manual selection, automated selection, or hybrid approaches. Different Multi-Objective Evolutionary Algorithms (MOEAs) have been used in this domain to find an optimal solution to minimize the cost of executing a regression test suite while obtaining maximum fault detection coverage as if the entire test suite was executed. MOEAs achieve this by selecting set of test cases and determining the order of their execution. In this paper, three Multi Objective Evolutionary Algorithms, namely, NSGA-II, IBEA and MoCell are used to solve test case prioritization problems using the fault detection rate and branch coverage of each test case. The paper intends to find out what’s the most effective algorithm to be used in test cases prioritization problems, and which algorithm is the most efficient one, and finally we examined if changing the fitness function would impose a change in results. Our experiment revealed that NSGA-II is the most effective and efficient MOEA; moreover, we found that changing the fitness function caused a significant reduction in evolution time, although it did not affect the coverage metric.

Test Pair Selection for Test Case Prioritization in Regression Testing for WS-BPEL Programs

2013 ◽  
Vol 10 (1) ◽  
pp. 73-102 ◽  
Author(s):  
Lijun Mei ◽  
Yan Cai ◽  
Changjiang Jia ◽  
Bo Jiang ◽  
W.K. Chan
Keyword(s):  
Web Services ◽  
Structural Similarity ◽  
Regression Testing ◽  
Test Pair ◽  
Test Suite ◽  
Test Case ◽  
Test Cases ◽  
Test Case Prioritization ◽  
Empirical Results ◽  
Selection For

Many web services not only communicate through XML-based messages, but also may dynamically modify their behaviors by applying different interpretations on XML messages through updating the associated XML Schemas or XML-based interface specifications. Such artifacts are usually complex, allowing XML-based messages conforming to these specifications structurally complex. Testing should cost-effectively cover all scenarios. Test case prioritization is a dimension of regression testing that assures a program from unintended modifications by reordering the test cases within a test suite. However, many existing test case prioritization techniques for regression testing treat test cases of different complexity generically. In this paper, the authors exploit the insights on the structural similarity of XML-based artifacts between test cases in both static and dynamic dimensions, and propose a family of test case prioritization techniques that selects pairs of test case without replacement in turn. To the best of their knowledge, it is the first test case prioritization proposal that selects test case pairs for prioritization. The authors validate their techniques by a suite of benchmarks. The empirical results show that when incorporating all dimensions, some members of our technique family can be more effective than conventional coverage-based techniques.

An Effective Approach to Test Suite Reduction and Fault Detection Using Data Mining Techniques

2017 ◽  
Vol 8 (4) ◽  
pp. 1-31 ◽  
Author(s):  
B. Subashini ◽  
D. Jeya Mala
Keyword(s):  
Data Mining ◽  
Attribute Reduction ◽  
Test Suite ◽  
Test Cases ◽  
Data Mining Techniques ◽  
Coverage Analysis ◽  
Quality Product ◽  
Irrelevant Attributes ◽  
Test Suite Reduction ◽  
Using Data

Software testing is used to find bugs in the software to provide a quality product to the end users. Test suites are used to detect failures in software but it may be redundant and it takes a lot of time for the execution of software. In this article, an enormous number of test cases are created using combinatorial test design algorithms. Attribute reduction is an important preprocessing task in data mining. Attributes are selected by removing all weak and irrelevant attributes to reduce complexity in data mining. After preprocessing, it is not necessary to test the software with every combination of test cases, since the test cases are large and redundant, the healthier test cases are identified using a data mining techniques algorithm. This is healthier and the final test suite will identify the defects in the software, it will provide better coverage analysis and reduces execution time on the software.

scholarly journals Enhancing Accuracy in a Touch Operation Biometric System: A Case on the Android Pattern Lock Scheme

2020 ◽  
Vol 2020 ◽  
pp. 1-12
Author(s):  
Allan Ng’ang’a ◽  
Paula M. W. Musuva
Keyword(s):  
Repeated Measures ◽  
Smart Devices ◽  
Biometric System ◽  
Data Set ◽  
Pattern Complexity ◽  
User Training ◽  
Average Accuracy ◽  
Within Subjects ◽  
Dynamic Time ◽  
Future Work

The main objective of this research study is to enhance the functionality of an Android pattern lock application by determining whether the time elements of a touch operation, in particular time on dot (TOD) and time between dot (TBD), can be accurately used as a biometric identifier. The three hypotheses that were tested through this study were the following–H1: there is a correlation between the number of touch stroke features used and the accuracy of the touch operation biometric system; H2: there is a correlation between pattern complexity and accuracy of the touch operation biometric system; H3: there is a correlation between user training and accuracy of the touch operation biometric system. Convenience sampling and a within-subjects design involving repeated measures were incorporated when testing an overall sample size of 12 subjects drawn from a university population who gave a total of 2,096 feature extracted data. Analysis was done using the Dynamic Time Warping (DTW) Algorithm. Through this study, it was shown that the extraction of one-touch stroke biometric feature coupled with user training was able to yield high average accuracy levels of up to 82%. This helps build a case for the introduction of biometrics into smart devices with average processing capabilities as they would be able to handle a biometric system without it compromising on the overall system performance. For future work, it is recommended that more work be done by applying other classification algorithms to the existing data set and comparing their results with those obtained with DTW.

Optimal Test Case Selection Using Ant Colony and Rough Sets

2020 ◽  
Vol 11 (2) ◽  
pp. 1-14
Author(s):  
Angelin Gladston ◽  
Niranjana Devi N.
Keyword(s):  
Optimization Technique ◽  
Computational Effort ◽  
Ant Colony ◽  
Test Suite ◽  
Test Case ◽  
Test Cases ◽  
Optimal Test ◽  
Average Percentage ◽  
Test Case Selection ◽  
Case Selection

Test case selection helps in improving quality of test suites by removing ambiguous, redundant test cases, thereby reducing the cost of software testing. Various works carried out have chosen test cases based on single parameter and optimized the test cases using single objective employing single strategies. In this article, a parameter selection technique is combined with an optimization technique for optimizing the selection of test cases. A two-step approach has been employed. In first step, the fuzzy entropy-based filtration is used for test case fitness evaluation and selection. In second step, the improvised ant colony optimization is employed to select test cases from the previously reduced test suite. The experimental evaluation using coverage parameters namely, average percentage statement coverage and average percentage decision coverage along with suite size reduction, demonstrate that by using this proposed approach, test suite size can be reduced, reducing further the computational effort incurred.

A Critical Review on Automated Test Case Generation for Conducting Combinatorial Testing Using Particle Swarm Optimization

2018 ◽  
Vol 7 (3.8) ◽  
pp. 22 ◽  
Author(s):  
Dr V. Chandra Prakash ◽  
Subhash Tatale ◽  
Vrushali Kondhalkar ◽  
Laxmi Bewoor
Keyword(s):  
Particle Swarm Optimization ◽  
Critical Review ◽  
Optimization Problem ◽  
Test Suite ◽  
Test Case Generation ◽  
Test Case ◽  
Test Cases ◽  
Combinatorial Testing ◽  
Swarm Optimization ◽  
Automated Test Case Generation

In software development life cycle, testing plays the significant role to verify requirement specification, analysis, design, coding and to estimate the reliability of software system. A test manager can write a set of test cases manually for the smaller software systems. However, for the extensive software system, normally the size of test suite is large, and the test suite is prone to an error committed like omissions of important test cases, duplication of some test cases and contradicting test cases etc. When test cases are generated automatically by a tool in an intelligent way, test errors can be eliminated. In addition, it is even possible to reduce the size of test suite and thereby to decrease the cost & time of software testing.It is a challenging job to reduce test suite size. When there are interacting inputs of Software under Test (SUT), combinatorial testing is highly essential to ensure higher reliability from 72 % to 91 % or even more than that. A meta-heuristic algorithm like Particle Swarm Optimization (PSO) solves optimization problem of automated combinatorial test case generation. Many authors have contributed in the field of combinatorial test case generation using PSO algorithms.We have reviewed some important research papers on automated test case generation for combinatorial testing using PSO. This paper provides a critical review of use of PSO and its variants for solving the classical optimization problem of automatic test case generation for conducting combinatorial testing.   

Regression Test Reduction for Object-Oriented Software: A Control Call Graph Based Technique and Associated Tool

2013 ◽  
Vol 2013 ◽  
pp. 1-10 ◽  
Author(s):  
Nicolas Frechette ◽  
Linda Badri ◽  
Mourad Badri
Keyword(s):  
Object Oriented ◽  
Control Flow ◽  
Regression Testing ◽  
Test Suite ◽  
Reduced Form ◽  
Test Cases ◽  
Call Graph ◽  
Call Graphs ◽  
Incremental Update ◽  
Flow Graphs

This paper presents a selective regression testing technique and an associated tool for object-oriented software. The technique is based on the concept of Control Call Graphs, which are a reduced form of traditional Control Flow Graphs. It uses static analysis of the source code of the program. The developed tool (1) identifies the Control Call Paths potentially impacted by changes, (2) selects, from an existing test suite, the appropriate test cases, and (3) generates new JUnit test cases for control call paths that are not covered by existing tests (new ones, or those whose structure has been modified after changes). In this way, the approach supports an incremental update of the test suite. The selected JUnit test cases, including the new ones, are automatically executed. Three concrete case studies are reported to provide evidence of the feasibility of the approach and its benefits in terms of reduction of regression testing effort.

Minimum Test Case for Discrete-Type Software Reliability Testing by Stratified Sampling

2012 ◽  
Vol 263-266 ◽  
pp. 1874-1879
Author(s):  
Qiu Ying Li ◽  
Lei Luo
Keyword(s):  
Software Reliability ◽  
Traditional Approach ◽  
Simple Random Sampling ◽  
Stratified Sampling ◽  
Necessary Condition ◽  
Test Case ◽  
Test Cases ◽  
Operational Profile ◽  
Future Work ◽  
Discrete Type

The traditional approach for generating software reliability test cases according to operational profile (OP) can be regarded as a simple random sampling. The stratified sampling is a complex but more accurate sampling approach which can make the sampling results more accurate and less variance by dividing the population into several subsets and randomly sampling from each subset respectively. Firstly, this paper introduced the traditional method which can be used to determine the number of test cases in the discrete-type software reliability demonstration testing. Secondly, the necessary condition for the minimum test case based on the stratified sampling was studied. Then a new approach was proposed and its principle was analyzed for determining the minimum test case of the discrete-type software by using the stratified sampling. Finally, the future work was discussed.

Sign in / Sign up

Export Citation Format

Share Document