Detecting Overly Strong Preconditions in Refactoring Engines
Melina Mongiovi, Rohit Gheyi, Gustavo Soares, Márcio Ribeiro, Paulo Borba, and Leopoldo Teixeira
Refactoring engines may have overly strong preconditions preventing developers from applying useful transformations. We find that 32% of the assertions used in the Eclipse and JRRT refactoring test suites are concerned to overly strong preconditions. In general, developers manually write test cases, which is costly and error prone. Our previous technique automatically detects overly strong preconditions using differential testing. However, it needs at least two engines. In this work, we propose a new technique to identify overly strong preconditions in refactoring engines. We automatically generate programs and attempt to refactor each one. Next, we disable the execution of the preconditions that lead the engine to reject transformations. For each rejected transformation, we try to apply the transformation using the refactoring engine with some preconditions disabled. If it applies a behavior preserving transformation, we consider the disabled preconditions as overly strong. We evaluate 10 refactoring implementations of Eclipse and 10 of JRRT by generating 153,444 input programs. We find 30 overly strong preconditions (15 in each refactoring engine). Our new technique detects 11 new bugs that our previous technique cannot detect while missing 5 bugs. They are complementary in terms of bug detection. Finally, we evaluate our technique by replacing the generated programs with the input programs of Eclipse and JRRT test suites. We detect 15 overly strong preconditions in the refactoring implementations of Eclipse and 4 in the JRRT ones. Therefore, developers of refactoring engines may not have a systematic strategy to detect the overly strong preconditions.
Evaluation of our technique to identify overly strong preconditions by disabling some preconditions (DP technique)
Table I: Summary of the evaluation results of JRRT and Eclipse refactoring implementations; Refact. = Kind of Refactoring; Skip = Skip value used by JDOLLY to reduce the number of generated programs; GP = Number of Generated Programs by JDOLLY; CP = number of compilable programs (%); LOC coverage = rate of Lines of Code coverage for the set of generated programs (%); Branch coverage = rate of Branch coverage for the set of generated programs (%); No of assessed preconditions = Number of assessed refactoring preconditions in our study; LOC added in the refactoring engine = Lines of Code added in the refactoring engine to disable the refactoring preconditions; Overly Strong Preconditions = Number of detected overly strong preconditions in the refactoring implementations; Time (hr) = Total time to evaluate the refactoring implementations in hours; Time to First Failure (min) = Time to find the first failure in minutes; "na" = not assessed.
Table II: Summary of the evaluation results of the comparison of DP and DT techniques using input programs generated by JDOLLY; Refact. = Kind of Refactoring; Skip = Skip value used by JDOLLY to reduce the number of generated programs; GP = Number of Generated Programs by JDOLLY; DP Tech. = DP Technique; DT Tech. = DT Technique; Overly Strong Preconditions = Number of detected overly strong preconditions in the refactoring implementations; "na" = not assessed.
Table III: Summary of the evaluation results of the comparison of DP and DT techniques using input programs of Eclipse and JRRT refactoring test suite; Refactoring = Kind of Refactoring; Input programs = Number of selected input programs of the JRRT and Eclipse refactoring test suite; No of assessed conditions = Number of assessed refactoring preconditions in our study; LOC added in the refactoring engine = Lines of Code added in the refactoring engine to disable the refactoring preconditions; Overly Strong Preconditions = Number of detected overly strong preconditions in the refactoring implementations; DP = DP Technique; DT = DT Technique.
Table IV: Assessed conditions of JRRT. Refactoring = Kind of refactoring; Precondition = precondition checking; Message = reported message when the precondition is unsatisfied; OS (DP) = yes if the DP technique found this precondition as overly strong in this experiment, otherwise no.
Table V: Assessed conditions of Eclipse. Refactoring = Kind of refactoring; Precondition = precondition checking; Message = reported message when the precondition is unsatisfied; OS (DP) = yes if the DP technique found this precondition as overly strong in this experiment, otherwise no.
Figure I. Class diagram describing the classes that we create to allow disabling refactoring preconditions.
Alloy Specification:
Java meta-model specified in Alloy
Tools used in our experiment:
SafeRefactor
JDolly
Templates and aspects used to disable preconditions of Eclipse and JRRT:
Templates and Aspects
Eclipse and JRRT source codes:
Eclipse source code with all changes applied manually to allow disabling the execution of the assessed preconditions of our experiment
Eclipse source code with with aspects to disable the same code fragments of all preconditions we manually disabled using the DP transformations
JRRT source code with all changes applied manually to allow disabling the execution of the assessed preconditions of our experiment
JRRT source code with aspects to disable the same code fragments of all preconditions we manually disabled using the DP transformations
Bugs of Eclipse JDT detected by the DP technique using the programas generated by JDolly as test inputs:
434881: Move Method (New Bug)
Status: CLOSED DUPLICATE
434886: Move Method (New Bug)
Status: ASSIGNED
486694: Move Method (New Bug) Status: NEW (They did not answer yet)
399350: Pull Up Method Status: ASSIGNED
399788: Pull Up Method Status: ASSIGNED
462994: Pull Up Field (New Bug) Status: RESOLVED INVALID
391715: Rename Method Status: NEW (They did not answer yet)
391713: Rename Method Status: ASSIGNED
399181: Rename Method Status: RESOLVED INVALID
486693: Encapsulate Field (New Bug) Status: NEW (They did not answer yet)
399789: Add Parameter Status: ASSIGNED
486692: Add Parameter (New Bug) Status: NEW (They did not answer yet)
399347: Push Down Method Status: ASSIGNED
391710 : Rename Type Status: ASSIGNED
399183: Rename Type Status: CLOSED DUPLICATE
Bugs of Eclipse JDT detected by the DP technique using the test inputs of Eclipse refactoring test suite:
488811: Move Method (New Bug) Status: NEW (They did not answer yet)
488813: Move Method (New Bug) Status: NEW (They did not answer yet)
488812: Move Method (New Bug) Status: NEW (They did not answer yet)
491071: Pull Up Method Status: NEW (They did not answer yet)
499312: Rename Field (New Bug) Status: NEW (They did not answer yet)
499313: Rename Field (New Bug) Status: NEW (They did not answer yet)
499314: Rename Method Status: NEW (They did not answer yet)
499315: Rename Method (New Bug) (Two refactoring preconditions) Status: NEW (They did not answer yet)
499316: Rename Method Status: NEW (They did not answer yet)
499317: Add Parameter (New Bug) Status: NEW (They did not answer yet)
507614: Encapsulate Field (New Bug) Status: NEW (They did not answer yet)
499311: Pull Up Field (New Bug) Status: NEW (They did not answer yet)
507615: Pull Up Field (New Bug) Status: NEW (They did not answer yet)
New Bugs of JRRT:
1. Move Method (overly strong condition)
2. Move Method (overly strong condition)
Catalog of bugs detected by the MT technique
Bugs related to overly strong preconditions found in the bug tracker of Eclipse:
Bugs