* The framework offers various annotations to control how your test code should be invoked and being checked. There are * three kinds of tests depending on how much control is needed over the test invocation: * Base tests (see {@link Test}), checked tests (see {@link Check}), and custom run tests * (see {@link Run}). Each type of test needs to define a unique test method that specifies a {@link Test @Test} * annotation which represents the test code that is eventually executed by the test framework. More information about * the usage and how to write different tests can be found in {@link Test}, {@link Check}, and {@link Run}. *
* Each test method can specify an arbitrary number of IR rules. This is done by using {@link IR @IR} annotations which * can define regex strings that are matched on the output of {@code -XX:+PrintIdeal} and {@code -XX:+PrintOptoAssembly}. * The matching is done after the test method was (optionally) warmed up and compiled. More information about the usage * and how to write different IR rules can be found at {@link IR}. *
* This framework should be used with the following JTreg setup in your Test.java file in package some.package: *
* {@literal @}library /test/lib
* {@literal @}run driver some.package.Test
*
* Note that even though the framework uses the Whitebox API internally, it is not required to build and enabel it in the
* JTreg test if the test itself is not utilizing any Whitebox features directly.
* * To specify additional flags, use {@link #runWithFlags(String...)}, {@link #addFlags(String...)}, or * {@link #addScenarios(Scenario...)} where the scenarios can also be used to run different flag combinations * (instead of specifying multiple JTreg {@code @run} entries). *
* After annotating your test code with the framework specific annotations, the framework needs to be invoked from the * {@code main()} method of your JTreg test. There are two ways to do so. The first way is by calling the various * {@code runXX()} methods of {@link TestFramework}. The second way, which gives more control, is to create a new * {@code TestFramework} builder object on which {@link #start()} needs to be eventually called to start the testing. *
* The framework is called from the driver VM in which the JTreg test is initially run by specifying {@code * @run driver} in the JTreg header. This strips all additionally specified JTreg VM and Javaoptions. * The framework creates a new flag VM with all these flags added again in order to figure out which flags are * required to run the tests specified in the test class (e.g. {@code -XX:+PrintIdeal} and {@code -XX:+PrintOptoAssembly} * for IR matching). *
* After the flag VM terminates, it starts a new test VM which performs the execution of the specified * tests in the test class as described in {@link Test}, {@link Check}, and {@link Run}. *
* In a last step, once the test VM has terminated without exceptions, IR matching is performed if there are any IR * rules and if no VM flags disable it (e.g. not running with {@code -Xint}, see {@link IR} for more details). * The IR regex matching is done on the output of {@code -XX:+PrintIdeal} and {@code -XX:+PrintOptoAssembly} by parsing * the hotspot_pid file of the test VM. Failing IR rules are reported by throwing a {@link IRViolationException}. * * @see Test * @see Check * @see Run * @see IR */ public class TestFramework { /** * JTreg can define additional VM (-Dtest.vm.opts) and Javaoptions (-Dtest.java.opts) flags. IR verification is only * performed when all these additional JTreg flags (does not include additionally added framework and scenario flags * by user code) are whitelisted. * *
* A flag is whitelisted if it is a property flag (starting with -D), -ea, -esa, or if the flag name contains any of
* the entries of this list as a substring (partial match).
*/
public static final Set The {@code flags} override any set VM or Javaoptions flags by JTreg by default.
* Use {@code -DPreferCommandLineFlags=true} if you want to prefer the JTreg VM and Javaoptions flags over
* the specified {@code flags} of this method. If you want to run your entire JTreg test with additional flags, use this method. If you want to run your entire JTreg test with additional flags but for another test class then the one
* from which this method was called from, use {@link #addFlags(String...)}, use this method. If you want to run your JTreg test with multiple flag combinations, use
* {@link #addScenarios(Scenario...)}
* Use {@code -DPreferCommandLineFlags=true} if you want to prefer the VM or Javaoptions over the scenario flags.
*
*
* The testing can be started by invoking {@link #start()}
*
* @param flags VM options to be applied to the test VM.
* @return the same framework instance.
*/
public TestFramework addFlags(String... flags) {
TestRun.check(flags != null && Arrays.stream(flags).noneMatch(Objects::isNull), "A flag cannot be null");
if (this.flags == null) {
this.flags = new ArrayList<>();
}
this.flags.addAll(Arrays.asList(flags));
return this;
}
/**
* Add helper classes that can specify additional compile command annotations ({@link ForceCompile @ForceCompile},
* {@link DontCompile @DontCompile}, {@link ForceInline @ForceInline}, {@link DontInline @DontInline}) to be applied
* while testing {@code testClass} (also see description of {@link TestFramework}).
*
*
* Duplicates in {@code helperClasses} are ignored. If a class is used by the test class that does not specify any
* compile command annotations, you do not need to include it with this method. If no helper class specifies any
* compile commands, you do not need to call this method at all.
*
*
* The testing can be started by invoking {@link #start()}.
*
* @param helperClasses helper classes containing compile command annotations ({@link ForceCompile},
* {@link DontCompile}, {@link ForceInline}, {@link DontInline}) to be applied
* while testing {@code testClass} (also see description of {@link TestFramework}).
* @return the same framework instance.
*/
public TestFramework addHelperClasses(Class... helperClasses) {
TestRun.check(helperClasses != null && Arrays.stream(helperClasses).noneMatch(Objects::isNull),
"A Helper class cannot be null");
if (this.helperClasses == null) {
this.helperClasses = new HashSet<>();
}
this.helperClasses.addAll(Arrays.asList(helperClasses));
return this;
}
/**
* Add scenarios to be used for the test VM. A test VM is called for each scenario in {@code scenarios} by using the
* specified VM flags in the scenario. The scenario flags override any flags set by {@link #addFlags(String...)}
* and thus also override any VM or Javaoptions set by JTreg by default.
* Use {@code -DPreferCommandLineFlags=true} if you want to prefer the VM and Javaoptions over the scenario flags.
*
*
* The testing can be started by invoking {@link #start()}
*
* @param scenarios scenarios which specify specific flags for the test VM.
* @return the same framework instance.
*/
public TestFramework addScenarios(Scenario... scenarios) {
TestFormat.checkAndReport(scenarios != null && Arrays.stream(scenarios).noneMatch(Objects::isNull),
"A scenario cannot be null");
if (this.scenarios == null) {
this.scenarios = new ArrayList<>();
this.scenarioIndices = new HashSet<>();
}
for (Scenario scenario : scenarios) {
int scenarioIndex = scenario.getIndex();
TestFormat.checkNoThrow(scenarioIndices.add(scenarioIndex),
"Cannot define two scenarios with the same index " + scenarioIndex);
this.scenarios.add(scenario);
}
TestFormat.throwIfAnyFailures();
return this;
}
/**
* Add test classes to boot classpath. This adds all classes found on path {@link jdk.test.lib.Utils#TEST_CLASSES}
* to the boot classpath with "-Xbootclasspath/a". This is useful when trying to run tests in a privileged mode.
*/
public TestFramework addTestClassesToBootClassPath() {
this.testClassesOnBootClassPath = true;
return this;
}
/**
* Start the testing of the implicitly (by {@link #TestFramework()}) or explicitly (by {@link #TestFramework(Class)})
* set test class.
*/
public void start() {
if (shouldInstallWhiteBox()) {
installWhiteBox();
}
checkIRRuleCompilePhasesFormat();
disableIRVerificationIfNotFeasible();
if (scenarios == null) {
try {
start(null);
} catch (TestVMException e) {
System.err.println(System.lineSeparator() + e.getExceptionInfo() + RERUN_HINT);
throw e;
} catch (IRViolationException e) {
System.out.println(e.getCompilations());
System.err.println(System.lineSeparator() + e.getExceptionInfo() + System.lineSeparator() + RERUN_HINT);
throw e;
}
} else {
startWithScenarios();
}
}
private void checkIRRuleCompilePhasesFormat() {
for (Method method : testClass.getDeclaredMethods()) {
for (IR irAnno : method.getAnnotationsByType(IR.class)) {
TestFormat.checkNoThrow(irAnno.phase().length > 0,
"@IR rule " + irAnno + " must specify a non-empty list of compile " +
"phases \"phase\" at " + method);
}
}
TestFormat.throwIfAnyFailures();
}
/**
* Try to load the Whitebox class from the user directory with a custom class loader. If the user has already built the
* Whitebox, we can load it. Otherwise, the framework needs to install it.
*
* @return true if the framework needs to install the Whitebox
*/
private boolean shouldInstallWhiteBox() {
try {
URL url = Path.of(System.getProperty("user.dir")).toUri().toURL();
URLClassLoader userDirClassLoader =
URLClassLoader.newInstance(new URL[] {url}, TestFramework.class.getClassLoader().getParent());
Class.forName(WhiteBox.class.getName(), false, userDirClassLoader);
} catch (MalformedURLException e) {
throw new TestFrameworkException("corrupted user.dir property", e);
} catch (ClassNotFoundException e) {
// We need to manually install the WhiteBox if we cannot load the WhiteBox class from the user directory.
// This happens when the user test does not explicitly install the WhiteBox as part of the test.
return true;
}
return false;
}
/**
* Set a new default warm-up (overriding the framework default of 2000 at
* {@link TestVM#WARMUP_ITERATIONS}) to be applied for all tests that do not specify an explicit
* warm-up with {@link Warmup @Warmup}.
*
* @param defaultWarmup a new non-negative default warm-up.
* @return the same framework instance.
*/
public TestFramework setDefaultWarmup(int defaultWarmup) {
TestFormat.checkAndReport(defaultWarmup >= 0, "Cannot specify a negative default warm-up");
this.defaultWarmup = defaultWarmup;
return this;
}
/**
* In rare cases, methods may not be compilable because of a compilation bailout. By default, this leads to a
* test failure. However, if such cases are expected in multiple methods in a test class, this flag can be set to
* true, which allows any test to pass even if there is a compilation bailout. If only selected methods are prone
* to bail out, it is preferred to use {@link Test#allowNotCompilable()} instead for more fine-grained control.
* By setting this flag, any associated {@link IR} rule of a test is only executed if the test method was compiled,
* and else it is ignored silently.
*/
public TestFramework allowNotCompilable() {
this.isAllowNotCompilable = true;
return this;
}
/**
* Get the VM output of the test VM. Use {@code -DVerbose=true} to enable more debug information. If scenarios
* were run, use {@link Scenario#getTestVMOutput()}.
*
* @return the last test VM output.
*/
public static String getLastTestVMOutput() {
return TestVMProcess.getLastTestVMOutput();
}
/*
* The following methods are only intended to be called from actual @Test methods and not from the main() method of
* a JTreg test. Calling these methods from main() results in a linking exception (Whitebox not yet loaded and enabled).
*/
/**
* Compile {@code m} at compilation level {@code compLevel}. {@code m} is first enqueued and might not be compiled,
* yet, upon returning from this method.
*
* @param m the method to be compiled.
* @param compLevel the (valid) compilation level at which the method should be compiled.
* @throws TestRunException if compilation level is {@link CompLevel#SKIP} or {@link CompLevel#WAIT_FOR_COMPILATION}.
*/
public static void compile(Method m, CompLevel compLevel) {
TestVM.compile(m, compLevel);
}
/**
* Deoptimize {@code m}.
*
* @param m the method to be deoptimized.
*/
public static void deoptimize(Method m) {
TestVM.deoptimize(m);
}
/**
* Returns a boolean indicating if {@code m} is compiled at any level.
*
* @param m the method to be checked.
* @return {@code true} if {@code m} is compiled at any level;
* {@code false} otherwise.
*/
public static boolean isCompiled(Method m) {
return TestVM.isCompiled(m);
}
/**
* Returns a boolean indicating if {@code m} is compiled with C1.
*
* @param m the method to be checked.
* @return {@code true} if {@code m} is compiled with C1;
* {@code false} otherwise.
*/
public static boolean isC1Compiled(Method m) {
return TestVM.isC1Compiled(m);
}
/**
* Returns a boolean indicating if {@code m} is compiled with C2.
*
* @param m the method to be checked.
* @return {@code true} if {@code m} is compiled with C2;
* {@code false} otherwise.
*/
public static boolean isC2Compiled(Method m) {
return TestVM.isC2Compiled(m);
}
/**
* Returns a boolean indicating if {@code m} is compiled at the specified {@code compLevel}.
*
* @param m the method to be checked.
* @param compLevel the compilation level.
* @return {@code true} if {@code m} is compiled at {@code compLevel};
* {@code false} otherwise.
*/
public static boolean isCompiledAtLevel(Method m, CompLevel compLevel) {
return TestVM.isCompiledAtLevel(m, compLevel);
}
/**
* Checks if {@code m} is compiled at any level.
*
* @param m the method to be checked.
* @throws TestRunException if {@code m} is not compiled at any level.
*/
public static void assertCompiled(Method m) {
TestVM.assertCompiled(m);
}
/**
* Checks if {@code m} is not compiled at any level.
*
* @param m the method to be checked.
* @throws TestRunException if {@code m} is compiled at any level.
*/
public static void assertNotCompiled(Method m) {
TestVM.assertNotCompiled(m);
}
/**
* Verifies that {@code m} is compiled with C1.
*
* @param m the method to be verified.
* @throws TestRunException if {@code m} is not compiled with C1.
*/
public static void assertCompiledByC1(Method m) {
TestVM.assertCompiledByC1(m);
}
/**
* Verifies that {@code m} is compiled with C2.
*
* @param m the method to be checked.
* @throws TestRunException if {@code m} is not compiled with C2.
*/
public static void assertCompiledByC2(Method m) {
TestVM.assertCompiledByC2(m);
}
/**
* Verifies that {@code m} is compiled at the specified {@code compLevel}.
*
* @param m the method to be checked.
* @param compLevel the compilation level.
* @throws TestRunException if {@code m} is not compiled at {@code compLevel}.
*/
public static void assertCompiledAtLevel(Method m, CompLevel compLevel) {
TestVM.assertCompiledAtLevel(m, compLevel);
}
/**
* Verifies that {@code m} was deoptimized after being C1 compiled.
*
* @param m the method to be checked.
* @throws TestRunException if {@code m} is was not deoptimized after being C1 compiled.
*/
public static void assertDeoptimizedByC1(Method m) {
TestVM.assertDeoptimizedByC1(m);
}
/**
* Verifies that {@code m} was deoptimized after being C2 compiled.
*
* @param m the method to be checked.
* @throws TestRunException if {@code m} is was not deoptimized after being C2 compiled.
*/
public static void assertDeoptimizedByC2(Method m) {
TestVM.assertDeoptimizedByC2(m);
}
/**
* Returns a different boolean each time this method is invoked (switching between {@code false} and {@code true}).
* The very first invocation returns {@code false}. Note that this method could be used by different tests and
* thus the first invocation for a test could be {@code true} or {@code false} depending on how many times
* other tests have already invoked this method.
*
* @return an inverted boolean of the result of the last invocation of this method.
*/
public static boolean toggleBoolean() {
toggleBool = !toggleBool;
return toggleBool;
}
/*
* End of public interface methods
*/
/**
* Used to move Whitebox class to the right folder in the JTreg test
*/
private void installWhiteBox() {
try {
ClassFileInstaller.main(WhiteBox.class.getName());
} catch (Exception e) {
throw new Error("failed to install whitebox classes", e);
}
}
/**
* Disable IR verification completely in certain cases.
*/
private void disableIRVerificationIfNotFeasible() {
if (!irVerificationPossible) {
return;
}
boolean debugTest = Platform.isDebugBuild();
boolean intTest = !Platform.isInt();
boolean compTest = !Platform.isComp();
boolean irTest = hasIRAnnotations();
// No IR verification is done if additional non-whitelisted JTreg VM or Javaoptions flag is specified.
List
*
*
* @param flags VM flags to be used for the test VM.
*/
public static void runWithFlags(String... flags) {
StackWalker walker = StackWalker.getInstance(StackWalker.Option.RETAIN_CLASS_REFERENCE);
TestFramework framework = new TestFramework(walker.getCallerClass());
framework.addFlags(flags);
framework.start();
}
/**
* Add VM flags to be used for the test VM. These flags override any VM or Javaoptions set by JTreg by default.